[ { "path": ".clang-format", "content": "AlignEscapedNewlinesLeft: false\nAlignTrailingComments: true\nAllowAllParametersOfDeclarationOnNextLine: true\nAllowShortBlocksOnASingleLine: false\nAllowShortCaseLabelsOnASingleLine: false\nAllowShortFunctionsOnASingleLine: None\nAllowShortIfStatementsOnASingleLine: false\nAllowShortLoopsOnASingleLine: false\nAlwaysBreakAfterDefinitionReturnType: false\nAlwaysBreakBeforeMultilineStrings: false\nBinPackArguments: true\nBinPackParameters: true\nBreakBeforeBinaryOperators: None\nBreakBeforeBraces: Attach\nBreakBeforeTernaryOperators: true\nColumnLimit: 0\nContinuationIndentWidth: 4\nCpp11BracedListStyle: false\nDerivePointerAlignment: false\nDisableFormat: false\nExperimentalAutoDetectBinPacking: false\nIndentCaseLabels: true\nIndentFunctionDeclarationAfterType: false\nIndentWidth: 4\nIndentWrappedFunctionNames: false\nKeepEmptyLinesAtTheStartOfBlocks: false\nLanguage: Cpp\nMaxEmptyLinesToKeep: 2\nPointerAlignment: Right\nSpaceAfterCStyleCast: false\nSpaceBeforeAssignmentOperators: true\nSpaceBeforeParens: ControlStatements\nSpaceInEmptyParentheses: false\nSpacesBeforeTrailingComments: 1\nSpacesInCStyleCastParentheses: false\nSpacesInParentheses: false\nSpacesInSquareBrackets: false\nUseTab: Never\n" }, { "path": ".gitignore", "content": "*.dSYM\n*.gcda\n*.o\n*.plist\n.deps\n.dirstamp\n.DS_Store\naclocal.m4\nag\nag.exe\nautom4te.cache\ncachegrind.out.*\ncallgrind.out.*\nclang_output_*\ncompile\nconfig.guess\nconfig.log\nconfig.status\nconfig.sub\nconfigure\ndepcomp\ngmon.out\ninstall-sh\nMakefile\nMakefile.in\nmissing\nsrc/config.h*\nstamp-h1\ntests/*.err\ntests/big/*.err\ntests/big/big_file.txt\nthe_silver_searcher.spec\n" }, { "path": ".travis.yml", "content": "language: c\ndist: xenial\nsudo: false\n\nbranches:\n only:\n - master\n - ppc64le\narch:\n - amd64\n - ppc64le\n\ncompiler:\n - clang\n - gcc\n\naddons:\n apt:\n sources:\n - ubuntu-toolchain-r-test\n packages:\n - automake\n - liblzma-dev\n - libpcre3-dev\n - pkg-config\n - zlib1g-dev\n\nenv:\n global:\n - LLVM_VERSION=6.0.1\n - LLVM_PATH=$HOME/clang+llvm\n - CLANG_FORMAT=$LLVM_PATH/bin/clang-format\n\nbefore_install:\n - wget http://llvm.org/releases/$LLVM_VERSION/clang+llvm-$LLVM_VERSION-x86_64-linux-gnu-ubuntu-16.04.tar.xz -O $LLVM_PATH.tar.xz\n - mkdir $LLVM_PATH\n - tar xf $LLVM_PATH.tar.xz -C $LLVM_PATH --strip-components=1\n - export PATH=$HOME/.local/bin:$PATH\n\ninstall:\n - pip install --user cram\n\nscript:\n - ./build.sh && make test\n\nnotifications:\n irc: 'chat.freenode.net#ag'\n on_success: change\n on_failure: always\n use_notice: true\n" }, { "path": "CONTRIBUTING.md", "content": "## Contributing\n\nI like when people send pull requests. It validates my existence. If you want to help out, check the [issue list](https://github.com/ggreer/the_silver_searcher/issues?sort=updated&state=open) or search the codebase for `TODO`. Don't worry if you lack experience writing C. If I think a pull request isn't ready to be merged, I'll give feedback in comments. Once everything looks good, I'll comment on your pull request with a cool animated gif and hit the merge button.\n\n### Running the test suite\n\nIf you contribute, you might want to run the test suite before and after writing\nsome code, just to make sure you did not break anything. Adding tests along with\nyour code is nice to have, because it makes regressions less likely to happen.\nAlso, if you think you have found a bug, contributing a failing test case is a\ngood way of making your point and adding value at the same time.\n\nThe test suite uses [Cram](https://bitheap.org/cram/). You'll need to build ag\nfirst, and then you can run the suite from the root of the repository :\n\n make test\n\n### Adding filetypes\n\nAg can search files which belong to a certain class for example `ag --html test` \nsearches all files with the extension defined in [lang.c](src/lang.c).\n\nIf you want to add a new file 'class' to ag please modify [lang.c](src/lang.c) and [list_file_types.t](tests/list_file_types.t).\n\n`lang.c` adds the functionality and `list_file_types.t` adds the test case. \nWithout adding a test case the test __will__ fail.\n" }, { "path": "LICENSE", "content": "\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n" }, { "path": "Makefile.am", "content": "ACLOCAL_AMFLAGS = ${ACLOCAL_FLAGS}\n\nbin_PROGRAMS = ag\nag_SOURCES = src/ignore.c src/ignore.h src/log.c src/log.h src/options.c src/options.h src/print.c src/print_w32.c src/print.h src/scandir.c src/scandir.h src/search.c src/search.h src/lang.c src/lang.h src/util.c src/util.h src/decompress.c src/decompress.h src/uthash.h src/main.c src/zfile.c\nag_LDADD = ${PCRE_LIBS} ${LZMA_LIBS} ${ZLIB_LIBS} $(PTHREAD_LIBS)\n\ndist_man_MANS = doc/ag.1\n\nbashcompdir = $(pkgdatadir)/completions\ndist_bashcomp_DATA = ag.bashcomp.sh\nzshcompdir = $(datadir)/zsh/site-functions\ndist_zshcomp_DATA = _the_silver_searcher\n\nEXTRA_DIST = Makefile.w32 LICENSE NOTICE the_silver_searcher.spec README.md\n\nall:\n\t@$(MAKE) ag -r\n\ntest: ag\n\tcram -v tests/*.t\nif HAS_CLANG_FORMAT\n\tCLANG_FORMAT=${CLANG_FORMAT} ./format.sh test\nelse\n\t@echo \"clang-format is not available. Skipped clang-format test.\"\nendif\n\ntest_big: ag\n\tcram -v tests/big/*.t\n\ntest_fail: ag\n\tcram -v tests/fail/*.t\n\n.PHONY : all clean test test_big test_fail\n" }, { "path": "Makefile.w32", "content": "SED=sed\nVERSION:=$(shell \"$(SED)\" -n \"s/[^[]*\\[\\([0-9]\\+\\.[0-9]\\+\\.[0-9]\\+\\)\\],/\\1/p\" configure.ac)\n\nCC=gcc\nRM=/bin/rm\n\nSRCS = \\\n\tsrc/decompress.c \\\n\tsrc/ignore.c \\\n\tsrc/lang.c \\\n\tsrc/log.c \\\n\tsrc/main.c \\\n\tsrc/options.c \\\n\tsrc/print.c \\\n\tsrc/scandir.c \\\n\tsrc/search.c \\\n\tsrc/util.c \\\n\tsrc/print_w32.c\nOBJS = $(subst .c,.o,$(SRCS))\n\nCFLAGS = -O2 -Isrc/win32 -DPACKAGE_VERSION=\\\"$(VERSION)\\\"\nLIBS = -lz -lpthread -lpcre -llzma -lshlwapi\nTARGET = ag.exe\n\nall : $(TARGET)\n\n# depend on configure.ac to account for version changes\n$(TARGET) : $(OBJS) configure.ac\n\t$(CC) -o $@ $(OBJS) $(LIBS)\n\n.c.o :\n\t$(CC) -c $(CFLAGS) -Isrc $< -o $@\n\nclean :\n\t$(RM) -f src/*.o $(TARGET)\n" }, { "path": "NOTICE", "content": "The Silver Searcher\nCopyright 2011-2016 Geoff Greer\n" }, { "path": "README.md", "content": "# The Silver Searcher\n\nA code searching tool similar to `ack`, with a focus on speed.\n\n[![Build Status](https://travis-ci.org/ggreer/the_silver_searcher.svg?branch=master)](https://travis-ci.org/ggreer/the_silver_searcher)\n\n[![Floobits Status](https://floobits.com/ggreer/ag.svg)](https://floobits.com/ggreer/ag/redirect)\n\n[![#ag on Freenode](https://img.shields.io/badge/Freenode-%23ag-brightgreen.svg)](https://webchat.freenode.net/?channels=ag)\n\nDo you know C? Want to improve ag? [I invite you to pair with me](http://geoff.greer.fm/2014/10/13/help-me-get-to-ag-10/).\n\n\n## What's so great about Ag?\n\n* It is an order of magnitude faster than `ack`.\n* It ignores file patterns from your `.gitignore` and `.hgignore`.\n* If there are files in your source repo you don't want to search, just add their patterns to a `.ignore` file. (\\*cough\\* `*.min.js` \\*cough\\*)\n* The command name is 33% shorter than `ack`, and all keys are on the home row!\n\nAg is quite stable now. Most changes are new features, minor bug fixes, or performance improvements. It's much faster than Ack in my benchmarks:\n\n ack test_blah ~/code/ 104.66s user 4.82s system 99% cpu 1:50.03 total\n\n ag test_blah ~/code/ 4.67s user 4.58s system 286% cpu 3.227 total\n\nAck and Ag found the same results, but Ag was 34x faster (3.2 seconds vs 110 seconds). My `~/code` directory is about 8GB. Thanks to git/hg/ignore, Ag only searched 700MB of that.\n\nThere are also [graphs of performance across releases](http://geoff.greer.fm/ag/speed/).\n\n## How is it so fast?\n\n* Ag uses [Pthreads](https://en.wikipedia.org/wiki/POSIX_Threads) to take advantage of multiple CPU cores and search files in parallel.\n* Files are `mmap()`ed instead of read into a buffer.\n* Literal string searching uses [Boyer-Moore strstr](https://en.wikipedia.org/wiki/Boyer%E2%80%93Moore_string_search_algorithm).\n* Regex searching uses [PCRE's JIT compiler](http://sljit.sourceforge.net/pcre.html) (if Ag is built with PCRE >=8.21).\n* Ag calls `pcre_study()` before executing the same regex on every file.\n* Instead of calling `fnmatch()` on every pattern in your ignore files, non-regex patterns are loaded into arrays and binary searched.\n\nI've written several blog posts showing how I've improved performance. These include how I [added pthreads](http://geoff.greer.fm/2012/09/07/the-silver-searcher-adding-pthreads/), [wrote my own `scandir()`](http://geoff.greer.fm/2012/09/03/profiling-ag-writing-my-own-scandir/), [benchmarked every revision to find performance regressions](http://geoff.greer.fm/2012/08/25/the-silver-searcher-benchmarking-revisions/), and profiled with [gprof](http://geoff.greer.fm/2012/02/08/profiling-with-gprof/) and [Valgrind](http://geoff.greer.fm/2012/01/23/making-programs-faster-profiling/).\n\n\n## Installing\n\n### macOS\n\n brew install the_silver_searcher\n\nor\n\n port install the_silver_searcher\n\n\n### Linux\n\n* Ubuntu >= 13.10 (Saucy) or Debian >= 8 (Jessie)\n\n apt-get install silversearcher-ag\n* Fedora 21 and lower\n\n yum install the_silver_searcher\n* Fedora 22+\n\n dnf install the_silver_searcher\n* RHEL7+\n\n yum install epel-release.noarch the_silver_searcher\n* Gentoo\n\n emerge -a sys-apps/the_silver_searcher\n* Arch\n\n pacman -S the_silver_searcher\n\n* Slackware\n\n sbopkg -i the_silver_searcher\n\n* openSUSE\n\n zypper install the_silver_searcher\n\n* CentOS\n\n yum install the_silver_searcher\n\n* NixOS/Nix/Nixpkgs\n\n nix-env -iA silver-searcher\n\n* SUSE Linux Enterprise: Follow [these simple instructions](https://software.opensuse.org/download.html?project=utilities&package=the_silver_searcher).\n\n\n### BSD\n\n* FreeBSD\n\n pkg install the_silver_searcher\n* OpenBSD/NetBSD\n\n pkg_add the_silver_searcher\n\n### Windows\n\n* Win32/64\n\n Unofficial daily builds are [available](https://github.com/k-takata/the_silver_searcher-win32).\n \n* winget\n\n winget install \"The Silver Searcher\"\n \n Notes:\n - This installs a [release](https://github.com/JFLarvoire/the_silver_searcher/releases) of ag.exe optimized for Windows.\n - winget is intended to become the default package manager client for Windows. \n As of June 2020, it's still in beta, and can be installed using instructions [there](https://github.com/microsoft/winget-cli).\n - The setup script in the Ag's winget package installs ag.exe in the first directory that matches one of these criteria:\n 1. Over a previous instance of ag.exe *from the same [origin](https://github.com/JFLarvoire/the_silver_searcher)* found in the PATH\n 2. In the directory defined in environment variable bindir_%PROCESSOR_ARCHITECTURE%\n 3. In the directory defined in environment variable bindir\n 4. In the directory defined in environment variable windir\n \n* Chocolatey\n\n choco install ag\n* MSYS2\n\n pacman -S mingw-w64-{i686,x86_64}-ag\n* Cygwin\n\n Run the relevant [`setup-*.exe`](https://cygwin.com/install.html), and select \"the\\_silver\\_searcher\" in the \"Utils\" category.\n\n## Building from source\n\n### Building master\n\n1. Install dependencies (Automake, pkg-config, PCRE, LZMA):\n * macOS:\n\n brew install automake pkg-config pcre xz\n or\n\n port install automake pkgconfig pcre xz\n * Ubuntu/Debian:\n\n apt-get install -y automake pkg-config libpcre3-dev zlib1g-dev liblzma-dev\n * Fedora:\n\n yum -y install pkgconfig automake gcc zlib-devel pcre-devel xz-devel\n * CentOS:\n\n yum -y groupinstall \"Development Tools\"\n yum -y install pcre-devel xz-devel zlib-devel\n * openSUSE:\n\n zypper source-install --build-deps-only the_silver_searcher\n\n * Windows: It's complicated. See [this wiki page](https://github.com/ggreer/the_silver_searcher/wiki/Windows).\n2. Run the build script (which just runs aclocal, automake, etc):\n\n ./build.sh\n\n On Windows (inside an msys/MinGW shell):\n\n make -f Makefile.w32\n3. Make install:\n\n sudo make install\n\n\n### Building a release tarball\n\nGPG-signed releases are available [here](http://geoff.greer.fm/ag).\n\nBuilding release tarballs requires the same dependencies, except for automake and pkg-config. Once you've installed the dependencies, just run:\n\n ./configure\n make\n make install\n\nYou may need to use `sudo` or run as root for the make install.\n\n\n## Editor Integration\n\n### Vim\n\nYou can use Ag with [ack.vim](https://github.com/mileszs/ack.vim) by adding the following line to your `.vimrc`:\n\n let g:ackprg = 'ag --nogroup --nocolor --column'\n\nor:\n\n let g:ackprg = 'ag --vimgrep'\n\nWhich has the same effect but will report every match on the line.\n\n### Emacs\n\nYou can use [ag.el][] as an Emacs front-end to Ag. See also: [helm-ag].\n\n[ag.el]: https://github.com/Wilfred/ag.el\n[helm-ag]: https://github.com/syohex/emacs-helm-ag\n\n### TextMate\n\nTextMate users can use Ag with [my fork](https://github.com/ggreer/AckMate) of the popular AckMate plugin, which lets you use both Ack and Ag for searching. If you already have AckMate you just want to replace Ack with Ag, move or delete `\"~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack\"` and run `ln -s /usr/local/bin/ag \"~/Library/Application Support/TextMate/PlugIns/AckMate.tmplugin/Contents/Resources/ackmate_ack\"`\n\n## Other stuff you might like\n\n* [Ack](https://github.com/petdance/ack3) - Better than grep. Without Ack, Ag would not exist.\n* [ack.vim](https://github.com/mileszs/ack.vim)\n* [Exuberant Ctags](http://ctags.sourceforge.net/) - Faster than Ag, but it builds an index beforehand. Good for *really* big codebases.\n* [Git-grep](http://git-scm.com/docs/git-grep) - As fast as Ag but only works on git repos.\n* [fzf](https://github.com/junegunn/fzf) - A command-line fuzzy finder \n* [ripgrep](https://github.com/BurntSushi/ripgrep)\n* [Sack](https://github.com/sampson-chen/sack) - A utility that wraps Ack and Ag. It removes a lot of repetition from searching and opening matching files.\n" }, { "path": "_the_silver_searcher", "content": "#compdef ag\n\n# Completion function for zsh\n\nlocal ret=1\nlocal -a args expl\n\n# Intentionally avoided many possible mutual exlusions because it is\n# likely that earlier options come from an alias. In line with this\n# the following conditionally adds options that assert defaults.\n[[ -n $words[(r)(-[is]|--ignore-case|--case-sensitive)] ]] && args+=(\n '(-S --smart-case -s -s --ignore-case --case-sensitive)'{-S,--smart-case}'[insensitive match unless pattern includes uppercase]'\n)\n[[ -n $words[(r)--nobreak] ]] && args+=(\n \"(--nobreak)--break[print newlines between matches in different files]\"\n)\n[[ -n $words[(r)--nogroup] ]] && args+=(\n \"(--nogroup)--group[don't repeat filename for each match line]\"\n)\n\n_tags normal-options file-types\nwhile _tags; do\n _requested normal-options && _arguments -S -s $args \\\n '--ackmate[print results in AckMate-parseable format]' \\\n '(--after -A)'{--after=-,-A+}'[specify lines of trailing context]::lines [2]' \\\n '(--before -B)'{--before=-,-B+}'[specify lines of leading context]::lines [2]' \\\n \"--nobreak[don't print newlines between matches in different files]\" \\\n '(--count -c)'{--count,-c}'[only print a count of matching lines]' \\\n '--color[enable color highlighting of output]' \\\n '(--color-line-number --color-match --color-path)--nocolor[disable color highlighting of output]' \\\n '--color-line-number=[specify color for line numbers]:color [1;33]' \\\n '--color-match=[specify color for result match numbers]:color [30;43]' \\\n '--color-path=[specify color for path names]:color [1;32]' \\\n '--column[print column numbers in results]' \\\n '(--context -C)'{--context=-,-C+}'[specify lines of context]::lines' \\\n '(--debug -D)'{--debug,-D}'[output debug information]' \\\n '--depth=[specify directory levels to descend when searching]:levels [25]' \\\n '(--noheading)--nofilename[suppress printing of filenames]' \\\n '(-f --follow)'{-f,--follow}'[follow symlinks]' \\\n '(-F --fixed-strings --literal -Q)'{--fixed-strings,-F,--literal,-Q}'[use literal strings]' \\\n '--nogroup[repeat filename for each match line]' \\\n '(1 -G --file-search-regex)-g+[print filenames matching a pattern]:regex' \\\n '(-G --file-search-regex)'{-G+,--file-search-regex=}'[limit search to filenames matching pattern]:regex' \\\n '(-H --heading --noheading)'{-H,--heading}'[print filename with each match]' \\\n '(-H --heading --noheading --nofilename)--noheading[suppress printing of filenames]' \\\n '--hidden[search hidden files (obeying .*ignore files)]' \\\n {--ignore=,--ignore-dir=}'[ignore files/directories matching pattern]:regex' \\\n '(-i --ignore-case)'{-i,--ignore-case}'[match case-insensitively]' \\\n '(-l --files-with-matches)'{-l,--files-with-matches}\"[output matching files' names only]\" \\\n '(-L --files-without-matches)'{-L,--files-without-matches}\"[output non-matching files' names only]\" \\\n '(--max-count -m)'{--max-count=,-m+}'[stop after specified no of matches in each file]:max number of matches' \\\n '--numbers[prefix output with line numbers, even for streams]' \\\n '--nonumbers[suppress printing of line numbers]' \\\n '(--only-matching -o)'{--only-matching,-o}'[show only matching part of line]' \\\n '(-p --path-to-ignore)'{-p+,--path-to-ignore=}'[use specified .ignore file]:file:_files' \\\n '--print-long-lines[print matches on very long lines]' \\\n \"--passthrough[when searching a stream, print all lines even if they don't match]\" \\\n '(-s --case-sensitive)'{-s,--case-sensitive}'[match case]' \\\n '--silent[suppress all log messages, including errors]' \\\n '(--stats-only)--stats[print stats (files scanned, time taken, etc.)]' \\\n '(--stats)--stats-only[print stats and nothing else]' \\\n '(-U --skip-vcs-ignores)'{-U,--skip-vcs-ignores}'[ignore VCS files (stil obey .ignore)]' \\\n '(-v --invert-match)'{-v,--invert-match}'[select non-matching lines]' \\\n '--vimgrep[output results like vim :vimgrep /pattern/g would]' \\\n '(-w --word-regexp)'{-w,--word-regexp}'[force pattern to match only whole words]' \\\n '(-z --search-zip)'{-z,--search-zip}'[search contents of compressed files]' \\\n '(-0 --null)'{-0,--null}'[separate filenames with null]' \\\n ': :_guard \"^-*\" pattern' \\\n '*:file:_files' \\\n '(- :)--list-file-types[list supported file types]' \\\n '(- :)'{-h,--help}'[display help information]' \\\n '(- :)'{-V,--version}'[display version information]' \\\n - '(ignores)' \\\n '(-a --all-types)'{-a,--all-types}'[search all files]' \\\n '--search-binary[search binary files for matches]' \\\n {-t,--all-text}'[search all text files (not including hidden files)]' \\\n {-u,--unrestricted}'[search all files]' && ret=0\n\n _requested file-types && { ! zstyle -T \":completion:${curcontext}:options\" prefix-needed ||\n [[ -prefix - ]] } && _all_labels file-types expl 'file type' \\\n compadd - ${(M)$(_call_program file-types $words[1] --list-file-types):#--*} && ret=0\n\n (( ret )) || break\ndone\n\nreturn ret\n" }, { "path": "ag.bashcomp.sh", "content": "_ag() {\n local lngopt shtopt split=false\n local cur prev\n\n COMPREPLY=()\n cur=$(_get_cword \"=\")\n prev=\"${COMP_WORDS[COMP_CWORD-1]}\"\n\n _expand || return 0\n\n lngopt='\n --ackmate\n --ackmate-dir-filter\n --affinity\n --after\n --all-text\n --all-types\n --before\n --break\n --case-sensitive\n --color\n --color-line-number\n --color-match\n --color-path\n --color-win-ansi\n --column\n --context\n --count\n --debug\n --depth\n --file-search-regex\n --filename\n --files-with-matches\n --files-without-matches\n --fixed-strings\n --follow\n --group\n --heading\n --help\n --hidden\n --ignore\n --ignore-case\n --ignore-dir\n --invert-match\n --line-numbers\n --list-file-types\n --literal\n --match\n --max-count\n --no-numbers\n --no-recurse\n --noaffinity\n --nobreak\n --nocolor\n --nofilename\n --nofollow\n --nogroup\n --noheading\n --nonumbers\n --nopager\n --norecurse\n --null\n --numbers\n --one-device\n --only-matching\n --pager\n --parallel\n --passthrough\n --passthru\n --path-to-ignore\n --print-long-lines\n --print0\n --recurse\n --search-binary\n --search-files\n --search-zip\n --silent\n --skip-vcs-ignores\n --smart-case\n --stats\n --unrestricted\n --version\n --vimgrep\n --word-regexp\n --workers\n '\n shtopt='\n -a -A -B -C -D\n -f -F -g -G -h\n -i -l -L -m -n\n -p -Q -r -R -s\n -S -t -u -U -v\n -V -w -z\n '\n\n types=$(ag --list-file-types |grep -- '--')\n\n # these options require an argument\n if [[ \"${prev}\" == -[ABCGgm] ]] ; then\n return 0\n fi\n\n _split_longopt && split=true\n\n case \"${prev}\" in\n --ignore-dir) # directory completion\n _filedir -d\n return 0;;\n --path-to-ignore) # file completion\n _filedir\n return 0;;\n --pager) # command completion\n COMPREPLY=( $(compgen -c -- \"${cur}\") )\n return 0;;\n --ackmate-dir-filter|--after|--before|--color-*|--context|--depth\\\n |--file-search-regex|--ignore|--max-count|--workers)\n return 0;;\n esac\n\n $split && return 0\n\n case \"${cur}\" in\n -*)\n COMPREPLY=( $(compgen -W \\\n \"${lngopt} ${shtopt} ${types}\" -- \"${cur}\") )\n return 0;;\n *)\n _filedir\n return 0;;\n esac\n} &&\n\n# shellcheck disable=SC2086\n# shellcheck disable=SC2154,SC2086\ncomplete -F _ag ${nospace} ag\n" }, { "path": "autogen.sh", "content": "#!/bin/sh\n\nset -e\ncd \"$(dirname \"$0\")\"\n\nAC_SEARCH_OPTS=\"\"\n# For those of us with pkg-config and other tools in /usr/local\nPATH=$PATH:/usr/local/bin\n\n# This is to make life easier for people who installed pkg-config in /usr/local\n# but have autoconf/make/etc in /usr/. AKA most mac users\nif [ -d \"/usr/local/share/aclocal\" ]\nthen\n AC_SEARCH_OPTS=\"-I /usr/local/share/aclocal\"\nfi\n\n# shellcheck disable=2086\naclocal $AC_SEARCH_OPTS\nautoconf\nautoheader\nautomake --add-missing\n" }, { "path": "build.sh", "content": "#!/bin/sh\n\nset -e\ncd \"$(dirname \"$0\")\"\n\n./autogen.sh\n./configure \"$@\"\nmake -j4\n" }, { "path": "configure.ac", "content": "AC_INIT(\n [the_silver_searcher],\n [2.2.0],\n [https://github.com/ggreer/the_silver_searcher/issues],\n [the_silver_searcher],\n [https://github.com/ggreer/the_silver_searcher])\n\nAM_INIT_AUTOMAKE([no-define foreign subdir-objects])\n\nAC_PROG_CC\nAM_PROG_CC_C_O\nAC_PREREQ([2.59])\nAC_PROG_GREP\n\nm4_ifdef(\n [AM_SILENT_RULES],\n [AM_SILENT_RULES([yes])])\n\nPKG_CHECK_MODULES([PCRE], [libpcre])\n\nm4_include([m4/ax_pthread.m4])\nAX_PTHREAD(\n [AC_CHECK_HEADERS([pthread.h])],\n [AC_MSG_WARN([No pthread support. Ag will be slower due to running single-threaded.])]\n)\n\n# Run CFLAGS=\"-pg\" ./configure if you want debug symbols\nif ! echo \"$CFLAGS\" | \"$GREP\" '\\(^\\|[[[:space:]]]\\)-O' > /dev/null; then\n CFLAGS=\"$CFLAGS -O2\"\nfi\n\nCFLAGS=\"$CFLAGS $PTHREAD_CFLAGS $PCRE_CFLAGS -Wall -Wextra -Wformat=2 -Wno-format-nonliteral -Wshadow\"\nCFLAGS=\"$CFLAGS -Wpointer-arith -Wcast-qual -Wmissing-prototypes -Wno-missing-braces -std=gnu89 -D_GNU_SOURCE\"\nLDFLAGS=\"$LDFLAGS\"\n\ncase $host in\n*mingw*)\n AC_CHECK_LIB(shlwapi, main,, AC_MSG_ERROR(libshlwapi missing))\nesac\n\nLIBS=\"$PTHREAD_LIBS $LIBS\"\n\nAC_ARG_ENABLE([zlib],\n AS_HELP_STRING([--disable-zlib], [Disable zlib compressed search support]))\n\nAS_IF([test \"x$enable_zlib\" != \"xno\"], [\n AC_CHECK_HEADERS([zlib.h])\n AC_SEARCH_LIBS([inflate], [zlib, z])\n])\n\nAC_ARG_ENABLE([lzma],\n AS_HELP_STRING([--disable-lzma], [Disable lzma compressed search support]))\n\nAS_IF([test \"x$enable_lzma\" != \"xno\"], [\n AC_CHECK_HEADERS([lzma.h])\n PKG_CHECK_MODULES([LZMA], [liblzma])\n])\n\nAC_CHECK_DECL([PCRE_CONFIG_JIT], [AC_DEFINE([USE_PCRE_JIT], [], [Use PCRE JIT])], [], [#include ])\n\nAC_CHECK_DECL([CPU_ZERO, CPU_SET], [AC_DEFINE([USE_CPU_SET], [], [Use CPU_SET macros])] , [], [#include ])\nAC_CHECK_HEADERS([sys/cpuset.h err.h])\n\nAC_CHECK_MEMBER([struct dirent.d_type], [AC_DEFINE([HAVE_DIRENT_DTYPE], [], [Have dirent struct member d_type])], [], [[#include ]])\nAC_CHECK_MEMBER([struct dirent.d_namlen], [AC_DEFINE([HAVE_DIRENT_DNAMLEN], [], [Have dirent struct member d_namlen])], [], [[#include ]])\n\nAC_CHECK_FUNCS(fgetln fopencookie getline realpath strlcpy strndup vasprintf madvise posix_fadvise pthread_setaffinity_np pledge)\n\nAC_CONFIG_FILES([Makefile the_silver_searcher.spec])\nAC_CONFIG_HEADERS([src/config.h])\n\nAC_CHECK_PROGS(\n [CLANG_FORMAT],\n [clang-format-3.8 clang-format-3.7 clang-format-3.6 clang-format],\n [no]\n)\nAM_CONDITIONAL([HAS_CLANG_FORMAT], [test x$CLANG_FORMAT != xno])\nAM_COND_IF(\n [HAS_CLANG_FORMAT],\n [AC_MSG_NOTICE([clang-format found. 'make test' will detect improperly-formatted files.])],\n [AC_MSG_WARN([clang-format not found. 'make test' will not detect improperly-formatted files.])]\n)\n\nAC_OUTPUT\n" }, { "path": "doc/ag.1", "content": ".\\\" generated with Ronn/v0.7.3\n.\\\" http://github.com/rtomayko/ronn/tree/0.7.3\n.\n.TH \"AG\" \"1\" \"December 2016\" \"\" \"\"\n.\n.SH \"NAME\"\n\\fBag\\fR \\- The Silver Searcher\\. Like ack, but faster\\.\n.\n.SH \"SYNOPSIS\"\n\\fBag\\fR [\\fIoptions\\fR] \\fIpattern\\fR [\\fIpath \\.\\.\\.\\fR]\n.\n.SH \"DESCRIPTION\"\nRecursively search for PATTERN in PATH\\. Like grep or ack, but faster\\.\n.\n.SH \"OPTIONS\"\n.\n.TP\n\\fB\\-\\-ackmate\\fR\nOutput results in a format parseable by AckMate \\fIhttps://github\\.com/protocool/AckMate\\fR\\.\n.\n.TP\n\\fB\\-\\-[no]affinity\\fR\nSet thread affinity (if platform supports it)\\. Default is true\\.\n.\n.TP\n\\fB\\-a \\-\\-all\\-types\\fR\nSearch all files\\. This doesn\\'t include hidden files, and doesn\\'t respect any ignore files\\.\n.\n.TP\n\\fB\\-A \\-\\-after [LINES]\\fR\nPrint lines after match\\. If not provided, LINES defaults to 2\\.\n.\n.TP\n\\fB\\-B \\-\\-before [LINES]\\fR\nPrint lines before match\\. If not provided, LINES defaults to 2\\.\n.\n.TP\n\\fB\\-\\-[no]break\\fR\nPrint a newline between matches in different files\\. Enabled by default\\.\n.\n.TP\n\\fB\\-c \\-\\-count\\fR\nOnly print the number of matches in each file\\. Note: This is the number of matches, \\fBnot\\fR the number of matching lines\\. Pipe output to \\fBwc \\-l\\fR if you want the number of matching lines\\.\n.\n.TP\n\\fB\\-\\-[no]color\\fR\nPrint color codes in results\\. Enabled by default\\.\n.\n.TP\n\\fB\\-\\-color\\-line\\-number\\fR\nColor codes for line numbers\\. Default is 1;33\\.\n.\n.TP\n\\fB\\-\\-color\\-match\\fR\nColor codes for result match numbers\\. Default is 30;43\\.\n.\n.TP\n\\fB\\-\\-color\\-path\\fR\nColor codes for path names\\. Default is 1;32\\.\n.\n.TP\n\\fB\\-\\-column\\fR\nPrint column numbers in results\\.\n.\n.TP\n\\fB\\-C \\-\\-context [LINES]\\fR\nPrint lines before and after matches\\. Default is 2\\.\n.\n.TP\n\\fB\\-D \\-\\-debug\\fR\nOutput ridiculous amounts of debugging info\\. Not useful unless you\\'re actually debugging\\.\n.\n.TP\n\\fB\\-\\-depth NUM\\fR\nSearch up to NUM directories deep, \\-1 for unlimited\\. Default is 25\\.\n.\n.TP\n\\fB\\-\\-[no]filename\\fR\nPrint file names\\. Enabled by default, except when searching a single file\\.\n.\n.TP\n\\fB\\-f \\-\\-[no]follow\\fR\nFollow symlinks\\. Default is false\\.\n.\n.TP\n\\fB\\-F \\-\\-fixed\\-strings\\fR\nAlias for \\-\\-literal for compatibility with grep\\.\n.\n.TP\n\\fB\\-\\-[no]group\\fR\nThe default, \\fB\\-\\-group\\fR, lumps multiple matches in the same file together, and presents them under a single occurrence of the filename\\. \\fB\\-\\-nogroup\\fR refrains from this, and instead places the filename at the start of each match line\\.\n.\n.TP\n\\fB\\-g PATTERN\\fR\nPrint filenames matching PATTERN\\.\n.\n.TP\n\\fB\\-G \\-\\-file\\-search\\-regex PATTERN\\fR\nOnly search files whose names match PATTERN\\.\n.\n.TP\n\\fB\\-H \\-\\-[no]heading\\fR\nPrint filenames above matching contents\\.\n.\n.TP\n\\fB\\-\\-hidden\\fR\nSearch hidden files\\. This option obeys ignored files\\.\n.\n.TP\n\\fB\\-\\-ignore PATTERN\\fR\nIgnore files/directories whose names match this pattern\\. Literal file and directory names are also allowed\\.\n.\n.TP\n\\fB\\-\\-ignore\\-dir NAME\\fR\nAlias for \\-\\-ignore for compatibility with ack\\.\n.\n.TP\n\\fB\\-i \\-\\-ignore\\-case\\fR\nMatch case\\-insensitively\\.\n.\n.TP\n\\fB\\-l \\-\\-files\\-with\\-matches\\fR\nOnly print the names of files containing matches, not the matching lines\\. An empty query will print all files that would be searched\\.\n.\n.TP\n\\fB\\-L \\-\\-files\\-without\\-matches\\fR\nOnly print the names of files that don\\'t contain matches\\.\n.\n.TP\n\\fB\\-\\-list\\-file\\-types\\fR\nSee \\fBFILE TYPES\\fR below\\.\n.\n.TP\n\\fB\\-m \\-\\-max\\-count NUM\\fR\nSkip the rest of a file after NUM matches\\. Default is 0, which never skips\\.\n.\n.TP\n\\fB\\-\\-[no]mmap\\fR\nToggle use of memory\\-mapped I/O\\. Defaults to true on platforms where \\fBmmap()\\fR is faster than \\fBread()\\fR\\. (All but macOS\\.)\n.\n.TP\n\\fB\\-\\-[no]multiline\\fR\nMatch regexes across newlines\\. Enabled by default\\.\n.\n.TP\n\\fB\\-n \\-\\-norecurse\\fR\nDon\\'t recurse into directories\\.\n.\n.TP\n\\fB\\-\\-[no]numbers\\fR\nPrint line numbers\\. Default is to omit line numbers when searching streams\\.\n.\n.TP\n\\fB\\-o \\-\\-only\\-matching\\fR\nPrint only the matching part of the lines\\.\n.\n.TP\n\\fB\\-\\-one\\-device\\fR\nWhen recursing directories, don\\'t scan dirs that reside on other storage devices\\. This lets you avoid scanning slow network mounts\\. This feature is not supported on all platforms\\.\n.\n.TP\n\\fB\\-p \\-\\-path\\-to\\-ignore STRING\\fR\nProvide a path to a specific \\.ignore file\\.\n.\n.TP\n\\fB\\-\\-pager COMMAND\\fR\nUse a pager such as \\fBless\\fR\\. Use \\fB\\-\\-nopager\\fR to override\\. This option is also ignored if output is piped to another program\\.\n.\n.TP\n\\fB\\-\\-parallel\\fR\nParse the input stream as a search term, not data to search\\. This is meant to be used with tools such as GNU parallel\\. For example: \\fBecho \"foo\\enbar\\enbaz\" | parallel \"ag {} \\.\"\\fR will run 3 instances of ag, searching the current directory for \"foo\", \"bar\", and \"baz\"\\.\n.\n.TP\n\\fB\\-\\-print\\-long\\-lines\\fR\nPrint matches on very long lines (> 2k characters by default)\\.\n.\n.TP\n\\fB\\-\\-passthrough \\-\\-passthru\\fR\nWhen searching a stream, print all lines even if they don\\'t match\\.\n.\n.TP\n\\fB\\-Q \\-\\-literal\\fR\nDo not parse PATTERN as a regular expression\\. Try to match it literally\\.\n.\n.TP\n\\fB\\-r \\-\\-recurse\\fR\nRecurse into directories when searching\\. Default is true\\.\n.\n.TP\n\\fB\\-s \\-\\-case\\-sensitive\\fR\nMatch case\\-sensitively\\.\n.\n.TP\n\\fB\\-S \\-\\-smart\\-case\\fR\nMatch case\\-sensitively if there are any uppercase letters in PATTERN, case\\-insensitively otherwise\\. Enabled by default\\.\n.\n.TP\n\\fB\\-\\-search\\-binary\\fR\nSearch binary files for matches\\.\n.\n.TP\n\\fB\\-\\-silent\\fR\nSuppress all log messages, including errors\\.\n.\n.TP\n\\fB\\-\\-stats\\fR\nPrint stats (files scanned, time taken, etc)\\.\n.\n.TP\n\\fB\\-\\-stats\\-only\\fR\nPrint stats (files scanned, time taken, etc) and nothing else\\.\n.\n.TP\n\\fB\\-t \\-\\-all\\-text\\fR\nSearch all text files\\. This doesn\\'t include hidden files\\.\n.\n.TP\n\\fB\\-u \\-\\-unrestricted\\fR\nSearch \\fIall\\fR files\\. This ignores \\.ignore, \\.gitignore, etc\\. It searches binary and hidden files as well\\.\n.\n.TP\n\\fB\\-U \\-\\-skip\\-vcs\\-ignores\\fR\nIgnore VCS ignore files (\\.gitignore, \\.hgignore), but still use \\.ignore\\.\n.\n.TP\n\\fB\\-v \\-\\-invert\\-match\\fR\nMatch every line \\fInot\\fR containing the specified pattern\\.\n.\n.TP\n\\fB\\-V \\-\\-version\\fR\nPrint version info\\.\n.\n.TP\n\\fB\\-\\-vimgrep\\fR\nOutput results in the same form as Vim\\'s \\fB:vimgrep /pattern/g\\fR\n.\n.IP\nHere is a ~/\\.vimrc configuration example:\n.\n.IP\n\\fBset grepprg=ag\\e \\-\\-vimgrep\\e $*\\fR \\fBset grepformat=%f:%l:%c:%m\\fR\n.\n.IP\nThen use \\fB:grep\\fR to grep for something\\. Then use \\fB:copen\\fR, \\fB:cn\\fR, \\fB:cp\\fR, etc\\. to navigate through the matches\\.\n.\n.TP\n\\fB\\-w \\-\\-word\\-regexp\\fR\nOnly match whole words\\.\n.\n.TP\n\\fB\\-\\-workers NUM\\fR\nUse NUM worker threads\\. Default is the number of CPU cores, with a max of 8\\.\n.\n.TP\n\\fB\\-z \\-\\-search\\-zip\\fR\nSearch contents of compressed files\\. Currently, gz and xz are supported\\. This option requires that ag is built with lzma and zlib\\.\n.\n.TP\n\\fB\\-0 \\-\\-null \\-\\-print0\\fR\nSeparate the filenames with \\fB\\e0\\fR, rather than \\fB\\en\\fR: this allows \\fBxargs \\-0 \\fR to correctly process filenames containing spaces or newlines\\.\n.\n.SH \"FILE TYPES\"\nIt is possible to restrict the types of files searched\\. For example, passing \\fB\\-\\-html\\fR will search only files with the extensions \\fBhtm\\fR, \\fBhtml\\fR, \\fBshtml\\fR or \\fBxhtml\\fR\\. For a list of supported types, run \\fBag \\-\\-list\\-file\\-types\\fR\\.\n.\n.SH \"IGNORING FILES\"\nBy default, ag will ignore files whose names match patterns in \\.gitignore, \\.hgignore, or \\.ignore\\. These files can be anywhere in the directories being searched\\. Binary files are ignored by default as well\\. Finally, ag looks in $HOME/\\.agignore for ignore patterns\\.\n.\n.P\nIf you want to ignore \\.gitignore and \\.hgignore, but still take \\.ignore into account, use \\fB\\-U\\fR\\.\n.\n.P\nUse the \\fB\\-t\\fR option to search all text files; \\fB\\-a\\fR to search all files; and \\fB\\-u\\fR to search all, including hidden files\\.\n.\n.SH \"EXAMPLES\"\n\\fBag printf\\fR: Find matches for \"printf\" in the current directory\\.\n.\n.P\n\\fBag foo /bar/\\fR: Find matches for \"foo\" in path /bar/\\.\n.\n.P\n\\fBag \\-\\- \\-\\-foo\\fR: Find matches for \"\\-\\-foo\" in the current directory\\. (As with most UNIX command line utilities, \"\\-\\-\" is used to signify that the remaining arguments should not be treated as options\\.)\n.\n.SH \"ABOUT\"\nag was originally created by Geoff Greer\\. More information (and the latest release) can be found at http://geoff\\.greer\\.fm/ag\n.\n.SH \"SEE ALSO\"\ngrep(1)\n" }, { "path": "doc/ag.1.md", "content": "ag(1) -- The Silver Searcher. Like ack, but faster.\n=============================================\n\n## SYNOPSIS\n\n`ag` [_options_] _pattern_ [_path ..._]\n\n## DESCRIPTION\n\nRecursively search for PATTERN in PATH. Like grep or ack, but faster.\n\n## OPTIONS\n\n * `--ackmate`:\n Output results in a format parseable by [AckMate](https://github.com/protocool/AckMate).\n\n * `--[no]affinity`:\n Set thread affinity (if platform supports it). Default is true.\n\n * `-a --all-types`:\n Search all files. This doesn't include hidden files, and doesn't respect any ignore files.\n\n * `-A --after [LINES]`:\n Print lines after match. If not provided, LINES defaults to 2.\n\n * `-B --before [LINES]`:\n Print lines before match. If not provided, LINES defaults to 2.\n\n * `--[no]break`:\n Print a newline between matches in different files. Enabled by default.\n\n * `-c --count`:\n Only print the number of matches in each file.\n Note: This is the number of matches, **not** the number of matching lines.\n Pipe output to `wc -l` if you want the number of matching lines.\n\n * `--[no]color`:\n Print color codes in results. Enabled by default.\n\n * `--color-line-number`:\n Color codes for line numbers. Default is 1;33.\n\n * `--color-match`:\n Color codes for result match numbers. Default is 30;43.\n\n * `--color-path`:\n Color codes for path names. Default is 1;32.\n\n * `--column`:\n Print column numbers in results.\n\n * `-C --context [LINES]`:\n Print lines before and after matches. Default is 2.\n\n * `-D --debug`:\n Output ridiculous amounts of debugging info. Not useful unless you're actually debugging.\n\n * `--depth NUM`:\n Search up to NUM directories deep, -1 for unlimited. Default is 25.\n\n * `--[no]filename`:\n Print file names. Enabled by default, except when searching a single file.\n\n * `-f --[no]follow`:\n Follow symlinks. Default is false.\n\n * `-F --fixed-strings`:\n Alias for --literal for compatibility with grep.\n\n * `--[no]group`:\n The default, `--group`, lumps multiple matches in the same file\n together, and presents them under a single occurrence of the\n filename. `--nogroup` refrains from this, and instead places the\n filename at the start of each match line.\n\n * `-g PATTERN`:\n Print filenames matching PATTERN.\n\n * `-G --file-search-regex PATTERN`:\n Only search files whose names match PATTERN.\n\n * `-H --[no]heading`:\n Print filenames above matching contents.\n\n * `--hidden`:\n Search hidden files. This option obeys ignored files.\n\n * `--ignore PATTERN`:\n Ignore files/directories whose names match this pattern. Literal\n file and directory names are also allowed.\n\n * `--ignore-dir NAME`:\n Alias for --ignore for compatibility with ack.\n\n * `-i --ignore-case`:\n Match case-insensitively.\n\n * `-l --files-with-matches`:\n Only print the names of files containing matches, not the matching\n lines. An empty query will print all files that would be searched.\n\n * `-L --files-without-matches`:\n Only print the names of files that don't contain matches.\n\n * `--list-file-types`:\n See `FILE TYPES` below.\n\n * `-m --max-count NUM`:\n Skip the rest of a file after NUM matches. Default is 0, which never skips.\n\n * `--[no]mmap`:\n Toggle use of memory-mapped I/O. Defaults to true on platforms where\n `mmap()` is faster than `read()`. (All but macOS.)\n\n * `--[no]multiline`:\n Match regexes across newlines. Enabled by default.\n\n * `-n --norecurse`:\n Don't recurse into directories.\n\n * `--[no]numbers`:\n Print line numbers. Default is to omit line numbers when searching streams.\n\n * `-o --only-matching`:\n Print only the matching part of the lines.\n\n * `--one-device`:\n When recursing directories, don't scan dirs that reside on other storage\n devices. This lets you avoid scanning slow network mounts.\n This feature is not supported on all platforms.\n\n * `-p --path-to-ignore STRING`:\n Provide a path to a specific .ignore file.\n\n * `--pager COMMAND`:\n Use a pager such as `less`. Use `--nopager` to override. This option\n is also ignored if output is piped to another program.\n\n * `--parallel`:\n Parse the input stream as a search term, not data to search. This is meant\n to be used with tools such as GNU parallel. For example:\n `echo \"foo\\nbar\\nbaz\" | parallel \"ag {} .\"` will run 3 instances of ag,\n searching the current directory for \"foo\", \"bar\", and \"baz\".\n\n * `--print-long-lines`:\n Print matches on very long lines (> 2k characters by default).\n\n * `--passthrough --passthru`:\n When searching a stream, print all lines even if they don't match.\n\n * `-Q --literal`:\n Do not parse PATTERN as a regular expression. Try to match it literally.\n\n * `-r --recurse`:\n Recurse into directories when searching. Default is true.\n\n * `-s --case-sensitive`:\n Match case-sensitively.\n\n * `-S --smart-case`:\n Match case-sensitively if there are any uppercase letters in PATTERN,\n case-insensitively otherwise. Enabled by default.\n\n * `--search-binary`:\n Search binary files for matches.\n\n * `--silent`:\n Suppress all log messages, including errors.\n\n * `--stats`:\n Print stats (files scanned, time taken, etc).\n\n * `--stats-only`:\n Print stats (files scanned, time taken, etc) and nothing else.\n\n * `-t --all-text`:\n Search all text files. This doesn't include hidden files.\n\n * `-u --unrestricted`:\n Search *all* files. This ignores .ignore, .gitignore, etc. It searches\n binary and hidden files as well.\n\n * `-U --skip-vcs-ignores`:\n Ignore VCS ignore files (.gitignore, .hgignore), but still\n use .ignore.\n\n * `-v --invert-match`:\n Match every line *not* containing the specified pattern.\n\n * `-V --version`:\n Print version info.\n\n * `--vimgrep`:\n Output results in the same form as Vim's `:vimgrep /pattern/g`\n\n Here is a ~/.vimrc configuration example:\n\n `set grepprg=ag\\ --vimgrep\\ $*`\n `set grepformat=%f:%l:%c:%m`\n\n Then use `:grep` to grep for something.\n Then use `:copen`, `:cn`, `:cp`, etc. to navigate through the matches.\n\n * `-w --word-regexp`:\n Only match whole words.\n\n * `--workers NUM`:\n Use NUM worker threads. Default is the number of CPU cores, with a max of 8.\n\n * `-W --width NUM`:\n Truncate match lines after NUM characters.\n\n * `-z --search-zip`:\n Search contents of compressed files. Currently, gz and xz are supported.\n This option requires that ag is built with lzma and zlib.\n\n * `-0 --null --print0`:\n Separate the filenames with `\\0`, rather than `\\n`:\n this allows `xargs -0 ` to correctly process filenames containing\n spaces or newlines.\n\n\n## FILE TYPES\n\nIt is possible to restrict the types of files searched. For example, passing\n`--html` will search only files with the extensions `htm`, `html`, `shtml`\nor `xhtml`. For a list of supported types, run `ag --list-file-types`.\n\n## IGNORING FILES\n\nBy default, ag will ignore files whose names match patterns in .gitignore,\n.hgignore, or .ignore. These files can be anywhere in the directories being\nsearched. Binary files are ignored by default as well. Finally, ag looks in\n$HOME/.agignore for ignore patterns.\n\nIf you want to ignore .gitignore and .hgignore, but still take .ignore into\naccount, use `-U`.\n\nUse the `-t` option to search all text files; `-a` to search all files; and `-u`\nto search all, including hidden files.\n\n## EXAMPLES\n\n`ag printf`:\n Find matches for \"printf\" in the current directory.\n\n`ag foo /bar/`:\n Find matches for \"foo\" in path /bar/.\n\n`ag -- --foo`:\n Find matches for \"--foo\" in the current directory. (As with most UNIX command\n line utilities, \"--\" is used to signify that the remaining arguments should\n not be treated as options.)\n\n## ABOUT\n\nag was originally created by Geoff Greer. More information (and the latest\nrelease) can be found at http://geoff.greer.fm/ag\n\n## SEE ALSO\n\ngrep(1)\n" }, { "path": "doc/generate_man.sh", "content": "#!/bin/sh\n\n# ronn is used to turn the markdown into a manpage.\n# Get ronn at https://github.com/rtomayko/ronn\n# Alternately, since ronn is a Ruby gem, you can just\n# `gem install ronn`\n\nsed -e 's/\\\\0/\\\\\\\\0/' ag.1.md.tmp\nronn -r ag.1.md.tmp\n\nrm -f ag.1.md.tmp\n" }, { "path": "format.sh", "content": "#!/bin/bash\n\nfunction usage() {\n echo \"Usage: $0 test|reformat\"\n}\n\nif [ $# -eq 0 ]\nthen\n usage\n exit 0\nfi\n\nif [ -z \"$CLANG_FORMAT\" ]\nthen\n CLANG_FORMAT=clang-format\n echo \"No CLANG_FORMAT set. Using $CLANG_FORMAT\"\nfi\n\nif ! type \"$CLANG_FORMAT\" &> /dev/null\nthen\n echo \"The command \\\"$CLANG_FORMAT\\\" was not found\"\n exit 1\nfi\n\nSOURCE_FILES=$(git ls-files src/)\n\nif [ \"$1\" == \"reformat\" ]\nthen\n echo \"Reformatting source files\"\n # shellcheck disable=2086\n echo $CLANG_FORMAT -style=file -i $SOURCE_FILES\n # shellcheck disable=2086\n $CLANG_FORMAT -style=file -i $SOURCE_FILES\n exit 0\nelif [ \"$1\" == \"test\" ]\nthen\n # shellcheck disable=2086\n RESULT=$($CLANG_FORMAT -style=file -output-replacements-xml $SOURCE_FILES | grep -c '\n# Copyright (c) 2011 Daniel Richard G. \n#\n# This program is free software: you can redistribute it and/or modify it\n# under the terms of the GNU General Public License as published by the\n# Free Software Foundation, either version 3 of the License, or (at your\n# option) any later version.\n#\n# This program is distributed in the hope that it will be useful, but\n# WITHOUT ANY WARRANTY; without even the implied warranty of\n# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General\n# Public License for more details.\n#\n# You should have received a copy of the GNU General Public License along\n# with this program. If not, see .\n#\n# As a special exception, the respective Autoconf Macro's copyright owner\n# gives unlimited permission to copy, distribute and modify the configure\n# scripts that are the output of Autoconf when processing the Macro. You\n# need not follow the terms of the GNU General Public License when using\n# or distributing such scripts, even though portions of the text of the\n# Macro appear in them. The GNU General Public License (GPL) does govern\n# all other use of the material that constitutes the Autoconf Macro.\n#\n# This special exception to the GPL applies to versions of the Autoconf\n# Macro released by the Autoconf Archive. When you make and distribute a\n# modified version of the Autoconf Macro, you may extend this special\n# exception to the GPL to apply to your modified version as well.\n\n#serial 21\n\nAU_ALIAS([ACX_PTHREAD], [AX_PTHREAD])\nAC_DEFUN([AX_PTHREAD], [\nAC_REQUIRE([AC_CANONICAL_HOST])\nAC_LANG_PUSH([C])\nax_pthread_ok=no\n\n# We used to check for pthread.h first, but this fails if pthread.h\n# requires special compiler flags (e.g. on True64 or Sequent).\n# It gets checked for in the link test anyway.\n\n# First of all, check if the user has set any of the PTHREAD_LIBS,\n# etcetera environment variables, and if threads linking works using\n# them:\nif test x\"$PTHREAD_LIBS$PTHREAD_CFLAGS\" != x; then\n save_CFLAGS=\"$CFLAGS\"\n CFLAGS=\"$CFLAGS $PTHREAD_CFLAGS\"\n save_LIBS=\"$LIBS\"\n LIBS=\"$PTHREAD_LIBS $LIBS\"\n AC_MSG_CHECKING([for pthread_join in LIBS=$PTHREAD_LIBS with CFLAGS=$PTHREAD_CFLAGS])\n AC_TRY_LINK_FUNC([pthread_join], [ax_pthread_ok=yes])\n AC_MSG_RESULT([$ax_pthread_ok])\n if test x\"$ax_pthread_ok\" = xno; then\n PTHREAD_LIBS=\"\"\n PTHREAD_CFLAGS=\"\"\n fi\n LIBS=\"$save_LIBS\"\n CFLAGS=\"$save_CFLAGS\"\nfi\n\n# We must check for the threads library under a number of different\n# names; the ordering is very important because some systems\n# (e.g. DEC) have both -lpthread and -lpthreads, where one of the\n# libraries is broken (non-POSIX).\n\n# Create a list of thread flags to try. Items starting with a \"-\" are\n# C compiler flags, and other items are library names, except for \"none\"\n# which indicates that we try without any flags at all, and \"pthread-config\"\n# which is a program returning the flags for the Pth emulation library.\n\nax_pthread_flags=\"pthreads none -Kthread -kthread lthread -pthread -pthreads -mthreads pthread --thread-safe -mt pthread-config\"\n\n# The ordering *is* (sometimes) important. Some notes on the\n# individual items follow:\n\n# pthreads: AIX (must check this before -lpthread)\n# none: in case threads are in libc; should be tried before -Kthread and\n# other compiler flags to prevent continual compiler warnings\n# -Kthread: Sequent (threads in libc, but -Kthread needed for pthread.h)\n# -kthread: FreeBSD kernel threads (preferred to -pthread since SMP-able)\n# lthread: LinuxThreads port on FreeBSD (also preferred to -pthread)\n# -pthread: Linux/gcc (kernel threads), BSD/gcc (userland threads)\n# -pthreads: Solaris/gcc\n# -mthreads: Mingw32/gcc, Lynx/gcc\n# -mt: Sun Workshop C (may only link SunOS threads [-lthread], but it\n# doesn't hurt to check since this sometimes defines pthreads too;\n# also defines -D_REENTRANT)\n# ... -mt is also the pthreads flag for HP/aCC\n# pthread: Linux, etcetera\n# --thread-safe: KAI C++\n# pthread-config: use pthread-config program (for GNU Pth library)\n\ncase ${host_os} in\n solaris*)\n\n # On Solaris (at least, for some versions), libc contains stubbed\n # (non-functional) versions of the pthreads routines, so link-based\n # tests will erroneously succeed. (We need to link with -pthreads/-mt/\n # -lpthread.) (The stubs are missing pthread_cleanup_push, or rather\n # a function called by this macro, so we could check for that, but\n # who knows whether they'll stub that too in a future libc.) So,\n # we'll just look for -pthreads and -lpthread first:\n\n ax_pthread_flags=\"-pthreads pthread -mt -pthread $ax_pthread_flags\"\n ;;\n\n darwin*)\n ax_pthread_flags=\"-pthread $ax_pthread_flags\"\n ;;\nesac\n\n# Clang doesn't consider unrecognized options an error unless we specify\n# -Werror. We throw in some extra Clang-specific options to ensure that\n# this doesn't happen for GCC, which also accepts -Werror.\n\nAC_MSG_CHECKING([if compiler needs -Werror to reject unknown flags])\nsave_CFLAGS=\"$CFLAGS\"\nax_pthread_extra_flags=\"-Werror\"\nCFLAGS=\"$CFLAGS $ax_pthread_extra_flags -Wunknown-warning-option -Wsizeof-array-argument\"\nAC_COMPILE_IFELSE([AC_LANG_PROGRAM([int foo(void);],[foo()])],\n [AC_MSG_RESULT([yes])],\n [ax_pthread_extra_flags=\n AC_MSG_RESULT([no])])\nCFLAGS=\"$save_CFLAGS\"\n\nif test x\"$ax_pthread_ok\" = xno; then\nfor flag in $ax_pthread_flags; do\n\n case $flag in\n none)\n AC_MSG_CHECKING([whether pthreads work without any flags])\n ;;\n\n -*)\n AC_MSG_CHECKING([whether pthreads work with $flag])\n PTHREAD_CFLAGS=\"$flag\"\n ;;\n\n pthread-config)\n AC_CHECK_PROG([ax_pthread_config], [pthread-config], [yes], [no])\n if test x\"$ax_pthread_config\" = xno; then continue; fi\n PTHREAD_CFLAGS=\"`pthread-config --cflags`\"\n PTHREAD_LIBS=\"`pthread-config --ldflags` `pthread-config --libs`\"\n ;;\n\n *)\n AC_MSG_CHECKING([for the pthreads library -l$flag])\n PTHREAD_LIBS=\"-l$flag\"\n ;;\n esac\n\n save_LIBS=\"$LIBS\"\n save_CFLAGS=\"$CFLAGS\"\n LIBS=\"$PTHREAD_LIBS $LIBS\"\n CFLAGS=\"$CFLAGS $PTHREAD_CFLAGS $ax_pthread_extra_flags\"\n\n # Check for various functions. We must include pthread.h,\n # since some functions may be macros. (On the Sequent, we\n # need a special flag -Kthread to make this header compile.)\n # We check for pthread_join because it is in -lpthread on IRIX\n # while pthread_create is in libc. We check for pthread_attr_init\n # due to DEC craziness with -lpthreads. We check for\n # pthread_cleanup_push because it is one of the few pthread\n # functions on Solaris that doesn't have a non-functional libc stub.\n # We try pthread_create on general principles.\n AC_LINK_IFELSE([AC_LANG_PROGRAM([#include \n static void routine(void *a) { a = 0; }\n static void *start_routine(void *a) { return a; }],\n [pthread_t th; pthread_attr_t attr;\n pthread_create(&th, 0, start_routine, 0);\n pthread_join(th, 0);\n pthread_attr_init(&attr);\n pthread_cleanup_push(routine, 0);\n pthread_cleanup_pop(0) /* ; */])],\n [ax_pthread_ok=yes],\n [])\n\n LIBS=\"$save_LIBS\"\n CFLAGS=\"$save_CFLAGS\"\n\n AC_MSG_RESULT([$ax_pthread_ok])\n if test \"x$ax_pthread_ok\" = xyes; then\n break;\n fi\n\n PTHREAD_LIBS=\"\"\n PTHREAD_CFLAGS=\"\"\ndone\nfi\n\n# Various other checks:\nif test \"x$ax_pthread_ok\" = xyes; then\n save_LIBS=\"$LIBS\"\n LIBS=\"$PTHREAD_LIBS $LIBS\"\n save_CFLAGS=\"$CFLAGS\"\n CFLAGS=\"$CFLAGS $PTHREAD_CFLAGS\"\n\n # Detect AIX lossage: JOINABLE attribute is called UNDETACHED.\n AC_MSG_CHECKING([for joinable pthread attribute])\n attr_name=unknown\n for attr in PTHREAD_CREATE_JOINABLE PTHREAD_CREATE_UNDETACHED; do\n AC_LINK_IFELSE([AC_LANG_PROGRAM([#include ],\n [int attr = $attr; return attr /* ; */])],\n [attr_name=$attr; break],\n [])\n done\n AC_MSG_RESULT([$attr_name])\n if test \"$attr_name\" != PTHREAD_CREATE_JOINABLE; then\n AC_DEFINE_UNQUOTED([PTHREAD_CREATE_JOINABLE], [$attr_name],\n [Define to necessary symbol if this constant\n uses a non-standard name on your system.])\n fi\n\n AC_MSG_CHECKING([if more special flags are required for pthreads])\n flag=no\n case ${host_os} in\n aix* | freebsd* | darwin*) flag=\"-D_THREAD_SAFE\";;\n osf* | hpux*) flag=\"-D_REENTRANT\";;\n solaris*)\n if test \"$GCC\" = \"yes\"; then\n flag=\"-D_REENTRANT\"\n else\n # TODO: What about Clang on Solaris?\n flag=\"-mt -D_REENTRANT\"\n fi\n ;;\n esac\n AC_MSG_RESULT([$flag])\n if test \"x$flag\" != xno; then\n PTHREAD_CFLAGS=\"$flag $PTHREAD_CFLAGS\"\n fi\n\n AC_CACHE_CHECK([for PTHREAD_PRIO_INHERIT],\n [ax_cv_PTHREAD_PRIO_INHERIT], [\n AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include ]],\n [[int i = PTHREAD_PRIO_INHERIT;]])],\n [ax_cv_PTHREAD_PRIO_INHERIT=yes],\n [ax_cv_PTHREAD_PRIO_INHERIT=no])\n ])\n AS_IF([test \"x$ax_cv_PTHREAD_PRIO_INHERIT\" = \"xyes\"],\n [AC_DEFINE([HAVE_PTHREAD_PRIO_INHERIT], [1], [Have PTHREAD_PRIO_INHERIT.])])\n\n LIBS=\"$save_LIBS\"\n CFLAGS=\"$save_CFLAGS\"\n\n # More AIX lossage: compile with *_r variant\n if test \"x$GCC\" != xyes; then\n case $host_os in\n aix*)\n AS_CASE([\"x/$CC\"],\n [x*/c89|x*/c89_128|x*/c99|x*/c99_128|x*/cc|x*/cc128|x*/xlc|x*/xlc_v6|x*/xlc128|x*/xlc128_v6],\n [#handle absolute path differently from PATH based program lookup\n AS_CASE([\"x$CC\"],\n [x/*],\n [AS_IF([AS_EXECUTABLE_P([${CC}_r])],[PTHREAD_CC=\"${CC}_r\"])],\n [AC_CHECK_PROGS([PTHREAD_CC],[${CC}_r],[$CC])])])\n ;;\n esac\n fi\nfi\n\ntest -n \"$PTHREAD_CC\" || PTHREAD_CC=\"$CC\"\n\nAC_SUBST([PTHREAD_LIBS])\nAC_SUBST([PTHREAD_CFLAGS])\nAC_SUBST([PTHREAD_CC])\n\n# Finally, execute ACTION-IF-FOUND/ACTION-IF-NOT-FOUND:\nif test x\"$ax_pthread_ok\" = xyes; then\n ifelse([$1],,[AC_DEFINE([HAVE_PTHREAD],[1],[Define if you have POSIX threads libraries and header files.])],[$1])\n :\nelse\n ax_pthread_ok=no\n $2\nfi\nAC_LANG_POP\n])dnl AX_PTHREAD\n" }, { "path": "pgo.sh", "content": "#!/bin/sh\n\nset -e\ncd \"$(dirname \"$0\")\"\n\nmake clean\n./build.sh CFLAGS=\"$CFLAGS -fprofile-generate\"\n./ag example ..\nmake clean\n./build.sh CFLAGS=\"$CFLAGS -fprofile-correction -fprofile-use\"\n" }, { "path": "sanitize.sh", "content": "#!/bin/bash\n# Copyright 2016 Allen Wild\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\nAVAILABLE_SANITIZERS=(\n address\n thread\n undefined\n valgrind\n)\n\nDEFAULT_SANITIZERS=(\n address\n thread\n undefined\n)\n\nusage() {\n cat < and then runs the test suite.\n Memory leaks or other errors will be printed in ag's output, thus failing the test.\n\n Available LLVM sanitizers are: ${AVAILABLE_SANITIZERS[*]}\n\n The compile-time sanitizers are supported in clang/llvm >= 3.1 and gcc >= 4.8\n for x86_64 Linux only. clang is preferred and will be used, if available.\n\n For function names and line numbers in error output traces, llvm-symbolizer needs\n to be available in PATH or set through ASAN_SYMBOLIZER_PATH.\n\n If 'valgrind' is passed as the sanitizer, then ag will be run through valgrind\n without recompiling. If $(dirname $0)/ag doesn't exist, then it will be built.\n\n WARNING: This script will run \"make distclean\" and \"./configure\" to recompile ag\n once per sanitizer (except for valgrind). If you need to pass additional\n options to ./configure, put them in the CONFIGOPTS environment variable.\nEOF\n}\n\nvrun() {\n echo \"Running: $*\"\n \"$@\"\n}\n\ndie() {\n echo \"Fatal: $*\"\n exit 1\n}\n\nvalid_sanitizer() {\n for san in \"${AVAILABLE_SANITIZERS[@]}\"; do\n if [[ \"$1\" == \"$san\" ]]; then\n return 0\n fi\n done\n return 1\n}\n\nrun_sanitizer() {\n sanitizer=$1\n if [[ \"$sanitizer\" == \"valgrind\" ]]; then\n run_valgrind\n return $?\n fi\n\n echo -e \"\\nCompiling for sanitizer '$sanitizer'\"\n [[ -f Makefile ]] && vrun make distclean\n vrun ./configure $CONFIGOPTS CC=$SANITIZE_CC \\\n CFLAGS=\"-g -O0 -fsanitize=$sanitizer $EXTRA_CFLAGS\"\n if [[ $? != 0 ]]; then\n echo \"ERROR: Failed to configure. Try setting CONFIGOPTS?\"\n return 1\n fi\n\n vrun make\n if [[ $? != 0 ]]; then\n echo \"ERROR: failed to build\"\n return 1\n fi\n\n echo \"Testing with sanitizer '$sanitizer'\"\n vrun make test\n if [[ $? != 0 ]]; then\n echo \"Tests for sanitizer '$sanitizer' FAIL!\"\n echo \"Check the above output for failure information\"\n return 2\n else\n echo \"Tests for sanitizer '$sanitizer' PASS!\"\n return 0\n fi\n}\n\nrun_valgrind() {\n echo \"Compiling ag normally for use with valgrind\"\n [[ -f Makefile ]] && vrun make distclean\n vrun ./configure $CONFIGOPTS\n if [[ $? != 0 ]]; then\n echo \"ERROR: Failed to configure. Try setting CONFIGOPTS?\"\n return 1\n fi\n\n vrun make\n if [[ $? != 0 ]]; then\n echo \"ERROR: failed to build\"\n return 1\n fi\n\n echo \"Running: AGPROG=\\\"valgrind -q $PWD/ag\\\" make test\"\n AGPROG=\"valgrind -q $PWD/ag\" make test\n if [[ $? != 0 ]]; then\n echo \"Valgrind tests FAIL!\"\n return 1\n else\n echo \"Valgrind tests PASS!\"\n return 0\n fi\n}\n\n#### MAIN ####\nrun_sanitizers=()\nfor opt in \"$@\"; do\n if [[ \"$opt\" == -* ]]; then\n case opt in\n -h|--help)\n usage\n exit 0\n ;;\n *)\n echo \"Unknown option: '$opt'\"\n usage\n exit 1\n ;;\n esac\n else\n if valid_sanitizer \"$opt\"; then\n run_sanitizers+=(\"$opt\")\n else\n echo \"Invalid Sanitizer: '$opt'\"\n usage\n exit 1\n fi\n fi\ndone\n\nif [[ ${#run_sanitizers[@]} == 0 ]]; then\n run_sanitizers=(${DEFAULT_SANITIZERS[@]})\nfi\n\nif [[ -n $CC ]]; then\n echo \"Using CC=$CC\"\n SANITIZE_CC=\"$CC\"\nelif which clang &>/dev/null; then\n SANITIZE_CC=\"clang\"\nelse\n echo \"Warning: CC unset and clang not found\"\nfi\n\nif [[ -n $CFLAGS ]]; then\n EXTRA_CFLAGS=\"$CFLAGS\"\n unset CFLAGS\nfi\n\nif [[ ! -e ./configure ]]; then\n echo \"Warning: ./configure not found. Running autogen\"\n vrun ./autogen.sh || die \"autogen.sh failed\"\nfi\n\necho \"Running sanitizers: ${run_sanitizers[*]}\"\nfailedsan=()\nfor san in \"${run_sanitizers[@]}\"; do\n run_sanitizer $san\n if [[ $? != 0 ]]; then\n failedsan+=($san)\n fi\ndone\n\nif [[ ${#failedsan[@]} == 0 ]]; then\n echo \"All sanitizers PASSED\"\n exit 0\nelse\n echo \"The following sanitizers FAILED: ${failedsan[*]}\"\n exit ${#failedsan[@]}\nfi\n" }, { "path": "src/decompress.c", "content": "#include \n#include \n\n#include \"decompress.h\"\n\n#ifdef HAVE_LZMA_H\n#include \n\n/* http://tukaani.org/xz/xz-file-format.txt */\nconst uint8_t XZ_HEADER_MAGIC[6] = { 0xFD, '7', 'z', 'X', 'Z', 0x00 };\nconst uint8_t LZMA_HEADER_SOMETIMES[3] = { 0x5D, 0x00, 0x00 };\n#endif\n\n\n#ifdef HAVE_ZLIB_H\n#define ZLIB_CONST 1\n#include \n\n/* Code in decompress_zlib from\n *\n * https://raw.github.com/madler/zlib/master/examples/zpipe.c\n *\n * zpipe.c: example of proper use of zlib's inflate() and deflate()\n * Not copyrighted -- provided to the public domain\n * Version 1.4 11 December 2005 Mark Adler \n */\nstatic void *decompress_zlib(const void *buf, const int buf_len,\n const char *dir_full_path, int *new_buf_len) {\n int ret = 0;\n unsigned char *result = NULL;\n size_t result_size = 0;\n size_t pagesize = 0;\n z_stream stream;\n\n log_debug(\"Decompressing zlib file %s\", dir_full_path);\n\n /* allocate inflate state */\n stream.zalloc = Z_NULL;\n stream.zfree = Z_NULL;\n stream.opaque = Z_NULL;\n stream.avail_in = 0;\n stream.next_in = Z_NULL;\n\n /* Add 32 to allow zlib and gzip format detection */\n if (inflateInit2(&stream, 32 + 15) != Z_OK) {\n log_err(\"Unable to initialize zlib: %s\", stream.msg);\n goto error_out;\n }\n\n stream.avail_in = buf_len;\n /* Explicitly cast away the const-ness of buf */\n stream.next_in = (Bytef *)buf;\n\n pagesize = getpagesize();\n result_size = ((buf_len + pagesize - 1) & ~(pagesize - 1));\n do {\n do {\n unsigned char *tmp_result = result;\n /* Double the buffer size and realloc */\n result_size *= 2;\n result = (unsigned char *)realloc(result, result_size * sizeof(unsigned char));\n if (result == NULL) {\n free(tmp_result);\n log_err(\"Unable to allocate %d bytes to decompress file %s\", result_size * sizeof(unsigned char), dir_full_path);\n inflateEnd(&stream);\n goto error_out;\n }\n\n stream.avail_out = result_size / 2;\n stream.next_out = &result[stream.total_out];\n ret = inflate(&stream, Z_SYNC_FLUSH);\n log_debug(\"inflate ret = %d\", ret);\n switch (ret) {\n case Z_STREAM_ERROR: {\n log_err(\"Found stream error while decompressing zlib stream: %s\", stream.msg);\n inflateEnd(&stream);\n goto error_out;\n }\n case Z_NEED_DICT:\n case Z_DATA_ERROR:\n case Z_MEM_ERROR: {\n log_err(\"Found mem/data error while decompressing zlib stream: %s\", stream.msg);\n inflateEnd(&stream);\n goto error_out;\n }\n }\n } while (stream.avail_out == 0);\n } while (ret == Z_OK);\n\n *new_buf_len = stream.total_out;\n inflateEnd(&stream);\n\n if (ret == Z_STREAM_END) {\n return result;\n }\n\nerror_out:\n *new_buf_len = 0;\n return NULL;\n}\n#endif\n\n\nstatic void *decompress_lzw(const void *buf, const int buf_len,\n const char *dir_full_path, int *new_buf_len) {\n (void)buf;\n (void)buf_len;\n log_err(\"LZW (UNIX compress) files not yet supported: %s\", dir_full_path);\n *new_buf_len = 0;\n return NULL;\n}\n\n\nstatic void *decompress_zip(const void *buf, const int buf_len,\n const char *dir_full_path, int *new_buf_len) {\n (void)buf;\n (void)buf_len;\n log_err(\"Zip files not yet supported: %s\", dir_full_path);\n *new_buf_len = 0;\n return NULL;\n}\n\n\n#ifdef HAVE_LZMA_H\nstatic void *decompress_lzma(const void *buf, const int buf_len,\n const char *dir_full_path, int *new_buf_len) {\n lzma_stream stream = LZMA_STREAM_INIT;\n lzma_ret lzrt;\n unsigned char *result = NULL;\n size_t result_size = 0;\n size_t pagesize = 0;\n\n stream.avail_in = buf_len;\n stream.next_in = buf;\n\n lzrt = lzma_auto_decoder(&stream, -1, 0);\n\n if (lzrt != LZMA_OK) {\n log_err(\"Unable to initialize lzma_auto_decoder: %d\", lzrt);\n goto error_out;\n }\n\n pagesize = getpagesize();\n result_size = ((buf_len + pagesize - 1) & ~(pagesize - 1));\n do {\n do {\n unsigned char *tmp_result = result;\n /* Double the buffer size and realloc */\n result_size *= 2;\n result = (unsigned char *)realloc(result, result_size * sizeof(unsigned char));\n if (result == NULL) {\n free(tmp_result);\n log_err(\"Unable to allocate %d bytes to decompress file %s\", result_size * sizeof(unsigned char), dir_full_path);\n goto error_out;\n }\n\n stream.avail_out = result_size / 2;\n stream.next_out = &result[stream.total_out];\n lzrt = lzma_code(&stream, LZMA_RUN);\n log_debug(\"lzma_code ret = %d\", lzrt);\n switch (lzrt) {\n case LZMA_OK:\n case LZMA_STREAM_END:\n break;\n default:\n log_err(\"Found mem/data error while decompressing xz/lzma stream: %d\", lzrt);\n goto error_out;\n }\n } while (stream.avail_out == 0);\n } while (lzrt == LZMA_OK);\n\n *new_buf_len = stream.total_out;\n\n if (lzrt == LZMA_STREAM_END) {\n lzma_end(&stream);\n return result;\n }\n\n\nerror_out:\n lzma_end(&stream);\n *new_buf_len = 0;\n if (result) {\n free(result);\n }\n return NULL;\n}\n#endif\n\n\n/* This function is very hot. It's called on every file when zip is enabled. */\nvoid *decompress(const ag_compression_type zip_type, const void *buf, const int buf_len,\n const char *dir_full_path, int *new_buf_len) {\n switch (zip_type) {\n#ifdef HAVE_ZLIB_H\n case AG_GZIP:\n return decompress_zlib(buf, buf_len, dir_full_path, new_buf_len);\n#endif\n case AG_COMPRESS:\n return decompress_lzw(buf, buf_len, dir_full_path, new_buf_len);\n case AG_ZIP:\n return decompress_zip(buf, buf_len, dir_full_path, new_buf_len);\n#ifdef HAVE_LZMA_H\n case AG_XZ:\n return decompress_lzma(buf, buf_len, dir_full_path, new_buf_len);\n#endif\n case AG_NO_COMPRESSION:\n log_err(\"File %s is not compressed\", dir_full_path);\n break;\n default:\n log_err(\"Unsupported compression type: %d\", zip_type);\n }\n\n *new_buf_len = 0;\n return NULL;\n}\n\n\n/* This function is very hot. It's called on every file. */\nag_compression_type is_zipped(const void *buf, const int buf_len) {\n /* Zip magic numbers\n * compressed file: { 0x1F, 0x9B }\n * http://en.wikipedia.org/wiki/Compress\n * \n * gzip file: { 0x1F, 0x8B }\n * http://www.gzip.org/zlib/rfc-gzip.html#file-format\n *\n * zip file: { 0x50, 0x4B, 0x03, 0x04 }\n * http://www.pkware.com/documents/casestudies/APPNOTE.TXT (Section 4.3)\n */\n\n const unsigned char *buf_c = buf;\n\n if (buf_len == 0)\n return AG_NO_COMPRESSION;\n\n /* Check for gzip & compress */\n if (buf_len >= 2) {\n if (buf_c[0] == 0x1F) {\n if (buf_c[1] == 0x8B) {\n#ifdef HAVE_ZLIB_H\n log_debug(\"Found gzip-based stream\");\n return AG_GZIP;\n#endif\n } else if (buf_c[1] == 0x9B) {\n log_debug(\"Found compress-based stream\");\n return AG_COMPRESS;\n }\n }\n }\n\n /* Check for zip */\n if (buf_len >= 4) {\n if (buf_c[0] == 0x50 && buf_c[1] == 0x4B && buf_c[2] == 0x03 && buf_c[3] == 0x04) {\n log_debug(\"Found zip-based stream\");\n return AG_ZIP;\n }\n }\n\n#ifdef HAVE_LZMA_H\n if (buf_len >= 6) {\n if (memcmp(XZ_HEADER_MAGIC, buf_c, 6) == 0) {\n log_debug(\"Found xz based stream\");\n return AG_XZ;\n }\n }\n\n /* LZMA doesn't really have a header: http://www.mail-archive.com/xz-devel@tukaani.org/msg00003.html */\n if (buf_len >= 3) {\n if (memcmp(LZMA_HEADER_SOMETIMES, buf_c, 3) == 0) {\n log_debug(\"Found lzma-based stream\");\n return AG_XZ;\n }\n }\n#endif\n\n return AG_NO_COMPRESSION;\n}\n" }, { "path": "src/decompress.h", "content": "#ifndef DECOMPRESS_H\n#define DECOMPRESS_H\n\n#include \n\n#include \"config.h\"\n#include \"log.h\"\n#include \"options.h\"\n\ntypedef enum {\n AG_NO_COMPRESSION,\n AG_GZIP,\n AG_COMPRESS,\n AG_ZIP,\n AG_XZ,\n} ag_compression_type;\n\nag_compression_type is_zipped(const void *buf, const int buf_len);\n\nvoid *decompress(const ag_compression_type zip_type, const void *buf, const int buf_len, const char *dir_full_path, int *new_buf_len);\n\n#if HAVE_FOPENCOOKIE\nFILE *decompress_open(int fd, const char *mode, ag_compression_type ctype);\n#endif\n\n#endif\n" }, { "path": "src/ignore.c", "content": "#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"ignore.h\"\n#include \"log.h\"\n#include \"options.h\"\n#include \"scandir.h\"\n#include \"util.h\"\n\n#ifdef _WIN32\n#include \n#define fnmatch(x, y, z) (!PathMatchSpec(y, x))\n#else\n#include \nconst int fnmatch_flags = FNM_PATHNAME;\n#endif\n\nignores *root_ignores;\n\n/* TODO: build a huge-ass list of files we want to ignore by default (build cache stuff, pyc files, etc) */\n\nconst char *evil_hardcoded_ignore_files[] = {\n \".\",\n \"..\",\n NULL\n};\n\n/* Warning: changing the first two strings will break skip_vcs_ignores. */\nconst char *ignore_pattern_files[] = {\n \".ignore\",\n \".gitignore\",\n \".git/info/exclude\",\n \".hgignore\",\n NULL\n};\n\nint is_empty(ignores *ig) {\n return (ig->extensions_len + ig->names_len + ig->slash_names_len + ig->regexes_len + ig->slash_regexes_len == 0);\n};\n\nignores *init_ignore(ignores *parent, const char *dirname, const size_t dirname_len) {\n ignores *ig = ag_malloc(sizeof(ignores));\n ig->extensions = NULL;\n ig->extensions_len = 0;\n ig->names = NULL;\n ig->names_len = 0;\n ig->slash_names = NULL;\n ig->slash_names_len = 0;\n ig->regexes = NULL;\n ig->regexes_len = 0;\n ig->invert_regexes = NULL;\n ig->invert_regexes_len = 0;\n ig->slash_regexes = NULL;\n ig->slash_regexes_len = 0;\n ig->dirname = dirname;\n ig->dirname_len = dirname_len;\n\n if (parent && is_empty(parent) && parent->parent) {\n ig->parent = parent->parent;\n } else {\n ig->parent = parent;\n }\n\n if (parent && parent->abs_path_len > 0) {\n ag_asprintf(&(ig->abs_path), \"%s/%s\", parent->abs_path, dirname);\n ig->abs_path_len = parent->abs_path_len + 1 + dirname_len;\n } else if (dirname_len == 1 && dirname[0] == '.') {\n ig->abs_path = ag_malloc(sizeof(char));\n ig->abs_path[0] = '\\0';\n ig->abs_path_len = 0;\n } else {\n ag_asprintf(&(ig->abs_path), \"%s\", dirname);\n ig->abs_path_len = dirname_len;\n }\n return ig;\n}\n\nvoid cleanup_ignore(ignores *ig) {\n if (ig == NULL) {\n return;\n }\n free_strings(ig->extensions, ig->extensions_len);\n free_strings(ig->names, ig->names_len);\n free_strings(ig->slash_names, ig->slash_names_len);\n free_strings(ig->regexes, ig->regexes_len);\n free_strings(ig->invert_regexes, ig->invert_regexes_len);\n free_strings(ig->slash_regexes, ig->slash_regexes_len);\n if (ig->abs_path) {\n free(ig->abs_path);\n }\n free(ig);\n}\n\nvoid add_ignore_pattern(ignores *ig, const char *pattern) {\n int i;\n int pattern_len;\n\n /* Strip off the leading dot so that matches are more likely. */\n if (strncmp(pattern, \"./\", 2) == 0) {\n pattern++;\n }\n\n /* Kill trailing whitespace */\n for (pattern_len = strlen(pattern); pattern_len > 0; pattern_len--) {\n if (!isspace(pattern[pattern_len - 1])) {\n break;\n }\n }\n\n if (pattern_len == 0) {\n log_debug(\"Pattern is empty. Not adding any ignores.\");\n return;\n }\n\n char ***patterns_p;\n size_t *patterns_len;\n if (is_fnmatch(pattern)) {\n if (pattern[0] == '*' && pattern[1] == '.' && strchr(pattern + 2, '.') && !is_fnmatch(pattern + 2)) {\n patterns_p = &(ig->extensions);\n patterns_len = &(ig->extensions_len);\n pattern += 2;\n pattern_len -= 2;\n } else if (pattern[0] == '/') {\n patterns_p = &(ig->slash_regexes);\n patterns_len = &(ig->slash_regexes_len);\n pattern++;\n pattern_len--;\n } else if (pattern[0] == '!') {\n patterns_p = &(ig->invert_regexes);\n patterns_len = &(ig->invert_regexes_len);\n pattern++;\n pattern_len--;\n } else {\n patterns_p = &(ig->regexes);\n patterns_len = &(ig->regexes_len);\n }\n } else {\n if (pattern[0] == '/') {\n patterns_p = &(ig->slash_names);\n patterns_len = &(ig->slash_names_len);\n pattern++;\n pattern_len--;\n } else {\n patterns_p = &(ig->names);\n patterns_len = &(ig->names_len);\n }\n }\n\n ++*patterns_len;\n\n char **patterns;\n\n /* a balanced binary tree is best for performance, but I'm lazy */\n *patterns_p = patterns = ag_realloc(*patterns_p, (*patterns_len) * sizeof(char *));\n /* TODO: de-dupe these patterns */\n for (i = *patterns_len - 1; i > 0; i--) {\n if (strcmp(pattern, patterns[i - 1]) > 0) {\n break;\n }\n patterns[i] = patterns[i - 1];\n }\n patterns[i] = ag_strndup(pattern, pattern_len);\n log_debug(\"added ignore pattern %s to %s\", pattern,\n ig == root_ignores ? \"root ignores\" : ig->abs_path);\n}\n\n/* For loading git/hg ignore patterns */\nvoid load_ignore_patterns(ignores *ig, const char *path) {\n FILE *fp = NULL;\n fp = fopen(path, \"r\");\n if (fp == NULL) {\n log_debug(\"Skipping ignore file %s: not readable\", path);\n return;\n }\n log_debug(\"Loading ignore file %s.\", path);\n\n char *line = NULL;\n ssize_t line_len = 0;\n size_t line_cap = 0;\n\n while ((line_len = getline(&line, &line_cap, fp)) > 0) {\n if (line_len == 0 || line[0] == '\\n' || line[0] == '#') {\n continue;\n }\n if (line[line_len - 1] == '\\n') {\n line[line_len - 1] = '\\0'; /* kill the \\n */\n }\n add_ignore_pattern(ig, line);\n }\n\n free(line);\n fclose(fp);\n}\n\nstatic int ackmate_dir_match(const char *dir_name) {\n if (opts.ackmate_dir_filter == NULL) {\n return 0;\n }\n /* we just care about the match, not where the matches are */\n return pcre_exec(opts.ackmate_dir_filter, NULL, dir_name, strlen(dir_name), 0, 0, NULL, 0);\n}\n\n/* This is the hottest code in Ag. 10-15% of all execution time is spent here */\nstatic int path_ignore_search(const ignores *ig, const char *path, const char *filename) {\n char *temp;\n int temp_start_pos;\n size_t i;\n int match_pos;\n\n match_pos = binary_search(filename, ig->names, 0, ig->names_len);\n if (match_pos >= 0) {\n log_debug(\"file %s ignored because name matches static pattern %s\", filename, ig->names[match_pos]);\n return 1;\n }\n\n ag_asprintf(&temp, \"%s/%s\", path[0] == '.' ? path + 1 : path, filename);\n //ig->abs_path has its leading slash stripped, so we have to strip the leading slash\n //of temp as well\n temp_start_pos = (temp[0] == '/') ? 1 : 0;\n\n if (strncmp(temp + temp_start_pos, ig->abs_path, ig->abs_path_len) == 0) {\n char *slash_filename = temp + temp_start_pos + ig->abs_path_len;\n if (slash_filename[0] == '/') {\n slash_filename++;\n }\n match_pos = binary_search(slash_filename, ig->names, 0, ig->names_len);\n if (match_pos >= 0) {\n log_debug(\"file %s ignored because name matches static pattern %s\", temp, ig->names[match_pos]);\n free(temp);\n return 1;\n }\n\n match_pos = binary_search(slash_filename, ig->slash_names, 0, ig->slash_names_len);\n if (match_pos >= 0) {\n log_debug(\"file %s ignored because name matches slash static pattern %s\", slash_filename, ig->slash_names[match_pos]);\n free(temp);\n return 1;\n }\n\n for (i = 0; i < ig->names_len; i++) {\n char *pos = strstr(slash_filename, ig->names[i]);\n if (pos == slash_filename || (pos && *(pos - 1) == '/')) {\n pos += strlen(ig->names[i]);\n if (*pos == '\\0' || *pos == '/') {\n log_debug(\"file %s ignored because path somewhere matches name %s\", slash_filename, ig->names[i]);\n free(temp);\n return 1;\n }\n }\n log_debug(\"pattern %s doesn't match path %s\", ig->names[i], slash_filename);\n }\n\n for (i = 0; i < ig->slash_regexes_len; i++) {\n if (fnmatch(ig->slash_regexes[i], slash_filename, fnmatch_flags) == 0) {\n log_debug(\"file %s ignored because name matches slash regex pattern %s\", slash_filename, ig->slash_regexes[i]);\n free(temp);\n return 1;\n }\n log_debug(\"pattern %s doesn't match slash file %s\", ig->slash_regexes[i], slash_filename);\n }\n }\n\n for (i = 0; i < ig->invert_regexes_len; i++) {\n if (fnmatch(ig->invert_regexes[i], filename, fnmatch_flags) == 0) {\n log_debug(\"file %s not ignored because name matches regex pattern !%s\", filename, ig->invert_regexes[i]);\n free(temp);\n return 0;\n }\n log_debug(\"pattern !%s doesn't match file %s\", ig->invert_regexes[i], filename);\n }\n\n for (i = 0; i < ig->regexes_len; i++) {\n if (fnmatch(ig->regexes[i], filename, fnmatch_flags) == 0) {\n log_debug(\"file %s ignored because name matches regex pattern %s\", filename, ig->regexes[i]);\n free(temp);\n return 1;\n }\n log_debug(\"pattern %s doesn't match file %s\", ig->regexes[i], filename);\n }\n\n int rv = ackmate_dir_match(temp);\n free(temp);\n return rv;\n}\n\n/* This function is REALLY HOT. It gets called for every file */\nint filename_filter(const char *path, const struct dirent *dir, void *baton) {\n const char *filename = dir->d_name;\n if (!opts.search_hidden_files && filename[0] == '.') {\n return 0;\n }\n\n size_t i;\n for (i = 0; evil_hardcoded_ignore_files[i] != NULL; i++) {\n if (strcmp(filename, evil_hardcoded_ignore_files[i]) == 0) {\n return 0;\n }\n }\n\n if (!opts.follow_symlinks && is_symlink(path, dir)) {\n log_debug(\"File %s ignored becaused it's a symlink\", dir->d_name);\n return 0;\n }\n\n if (is_named_pipe(path, dir)) {\n log_debug(\"%s ignored because it's a named pipe or socket\", path);\n return 0;\n }\n\n if (opts.search_all_files && !opts.path_to_ignore) {\n return 1;\n }\n\n scandir_baton_t *scandir_baton = (scandir_baton_t *)baton;\n const char *path_start = scandir_baton->path_start;\n\n const char *extension = strchr(filename, '.');\n if (extension) {\n if (extension[1]) {\n // The dot is not the last character, extension starts at the next one\n ++extension;\n } else {\n // No extension\n extension = NULL;\n }\n }\n\n#ifdef HAVE_DIRENT_DNAMLEN\n size_t filename_len = dir->d_namlen;\n#else\n size_t filename_len = 0;\n#endif\n\n if (strncmp(filename, \"./\", 2) == 0) {\n#ifndef HAVE_DIRENT_DNAMLEN\n filename_len = strlen(filename);\n#endif\n filename++;\n filename_len--;\n }\n\n const ignores *ig = scandir_baton->ig;\n\n while (ig != NULL) {\n if (extension) {\n int match_pos = binary_search(extension, ig->extensions, 0, ig->extensions_len);\n if (match_pos >= 0) {\n log_debug(\"file %s ignored because name matches extension %s\", filename, ig->extensions[match_pos]);\n return 0;\n }\n }\n\n if (path_ignore_search(ig, path_start, filename)) {\n return 0;\n }\n\n if (is_directory(path, dir)) {\n#ifndef HAVE_DIRENT_DNAMLEN\n if (!filename_len) {\n filename_len = strlen(filename);\n }\n#endif\n if (filename[filename_len - 1] != '/') {\n char *temp;\n ag_asprintf(&temp, \"%s/\", filename);\n int rv = path_ignore_search(ig, path_start, temp);\n free(temp);\n if (rv) {\n return 0;\n }\n }\n }\n ig = ig->parent;\n }\n\n log_debug(\"%s not ignored\", filename);\n return 1;\n}\n" }, { "path": "src/ignore.h", "content": "#ifndef IGNORE_H\n#define IGNORE_H\n\n#include \n#include \n\nstruct ignores {\n char **extensions; /* File extensions to ignore */\n size_t extensions_len;\n\n char **names; /* Non-regex ignore lines. Sorted so we can binary search them. */\n size_t names_len;\n char **slash_names; /* Same but starts with a slash */\n size_t slash_names_len;\n\n char **regexes; /* For patterns that need fnmatch */\n size_t regexes_len;\n char **invert_regexes; /* For \"!\" patterns */\n size_t invert_regexes_len;\n char **slash_regexes;\n size_t slash_regexes_len;\n\n const char *dirname;\n size_t dirname_len;\n char *abs_path;\n size_t abs_path_len;\n\n struct ignores *parent;\n};\ntypedef struct ignores ignores;\n\nextern ignores *root_ignores;\n\nextern const char *evil_hardcoded_ignore_files[];\nextern const char *ignore_pattern_files[];\n\nignores *init_ignore(ignores *parent, const char *dirname, const size_t dirname_len);\nvoid cleanup_ignore(ignores *ig);\n\nvoid add_ignore_pattern(ignores *ig, const char *pattern);\n\nvoid load_ignore_patterns(ignores *ig, const char *path);\n\nint filename_filter(const char *path, const struct dirent *dir, void *baton);\n\nint is_empty(ignores *ig);\n\n#endif\n" }, { "path": "src/lang.c", "content": "#include \n#include \n\n#include \"lang.h\"\n#include \"util.h\"\n\nlang_spec_t langs[] = {\n { \"actionscript\", { \"as\", \"mxml\" } },\n { \"ada\", { \"ada\", \"adb\", \"ads\" } },\n { \"asciidoc\", { \"adoc\", \"ad\", \"asc\", \"asciidoc\" } },\n { \"apl\", { \"apl\" } },\n { \"asm\", { \"asm\", \"s\" } },\n { \"asp\", { \"asp\", \"asa\", \"aspx\", \"asax\", \"ashx\", \"ascx\", \"asmx\" } },\n { \"aspx\", { \"asp\", \"asa\", \"aspx\", \"asax\", \"ashx\", \"ascx\", \"asmx\" } },\n { \"batch\", { \"bat\", \"cmd\" } },\n { \"bazel\", { \"bazel\" } },\n { \"bitbake\", { \"bb\", \"bbappend\", \"bbclass\", \"inc\" } },\n { \"cc\", { \"c\", \"h\", \"xs\" } },\n { \"cfmx\", { \"cfc\", \"cfm\", \"cfml\" } },\n { \"chpl\", { \"chpl\" } },\n { \"clojure\", { \"clj\", \"cljs\", \"cljc\", \"cljx\", \"edn\" } },\n { \"coffee\", { \"coffee\", \"cjsx\" } },\n { \"config\", { \"config\" } },\n { \"coq\", { \"coq\", \"g\", \"v\" } },\n { \"cpp\", { \"cpp\", \"cc\", \"C\", \"cxx\", \"m\", \"hpp\", \"hh\", \"h\", \"H\", \"hxx\", \"tpp\" } },\n { \"crystal\", { \"cr\", \"ecr\" } },\n { \"csharp\", { \"cs\" } },\n { \"cshtml\", { \"cshtml\" } },\n { \"css\", { \"css\" } },\n { \"cython\", { \"pyx\", \"pxd\", \"pxi\" } },\n { \"delphi\", { \"pas\", \"int\", \"dfm\", \"nfm\", \"dof\", \"dpk\", \"dpr\", \"dproj\", \"groupproj\", \"bdsgroup\", \"bdsproj\" } },\n { \"dlang\", { \"d\", \"di\" } },\n { \"dot\", { \"dot\", \"gv\" } },\n { \"dts\", { \"dts\", \"dtsi\" } },\n { \"ebuild\", { \"ebuild\", \"eclass\" } },\n { \"elisp\", { \"el\" } },\n { \"elixir\", { \"ex\", \"eex\", \"exs\" } },\n { \"elm\", { \"elm\" } },\n { \"erlang\", { \"erl\", \"hrl\" } },\n { \"factor\", { \"factor\" } },\n { \"fortran\", { \"f\", \"F\", \"f77\", \"f90\", \"F90\", \"f95\", \"f03\", \"for\", \"ftn\", \"fpp\", \"FPP\" } },\n { \"fsharp\", { \"fs\", \"fsi\", \"fsx\" } },\n { \"gettext\", { \"po\", \"pot\", \"mo\" } },\n { \"glsl\", { \"vert\", \"tesc\", \"tese\", \"geom\", \"frag\", \"comp\" } },\n { \"go\", { \"go\" } },\n { \"gradle\", { \"gradle\" } },\n { \"groovy\", { \"groovy\", \"gtmpl\", \"gpp\", \"grunit\", \"gradle\" } },\n { \"haml\", { \"haml\" } },\n { \"handlebars\", { \"hbs\" } },\n { \"haskell\", { \"hs\", \"hsig\", \"lhs\" } },\n { \"haxe\", { \"hx\" } },\n { \"hh\", { \"h\" } },\n { \"html\", { \"htm\", \"html\", \"shtml\", \"xhtml\" } },\n { \"idris\", { \"idr\", \"ipkg\", \"lidr\" } },\n { \"ini\", { \"ini\" } },\n { \"ipython\", { \"ipynb\" } },\n { \"isabelle\", { \"thy\" } },\n { \"j\", { \"ijs\" } },\n { \"jade\", { \"jade\" } },\n { \"java\", { \"java\", \"properties\" } },\n { \"jinja2\", { \"j2\" } },\n { \"js\", { \"es6\", \"js\", \"jsx\", \"vue\" } },\n { \"json\", { \"json\" } },\n { \"jsp\", { \"jsp\", \"jspx\", \"jhtm\", \"jhtml\", \"jspf\", \"tag\", \"tagf\" } },\n { \"julia\", { \"jl\" } },\n { \"kotlin\", { \"kt\" } },\n { \"less\", { \"less\" } },\n { \"liquid\", { \"liquid\" } },\n { \"lisp\", { \"lisp\", \"lsp\" } },\n { \"log\", { \"log\" } },\n { \"lua\", { \"lua\" } },\n { \"m4\", { \"m4\" } },\n { \"make\", { \"Makefiles\", \"mk\", \"mak\" } },\n { \"mako\", { \"mako\" } },\n { \"markdown\", { \"markdown\", \"mdown\", \"mdwn\", \"mkdn\", \"mkd\", \"md\" } },\n { \"mason\", { \"mas\", \"mhtml\", \"mpl\", \"mtxt\" } },\n { \"matlab\", { \"m\" } },\n { \"mathematica\", { \"m\", \"wl\" } },\n { \"md\", { \"markdown\", \"mdown\", \"mdwn\", \"mkdn\", \"mkd\", \"md\" } },\n { \"mercury\", { \"m\", \"moo\" } },\n { \"naccess\", { \"asa\", \"rsa\" } },\n { \"nim\", { \"nim\" } },\n { \"nix\", { \"nix\" } },\n { \"objc\", { \"m\", \"h\" } },\n { \"objcpp\", { \"mm\", \"h\" } },\n { \"ocaml\", { \"ml\", \"mli\", \"mll\", \"mly\" } },\n { \"octave\", { \"m\" } },\n { \"org\", { \"org\" } },\n { \"parrot\", { \"pir\", \"pasm\", \"pmc\", \"ops\", \"pod\", \"pg\", \"tg\" } },\n { \"pdb\", { \"pdb\" } },\n { \"perl\", { \"pl\", \"pm\", \"pm6\", \"pod\", \"t\" } },\n { \"php\", { \"php\", \"phpt\", \"php3\", \"php4\", \"php5\", \"phtml\" } },\n { \"pike\", { \"pike\", \"pmod\" } },\n { \"plist\", { \"plist\" } },\n { \"plone\", { \"pt\", \"cpt\", \"metadata\", \"cpy\", \"py\", \"xml\", \"zcml\" } },\n { \"powershell\", { \"ps1\" } },\n { \"proto\", { \"proto\" } },\n { \"ps1\", { \"ps1\" } },\n { \"pug\", { \"pug\" } },\n { \"puppet\", { \"pp\" } },\n { \"python\", { \"py\" } },\n { \"qml\", { \"qml\" } },\n { \"racket\", { \"rkt\", \"ss\", \"scm\" } },\n { \"rake\", { \"Rakefile\" } },\n { \"razor\", { \"cshtml\" } },\n { \"restructuredtext\", { \"rst\" } },\n { \"rs\", { \"rs\" } },\n { \"r\", { \"r\", \"R\", \"Rmd\", \"Rnw\", \"Rtex\", \"Rrst\" } },\n { \"rdoc\", { \"rdoc\" } },\n { \"ruby\", { \"rb\", \"rhtml\", \"rjs\", \"rxml\", \"erb\", \"rake\", \"spec\" } },\n { \"rust\", { \"rs\" } },\n { \"salt\", { \"sls\" } },\n { \"sass\", { \"sass\", \"scss\" } },\n { \"scala\", { \"scala\" } },\n { \"scheme\", { \"scm\", \"ss\" } },\n { \"shell\", { \"sh\", \"bash\", \"csh\", \"tcsh\", \"ksh\", \"zsh\", \"fish\" } },\n { \"smalltalk\", { \"st\" } },\n { \"sml\", { \"sml\", \"fun\", \"mlb\", \"sig\" } },\n { \"sql\", { \"sql\", \"ctl\" } },\n { \"stata\", { \"do\", \"ado\" } },\n { \"stylus\", { \"styl\" } },\n { \"swift\", { \"swift\" } },\n { \"tcl\", { \"tcl\", \"itcl\", \"itk\" } },\n { \"terraform\", { \"tf\", \"tfvars\" } },\n { \"tex\", { \"tex\", \"cls\", \"sty\" } },\n { \"thrift\", { \"thrift\" } },\n { \"tla\", { \"tla\" } },\n { \"tt\", { \"tt\", \"tt2\", \"ttml\" } },\n { \"toml\", { \"toml\" } },\n { \"ts\", { \"ts\", \"tsx\" } },\n { \"twig\", { \"twig\" } },\n { \"vala\", { \"vala\", \"vapi\" } },\n { \"vb\", { \"bas\", \"cls\", \"frm\", \"ctl\", \"vb\", \"resx\" } },\n { \"velocity\", { \"vm\", \"vtl\", \"vsl\" } },\n { \"verilog\", { \"v\", \"vh\", \"sv\", \"svh\" } },\n { \"vhdl\", { \"vhd\", \"vhdl\" } },\n { \"vim\", { \"vim\" } },\n { \"vue\", { \"vue\" } },\n { \"wix\", { \"wxi\", \"wxs\" } },\n { \"wsdl\", { \"wsdl\" } },\n { \"wadl\", { \"wadl\" } },\n { \"xml\", { \"xml\", \"dtd\", \"xsl\", \"xslt\", \"xsd\", \"ent\", \"tld\", \"plist\", \"wsdl\" } },\n { \"yaml\", { \"yaml\", \"yml\" } },\n { \"zeek\", { \"zeek\", \"bro\", \"bif\" } },\n { \"zephir\", { \"zep\" } }\n};\n\nsize_t get_lang_count() {\n return sizeof(langs) / sizeof(lang_spec_t);\n}\n\nchar *make_lang_regex(char *ext_array, size_t num_exts) {\n int regex_capacity = 100;\n char *regex = ag_malloc(regex_capacity);\n int regex_length = 3;\n int subsequent = 0;\n char *extension;\n size_t i;\n\n strcpy(regex, \"\\\\.(\");\n\n for (i = 0; i < num_exts; ++i) {\n extension = ext_array + i * SINGLE_EXT_LEN;\n int extension_length = strlen(extension);\n while (regex_length + extension_length + 3 + subsequent > regex_capacity) {\n regex_capacity *= 2;\n regex = ag_realloc(regex, regex_capacity);\n }\n if (subsequent) {\n regex[regex_length++] = '|';\n } else {\n subsequent = 1;\n }\n strcpy(regex + regex_length, extension);\n regex_length += extension_length;\n }\n\n regex[regex_length++] = ')';\n regex[regex_length++] = '$';\n regex[regex_length++] = 0;\n return regex;\n}\n\nsize_t combine_file_extensions(size_t *extension_index, size_t len, char **exts) {\n /* Keep it fixed as 100 for the reason that if you have more than 100\n * file types to search, you'd better search all the files.\n * */\n size_t ext_capacity = 100;\n (*exts) = (char *)ag_malloc(ext_capacity * SINGLE_EXT_LEN);\n memset((*exts), 0, ext_capacity * SINGLE_EXT_LEN);\n size_t num_of_extensions = 0;\n\n size_t i;\n for (i = 0; i < len; ++i) {\n size_t j = 0;\n const char *ext = langs[extension_index[i]].extensions[j];\n do {\n if (num_of_extensions == ext_capacity) {\n break;\n }\n char *pos = (*exts) + num_of_extensions * SINGLE_EXT_LEN;\n strncpy(pos, ext, strlen(ext));\n ++num_of_extensions;\n ext = langs[extension_index[i]].extensions[++j];\n } while (ext);\n }\n\n return num_of_extensions;\n}\n" }, { "path": "src/lang.h", "content": "#ifndef LANG_H\n#define LANG_H\n\n#define MAX_EXTENSIONS 12\n#define SINGLE_EXT_LEN 20\n\ntypedef struct {\n const char *name;\n const char *extensions[MAX_EXTENSIONS];\n} lang_spec_t;\n\nextern lang_spec_t langs[];\n\n/**\n Return the language count.\n */\nsize_t get_lang_count(void);\n\n/**\nConvert a NULL-terminated array of language extensions\ninto a regular expression of the form \\.(extension1|extension2...)$\n\nCaller is responsible for freeing the returned string.\n*/\nchar *make_lang_regex(char *ext_array, size_t num_exts);\n\n\n/**\nCombine multiple file type extensions into one array.\n\nThe combined result is returned through *exts*;\n*exts* is one-dimension array, which can contain up to 100 extensions;\nThe number of extensions that *exts* actually contain is returned.\n*/\nsize_t combine_file_extensions(size_t *extension_index, size_t len, char **exts);\n#endif\n" }, { "path": "src/log.c", "content": "#include \n#include \n\n#include \"log.h\"\n#include \"util.h\"\n\npthread_mutex_t print_mtx = PTHREAD_MUTEX_INITIALIZER;\nstatic enum log_level log_threshold = LOG_LEVEL_ERR;\n\nvoid set_log_level(enum log_level threshold) {\n log_threshold = threshold;\n}\n\nvoid log_debug(const char *fmt, ...) {\n va_list args;\n va_start(args, fmt);\n vplog(LOG_LEVEL_DEBUG, fmt, args);\n va_end(args);\n}\n\nvoid log_msg(const char *fmt, ...) {\n va_list args;\n va_start(args, fmt);\n vplog(LOG_LEVEL_MSG, fmt, args);\n va_end(args);\n}\n\nvoid log_warn(const char *fmt, ...) {\n va_list args;\n va_start(args, fmt);\n vplog(LOG_LEVEL_WARN, fmt, args);\n va_end(args);\n}\n\nvoid log_err(const char *fmt, ...) {\n va_list args;\n va_start(args, fmt);\n vplog(LOG_LEVEL_ERR, fmt, args);\n va_end(args);\n}\n\nvoid vplog(const unsigned int level, const char *fmt, va_list args) {\n if (level < log_threshold) {\n return;\n }\n\n pthread_mutex_lock(&print_mtx);\n FILE *stream = out_fd;\n\n switch (level) {\n case LOG_LEVEL_DEBUG:\n fprintf(stream, \"DEBUG: \");\n break;\n case LOG_LEVEL_MSG:\n fprintf(stream, \"MSG: \");\n break;\n case LOG_LEVEL_WARN:\n fprintf(stream, \"WARN: \");\n break;\n case LOG_LEVEL_ERR:\n stream = stderr;\n fprintf(stream, \"ERR: \");\n break;\n }\n\n vfprintf(stream, fmt, args);\n fprintf(stream, \"\\n\");\n pthread_mutex_unlock(&print_mtx);\n}\n\nvoid plog(const unsigned int level, const char *fmt, ...) {\n va_list args;\n va_start(args, fmt);\n vplog(level, fmt, args);\n va_end(args);\n}\n" }, { "path": "src/log.h", "content": "#ifndef LOG_H\n#define LOG_H\n\n#include \n\n#include \"config.h\"\n\n#ifdef HAVE_PTHREAD_H\n#include \n#endif\n\nextern pthread_mutex_t print_mtx;\n\nenum log_level {\n LOG_LEVEL_DEBUG = 10,\n LOG_LEVEL_MSG = 20,\n LOG_LEVEL_WARN = 30,\n LOG_LEVEL_ERR = 40,\n LOG_LEVEL_NONE = 100\n};\n\nvoid set_log_level(enum log_level threshold);\n\nvoid log_debug(const char *fmt, ...);\nvoid log_msg(const char *fmt, ...);\nvoid log_warn(const char *fmt, ...);\nvoid log_err(const char *fmt, ...);\n\nvoid vplog(const unsigned int level, const char *fmt, va_list args);\nvoid plog(const unsigned int level, const char *fmt, ...);\n\n#endif\n" }, { "path": "src/main.c", "content": "#include \n#include \n#include \n#include \n#include \n#include \n#include \n#ifdef _WIN32\n#include \n#endif\n\n#include \"config.h\"\n\n#ifdef HAVE_SYS_CPUSET_H\n#include \n#endif\n\n#ifdef HAVE_PTHREAD_H\n#include \n#endif\n\n#if defined(HAVE_PTHREAD_SETAFFINITY_NP) && defined(__FreeBSD__)\n#include \n#endif\n\n#include \"log.h\"\n#include \"options.h\"\n#include \"search.h\"\n#include \"util.h\"\n\ntypedef struct {\n pthread_t thread;\n int id;\n} worker_t;\n\nint main(int argc, char **argv) {\n char **base_paths = NULL;\n char **paths = NULL;\n int i;\n int pcre_opts = PCRE_MULTILINE;\n int study_opts = 0;\n worker_t *workers = NULL;\n int workers_len;\n int num_cores;\n\n#ifdef HAVE_PLEDGE\n if (pledge(\"stdio rpath proc exec\", NULL) == -1) {\n die(\"pledge: %s\", strerror(errno));\n }\n#endif\n\n set_log_level(LOG_LEVEL_WARN);\n\n work_queue = NULL;\n work_queue_tail = NULL;\n root_ignores = init_ignore(NULL, \"\", 0);\n out_fd = stdout;\n\n parse_options(argc, argv, &base_paths, &paths);\n log_debug(\"PCRE Version: %s\", pcre_version());\n if (opts.stats) {\n memset(&stats, 0, sizeof(stats));\n gettimeofday(&(stats.time_start), NULL);\n }\n\n#ifdef USE_PCRE_JIT\n int has_jit = 0;\n pcre_config(PCRE_CONFIG_JIT, &has_jit);\n if (has_jit) {\n study_opts |= PCRE_STUDY_JIT_COMPILE;\n }\n#endif\n\n#ifdef _WIN32\n {\n SYSTEM_INFO si;\n GetSystemInfo(&si);\n num_cores = si.dwNumberOfProcessors;\n }\n#else\n num_cores = (int)sysconf(_SC_NPROCESSORS_ONLN);\n#endif\n\n workers_len = num_cores < 8 ? num_cores : 8;\n if (opts.literal) {\n workers_len--;\n }\n if (opts.workers) {\n workers_len = opts.workers;\n }\n if (workers_len < 1) {\n workers_len = 1;\n }\n\n log_debug(\"Using %i workers\", workers_len);\n done_adding_files = FALSE;\n workers = ag_calloc(workers_len, sizeof(worker_t));\n if (pthread_cond_init(&files_ready, NULL)) {\n die(\"pthread_cond_init failed!\");\n }\n if (pthread_mutex_init(&print_mtx, NULL)) {\n die(\"pthread_mutex_init failed!\");\n }\n if (opts.stats && pthread_mutex_init(&stats_mtx, NULL)) {\n die(\"pthread_mutex_init failed!\");\n }\n if (pthread_mutex_init(&work_queue_mtx, NULL)) {\n die(\"pthread_mutex_init failed!\");\n }\n\n if (opts.casing == CASE_SMART) {\n opts.casing = is_lowercase(opts.query) ? CASE_INSENSITIVE : CASE_SENSITIVE;\n }\n\n if (opts.literal) {\n if (opts.casing == CASE_INSENSITIVE) {\n /* Search routine needs the query to be lowercase */\n char *c = opts.query;\n for (; *c != '\\0'; ++c) {\n *c = (char)tolower(*c);\n }\n }\n generate_alpha_skip(opts.query, opts.query_len, alpha_skip_lookup, opts.casing == CASE_SENSITIVE);\n find_skip_lookup = NULL;\n generate_find_skip(opts.query, opts.query_len, &find_skip_lookup, opts.casing == CASE_SENSITIVE);\n generate_hash(opts.query, opts.query_len, h_table, opts.casing == CASE_SENSITIVE);\n if (opts.word_regexp) {\n init_wordchar_table();\n opts.literal_starts_wordchar = is_wordchar(opts.query[0]);\n opts.literal_ends_wordchar = is_wordchar(opts.query[opts.query_len - 1]);\n }\n } else {\n if (opts.casing == CASE_INSENSITIVE) {\n pcre_opts |= PCRE_CASELESS;\n }\n if (opts.word_regexp) {\n char *word_regexp_query;\n ag_asprintf(&word_regexp_query, \"\\\\b(?:%s)\\\\b\", opts.query);\n free(opts.query);\n opts.query = word_regexp_query;\n opts.query_len = strlen(opts.query);\n }\n compile_study(&opts.re, &opts.re_extra, opts.query, pcre_opts, study_opts);\n }\n\n if (opts.search_stream) {\n search_stream(stdin, \"\");\n } else {\n for (i = 0; i < workers_len; i++) {\n workers[i].id = i;\n int rv = pthread_create(&(workers[i].thread), NULL, &search_file_worker, &(workers[i].id));\n if (rv != 0) {\n die(\"Error in pthread_create(): %s\", strerror(rv));\n }\n#if defined(HAVE_PTHREAD_SETAFFINITY_NP) && (defined(USE_CPU_SET) || defined(HAVE_SYS_CPUSET_H))\n if (opts.use_thread_affinity) {\n#if defined(__linux__) || defined(__midipix__)\n cpu_set_t cpu_set;\n#elif __FreeBSD__\n cpuset_t cpu_set;\n#endif\n CPU_ZERO(&cpu_set);\n CPU_SET(i % num_cores, &cpu_set);\n rv = pthread_setaffinity_np(workers[i].thread, sizeof(cpu_set), &cpu_set);\n if (rv) {\n log_err(\"Error in pthread_setaffinity_np(): %s\", strerror(rv));\n log_err(\"Performance may be affected. Use --noaffinity to suppress this message.\");\n } else {\n log_debug(\"Thread %i set to CPU %i\", i, i);\n }\n } else {\n log_debug(\"Thread affinity disabled.\");\n }\n#else\n log_debug(\"No CPU affinity support.\");\n#endif\n }\n\n#ifdef HAVE_PLEDGE\n if (pledge(\"stdio rpath\", NULL) == -1) {\n die(\"pledge: %s\", strerror(errno));\n }\n#endif\n for (i = 0; paths[i] != NULL; i++) {\n log_debug(\"searching path %s for %s\", paths[i], opts.query);\n symhash = NULL;\n ignores *ig = init_ignore(root_ignores, \"\", 0);\n struct stat s = { .st_dev = 0 };\n#ifndef _WIN32\n /* The device is ignored if opts.one_dev is false, so it's fine\n * to leave it at the default 0\n */\n if (opts.one_dev && lstat(paths[i], &s) == -1) {\n log_err(\"Failed to get device information for path %s. Skipping...\", paths[i]);\n }\n#endif\n search_dir(ig, base_paths[i], paths[i], 0, s.st_dev);\n cleanup_ignore(ig);\n }\n pthread_mutex_lock(&work_queue_mtx);\n done_adding_files = TRUE;\n pthread_cond_broadcast(&files_ready);\n pthread_mutex_unlock(&work_queue_mtx);\n for (i = 0; i < workers_len; i++) {\n if (pthread_join(workers[i].thread, NULL)) {\n die(\"pthread_join failed!\");\n }\n }\n }\n\n if (opts.stats) {\n gettimeofday(&(stats.time_end), NULL);\n double time_diff = ((long)stats.time_end.tv_sec * 1000000 + stats.time_end.tv_usec) -\n ((long)stats.time_start.tv_sec * 1000000 + stats.time_start.tv_usec);\n time_diff /= 1000000;\n printf(\"%zu matches\\n%zu files contained matches\\n%zu files searched\\n%zu bytes searched\\n%f seconds\\n\",\n stats.total_matches, stats.total_file_matches, stats.total_files, stats.total_bytes, time_diff);\n pthread_mutex_destroy(&stats_mtx);\n }\n\n if (opts.pager) {\n pclose(out_fd);\n }\n cleanup_options();\n pthread_cond_destroy(&files_ready);\n pthread_mutex_destroy(&work_queue_mtx);\n pthread_mutex_destroy(&print_mtx);\n cleanup_ignore(root_ignores);\n free(workers);\n for (i = 0; paths[i] != NULL; i++) {\n free(paths[i]);\n free(base_paths[i]);\n }\n free(base_paths);\n free(paths);\n if (find_skip_lookup) {\n free(find_skip_lookup);\n }\n return !opts.match_found;\n}\n" }, { "path": "src/options.c", "content": "#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"config.h\"\n#include \"ignore.h\"\n#include \"lang.h\"\n#include \"log.h\"\n#include \"options.h\"\n#include \"print.h\"\n#include \"util.h\"\n\nconst char *color_line_number = \"\\033[1;33m\"; /* bold yellow */\nconst char *color_match = \"\\033[30;43m\"; /* black with yellow background */\nconst char *color_path = \"\\033[1;32m\"; /* bold green */\n\ncli_options opts;\n\n/* TODO: try to obey out_fd? */\nvoid usage(void) {\n printf(\"\\n\");\n printf(\"Usage: ag [FILE-TYPE] [OPTIONS] PATTERN [PATH]\\n\\n\");\n\n printf(\" Recursively search for PATTERN in PATH.\\n\");\n printf(\" Like grep or ack, but faster.\\n\\n\");\n\n printf(\"Example:\\n ag -i foo /bar/\\n\\n\");\n\n printf(\"\\\nOutput Options:\\n\\\n --ackmate Print results in AckMate-parseable format\\n\\\n -A --after [LINES] Print lines after match (Default: 2)\\n\\\n -B --before [LINES] Print lines before match (Default: 2)\\n\\\n --[no]break Print newlines between matches in different files\\n\\\n (Enabled by default)\\n\\\n -c --count Only print the number of matches in each file.\\n\\\n (This often differs from the number of matching lines)\\n\\\n --[no]color Print color codes in results (Enabled by default)\\n\\\n --color-line-number Color codes for line numbers (Default: 1;33)\\n\\\n --color-match Color codes for result match numbers (Default: 30;43)\\n\\\n --color-path Color codes for path names (Default: 1;32)\\n\\\n\");\n#ifdef _WIN32\n printf(\"\\\n --color-win-ansi Use ansi colors on Windows even where we can use native\\n\\\n (pager/pipe colors are ansi regardless) (Default: off)\\n\\\n\");\n#endif\n printf(\"\\\n --column Print column numbers in results\\n\\\n --[no]filename Print file names (Enabled unless searching a single file)\\n\\\n -H --[no]heading Print file names before each file's matches\\n\\\n (Enabled by default)\\n\\\n -C --context [LINES] Print lines before and after matches (Default: 2)\\n\\\n --[no]group Same as --[no]break --[no]heading\\n\\\n -g --filename-pattern PATTERN\\n\\\n Print filenames matching PATTERN\\n\\\n -l --files-with-matches Only print filenames that contain matches\\n\\\n (don't print the matching lines)\\n\\\n -L --files-without-matches\\n\\\n Only print filenames that don't contain matches\\n\\\n --print-all-files Print headings for all files searched, even those that\\n\\\n don't contain matches\\n\\\n --[no]numbers Print line numbers. Default is to omit line numbers\\n\\\n when searching streams\\n\\\n -o --only-matching Prints only the matching part of the lines\\n\\\n --print-long-lines Print matches on very long lines (Default: >2k characters)\\n\\\n --passthrough When searching a stream, print all lines even if they\\n\\\n don't match\\n\\\n --silent Suppress all log messages, including errors\\n\\\n --stats Print stats (files scanned, time taken, etc.)\\n\\\n --stats-only Print stats and nothing else.\\n\\\n (Same as --count when searching a single file)\\n\\\n --vimgrep Print results like vim's :vimgrep /pattern/g would\\n\\\n (it reports every match on the line)\\n\\\n -0 --null --print0 Separate filenames with null (for 'xargs -0')\\n\\\n\\n\\\nSearch Options:\\n\\\n -a --all-types Search all files (doesn't include hidden files\\n\\\n or patterns from ignore files)\\n\\\n -D --debug Ridiculous debugging (probably not useful)\\n\\\n --depth NUM Search up to NUM directories deep (Default: 25)\\n\\\n -f --follow Follow symlinks\\n\\\n -F --fixed-strings Alias for --literal for compatibility with grep\\n\\\n -G --file-search-regex PATTERN Limit search to filenames matching PATTERN\\n\\\n --hidden Search hidden files (obeys .*ignore files)\\n\\\n -i --ignore-case Match case insensitively\\n\\\n --ignore PATTERN Ignore files/directories matching PATTERN\\n\\\n (literal file/directory names also allowed)\\n\\\n --ignore-dir NAME Alias for --ignore for compatibility with ack.\\n\\\n -m --max-count NUM Skip the rest of a file after NUM matches (Default: 10,000)\\n\\\n --one-device Don't follow links to other devices.\\n\\\n -p --path-to-ignore STRING\\n\\\n Use .ignore file at STRING\\n\\\n -Q --literal Don't parse PATTERN as a regular expression\\n\\\n -s --case-sensitive Match case sensitively\\n\\\n -S --smart-case Match case insensitively unless PATTERN contains\\n\\\n uppercase characters (Enabled by default)\\n\\\n --search-binary Search binary files for matches\\n\\\n -t --all-text Search all text files (doesn't include hidden files)\\n\\\n -u --unrestricted Search all files (ignore .ignore, .gitignore, etc.;\\n\\\n searches binary and hidden files as well)\\n\\\n -U --skip-vcs-ignores Ignore VCS ignore files\\n\\\n (.gitignore, .hgignore; still obey .ignore)\\n\\\n -v --invert-match\\n\\\n -w --word-regexp Only match whole words\\n\\\n -W --width NUM Truncate match lines after NUM characters\\n\\\n -z --search-zip Search contents of compressed (e.g., gzip) files\\n\\\n\\n\");\n printf(\"File Types:\\n\\\nThe search can be restricted to certain types of files. Example:\\n\\\n ag --html needle\\n\\\n - Searches for 'needle' in files with suffix .htm, .html, .shtml or .xhtml.\\n\\\n\\n\\\nFor a list of supported file types run:\\n\\\n ag --list-file-types\\n\\n\\\nag was originally created by Geoff Greer. More information (and the latest release)\\n\\\ncan be found at http://geoff.greer.fm/ag\\n\");\n}\n\nvoid print_version(void) {\n char jit = '-';\n char lzma = '-';\n char zlib = '-';\n\n#ifdef USE_PCRE_JIT\n jit = '+';\n#endif\n#ifdef HAVE_LZMA_H\n lzma = '+';\n#endif\n#ifdef HAVE_ZLIB_H\n zlib = '+';\n#endif\n\n printf(\"ag version %s\\n\\n\", PACKAGE_VERSION);\n printf(\"Features:\\n\");\n printf(\" %cjit %clzma %czlib\\n\", jit, lzma, zlib);\n}\n\nvoid init_options(void) {\n char *term = getenv(\"TERM\");\n\n memset(&opts, 0, sizeof(opts));\n opts.casing = CASE_DEFAULT;\n opts.color = TRUE;\n if (term && !strcmp(term, \"dumb\")) {\n opts.color = FALSE;\n }\n opts.color_win_ansi = FALSE;\n opts.max_matches_per_file = 0;\n opts.max_search_depth = DEFAULT_MAX_SEARCH_DEPTH;\n#if defined(__APPLE__) || defined(__MACH__)\n /* mamp() is slower than normal read() on macos. default to off */\n opts.mmap = FALSE;\n#else\n opts.mmap = TRUE;\n#endif\n opts.multiline = TRUE;\n opts.width = 0;\n opts.path_sep = '\\n';\n opts.print_break = TRUE;\n opts.print_path = PATH_PRINT_DEFAULT;\n opts.print_all_paths = FALSE;\n opts.print_line_numbers = TRUE;\n opts.recurse_dirs = TRUE;\n opts.color_path = ag_strdup(color_path);\n opts.color_match = ag_strdup(color_match);\n opts.color_line_number = ag_strdup(color_line_number);\n opts.use_thread_affinity = TRUE;\n}\n\nvoid cleanup_options(void) {\n free(opts.color_path);\n free(opts.color_match);\n free(opts.color_line_number);\n\n if (opts.query) {\n free(opts.query);\n }\n\n pcre_free(opts.re);\n if (opts.re_extra) {\n /* Using pcre_free_study on pcre_extra* can segfault on some versions of PCRE */\n pcre_free(opts.re_extra);\n }\n\n if (opts.ackmate_dir_filter) {\n pcre_free(opts.ackmate_dir_filter);\n }\n if (opts.ackmate_dir_filter_extra) {\n pcre_free(opts.ackmate_dir_filter_extra);\n }\n\n if (opts.file_search_regex) {\n pcre_free(opts.file_search_regex);\n }\n if (opts.file_search_regex_extra) {\n pcre_free(opts.file_search_regex_extra);\n }\n}\n\nvoid parse_options(int argc, char **argv, char **base_paths[], char **paths[]) {\n int ch;\n size_t i;\n int path_len = 0;\n int base_path_len = 0;\n int useless = 0;\n int group = 1;\n int help = 0;\n int version = 0;\n int list_file_types = 0;\n int opt_index = 0;\n char *num_end;\n const char *home_dir = getenv(\"HOME\");\n char *ignore_file_path = NULL;\n int accepts_query = 1;\n int needs_query = 1;\n struct stat statbuf;\n int rv;\n size_t lang_count;\n size_t lang_num = 0;\n int has_filetype = 0;\n\n size_t longopts_len, full_len;\n option_t *longopts;\n char *lang_regex = NULL;\n size_t *ext_index = NULL;\n char *extensions = NULL;\n size_t num_exts = 0;\n\n init_options();\n\n option_t base_longopts[] = {\n { \"ackmate\", no_argument, &opts.ackmate, 1 },\n { \"ackmate-dir-filter\", required_argument, NULL, 0 },\n { \"affinity\", no_argument, &opts.use_thread_affinity, 1 },\n { \"after\", optional_argument, NULL, 'A' },\n { \"all-text\", no_argument, NULL, 't' },\n { \"all-types\", no_argument, NULL, 'a' },\n { \"before\", optional_argument, NULL, 'B' },\n { \"break\", no_argument, &opts.print_break, 1 },\n { \"case-sensitive\", no_argument, NULL, 's' },\n { \"color\", no_argument, &opts.color, 1 },\n { \"color-line-number\", required_argument, NULL, 0 },\n { \"color-match\", required_argument, NULL, 0 },\n { \"color-path\", required_argument, NULL, 0 },\n { \"color-win-ansi\", no_argument, &opts.color_win_ansi, TRUE },\n { \"column\", no_argument, &opts.column, 1 },\n { \"context\", optional_argument, NULL, 'C' },\n { \"count\", no_argument, NULL, 'c' },\n { \"debug\", no_argument, NULL, 'D' },\n { \"depth\", required_argument, NULL, 0 },\n { \"filename\", no_argument, NULL, 0 },\n { \"filename-pattern\", required_argument, NULL, 'g' },\n { \"file-search-regex\", required_argument, NULL, 'G' },\n { \"files-with-matches\", no_argument, NULL, 'l' },\n { \"files-without-matches\", no_argument, NULL, 'L' },\n { \"fixed-strings\", no_argument, NULL, 'F' },\n { \"follow\", no_argument, &opts.follow_symlinks, 1 },\n { \"group\", no_argument, &group, 1 },\n { \"heading\", no_argument, &opts.print_path, PATH_PRINT_TOP },\n { \"help\", no_argument, NULL, 'h' },\n { \"hidden\", no_argument, &opts.search_hidden_files, 1 },\n { \"ignore\", required_argument, NULL, 0 },\n { \"ignore-case\", no_argument, NULL, 'i' },\n { \"ignore-dir\", required_argument, NULL, 0 },\n { \"invert-match\", no_argument, NULL, 'v' },\n /* deprecated for --numbers. Remove eventually. */\n { \"line-numbers\", no_argument, &opts.print_line_numbers, 2 },\n { \"list-file-types\", no_argument, &list_file_types, 1 },\n { \"literal\", no_argument, NULL, 'Q' },\n { \"match\", no_argument, &useless, 0 },\n { \"max-count\", required_argument, NULL, 'm' },\n { \"mmap\", no_argument, &opts.mmap, TRUE },\n { \"multiline\", no_argument, &opts.multiline, TRUE },\n /* Accept both --no-* and --no* forms for convenience/BC */\n { \"no-affinity\", no_argument, &opts.use_thread_affinity, 0 },\n { \"noaffinity\", no_argument, &opts.use_thread_affinity, 0 },\n { \"no-break\", no_argument, &opts.print_break, 0 },\n { \"nobreak\", no_argument, &opts.print_break, 0 },\n { \"no-color\", no_argument, &opts.color, 0 },\n { \"nocolor\", no_argument, &opts.color, 0 },\n { \"no-filename\", no_argument, NULL, 0 },\n { \"nofilename\", no_argument, NULL, 0 },\n { \"no-follow\", no_argument, &opts.follow_symlinks, 0 },\n { \"nofollow\", no_argument, &opts.follow_symlinks, 0 },\n { \"no-group\", no_argument, &group, 0 },\n { \"nogroup\", no_argument, &group, 0 },\n { \"no-heading\", no_argument, &opts.print_path, PATH_PRINT_EACH_LINE },\n { \"noheading\", no_argument, &opts.print_path, PATH_PRINT_EACH_LINE },\n { \"no-mmap\", no_argument, &opts.mmap, FALSE },\n { \"nommap\", no_argument, &opts.mmap, FALSE },\n { \"no-multiline\", no_argument, &opts.multiline, FALSE },\n { \"nomultiline\", no_argument, &opts.multiline, FALSE },\n { \"no-numbers\", no_argument, &opts.print_line_numbers, FALSE },\n { \"nonumbers\", no_argument, &opts.print_line_numbers, FALSE },\n { \"no-pager\", no_argument, NULL, 0 },\n { \"nopager\", no_argument, NULL, 0 },\n { \"no-recurse\", no_argument, NULL, 'n' },\n { \"norecurse\", no_argument, NULL, 'n' },\n { \"null\", no_argument, NULL, '0' },\n { \"numbers\", no_argument, &opts.print_line_numbers, 2 },\n { \"only-matching\", no_argument, NULL, 'o' },\n { \"one-device\", no_argument, &opts.one_dev, 1 },\n { \"pager\", required_argument, NULL, 0 },\n { \"parallel\", no_argument, &opts.parallel, 1 },\n { \"passthrough\", no_argument, &opts.passthrough, 1 },\n { \"passthru\", no_argument, &opts.passthrough, 1 },\n { \"path-to-ignore\", required_argument, NULL, 'p' },\n { \"print0\", no_argument, NULL, '0' },\n { \"print-all-files\", no_argument, NULL, 0 },\n { \"print-long-lines\", no_argument, &opts.print_long_lines, 1 },\n { \"recurse\", no_argument, NULL, 'r' },\n { \"search-binary\", no_argument, &opts.search_binary_files, 1 },\n { \"search-files\", no_argument, &opts.search_stream, 0 },\n { \"search-zip\", no_argument, &opts.search_zip_files, 1 },\n { \"silent\", no_argument, NULL, 0 },\n { \"skip-vcs-ignores\", no_argument, NULL, 'U' },\n { \"smart-case\", no_argument, NULL, 'S' },\n { \"stats\", no_argument, &opts.stats, 1 },\n { \"stats-only\", no_argument, NULL, 0 },\n { \"unrestricted\", no_argument, NULL, 'u' },\n { \"version\", no_argument, &version, 1 },\n { \"vimgrep\", no_argument, &opts.vimgrep, 1 },\n { \"width\", required_argument, NULL, 'W' },\n { \"word-regexp\", no_argument, NULL, 'w' },\n { \"workers\", required_argument, NULL, 0 },\n };\n\n lang_count = get_lang_count();\n longopts_len = (sizeof(base_longopts) / sizeof(option_t));\n full_len = (longopts_len + lang_count + 1);\n longopts = ag_malloc(full_len * sizeof(option_t));\n memcpy(longopts, base_longopts, sizeof(base_longopts));\n ext_index = (size_t *)ag_malloc(sizeof(size_t) * lang_count);\n memset(ext_index, 0, sizeof(size_t) * lang_count);\n\n for (i = 0; i < lang_count; i++) {\n option_t opt = { langs[i].name, no_argument, NULL, 0 };\n longopts[i + longopts_len] = opt;\n }\n longopts[full_len - 1] = (option_t){ NULL, 0, NULL, 0 };\n\n if (argc < 2) {\n usage();\n cleanup_ignore(root_ignores);\n cleanup_options();\n exit(1);\n }\n\n rv = fstat(fileno(stdin), &statbuf);\n if (rv == 0) {\n if (S_ISFIFO(statbuf.st_mode) || S_ISREG(statbuf.st_mode)) {\n opts.search_stream = 1;\n }\n }\n\n /* If we're not outputting to a terminal. change output to:\n * turn off colors\n * print filenames on every line\n */\n if (!isatty(fileno(stdout))) {\n opts.color = 0;\n group = 0;\n\n /* Don't search the file that stdout is redirected to */\n rv = fstat(fileno(stdout), &statbuf);\n if (rv != 0) {\n die(\"Error fstat()ing stdout\");\n }\n opts.stdout_inode = statbuf.st_ino;\n }\n\n char *file_search_regex = NULL;\n while ((ch = getopt_long(argc, argv, \"A:aB:C:cDG:g:FfHhiLlm:nop:QRrSsvVtuUwW:z0\", longopts, &opt_index)) != -1) {\n switch (ch) {\n case 'A':\n if (optarg) {\n opts.after = strtol(optarg, &num_end, 10);\n if (num_end == optarg || *num_end != '\\0' || errno == ERANGE) {\n /* This arg must be the search string instead of the after length */\n optind--;\n opts.after = DEFAULT_AFTER_LEN;\n }\n } else {\n opts.after = DEFAULT_AFTER_LEN;\n }\n break;\n case 'a':\n opts.search_all_files = 1;\n opts.search_binary_files = 1;\n break;\n case 'B':\n if (optarg) {\n opts.before = strtol(optarg, &num_end, 10);\n if (num_end == optarg || *num_end != '\\0' || errno == ERANGE) {\n /* This arg must be the search string instead of the before length */\n optind--;\n opts.before = DEFAULT_BEFORE_LEN;\n }\n } else {\n opts.before = DEFAULT_BEFORE_LEN;\n }\n break;\n case 'C':\n if (optarg) {\n opts.context = strtol(optarg, &num_end, 10);\n if (num_end == optarg || *num_end != '\\0' || errno == ERANGE) {\n /* This arg must be the search string instead of the context length */\n optind--;\n opts.context = DEFAULT_CONTEXT_LEN;\n }\n } else {\n opts.context = DEFAULT_CONTEXT_LEN;\n }\n break;\n case 'c':\n opts.print_count = 1;\n opts.print_filename_only = 1;\n break;\n case 'D':\n set_log_level(LOG_LEVEL_DEBUG);\n break;\n case 'f':\n opts.follow_symlinks = 1;\n break;\n case 'g':\n needs_query = accepts_query = 0;\n opts.match_files = 1;\n /* fall through */\n case 'G':\n if (file_search_regex) {\n log_err(\"File search regex (-g or -G) already specified.\");\n usage();\n exit(1);\n }\n file_search_regex = ag_strdup(optarg);\n break;\n case 'H':\n opts.print_path = PATH_PRINT_TOP;\n break;\n case 'h':\n help = 1;\n break;\n case 'i':\n opts.casing = CASE_INSENSITIVE;\n break;\n case 'L':\n opts.print_nonmatching_files = 1;\n opts.print_path = PATH_PRINT_TOP;\n break;\n case 'l':\n needs_query = 0;\n opts.print_filename_only = 1;\n opts.print_path = PATH_PRINT_TOP;\n break;\n case 'm':\n opts.max_matches_per_file = atoi(optarg);\n break;\n case 'n':\n opts.recurse_dirs = 0;\n break;\n case 'p':\n opts.path_to_ignore = TRUE;\n load_ignore_patterns(root_ignores, optarg);\n break;\n case 'o':\n opts.only_matching = 1;\n break;\n case 'F':\n case 'Q':\n opts.literal = 1;\n break;\n case 'R':\n case 'r':\n opts.recurse_dirs = 1;\n break;\n case 'S':\n opts.casing = CASE_SMART;\n break;\n case 's':\n opts.casing = CASE_SENSITIVE;\n break;\n case 't':\n opts.search_all_files = 1;\n break;\n case 'u':\n opts.search_binary_files = 1;\n opts.search_all_files = 1;\n opts.search_hidden_files = 1;\n break;\n case 'U':\n opts.skip_vcs_ignores = 1;\n break;\n case 'v':\n opts.invert_match = 1;\n /* Color highlighting doesn't make sense when inverting matches */\n opts.color = 0;\n break;\n case 'V':\n version = 1;\n break;\n case 'w':\n opts.word_regexp = 1;\n break;\n case 'W':\n opts.width = strtol(optarg, &num_end, 10);\n if (num_end == optarg || *num_end != '\\0' || errno == ERANGE) {\n die(\"Invalid width\\n\");\n }\n break;\n case 'z':\n opts.search_zip_files = 1;\n break;\n case '0':\n opts.path_sep = '\\0';\n break;\n case 0: /* Long option */\n if (strcmp(longopts[opt_index].name, \"ackmate-dir-filter\") == 0) {\n compile_study(&opts.ackmate_dir_filter, &opts.ackmate_dir_filter_extra, optarg, 0, 0);\n break;\n } else if (strcmp(longopts[opt_index].name, \"depth\") == 0) {\n opts.max_search_depth = atoi(optarg);\n break;\n } else if (strcmp(longopts[opt_index].name, \"filename\") == 0) {\n opts.print_path = PATH_PRINT_DEFAULT;\n opts.print_line_numbers = TRUE;\n break;\n } else if (strcmp(longopts[opt_index].name, \"ignore-dir\") == 0) {\n add_ignore_pattern(root_ignores, optarg);\n break;\n } else if (strcmp(longopts[opt_index].name, \"ignore\") == 0) {\n add_ignore_pattern(root_ignores, optarg);\n break;\n } else if (strcmp(longopts[opt_index].name, \"no-filename\") == 0 ||\n strcmp(longopts[opt_index].name, \"nofilename\") == 0) {\n opts.print_path = PATH_PRINT_NOTHING;\n opts.print_line_numbers = FALSE;\n break;\n } else if (strcmp(longopts[opt_index].name, \"no-pager\") == 0 ||\n strcmp(longopts[opt_index].name, \"nopager\") == 0) {\n out_fd = stdout;\n opts.pager = NULL;\n break;\n } else if (strcmp(longopts[opt_index].name, \"pager\") == 0) {\n opts.pager = optarg;\n break;\n } else if (strcmp(longopts[opt_index].name, \"print-all-files\") == 0) {\n opts.print_all_paths = TRUE;\n break;\n } else if (strcmp(longopts[opt_index].name, \"workers\") == 0) {\n opts.workers = atoi(optarg);\n break;\n } else if (strcmp(longopts[opt_index].name, \"color-line-number\") == 0) {\n free(opts.color_line_number);\n ag_asprintf(&opts.color_line_number, \"\\033[%sm\", optarg);\n break;\n } else if (strcmp(longopts[opt_index].name, \"color-match\") == 0) {\n free(opts.color_match);\n ag_asprintf(&opts.color_match, \"\\033[%sm\", optarg);\n break;\n } else if (strcmp(longopts[opt_index].name, \"color-path\") == 0) {\n free(opts.color_path);\n ag_asprintf(&opts.color_path, \"\\033[%sm\", optarg);\n break;\n } else if (strcmp(longopts[opt_index].name, \"silent\") == 0) {\n set_log_level(LOG_LEVEL_NONE);\n break;\n } else if (strcmp(longopts[opt_index].name, \"stats-only\") == 0) {\n opts.print_filename_only = 1;\n opts.print_path = PATH_PRINT_NOTHING;\n opts.stats = 1;\n break;\n }\n\n /* Continue to usage if we don't recognize the option */\n if (longopts[opt_index].flag != 0) {\n break;\n }\n\n for (i = 0; i < lang_count; i++) {\n if (strcmp(longopts[opt_index].name, langs[i].name) == 0) {\n has_filetype = 1;\n ext_index[lang_num++] = i;\n break;\n }\n }\n if (i != lang_count) {\n break;\n }\n\n log_err(\"option %s does not take a value\", longopts[opt_index].name);\n /* fall through */\n default:\n usage();\n exit(1);\n }\n }\n\n if (opts.casing == CASE_DEFAULT) {\n opts.casing = CASE_SMART;\n }\n\n if (file_search_regex) {\n int pcre_opts = 0;\n if (opts.casing == CASE_INSENSITIVE || (opts.casing == CASE_SMART && is_lowercase(file_search_regex))) {\n pcre_opts |= PCRE_CASELESS;\n }\n if (opts.word_regexp) {\n char *old_file_search_regex = file_search_regex;\n ag_asprintf(&file_search_regex, \"\\\\b%s\\\\b\", file_search_regex);\n free(old_file_search_regex);\n }\n compile_study(&opts.file_search_regex, &opts.file_search_regex_extra, file_search_regex, pcre_opts, 0);\n free(file_search_regex);\n }\n\n if (has_filetype) {\n num_exts = combine_file_extensions(ext_index, lang_num, &extensions);\n lang_regex = make_lang_regex(extensions, num_exts);\n compile_study(&opts.file_search_regex, &opts.file_search_regex_extra, lang_regex, 0, 0);\n }\n\n if (extensions) {\n free(extensions);\n }\n free(ext_index);\n if (lang_regex) {\n free(lang_regex);\n }\n free(longopts);\n\n argc -= optind;\n argv += optind;\n\n if (opts.pager) {\n out_fd = popen(opts.pager, \"w\");\n if (!out_fd) {\n perror(\"Failed to run pager\");\n exit(1);\n }\n }\n\n#ifdef HAVE_PLEDGE\n if (opts.skip_vcs_ignores) {\n if (pledge(\"stdio rpath proc\", NULL) == -1) {\n die(\"pledge: %s\", strerror(errno));\n }\n }\n#endif\n\n if (help) {\n usage();\n exit(0);\n }\n\n if (version) {\n print_version();\n exit(0);\n }\n\n if (list_file_types) {\n size_t lang_index;\n printf(\"The following file types are supported:\\n\");\n for (lang_index = 0; lang_index < lang_count; lang_index++) {\n printf(\" --%s\\n \", langs[lang_index].name);\n int j;\n for (j = 0; j < MAX_EXTENSIONS && langs[lang_index].extensions[j]; j++) {\n printf(\" .%s\", langs[lang_index].extensions[j]);\n }\n printf(\"\\n\\n\");\n }\n exit(0);\n }\n\n if (needs_query && argc == 0) {\n log_err(\"What do you want to search for?\");\n exit(1);\n }\n\n if (home_dir && !opts.search_all_files) {\n log_debug(\"Found user's home dir: %s\", home_dir);\n ag_asprintf(&ignore_file_path, \"%s/.agignore\", home_dir);\n load_ignore_patterns(root_ignores, ignore_file_path);\n free(ignore_file_path);\n }\n\n if (!opts.skip_vcs_ignores) {\n FILE *gitconfig_file = NULL;\n size_t buf_len = 0;\n char *gitconfig_res = NULL;\n\n#ifdef _WIN32\n gitconfig_file = popen(\"git config -z --path --get core.excludesfile 2>NUL\", \"r\");\n#else\n gitconfig_file = popen(\"git config -z --path --get core.excludesfile 2>/dev/null\", \"r\");\n#endif\n if (gitconfig_file != NULL) {\n do {\n gitconfig_res = ag_realloc(gitconfig_res, buf_len + 65);\n buf_len += fread(gitconfig_res + buf_len, 1, 64, gitconfig_file);\n } while (!feof(gitconfig_file) && buf_len > 0 && buf_len % 64 == 0);\n gitconfig_res[buf_len] = '\\0';\n if (buf_len == 0) {\n free(gitconfig_res);\n const char *config_home = getenv(\"XDG_CONFIG_HOME\");\n if (config_home) {\n ag_asprintf(&gitconfig_res, \"%s/%s\", config_home, \"git/ignore\");\n } else if (home_dir) {\n ag_asprintf(&gitconfig_res, \"%s/%s\", home_dir, \".config/git/ignore\");\n } else {\n gitconfig_res = ag_strdup(\"\");\n }\n }\n log_debug(\"global core.excludesfile: %s\", gitconfig_res);\n load_ignore_patterns(root_ignores, gitconfig_res);\n free(gitconfig_res);\n pclose(gitconfig_file);\n }\n }\n\n#ifdef HAVE_PLEDGE\n if (pledge(\"stdio rpath proc\", NULL) == -1) {\n die(\"pledge: %s\", strerror(errno));\n }\n#endif\n\n if (opts.context > 0) {\n opts.before = opts.context;\n opts.after = opts.context;\n }\n\n if (opts.ackmate) {\n opts.color = 0;\n opts.print_break = 1;\n group = 1;\n opts.search_stream = 0;\n }\n\n if (opts.vimgrep) {\n opts.color = 0;\n opts.print_break = 0;\n group = 1;\n opts.search_stream = 0;\n opts.print_path = PATH_PRINT_NOTHING;\n }\n\n if (opts.parallel) {\n opts.search_stream = 0;\n }\n\n if (!(opts.print_path != PATH_PRINT_DEFAULT || opts.print_break == 0)) {\n if (group) {\n opts.print_break = 1;\n } else {\n opts.print_path = PATH_PRINT_DEFAULT_EACH_LINE;\n opts.print_break = 0;\n }\n }\n\n if (opts.search_stream) {\n opts.print_break = 0;\n opts.print_path = PATH_PRINT_NOTHING;\n if (opts.print_line_numbers != 2) {\n opts.print_line_numbers = 0;\n }\n }\n\n if (accepts_query && argc > 0) {\n if (!needs_query && strlen(argv[0]) == 0) {\n // use default query\n opts.query = ag_strdup(\".\");\n } else {\n // use the provided query\n opts.query = ag_strdup(argv[0]);\n }\n argc--;\n argv++;\n } else if (!needs_query) {\n // use default query\n opts.query = ag_strdup(\".\");\n }\n opts.query_len = strlen(opts.query);\n\n log_debug(\"Query is %s\", opts.query);\n\n if (opts.query_len == 0) {\n log_err(\"Error: No query. What do you want to search for?\");\n exit(1);\n }\n\n if (!is_regex(opts.query)) {\n opts.literal = 1;\n }\n\n char *path = NULL;\n char *base_path = NULL;\n#ifdef PATH_MAX\n char *tmp = NULL;\n#endif\n opts.paths_len = argc;\n if (argc > 0) {\n *paths = ag_calloc(sizeof(char *), argc + 1);\n *base_paths = ag_calloc(sizeof(char *), argc + 1);\n for (i = 0; i < (size_t)argc; i++) {\n path = ag_strdup(argv[i]);\n path_len = strlen(path);\n /* kill trailing slash */\n if (path_len > 1 && path[path_len - 1] == '/') {\n path[path_len - 1] = '\\0';\n }\n (*paths)[i] = path;\n#ifdef PATH_MAX\n tmp = ag_malloc(PATH_MAX);\n base_path = realpath(path, tmp);\n#else\n base_path = realpath(path, NULL);\n#endif\n if (base_path) {\n base_path_len = strlen(base_path);\n /* add trailing slash */\n if (base_path_len > 1 && base_path[base_path_len - 1] != '/') {\n base_path = ag_realloc(base_path, base_path_len + 2);\n base_path[base_path_len] = '/';\n base_path[base_path_len + 1] = '\\0';\n }\n }\n (*base_paths)[i] = base_path;\n }\n /* Make sure we search these paths instead of stdin. */\n opts.search_stream = 0;\n } else {\n path = ag_strdup(\".\");\n *paths = ag_malloc(sizeof(char *) * 2);\n *base_paths = ag_malloc(sizeof(char *) * 2);\n (*paths)[0] = path;\n#ifdef PATH_MAX\n tmp = ag_malloc(PATH_MAX);\n (*base_paths)[0] = realpath(path, tmp);\n#else\n (*base_paths)[0] = realpath(path, NULL);\n#endif\n i = 1;\n }\n (*paths)[i] = NULL;\n (*base_paths)[i] = NULL;\n\n#ifdef _WIN32\n windows_use_ansi(opts.color_win_ansi);\n#endif\n}\n" }, { "path": "src/options.h", "content": "#ifndef OPTIONS_H\n#define OPTIONS_H\n\n#include \n#include \n\n#include \n\n#define DEFAULT_AFTER_LEN 2\n#define DEFAULT_BEFORE_LEN 2\n#define DEFAULT_CONTEXT_LEN 2\n#define DEFAULT_MAX_SEARCH_DEPTH 25\nenum case_behavior {\n CASE_DEFAULT, /* Changes to CASE_SMART at the end of option parsing */\n CASE_SENSITIVE,\n CASE_INSENSITIVE,\n CASE_SMART,\n CASE_SENSITIVE_RETRY_INSENSITIVE /* for future use */\n};\n\nenum path_print_behavior {\n PATH_PRINT_DEFAULT, /* PRINT_TOP if > 1 file being searched, else PRINT_NOTHING */\n PATH_PRINT_DEFAULT_EACH_LINE, /* PRINT_EACH_LINE if > 1 file being searched, else PRINT_NOTHING */\n PATH_PRINT_TOP,\n PATH_PRINT_EACH_LINE,\n PATH_PRINT_NOTHING\n};\n\ntypedef struct {\n int ackmate;\n pcre *ackmate_dir_filter;\n pcre_extra *ackmate_dir_filter_extra;\n size_t after;\n size_t before;\n enum case_behavior casing;\n const char *file_search_string;\n int match_files;\n pcre *file_search_regex;\n pcre_extra *file_search_regex_extra;\n int color;\n char *color_line_number;\n char *color_match;\n char *color_path;\n int color_win_ansi;\n int column;\n int context;\n int follow_symlinks;\n int invert_match;\n int literal;\n int literal_starts_wordchar;\n int literal_ends_wordchar;\n size_t max_matches_per_file;\n int max_search_depth;\n int mmap;\n int multiline;\n int one_dev;\n int only_matching;\n char path_sep;\n int path_to_ignore;\n int print_break;\n int print_count;\n int print_filename_only;\n int print_nonmatching_files;\n int print_path;\n int print_all_paths;\n int print_line_numbers;\n int print_long_lines; /* TODO: support this in print.c */\n int passthrough;\n pcre *re;\n pcre_extra *re_extra;\n int recurse_dirs;\n int search_all_files;\n int skip_vcs_ignores;\n int search_binary_files;\n int search_zip_files;\n int search_hidden_files;\n int search_stream; /* true if tail -F blah | ag */\n int stats;\n size_t stream_line_num; /* This should totally not be in here */\n int match_found; /* This should totally not be in here */\n ino_t stdout_inode;\n char *query;\n int query_len;\n char *pager;\n int paths_len;\n int parallel;\n int use_thread_affinity;\n int vimgrep;\n size_t width;\n int word_regexp;\n int workers;\n} cli_options;\n\n/* global options. parse_options gives it sane values, everything else reads from it */\nextern cli_options opts;\n\ntypedef struct option option_t;\n\nvoid usage(void);\nvoid print_version(void);\n\nvoid init_options(void);\nvoid parse_options(int argc, char **argv, char **base_paths[], char **paths[]);\nvoid cleanup_options(void);\n\n#endif\n" }, { "path": "src/print.c", "content": "#include \n#include \n#include \n#include \n#include \n\n#include \"ignore.h\"\n#include \"log.h\"\n#include \"options.h\"\n#include \"print.h\"\n#include \"search.h\"\n#include \"util.h\"\n#ifdef _WIN32\n#define fprintf(...) fprintf_w32(__VA_ARGS__)\n#endif\n\nint first_file_match = 1;\n\nconst char *color_reset = \"\\033[0m\\033[K\";\n\nconst char *truncate_marker = \" [...]\";\n\n__thread struct print_context {\n size_t line;\n char **context_prev_lines;\n size_t prev_line;\n size_t last_prev_line;\n size_t prev_line_offset;\n size_t line_preceding_current_match_offset;\n size_t lines_since_last_match;\n size_t last_printed_match;\n int in_a_match;\n int printing_a_match;\n} print_context;\n\nvoid print_init_context(void) {\n if (print_context.context_prev_lines != NULL) {\n return;\n }\n print_context.context_prev_lines = ag_calloc(sizeof(char *), (opts.before + 1));\n print_context.line = 1;\n print_context.prev_line = 0;\n print_context.last_prev_line = 0;\n print_context.prev_line_offset = 0;\n print_context.line_preceding_current_match_offset = 0;\n print_context.lines_since_last_match = INT_MAX;\n print_context.last_printed_match = 0;\n print_context.in_a_match = FALSE;\n print_context.printing_a_match = FALSE;\n}\n\nvoid print_cleanup_context(void) {\n size_t i;\n\n if (print_context.context_prev_lines == NULL) {\n return;\n }\n\n for (i = 0; i < opts.before; i++) {\n if (print_context.context_prev_lines[i] != NULL) {\n free(print_context.context_prev_lines[i]);\n }\n }\n free(print_context.context_prev_lines);\n print_context.context_prev_lines = NULL;\n}\n\nvoid print_context_append(const char *line, size_t len) {\n if (opts.before == 0) {\n return;\n }\n if (print_context.context_prev_lines[print_context.last_prev_line] != NULL) {\n free(print_context.context_prev_lines[print_context.last_prev_line]);\n }\n print_context.context_prev_lines[print_context.last_prev_line] = ag_strndup(line, len);\n print_context.last_prev_line = (print_context.last_prev_line + 1) % opts.before;\n}\n\nvoid print_trailing_context(const char *path, const char *buf, size_t n) {\n char sep = '-';\n\n if (opts.ackmate || opts.vimgrep) {\n sep = ':';\n }\n\n if (print_context.lines_since_last_match != 0 &&\n print_context.lines_since_last_match <= opts.after) {\n if (opts.print_path == PATH_PRINT_EACH_LINE) {\n print_path(path, ':');\n }\n print_line_number(print_context.line, sep);\n\n fwrite(buf, 1, n, out_fd);\n fputc('\\n', out_fd);\n }\n\n print_context.line++;\n if (!print_context.in_a_match && print_context.lines_since_last_match < INT_MAX) {\n print_context.lines_since_last_match++;\n }\n}\n\nvoid print_path(const char *path, const char sep) {\n if (opts.print_path == PATH_PRINT_NOTHING && !opts.vimgrep) {\n return;\n }\n path = normalize_path(path);\n\n if (opts.ackmate) {\n fprintf(out_fd, \":%s%c\", path, sep);\n } else if (opts.vimgrep) {\n fprintf(out_fd, \"%s%c\", path, sep);\n } else {\n if (opts.color) {\n fprintf(out_fd, \"%s%s%s%c\", opts.color_path, path, color_reset, sep);\n } else {\n fprintf(out_fd, \"%s%c\", path, sep);\n }\n }\n}\n\nvoid print_path_count(const char *path, const char sep, const size_t count) {\n if (*path) {\n print_path(path, ':');\n }\n if (opts.color) {\n fprintf(out_fd, \"%s%lu%s%c\", opts.color_line_number, (unsigned long)count, color_reset, sep);\n } else {\n fprintf(out_fd, \"%lu%c\", (unsigned long)count, sep);\n }\n}\n\nvoid print_line(const char *buf, size_t buf_pos, size_t prev_line_offset) {\n size_t write_chars = buf_pos - prev_line_offset + 1;\n if (opts.width > 0 && opts.width < write_chars) {\n write_chars = opts.width;\n }\n\n fwrite(buf + prev_line_offset, 1, write_chars, out_fd);\n}\n\nvoid print_binary_file_matches(const char *path) {\n path = normalize_path(path);\n print_file_separator();\n fprintf(out_fd, \"Binary file %s matches.\\n\", path);\n}\n\nvoid print_file_matches(const char *path, const char *buf, const size_t buf_len, const match_t matches[], const size_t matches_len) {\n size_t cur_match = 0;\n ssize_t lines_to_print = 0;\n char sep = '-';\n size_t i, j;\n int blanks_between_matches = opts.context || opts.after || opts.before;\n\n if (opts.ackmate || opts.vimgrep) {\n sep = ':';\n }\n\n print_file_separator();\n\n if (opts.print_path == PATH_PRINT_DEFAULT) {\n opts.print_path = PATH_PRINT_TOP;\n } else if (opts.print_path == PATH_PRINT_DEFAULT_EACH_LINE) {\n opts.print_path = PATH_PRINT_EACH_LINE;\n }\n\n if (opts.print_path == PATH_PRINT_TOP) {\n if (opts.print_count) {\n print_path_count(path, opts.path_sep, matches_len);\n } else {\n print_path(path, opts.path_sep);\n }\n }\n\n for (i = 0; i <= buf_len && (cur_match < matches_len || print_context.lines_since_last_match <= opts.after); i++) {\n if (cur_match < matches_len && i == matches[cur_match].start) {\n print_context.in_a_match = TRUE;\n /* We found the start of a match */\n if (cur_match > 0 && blanks_between_matches && print_context.lines_since_last_match > (opts.before + opts.after + 1)) {\n fprintf(out_fd, \"--\\n\");\n }\n\n if (print_context.lines_since_last_match > 0 && opts.before > 0) {\n /* TODO: better, but still needs work */\n /* print the previous line(s) */\n lines_to_print = print_context.lines_since_last_match - (opts.after + 1);\n if (lines_to_print < 0) {\n lines_to_print = 0;\n } else if ((size_t)lines_to_print > opts.before) {\n lines_to_print = opts.before;\n }\n\n for (j = (opts.before - lines_to_print); j < opts.before; j++) {\n print_context.prev_line = (print_context.last_prev_line + j) % opts.before;\n if (print_context.context_prev_lines[print_context.prev_line] != NULL) {\n if (opts.print_path == PATH_PRINT_EACH_LINE) {\n print_path(path, ':');\n }\n print_line_number(print_context.line - (opts.before - j), sep);\n fprintf(out_fd, \"%s\\n\", print_context.context_prev_lines[print_context.prev_line]);\n }\n }\n }\n print_context.lines_since_last_match = 0;\n }\n\n if (cur_match < matches_len && i == matches[cur_match].end) {\n /* We found the end of a match. */\n cur_match++;\n print_context.in_a_match = FALSE;\n }\n\n /* We found the end of a line. */\n if ((i == buf_len || buf[i] == '\\n') && opts.before > 0) {\n /* We don't want to strcpy the \\n */\n print_context_append(&buf[print_context.prev_line_offset], i - print_context.prev_line_offset);\n }\n\n if (i == buf_len || buf[i] == '\\n') {\n if (print_context.lines_since_last_match == 0) {\n if (opts.print_path == PATH_PRINT_EACH_LINE && !opts.search_stream) {\n print_path(path, ':');\n }\n if (opts.ackmate) {\n /* print headers for ackmate to parse */\n print_line_number(print_context.line, ';');\n for (; print_context.last_printed_match < cur_match; print_context.last_printed_match++) {\n size_t start = matches[print_context.last_printed_match].start - print_context.line_preceding_current_match_offset;\n fprintf(out_fd, \"%lu %lu\",\n start,\n matches[print_context.last_printed_match].end - matches[print_context.last_printed_match].start);\n print_context.last_printed_match == cur_match - 1 ? fputc(':', out_fd) : fputc(',', out_fd);\n }\n print_line(buf, i, print_context.prev_line_offset);\n } else if (opts.vimgrep) {\n for (; print_context.last_printed_match < cur_match; print_context.last_printed_match++) {\n print_path(path, sep);\n print_line_number(print_context.line, sep);\n print_column_number(matches, print_context.last_printed_match, print_context.prev_line_offset, sep);\n print_line(buf, i, print_context.prev_line_offset);\n }\n } else {\n print_line_number(print_context.line, ':');\n int printed_match = FALSE;\n if (opts.column) {\n print_column_number(matches, print_context.last_printed_match, print_context.prev_line_offset, ':');\n }\n\n if (print_context.printing_a_match && opts.color) {\n fprintf(out_fd, \"%s\", opts.color_match);\n }\n for (j = print_context.prev_line_offset; j <= i; j++) {\n /* close highlight of match term */\n if (print_context.last_printed_match < matches_len && j == matches[print_context.last_printed_match].end) {\n if (opts.color) {\n fprintf(out_fd, \"%s\", color_reset);\n }\n print_context.printing_a_match = FALSE;\n print_context.last_printed_match++;\n printed_match = TRUE;\n if (opts.only_matching) {\n fputc('\\n', out_fd);\n }\n }\n /* skip remaining characters if truncation width exceeded, needs to be done\n * before highlight opening */\n if (j < buf_len && opts.width > 0 && j - print_context.prev_line_offset >= opts.width) {\n if (j < i) {\n fputs(truncate_marker, out_fd);\n }\n fputc('\\n', out_fd);\n\n /* prevent any more characters or highlights */\n j = i;\n print_context.last_printed_match = matches_len;\n }\n /* open highlight of match term */\n if (print_context.last_printed_match < matches_len && j == matches[print_context.last_printed_match].start) {\n if (opts.only_matching && printed_match) {\n if (opts.print_path == PATH_PRINT_EACH_LINE) {\n print_path(path, ':');\n }\n print_line_number(print_context.line, ':');\n if (opts.column) {\n print_column_number(matches, print_context.last_printed_match, print_context.prev_line_offset, ':');\n }\n }\n if (opts.color) {\n fprintf(out_fd, \"%s\", opts.color_match);\n }\n print_context.printing_a_match = TRUE;\n }\n /* Don't print the null terminator */\n if (j < buf_len) {\n /* if only_matching is set, print only matches and newlines */\n if (!opts.only_matching || print_context.printing_a_match) {\n if (opts.width == 0 || j - print_context.prev_line_offset < opts.width) {\n fputc(buf[j], out_fd);\n }\n }\n }\n }\n if (print_context.printing_a_match && opts.color) {\n fprintf(out_fd, \"%s\", color_reset);\n }\n }\n }\n\n if (opts.search_stream) {\n print_context.last_printed_match = 0;\n break;\n }\n\n /* print context after matching line */\n print_trailing_context(path, &buf[print_context.prev_line_offset], i - print_context.prev_line_offset);\n\n print_context.prev_line_offset = i + 1; /* skip the newline */\n if (!print_context.in_a_match) {\n print_context.line_preceding_current_match_offset = i + 1;\n }\n\n /* File doesn't end with a newline. Print one so the output is pretty. */\n if (i == buf_len && buf[i - 1] != '\\n') {\n fputc('\\n', out_fd);\n }\n }\n }\n /* Flush output if stdout is not a tty */\n if (opts.stdout_inode) {\n fflush(out_fd);\n }\n}\n\nvoid print_line_number(size_t line, const char sep) {\n if (!opts.print_line_numbers) {\n return;\n }\n if (opts.color) {\n fprintf(out_fd, \"%s%lu%s%c\", opts.color_line_number, (unsigned long)line, color_reset, sep);\n } else {\n fprintf(out_fd, \"%lu%c\", (unsigned long)line, sep);\n }\n}\n\nvoid print_column_number(const match_t matches[], size_t last_printed_match,\n size_t prev_line_offset, const char sep) {\n size_t column = 0;\n if (prev_line_offset <= matches[last_printed_match].start) {\n column = (matches[last_printed_match].start - prev_line_offset) + 1;\n }\n fprintf(out_fd, \"%lu%c\", (unsigned long)column, sep);\n}\n\nvoid print_file_separator(void) {\n if (first_file_match == 0 && opts.print_break) {\n fprintf(out_fd, \"\\n\");\n }\n first_file_match = 0;\n}\n\nconst char *normalize_path(const char *path) {\n if (strlen(path) < 3) {\n return path;\n }\n if (path[0] == '.' && path[1] == '/') {\n return path + 2;\n }\n if (path[0] == '/' && path[1] == '/') {\n return path + 1;\n }\n return path;\n}\n" }, { "path": "src/print.h", "content": "#ifndef PRINT_H\n#define PRINT_H\n\n#include \"util.h\"\n\nvoid print_init_context(void);\nvoid print_cleanup_context(void);\nvoid print_context_append(const char *line, size_t len);\nvoid print_trailing_context(const char *path, const char *buf, size_t n);\nvoid print_path(const char *path, const char sep);\nvoid print_path_count(const char *path, const char sep, const size_t count);\nvoid print_line(const char *buf, size_t buf_pos, size_t prev_line_offset);\nvoid print_binary_file_matches(const char *path);\nvoid print_file_matches(const char *path, const char *buf, const size_t buf_len, const match_t matches[], const size_t matches_len);\nvoid print_line_number(size_t line, const char sep);\nvoid print_column_number(const match_t matches[], size_t last_printed_match,\n size_t prev_line_offset, const char sep);\nvoid print_file_separator(void);\nconst char *normalize_path(const char *path);\n\n#ifdef _WIN32\nvoid windows_use_ansi(int use_ansi);\nint fprintf_w32(FILE *fp, const char *format, ...);\n#endif\n\n#endif\n" }, { "path": "src/print_w32.c", "content": "#ifdef _WIN32\n\n#include \"print.h\"\n#include \n#include \n#include \n#include \n\n#ifndef FOREGROUND_MASK\n#define FOREGROUND_MASK (FOREGROUND_RED | FOREGROUND_BLUE | FOREGROUND_GREEN | FOREGROUND_INTENSITY)\n#endif\n#ifndef BACKGROUND_MASK\n#define BACKGROUND_MASK (BACKGROUND_RED | BACKGROUND_BLUE | BACKGROUND_GREEN | BACKGROUND_INTENSITY)\n#endif\n\n#define FG_RGB (FOREGROUND_RED | FOREGROUND_BLUE | FOREGROUND_GREEN)\n#define BG_RGB (BACKGROUND_RED | BACKGROUND_BLUE | BACKGROUND_GREEN)\n\n// BUFSIZ is guarenteed to be \"at least 256 bytes\" which might\n// not be enough for us. Use an arbitrary but reasonably big\n// buffer. win32 colored output will be truncated to this length.\n#define BUF_SIZE (16 * 1024)\n\n// max consecutive ansi sequence values beyond which we're aborting\n// e.g. this is 3 values: \\e[0;1;33m\n#define MAX_VALUES 8\n\nstatic int g_use_ansi = 0;\nvoid windows_use_ansi(int use_ansi) {\n g_use_ansi = use_ansi;\n}\n\nint fprintf_w32(FILE *fp, const char *format, ...) {\n va_list args;\n char buf[BUF_SIZE] = { 0 }, *ptr = buf;\n static WORD attr_reset;\n static BOOL attr_initialized = FALSE;\n HANDLE stdo = INVALID_HANDLE_VALUE;\n WORD attr;\n DWORD written, csize;\n CONSOLE_CURSOR_INFO cci;\n CONSOLE_SCREEN_BUFFER_INFO csbi;\n COORD coord;\n\n // if we don't output to screen (tty) e.g. for pager/pipe or\n // if for other reason we can't access the screen info, of if\n // the user just prefers ansi, do plain passthrough.\n BOOL passthrough =\n g_use_ansi ||\n !isatty(fileno(fp)) ||\n INVALID_HANDLE_VALUE == (stdo = (HANDLE)_get_osfhandle(fileno(fp))) ||\n !GetConsoleScreenBufferInfo(stdo, &csbi);\n\n if (passthrough) {\n int rv;\n va_start(args, format);\n rv = vfprintf(fp, format, args);\n va_end(args);\n return rv;\n }\n\n va_start(args, format);\n // truncates to (null terminated) BUF_SIZE if too long.\n // if too long - vsnprintf will fill count chars without\n // terminating null. buf is zeroed, so make sure we don't fill it.\n vsnprintf(buf, BUF_SIZE - 1, format, args);\n va_end(args);\n\n attr = csbi.wAttributes;\n if (!attr_initialized) {\n // reset is defined to have all (non color) attributes off\n attr_reset = attr & (FG_RGB | BG_RGB);\n attr_initialized = TRUE;\n }\n\n while (*ptr) {\n if (*ptr == '\\033') {\n unsigned char c;\n int i, n = 0, m = '\\0', v[MAX_VALUES], w, h;\n for (i = 0; i < MAX_VALUES; i++)\n v[i] = -1;\n ptr++;\n retry:\n if ((c = *ptr++) == 0)\n break;\n if (isdigit(c)) {\n if (v[n] == -1)\n v[n] = c - '0';\n else\n v[n] = v[n] * 10 + c - '0';\n goto retry;\n }\n if (c == '[') {\n goto retry;\n }\n if (c == ';') {\n if (++n == MAX_VALUES)\n break;\n goto retry;\n }\n if (c == '>' || c == '?') {\n m = c;\n goto retry;\n }\n\n switch (c) {\n // n is the last occupied index, so we have n+1 values\n case 'h':\n if (m == '?') {\n for (i = 0; i <= n; i++) {\n switch (v[i]) {\n case 3:\n GetConsoleScreenBufferInfo(stdo, &csbi);\n w = csbi.dwSize.X;\n h = csbi.srWindow.Bottom - csbi.srWindow.Top + 1;\n csize = w * (h + 1);\n coord.X = 0;\n coord.Y = csbi.srWindow.Top;\n FillConsoleOutputCharacter(stdo, ' ', csize, coord, &written);\n FillConsoleOutputAttribute(stdo, csbi.wAttributes, csize, coord, &written);\n SetConsoleCursorPosition(stdo, csbi.dwCursorPosition);\n csbi.dwSize.X = 132;\n SetConsoleScreenBufferSize(stdo, csbi.dwSize);\n csbi.srWindow.Right = csbi.srWindow.Left + 131;\n SetConsoleWindowInfo(stdo, TRUE, &csbi.srWindow);\n break;\n case 5:\n attr =\n ((attr & FOREGROUND_MASK) << 4) |\n ((attr & BACKGROUND_MASK) >> 4);\n SetConsoleTextAttribute(stdo, attr);\n break;\n case 9:\n break;\n case 25:\n GetConsoleCursorInfo(stdo, &cci);\n cci.bVisible = TRUE;\n SetConsoleCursorInfo(stdo, &cci);\n break;\n case 47:\n coord.X = 0;\n coord.Y = 0;\n SetConsoleCursorPosition(stdo, coord);\n break;\n default:\n break;\n }\n }\n } else if (m == '>' && v[0] == 5) {\n GetConsoleCursorInfo(stdo, &cci);\n cci.bVisible = FALSE;\n SetConsoleCursorInfo(stdo, &cci);\n }\n break;\n case 'l':\n if (m == '?') {\n for (i = 0; i <= n; i++) {\n switch (v[i]) {\n case 3:\n GetConsoleScreenBufferInfo(stdo, &csbi);\n w = csbi.dwSize.X;\n h = csbi.srWindow.Bottom - csbi.srWindow.Top + 1;\n csize = w * (h + 1);\n coord.X = 0;\n coord.Y = csbi.srWindow.Top;\n FillConsoleOutputCharacter(stdo, ' ', csize, coord, &written);\n FillConsoleOutputAttribute(stdo, csbi.wAttributes, csize, coord, &written);\n SetConsoleCursorPosition(stdo, csbi.dwCursorPosition);\n csbi.srWindow.Right = csbi.srWindow.Left + 79;\n SetConsoleWindowInfo(stdo, TRUE, &csbi.srWindow);\n csbi.dwSize.X = 80;\n SetConsoleScreenBufferSize(stdo, csbi.dwSize);\n break;\n case 5:\n attr =\n ((attr & FOREGROUND_MASK) << 4) |\n ((attr & BACKGROUND_MASK) >> 4);\n SetConsoleTextAttribute(stdo, attr);\n break;\n case 25:\n GetConsoleCursorInfo(stdo, &cci);\n cci.bVisible = FALSE;\n SetConsoleCursorInfo(stdo, &cci);\n break;\n default:\n break;\n }\n }\n } else if (m == '>' && v[0] == 5) {\n GetConsoleCursorInfo(stdo, &cci);\n cci.bVisible = TRUE;\n SetConsoleCursorInfo(stdo, &cci);\n }\n break;\n case 'm':\n for (i = 0; i <= n; i++) {\n if (v[i] == -1 || v[i] == 0)\n attr = attr_reset;\n else if (v[i] == 1)\n attr |= FOREGROUND_INTENSITY;\n else if (v[i] == 4)\n attr |= FOREGROUND_INTENSITY;\n else if (v[i] == 5) // blink is typically applied as bg intensity\n attr |= BACKGROUND_INTENSITY;\n else if (v[i] == 7)\n attr =\n ((attr & FOREGROUND_MASK) << 4) |\n ((attr & BACKGROUND_MASK) >> 4);\n else if (v[i] == 10)\n ; // symbol on\n else if (v[i] == 11)\n ; // symbol off\n else if (v[i] == 22)\n attr &= ~FOREGROUND_INTENSITY;\n else if (v[i] == 24)\n attr &= ~FOREGROUND_INTENSITY;\n else if (v[i] == 25)\n attr &= ~BACKGROUND_INTENSITY;\n else if (v[i] == 27)\n attr =\n ((attr & FOREGROUND_MASK) << 4) |\n ((attr & BACKGROUND_MASK) >> 4);\n else if (v[i] >= 30 && v[i] <= 37) {\n attr &= ~FG_RGB; // doesn't affect attributes\n if ((v[i] - 30) & 1)\n attr |= FOREGROUND_RED;\n if ((v[i] - 30) & 2)\n attr |= FOREGROUND_GREEN;\n if ((v[i] - 30) & 4)\n attr |= FOREGROUND_BLUE;\n } else if (v[i] == 39) // reset fg color and attributes\n attr = (attr & ~FOREGROUND_MASK) | (attr_reset & FG_RGB);\n else if (v[i] >= 40 && v[i] <= 47) {\n attr &= ~BG_RGB;\n if ((v[i] - 40) & 1)\n attr |= BACKGROUND_RED;\n if ((v[i] - 40) & 2)\n attr |= BACKGROUND_GREEN;\n if ((v[i] - 40) & 4)\n attr |= BACKGROUND_BLUE;\n } else if (v[i] == 49) // reset bg color\n attr = (attr & ~BACKGROUND_MASK) | (attr_reset & BG_RGB);\n }\n SetConsoleTextAttribute(stdo, attr);\n break;\n case 'K':\n GetConsoleScreenBufferInfo(stdo, &csbi);\n coord = csbi.dwCursorPosition;\n switch (v[0]) {\n default:\n case 0:\n csize = csbi.dwSize.X - coord.X;\n break;\n case 1:\n csize = coord.X;\n coord.X = 0;\n break;\n case 2:\n csize = csbi.dwSize.X;\n coord.X = 0;\n break;\n }\n FillConsoleOutputCharacter(stdo, ' ', csize, coord, &written);\n FillConsoleOutputAttribute(stdo, csbi.wAttributes, csize, coord, &written);\n SetConsoleCursorPosition(stdo, csbi.dwCursorPosition);\n break;\n case 'J':\n GetConsoleScreenBufferInfo(stdo, &csbi);\n w = csbi.dwSize.X;\n h = csbi.srWindow.Bottom - csbi.srWindow.Top + 1;\n coord = csbi.dwCursorPosition;\n switch (v[0]) {\n default:\n case 0:\n csize = w * (h - coord.Y) - coord.X;\n coord.X = 0;\n break;\n case 1:\n csize = w * coord.Y + coord.X;\n coord.X = 0;\n coord.Y = csbi.srWindow.Top;\n break;\n case 2:\n csize = w * (h + 1);\n coord.X = 0;\n coord.Y = csbi.srWindow.Top;\n break;\n }\n FillConsoleOutputCharacter(stdo, ' ', csize, coord, &written);\n FillConsoleOutputAttribute(stdo, csbi.wAttributes, csize, coord, &written);\n SetConsoleCursorPosition(stdo, csbi.dwCursorPosition);\n break;\n case 'H':\n GetConsoleScreenBufferInfo(stdo, &csbi);\n coord = csbi.dwCursorPosition;\n if (v[0] != -1) {\n if (v[1] != -1) {\n coord.Y = csbi.srWindow.Top + v[0] - 1;\n coord.X = v[1] - 1;\n } else\n coord.X = v[0] - 1;\n } else {\n coord.X = 0;\n coord.Y = csbi.srWindow.Top;\n }\n if (coord.X < csbi.srWindow.Left)\n coord.X = csbi.srWindow.Left;\n else if (coord.X > csbi.srWindow.Right)\n coord.X = csbi.srWindow.Right;\n if (coord.Y < csbi.srWindow.Top)\n coord.Y = csbi.srWindow.Top;\n else if (coord.Y > csbi.srWindow.Bottom)\n coord.Y = csbi.srWindow.Bottom;\n SetConsoleCursorPosition(stdo, coord);\n break;\n default:\n break;\n }\n } else {\n putchar(*ptr);\n ptr++;\n }\n }\n return ptr - buf;\n}\n\n#endif /* _WIN32 */\n" }, { "path": "src/scandir.c", "content": "#include \n#include \n\n#include \"scandir.h\"\n#include \"util.h\"\n\nint ag_scandir(const char *dirname,\n struct dirent ***namelist,\n filter_fp filter,\n void *baton) {\n DIR *dirp = NULL;\n struct dirent **names = NULL;\n struct dirent *entry, *d;\n int names_len = 32;\n int results_len = 0;\n\n dirp = opendir(dirname);\n if (dirp == NULL) {\n goto fail;\n }\n\n names = malloc(sizeof(struct dirent *) * names_len);\n if (names == NULL) {\n goto fail;\n }\n\n while ((entry = readdir(dirp)) != NULL) {\n if ((*filter)(dirname, entry, baton) == FALSE) {\n continue;\n }\n if (results_len >= names_len) {\n struct dirent **tmp_names = names;\n names_len *= 2;\n names = realloc(names, sizeof(struct dirent *) * names_len);\n if (names == NULL) {\n free(tmp_names);\n goto fail;\n }\n }\n\n#if defined(__MINGW32__) || defined(__CYGWIN__)\n d = malloc(sizeof(struct dirent));\n#else\n d = malloc(entry->d_reclen);\n#endif\n\n if (d == NULL) {\n goto fail;\n }\n#if defined(__MINGW32__) || defined(__CYGWIN__)\n memcpy(d, entry, sizeof(struct dirent));\n#else\n memcpy(d, entry, entry->d_reclen);\n#endif\n\n names[results_len] = d;\n results_len++;\n }\n\n closedir(dirp);\n *namelist = names;\n return results_len;\n\nfail:\n if (dirp) {\n closedir(dirp);\n }\n\n if (names != NULL) {\n int i;\n for (i = 0; i < results_len; i++) {\n free(names[i]);\n }\n free(names);\n }\n return -1;\n}\n" }, { "path": "src/scandir.h", "content": "#ifndef SCANDIR_H\n#define SCANDIR_H\n\n#include \"ignore.h\"\n\ntypedef struct {\n const ignores *ig;\n const char *base_path;\n size_t base_path_len;\n const char *path_start;\n} scandir_baton_t;\n\ntypedef int (*filter_fp)(const char *path, const struct dirent *, void *);\n\nint ag_scandir(const char *dirname,\n struct dirent ***namelist,\n filter_fp filter,\n void *baton);\n\n#endif\n" }, { "path": "src/search.c", "content": "#include \"search.h\"\n#include \"print.h\"\n#include \"scandir.h\"\n\nsize_t alpha_skip_lookup[256];\nsize_t *find_skip_lookup;\nuint8_t h_table[H_SIZE] __attribute__((aligned(64)));\n\nwork_queue_t *work_queue = NULL;\nwork_queue_t *work_queue_tail = NULL;\nint done_adding_files = 0;\npthread_cond_t files_ready = PTHREAD_COND_INITIALIZER;\npthread_mutex_t stats_mtx = PTHREAD_MUTEX_INITIALIZER;\npthread_mutex_t work_queue_mtx = PTHREAD_MUTEX_INITIALIZER;\n\nsymdir_t *symhash = NULL;\n\n/* Returns: -1 if skipped, otherwise # of matches */\nssize_t search_buf(const char *buf, const size_t buf_len,\n const char *dir_full_path) {\n int binary = -1; /* 1 = yes, 0 = no, -1 = don't know */\n size_t buf_offset = 0;\n\n if (opts.search_stream) {\n binary = 0;\n } else if (!opts.search_binary_files && opts.mmap) { /* if not using mmap, binary files have already been skipped */\n binary = is_binary((const void *)buf, buf_len);\n if (binary) {\n log_debug(\"File %s is binary. Skipping...\", dir_full_path);\n return -1;\n }\n }\n\n size_t matches_len = 0;\n match_t *matches;\n size_t matches_size;\n size_t matches_spare;\n\n if (opts.invert_match) {\n /* If we are going to invert the set of matches at the end, we will need\n * one extra match struct, even if there are no matches at all. So make\n * sure we have a nonempty array; and make sure we always have spare\n * capacity for one extra.\n */\n matches_size = 100;\n matches = ag_malloc(matches_size * sizeof(match_t));\n matches_spare = 1;\n } else {\n matches_size = 0;\n matches = NULL;\n matches_spare = 0;\n }\n\n if (!opts.literal && opts.query_len == 1 && opts.query[0] == '.') {\n matches_size = 1;\n matches = matches == NULL ? ag_malloc(matches_size * sizeof(match_t)) : matches;\n matches[0].start = 0;\n matches[0].end = buf_len;\n matches_len = 1;\n } else if (opts.literal) {\n const char *match_ptr = buf;\n\n while (buf_offset < buf_len) {\n/* hash_strnstr only for little-endian platforms that allow unaligned access */\n#if defined(__i386__) || defined(__x86_64__)\n /* Decide whether to fall back on boyer-moore */\n if ((size_t)opts.query_len < 2 * sizeof(uint16_t) - 1 || opts.query_len >= UCHAR_MAX) {\n match_ptr = boyer_moore_strnstr(match_ptr, opts.query, buf_len - buf_offset, opts.query_len, alpha_skip_lookup, find_skip_lookup, opts.casing == CASE_INSENSITIVE);\n } else {\n match_ptr = hash_strnstr(match_ptr, opts.query, buf_len - buf_offset, opts.query_len, h_table, opts.casing == CASE_SENSITIVE);\n }\n#else\n match_ptr = boyer_moore_strnstr(match_ptr, opts.query, buf_len - buf_offset, opts.query_len, alpha_skip_lookup, find_skip_lookup, opts.casing == CASE_INSENSITIVE);\n#endif\n\n if (match_ptr == NULL) {\n break;\n }\n\n if (opts.word_regexp) {\n const char *start = match_ptr;\n const char *end = match_ptr + opts.query_len;\n\n /* Check whether both start and end of the match lie on a word\n * boundary\n */\n if ((start == buf ||\n is_wordchar(*(start - 1)) != opts.literal_starts_wordchar) &&\n (end == buf + buf_len ||\n is_wordchar(*end) != opts.literal_ends_wordchar)) {\n /* It's a match */\n } else {\n /* It's not a match */\n match_ptr += find_skip_lookup[0] - opts.query_len + 1;\n buf_offset = match_ptr - buf;\n continue;\n }\n }\n\n realloc_matches(&matches, &matches_size, matches_len + matches_spare);\n\n matches[matches_len].start = match_ptr - buf;\n matches[matches_len].end = matches[matches_len].start + opts.query_len;\n buf_offset = matches[matches_len].end;\n log_debug(\"Match found. File %s, offset %lu bytes.\", dir_full_path, matches[matches_len].start);\n matches_len++;\n match_ptr += opts.query_len;\n\n if (opts.max_matches_per_file > 0 && matches_len >= opts.max_matches_per_file) {\n log_err(\"Too many matches in %s. Skipping the rest of this file.\", dir_full_path);\n break;\n }\n }\n } else {\n int offset_vector[3];\n if (opts.multiline) {\n while (buf_offset < buf_len &&\n (pcre_exec(opts.re, opts.re_extra, buf, buf_len, buf_offset, 0, offset_vector, 3)) >= 0) {\n log_debug(\"Regex match found. File %s, offset %i bytes.\", dir_full_path, offset_vector[0]);\n buf_offset = offset_vector[1];\n if (offset_vector[0] == offset_vector[1]) {\n ++buf_offset;\n log_debug(\"Regex match is of length zero. Advancing offset one byte.\");\n }\n\n realloc_matches(&matches, &matches_size, matches_len + matches_spare);\n\n matches[matches_len].start = offset_vector[0];\n matches[matches_len].end = offset_vector[1];\n matches_len++;\n\n if (opts.max_matches_per_file > 0 && matches_len >= opts.max_matches_per_file) {\n log_err(\"Too many matches in %s. Skipping the rest of this file.\", dir_full_path);\n break;\n }\n }\n } else {\n while (buf_offset < buf_len) {\n const char *line;\n size_t line_len = buf_getline(&line, buf, buf_len, buf_offset);\n if (!line) {\n break;\n }\n size_t line_offset = 0;\n while (line_offset < line_len) {\n int rv = pcre_exec(opts.re, opts.re_extra, line, line_len, line_offset, 0, offset_vector, 3);\n if (rv < 0) {\n break;\n }\n size_t line_to_buf = buf_offset + line_offset;\n log_debug(\"Regex match found. File %s, offset %i bytes.\", dir_full_path, offset_vector[0]);\n line_offset = offset_vector[1];\n if (offset_vector[0] == offset_vector[1]) {\n ++line_offset;\n log_debug(\"Regex match is of length zero. Advancing offset one byte.\");\n }\n\n realloc_matches(&matches, &matches_size, matches_len + matches_spare);\n\n matches[matches_len].start = offset_vector[0] + line_to_buf;\n matches[matches_len].end = offset_vector[1] + line_to_buf;\n matches_len++;\n\n if (opts.max_matches_per_file > 0 && matches_len >= opts.max_matches_per_file) {\n log_err(\"Too many matches in %s. Skipping the rest of this file.\", dir_full_path);\n goto multiline_done;\n }\n }\n buf_offset += line_len + 1;\n }\n }\n }\n\nmultiline_done:\n\n if (opts.invert_match) {\n matches_len = invert_matches(buf, buf_len, matches, matches_len);\n }\n\n if (opts.stats) {\n pthread_mutex_lock(&stats_mtx);\n stats.total_bytes += buf_len;\n stats.total_files++;\n stats.total_matches += matches_len;\n if (matches_len > 0) {\n stats.total_file_matches++;\n }\n pthread_mutex_unlock(&stats_mtx);\n }\n\n if (!opts.print_nonmatching_files && (matches_len > 0 || opts.print_all_paths)) {\n if (binary == -1 && !opts.print_filename_only) {\n binary = is_binary((const void *)buf, buf_len);\n }\n pthread_mutex_lock(&print_mtx);\n if (opts.print_filename_only) {\n if (opts.print_count) {\n print_path_count(dir_full_path, opts.path_sep, (size_t)matches_len);\n } else {\n print_path(dir_full_path, opts.path_sep);\n }\n } else if (binary) {\n print_binary_file_matches(dir_full_path);\n } else {\n print_file_matches(dir_full_path, buf, buf_len, matches, matches_len);\n }\n pthread_mutex_unlock(&print_mtx);\n opts.match_found = 1;\n } else if (opts.search_stream && opts.passthrough) {\n fprintf(out_fd, \"%s\", buf);\n } else {\n log_debug(\"No match in %s\", dir_full_path);\n }\n\n if (matches_len == 0 && opts.search_stream) {\n print_context_append(buf, buf_len - 1);\n }\n\n if (matches_size > 0) {\n free(matches);\n }\n\n /* FIXME: handle case where matches_len > SSIZE_MAX */\n return (ssize_t)matches_len;\n}\n\n/* Return value: -1 if skipped, otherwise # of matches */\n/* TODO: this will only match single lines. multi-line regexes silently don't match */\nssize_t search_stream(FILE *stream, const char *path) {\n char *line = NULL;\n ssize_t matches_count = 0;\n ssize_t line_len = 0;\n size_t line_cap = 0;\n size_t i;\n\n print_init_context();\n\n for (i = 1; (line_len = getline(&line, &line_cap, stream)) > 0; i++) {\n ssize_t result;\n opts.stream_line_num = i;\n result = search_buf(line, line_len, path);\n if (result > 0) {\n if (matches_count == -1) {\n matches_count = 0;\n }\n matches_count += result;\n } else if (matches_count <= 0 && result == -1) {\n matches_count = -1;\n }\n if (line[line_len - 1] == '\\n') {\n line_len--;\n }\n print_trailing_context(path, line, line_len);\n }\n\n free(line);\n print_cleanup_context();\n return matches_count;\n}\n\nvoid search_file(const char *file_full_path) {\n int fd = -1;\n off_t f_len = 0;\n char *buf = NULL;\n struct stat statbuf;\n int rv = 0;\n int matches_count = -1;\n FILE *fp = NULL;\n\n rv = stat(file_full_path, &statbuf);\n if (rv != 0) {\n log_err(\"Skipping %s: Error fstat()ing file.\", file_full_path);\n goto cleanup;\n }\n\n if (opts.stdout_inode != 0 && opts.stdout_inode == statbuf.st_ino) {\n log_debug(\"Skipping %s: stdout is redirected to it\", file_full_path);\n goto cleanup;\n }\n\n // handling only regular files and FIFOs\n if (!S_ISREG(statbuf.st_mode) && !S_ISFIFO(statbuf.st_mode)) {\n log_err(\"Skipping %s: Mode %u is not a file.\", file_full_path, statbuf.st_mode);\n goto cleanup;\n }\n\n fd = open(file_full_path, O_RDONLY);\n if (fd < 0) {\n /* XXXX: strerror is not thread-safe */\n log_err(\"Skipping %s: Error opening file: %s\", file_full_path, strerror(errno));\n goto cleanup;\n }\n\n // repeating stat check with file handle to prevent TOCTOU issue\n rv = fstat(fd, &statbuf);\n if (rv != 0) {\n log_err(\"Skipping %s: Error fstat()ing file.\", file_full_path);\n goto cleanup;\n }\n\n if (opts.stdout_inode != 0 && opts.stdout_inode == statbuf.st_ino) {\n log_debug(\"Skipping %s: stdout is redirected to it\", file_full_path);\n goto cleanup;\n }\n\n // handling only regular files and FIFOs\n if (!S_ISREG(statbuf.st_mode) && !S_ISFIFO(statbuf.st_mode)) {\n log_err(\"Skipping %s: Mode %u is not a file.\", file_full_path, statbuf.st_mode);\n goto cleanup;\n }\n\n print_init_context();\n\n if (statbuf.st_mode & S_IFIFO) {\n log_debug(\"%s is a named pipe. stream searching\", file_full_path);\n fp = fdopen(fd, \"r\");\n matches_count = search_stream(fp, file_full_path);\n fclose(fp);\n goto cleanup;\n }\n\n f_len = statbuf.st_size;\n\n if (f_len == 0) {\n if (opts.query[0] == '.' && opts.query_len == 1 && !opts.literal && opts.search_all_files) {\n matches_count = search_buf(buf, f_len, file_full_path);\n } else {\n log_debug(\"Skipping %s: file is empty.\", file_full_path);\n }\n goto cleanup;\n }\n\n if (!opts.literal && f_len > INT_MAX) {\n log_err(\"Skipping %s: pcre_exec() can't handle files larger than %i bytes.\", file_full_path, INT_MAX);\n goto cleanup;\n }\n\n#ifdef _WIN32\n {\n HANDLE hmmap = CreateFileMapping(\n (HANDLE)_get_osfhandle(fd), 0, PAGE_READONLY, 0, f_len, NULL);\n buf = (char *)MapViewOfFile(hmmap, FILE_SHARE_READ, 0, 0, f_len);\n if (hmmap != NULL)\n CloseHandle(hmmap);\n }\n if (buf == NULL) {\n FormatMessageA(\n FORMAT_MESSAGE_ALLOCATE_BUFFER |\n FORMAT_MESSAGE_FROM_SYSTEM |\n FORMAT_MESSAGE_IGNORE_INSERTS,\n NULL, GetLastError(), 0, (void *)&buf, 0, NULL);\n log_err(\"File %s failed to load: %s.\", file_full_path, buf);\n LocalFree((void *)buf);\n goto cleanup;\n }\n#else\n\n if (opts.mmap) {\n buf = mmap(0, f_len, PROT_READ, MAP_PRIVATE, fd, 0);\n if (buf == MAP_FAILED) {\n log_err(\"File %s failed to load: %s.\", file_full_path, strerror(errno));\n goto cleanup;\n }\n#if HAVE_MADVISE\n madvise(buf, f_len, MADV_SEQUENTIAL);\n#elif HAVE_POSIX_FADVISE\n posix_fadvise(fd, 0, f_len, POSIX_MADV_SEQUENTIAL);\n#endif\n } else {\n buf = ag_malloc(f_len);\n\n ssize_t bytes_read = 0;\n\n if (!opts.search_binary_files) {\n bytes_read += read(fd, buf, ag_min(f_len, 512));\n // Optimization: If skipping binary files, don't read the whole buffer before checking if binary or not.\n if (is_binary(buf, f_len)) {\n log_debug(\"File %s is binary. Skipping...\", file_full_path);\n goto cleanup;\n }\n }\n\n while (bytes_read < f_len) {\n bytes_read += read(fd, buf + bytes_read, f_len);\n }\n if (bytes_read != f_len) {\n die(\"File %s read(): expected to read %u bytes but read %u\", file_full_path, f_len, bytes_read);\n }\n }\n#endif\n\n if (opts.search_zip_files) {\n ag_compression_type zip_type = is_zipped(buf, f_len);\n if (zip_type != AG_NO_COMPRESSION) {\n#if HAVE_FOPENCOOKIE\n log_debug(\"%s is a compressed file. stream searching\", file_full_path);\n fp = decompress_open(fd, \"r\", zip_type);\n matches_count = search_stream(fp, file_full_path);\n fclose(fp);\n#else\n int _buf_len = (int)f_len;\n char *_buf = decompress(zip_type, buf, f_len, file_full_path, &_buf_len);\n if (_buf == NULL || _buf_len == 0) {\n log_err(\"Cannot decompress zipped file %s\", file_full_path);\n goto cleanup;\n }\n matches_count = search_buf(_buf, _buf_len, file_full_path);\n free(_buf);\n#endif\n goto cleanup;\n }\n }\n\n matches_count = search_buf(buf, f_len, file_full_path);\n\ncleanup:\n\n if (opts.print_nonmatching_files && matches_count == 0) {\n pthread_mutex_lock(&print_mtx);\n print_path(file_full_path, opts.path_sep);\n pthread_mutex_unlock(&print_mtx);\n opts.match_found = 1;\n }\n\n print_cleanup_context();\n if (buf != NULL) {\n#ifdef _WIN32\n UnmapViewOfFile(buf);\n#else\n if (opts.mmap) {\n if (buf != MAP_FAILED) {\n munmap(buf, f_len);\n }\n } else {\n free(buf);\n }\n#endif\n }\n if (fd != -1) {\n close(fd);\n }\n}\n\nvoid *search_file_worker(void *i) {\n work_queue_t *queue_item;\n int worker_id = *(int *)i;\n\n log_debug(\"Worker %i started\", worker_id);\n while (TRUE) {\n pthread_mutex_lock(&work_queue_mtx);\n while (work_queue == NULL) {\n if (done_adding_files) {\n pthread_mutex_unlock(&work_queue_mtx);\n log_debug(\"Worker %i finished.\", worker_id);\n pthread_exit(NULL);\n }\n pthread_cond_wait(&files_ready, &work_queue_mtx);\n }\n queue_item = work_queue;\n work_queue = work_queue->next;\n if (work_queue == NULL) {\n work_queue_tail = NULL;\n }\n pthread_mutex_unlock(&work_queue_mtx);\n\n search_file(queue_item->path);\n free(queue_item->path);\n free(queue_item);\n }\n}\n\nstatic int check_symloop_enter(const char *path, dirkey_t *outkey) {\n#ifdef _WIN32\n return SYMLOOP_OK;\n#else\n struct stat buf;\n symdir_t *item_found = NULL;\n symdir_t *new_item = NULL;\n\n memset(outkey, 0, sizeof(dirkey_t));\n outkey->dev = 0;\n outkey->ino = 0;\n\n int res = stat(path, &buf);\n if (res != 0) {\n log_err(\"Error stat()ing: %s\", path);\n return SYMLOOP_ERROR;\n }\n\n outkey->dev = buf.st_dev;\n outkey->ino = buf.st_ino;\n\n HASH_FIND(hh, symhash, outkey, sizeof(dirkey_t), item_found);\n if (item_found) {\n return SYMLOOP_LOOP;\n }\n\n new_item = (symdir_t *)ag_malloc(sizeof(symdir_t));\n memcpy(&new_item->key, outkey, sizeof(dirkey_t));\n HASH_ADD(hh, symhash, key, sizeof(dirkey_t), new_item);\n return SYMLOOP_OK;\n#endif\n}\n\nstatic int check_symloop_leave(dirkey_t *dirkey) {\n#ifdef _WIN32\n return SYMLOOP_OK;\n#else\n symdir_t *item_found = NULL;\n\n if (dirkey->dev == 0 && dirkey->ino == 0) {\n return SYMLOOP_ERROR;\n }\n\n HASH_FIND(hh, symhash, dirkey, sizeof(dirkey_t), item_found);\n if (!item_found) {\n log_err(\"item not found! weird stuff...\\n\");\n return SYMLOOP_ERROR;\n }\n\n HASH_DELETE(hh, symhash, item_found);\n free(item_found);\n return SYMLOOP_OK;\n#endif\n}\n\n/* TODO: Append matches to some data structure instead of just printing them out.\n * Then ag can have sweet summaries of matches/files scanned/time/etc.\n */\nvoid search_dir(ignores *ig, const char *base_path, const char *path, const int depth,\n dev_t original_dev) {\n struct dirent **dir_list = NULL;\n struct dirent *dir = NULL;\n scandir_baton_t scandir_baton;\n int results = 0;\n size_t base_path_len = 0;\n const char *path_start = path;\n\n char *dir_full_path = NULL;\n const char *ignore_file = NULL;\n int i;\n\n int symres;\n dirkey_t current_dirkey;\n\n symres = check_symloop_enter(path, ¤t_dirkey);\n if (symres == SYMLOOP_LOOP) {\n log_err(\"Recursive directory loop: %s\", path);\n return;\n }\n\n /* find .*ignore files to load ignore patterns from */\n for (i = 0; opts.skip_vcs_ignores ? (i == 0) : (ignore_pattern_files[i] != NULL); i++) {\n ignore_file = ignore_pattern_files[i];\n ag_asprintf(&dir_full_path, \"%s/%s\", path, ignore_file);\n load_ignore_patterns(ig, dir_full_path);\n free(dir_full_path);\n dir_full_path = NULL;\n }\n\n /* path_start is the part of path that isn't in base_path\n * base_path will have a trailing '/' because we put it there in parse_options\n */\n base_path_len = base_path ? strlen(base_path) : 0;\n for (i = 0; ((size_t)i < base_path_len) && (path[i]) && (base_path[i] == path[i]); i++) {\n path_start = path + i + 1;\n }\n log_debug(\"search_dir: path is '%s', base_path is '%s', path_start is '%s'\", path, base_path, path_start);\n\n scandir_baton.ig = ig;\n scandir_baton.base_path = base_path;\n scandir_baton.base_path_len = base_path_len;\n scandir_baton.path_start = path_start;\n\n results = ag_scandir(path, &dir_list, &filename_filter, &scandir_baton);\n if (results == 0) {\n log_debug(\"No results found in directory %s\", path);\n goto search_dir_cleanup;\n } else if (results == -1) {\n if (errno == ENOTDIR) {\n /* Not a directory. Probably a file. */\n if (depth == 0 && opts.paths_len == 1) {\n /* If we're only searching one file, don't print the filename header at the top. */\n if (opts.print_path == PATH_PRINT_DEFAULT || opts.print_path == PATH_PRINT_DEFAULT_EACH_LINE) {\n opts.print_path = PATH_PRINT_NOTHING;\n }\n /* If we're only searching one file and --only-matching is specified, disable line numbers too. */\n if (opts.only_matching && opts.print_path == PATH_PRINT_NOTHING) {\n opts.print_line_numbers = FALSE;\n }\n }\n search_file(path);\n } else {\n log_err(\"Error opening directory %s: %s\", path, strerror(errno));\n }\n goto search_dir_cleanup;\n }\n\n int offset_vector[3];\n int rc = 0;\n work_queue_t *queue_item;\n\n for (i = 0; i < results; i++) {\n queue_item = NULL;\n dir = dir_list[i];\n ag_asprintf(&dir_full_path, \"%s/%s\", path, dir->d_name);\n#ifndef _WIN32\n if (opts.one_dev) {\n struct stat s;\n if (lstat(dir_full_path, &s) != 0) {\n log_err(\"Failed to get device information for %s. Skipping...\", dir->d_name);\n goto cleanup;\n }\n if (s.st_dev != original_dev) {\n log_debug(\"File %s crosses a device boundary (is probably a mount point.) Skipping...\", dir->d_name);\n goto cleanup;\n }\n }\n#endif\n\n /* If a link points to a directory then we need to treat it as a directory. */\n if (!opts.follow_symlinks && is_symlink(path, dir)) {\n log_debug(\"File %s ignored becaused it's a symlink\", dir->d_name);\n goto cleanup;\n }\n\n if (!is_directory(path, dir)) {\n if (opts.file_search_regex) {\n rc = pcre_exec(opts.file_search_regex, NULL, dir_full_path, strlen(dir_full_path),\n 0, 0, offset_vector, 3);\n if (rc < 0) { /* no match */\n log_debug(\"Skipping %s due to file_search_regex.\", dir_full_path);\n goto cleanup;\n } else if (opts.match_files) {\n log_debug(\"match_files: file_search_regex matched for %s.\", dir_full_path);\n pthread_mutex_lock(&print_mtx);\n print_path(dir_full_path, opts.path_sep);\n pthread_mutex_unlock(&print_mtx);\n opts.match_found = 1;\n goto cleanup;\n }\n }\n\n queue_item = ag_malloc(sizeof(work_queue_t));\n queue_item->path = dir_full_path;\n queue_item->next = NULL;\n pthread_mutex_lock(&work_queue_mtx);\n if (work_queue_tail == NULL) {\n work_queue = queue_item;\n } else {\n work_queue_tail->next = queue_item;\n }\n work_queue_tail = queue_item;\n pthread_cond_signal(&files_ready);\n pthread_mutex_unlock(&work_queue_mtx);\n log_debug(\"%s added to work queue\", dir_full_path);\n } else if (opts.recurse_dirs) {\n if (depth < opts.max_search_depth || opts.max_search_depth == -1) {\n log_debug(\"Searching dir %s\", dir_full_path);\n ignores *child_ig;\n#ifdef HAVE_DIRENT_DNAMLEN\n child_ig = init_ignore(ig, dir->d_name, dir->d_namlen);\n#else\n child_ig = init_ignore(ig, dir->d_name, strlen(dir->d_name));\n#endif\n search_dir(child_ig, base_path, dir_full_path, depth + 1,\n original_dev);\n cleanup_ignore(child_ig);\n } else {\n if (opts.max_search_depth == DEFAULT_MAX_SEARCH_DEPTH) {\n /*\n * If the user didn't intentionally specify a particular depth,\n * this is a warning...\n */\n log_err(\"Skipping %s. Use the --depth option to search deeper.\", dir_full_path);\n } else {\n /* ... if they did, let's settle for debug. */\n log_debug(\"Skipping %s. Use the --depth option to search deeper.\", dir_full_path);\n }\n }\n }\n\n cleanup:\n free(dir);\n dir = NULL;\n if (queue_item == NULL) {\n free(dir_full_path);\n dir_full_path = NULL;\n }\n }\n\nsearch_dir_cleanup:\n check_symloop_leave(¤t_dirkey);\n free(dir_list);\n dir_list = NULL;\n}\n" }, { "path": "src/search.h", "content": "#ifndef SEARCH_H\n#define SEARCH_H\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#ifdef _WIN32\n#include \n#else\n#include \n#endif\n#include \n#include \n\n#include \"config.h\"\n\n#ifdef HAVE_PTHREAD_H\n#include \n#endif\n\n#include \"decompress.h\"\n#include \"ignore.h\"\n#include \"log.h\"\n#include \"options.h\"\n#include \"print.h\"\n#include \"uthash.h\"\n#include \"util.h\"\n\nextern size_t alpha_skip_lookup[256];\nextern size_t *find_skip_lookup;\nextern uint8_t h_table[H_SIZE] __attribute__((aligned(64)));\n\nstruct work_queue_t {\n char *path;\n struct work_queue_t *next;\n};\ntypedef struct work_queue_t work_queue_t;\n\nextern work_queue_t *work_queue;\nextern work_queue_t *work_queue_tail;\nextern int done_adding_files;\nextern pthread_cond_t files_ready;\nextern pthread_mutex_t stats_mtx;\nextern pthread_mutex_t work_queue_mtx;\n\n\n/* For symlink loop detection */\n#define SYMLOOP_ERROR (-1)\n#define SYMLOOP_OK (0)\n#define SYMLOOP_LOOP (1)\n\ntypedef struct {\n dev_t dev;\n ino_t ino;\n} dirkey_t;\n\ntypedef struct {\n dirkey_t key;\n UT_hash_handle hh;\n} symdir_t;\n\nextern symdir_t *symhash;\n\nssize_t search_buf(const char *buf, const size_t buf_len,\n const char *dir_full_path);\nssize_t search_stream(FILE *stream, const char *path);\nvoid search_file(const char *file_full_path);\n\nvoid *search_file_worker(void *i);\n\nvoid search_dir(ignores *ig, const char *base_path, const char *path, const int depth, dev_t original_dev);\n\n#endif\n" }, { "path": "src/uthash.h", "content": "/*\nCopyright (c) 2003-2014, Troy D. Hanson http://troydhanson.github.com/uthash/\nAll rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n * Redistributions of source code must retain the above copyright\n notice, this list of conditions and the following disclaimer.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS\nIS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED\nTO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A\nPARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER\nOR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\nEXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\nPROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\nPROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF\nLIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING\nNEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\nSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n*/\n\n#ifndef UTHASH_H\n#define UTHASH_H\n\n#include /* ptrdiff_t */\n#include /* exit() */\n#include /* memcmp,strlen */\n\n/* These macros use decltype or the earlier __typeof GNU extension.\n As decltype is only available in newer compilers (VS2010 or gcc 4.3+\n when compiling c++ source) this code uses whatever method is needed\n or, for VS2008 where neither is available, uses casting workarounds. */\n#if defined(_MSC_VER) /* MS compiler */\n#if _MSC_VER >= 1600 && defined(__cplusplus) /* VS2010 or newer in C++ mode */\n#define DECLTYPE(x) (decltype(x))\n#else /* VS2008 or older (or VS2010 in C mode) */\n#define NO_DECLTYPE\n#define DECLTYPE(x)\n#endif\n#elif defined(__BORLANDC__) || defined(__LCC__) || defined(__WATCOMC__)\n#define NO_DECLTYPE\n#define DECLTYPE(x)\n#else /* GNU, Sun and other compilers */\n#define DECLTYPE(x) (__typeof(x))\n#endif\n\n#ifdef NO_DECLTYPE\n#define DECLTYPE_ASSIGN(dst, src) \\\n do { \\\n char **_da_dst = (char **)(&(dst)); \\\n *_da_dst = (char *)(src); \\\n } while (0)\n#else\n#define DECLTYPE_ASSIGN(dst, src) \\\n do { \\\n (dst) = DECLTYPE(dst)(src); \\\n } while (0)\n#endif\n\n/* a number of the hash function use uint32_t which isn't defined on Pre VS2010 */\n#if defined(_WIN32)\n#if defined(_MSC_VER) && _MSC_VER >= 1600\n#include \n#elif defined(__WATCOMC__)\n#include \n#else\ntypedef unsigned int uint32_t;\ntypedef unsigned char uint8_t;\n#endif\n#else\n#include \n#endif\n\n#define UTHASH_VERSION 1.9.9\n\n#ifndef uthash_fatal\n#define uthash_fatal(msg) exit(-1) /* fatal error (out of memory,etc) */\n#endif\n#ifndef uthash_malloc\n#define uthash_malloc(sz) malloc(sz) /* malloc fcn */\n#endif\n#ifndef uthash_free\n#define uthash_free(ptr, sz) free(ptr) /* free fcn */\n#endif\n\n#ifndef uthash_noexpand_fyi\n#define uthash_noexpand_fyi(tbl) /* can be defined to log noexpand */\n#endif\n#ifndef uthash_expand_fyi\n#define uthash_expand_fyi(tbl) /* can be defined to log expands */\n#endif\n\n/* initial number of buckets */\n#define HASH_INITIAL_NUM_BUCKETS 32 /* initial number of buckets */\n#define HASH_INITIAL_NUM_BUCKETS_LOG2 5 /* lg2 of initial number of buckets */\n#define HASH_BKT_CAPACITY_THRESH 10 /* expand when bucket count reaches */\n\n/* calculate the element whose hash handle address is hhe */\n#define ELMT_FROM_HH(tbl, hhp) ((void *)(((char *)(hhp)) - ((tbl)->hho)))\n\n#define HASH_FIND(hh, head, keyptr, keylen, out) \\\n do { \\\n unsigned _hf_bkt, _hf_hashv; \\\n out = NULL; \\\n if (head) { \\\n HASH_FCN(keyptr, keylen, (head)->hh.tbl->num_buckets, _hf_hashv, _hf_bkt); \\\n if (HASH_BLOOM_TEST((head)->hh.tbl, _hf_hashv)) { \\\n HASH_FIND_IN_BKT((head)->hh.tbl, hh, (head)->hh.tbl->buckets[_hf_bkt], keyptr, keylen, out); \\\n } \\\n } \\\n } while (0)\n\n#ifdef HASH_BLOOM\n#define HASH_BLOOM_BITLEN (1ULL << HASH_BLOOM)\n#define HASH_BLOOM_BYTELEN (HASH_BLOOM_BITLEN / 8) + ((HASH_BLOOM_BITLEN % 8) ? 1 : 0)\n#define HASH_BLOOM_MAKE(tbl) \\\n do { \\\n (tbl)->bloom_nbits = HASH_BLOOM; \\\n (tbl)->bloom_bv = (uint8_t *)uthash_malloc(HASH_BLOOM_BYTELEN); \\\n if (!((tbl)->bloom_bv)) { \\\n uthash_fatal(\"out of memory\"); \\\n } \\\n memset((tbl)->bloom_bv, 0, HASH_BLOOM_BYTELEN); \\\n (tbl)->bloom_sig = HASH_BLOOM_SIGNATURE; \\\n } while (0)\n\n#define HASH_BLOOM_FREE(tbl) \\\n do { \\\n uthash_free((tbl)->bloom_bv, HASH_BLOOM_BYTELEN); \\\n } while (0)\n\n#define HASH_BLOOM_BITSET(bv, idx) (bv[(idx) / 8] |= (1U << ((idx) % 8)))\n#define HASH_BLOOM_BITTEST(bv, idx) (bv[(idx) / 8] & (1U << ((idx) % 8)))\n\n#define HASH_BLOOM_ADD(tbl, hashv) \\\n HASH_BLOOM_BITSET((tbl)->bloom_bv, (hashv & (uint32_t)((1ULL << (tbl)->bloom_nbits) - 1)))\n\n#define HASH_BLOOM_TEST(tbl, hashv) \\\n HASH_BLOOM_BITTEST((tbl)->bloom_bv, (hashv & (uint32_t)((1ULL << (tbl)->bloom_nbits) - 1)))\n\n#else\n#define HASH_BLOOM_MAKE(tbl)\n#define HASH_BLOOM_FREE(tbl)\n#define HASH_BLOOM_ADD(tbl, hashv)\n#define HASH_BLOOM_TEST(tbl, hashv) (1)\n#define HASH_BLOOM_BYTELEN 0\n#endif\n\n#define HASH_MAKE_TABLE(hh, head) \\\n do { \\\n (head)->hh.tbl = (UT_hash_table *)uthash_malloc(sizeof(UT_hash_table)); \\\n if (!((head)->hh.tbl)) { \\\n uthash_fatal(\"out of memory\"); \\\n } \\\n memset((head)->hh.tbl, 0, sizeof(UT_hash_table)); \\\n (head)->hh.tbl->tail = &((head)->hh); \\\n (head)->hh.tbl->num_buckets = HASH_INITIAL_NUM_BUCKETS; \\\n (head)->hh.tbl->log2_num_buckets = HASH_INITIAL_NUM_BUCKETS_LOG2; \\\n (head)->hh.tbl->hho = (char *)(&(head)->hh) - (char *)(head); \\\n (head)->hh.tbl->buckets = (UT_hash_bucket *)uthash_malloc(HASH_INITIAL_NUM_BUCKETS * sizeof(struct UT_hash_bucket)); \\\n if (!(head)->hh.tbl->buckets) { \\\n uthash_fatal(\"out of memory\"); \\\n } \\\n memset((head)->hh.tbl->buckets, 0, HASH_INITIAL_NUM_BUCKETS * sizeof(struct UT_hash_bucket)); \\\n HASH_BLOOM_MAKE((head)->hh.tbl); \\\n (head)->hh.tbl->signature = HASH_SIGNATURE; \\\n } while (0)\n\n#define HASH_ADD(hh, head, fieldname, keylen_in, add) \\\n HASH_ADD_KEYPTR(hh, head, &((add)->fieldname), keylen_in, add)\n\n#define HASH_REPLACE(hh, head, fieldname, keylen_in, add, replaced) \\\n do { \\\n replaced = NULL; \\\n HASH_FIND(hh, head, &((add)->fieldname), keylen_in, replaced); \\\n if (replaced != NULL) { \\\n HASH_DELETE(hh, head, replaced); \\\n }; \\\n HASH_ADD(hh, head, fieldname, keylen_in, add); \\\n } while (0)\n\n#define HASH_ADD_KEYPTR(hh, head, keyptr, keylen_in, add) \\\n do { \\\n unsigned _ha_bkt; \\\n (add)->hh.next = NULL; \\\n (add)->hh.key = (char *)(keyptr); \\\n (add)->hh.keylen = (unsigned)(keylen_in); \\\n if (!(head)) { \\\n head = (add); \\\n (head)->hh.prev = NULL; \\\n HASH_MAKE_TABLE(hh, head); \\\n } else { \\\n (head)->hh.tbl->tail->next = (add); \\\n (add)->hh.prev = ELMT_FROM_HH((head)->hh.tbl, (head)->hh.tbl->tail); \\\n (head)->hh.tbl->tail = &((add)->hh); \\\n } \\\n (head)->hh.tbl->num_items++; \\\n (add)->hh.tbl = (head)->hh.tbl; \\\n HASH_FCN(keyptr, keylen_in, (head)->hh.tbl->num_buckets, (add)->hh.hashv, _ha_bkt); \\\n HASH_ADD_TO_BKT((head)->hh.tbl->buckets[_ha_bkt], &(add)->hh); \\\n HASH_BLOOM_ADD((head)->hh.tbl, (add)->hh.hashv); \\\n HASH_EMIT_KEY(hh, head, keyptr, keylen_in); \\\n HASH_FSCK(hh, head); \\\n } while (0)\n\n#define HASH_TO_BKT(hashv, num_bkts, bkt) \\\n do { \\\n bkt = ((hashv) & (num_bkts - 1)); \\\n } while (0)\n\n/* delete \"delptr\" from the hash table.\n * \"the usual\" patch-up process for the app-order doubly-linked-list.\n * The use of _hd_hh_del below deserves special explanation.\n * These used to be expressed using (delptr) but that led to a bug\n * if someone used the same symbol for the head and deletee, like\n * HASH_DELETE(hh,users,users);\n * We want that to work, but by changing the head (users) below\n * we were forfeiting our ability to further refer to the deletee (users)\n * in the patch-up process. Solution: use scratch space to\n * copy the deletee pointer, then the latter references are via that\n * scratch pointer rather than through the repointed (users) symbol.\n */\n#define HASH_DELETE(hh, head, delptr) \\\n do { \\\n unsigned _hd_bkt; \\\n struct UT_hash_handle *_hd_hh_del; \\\n if (((delptr)->hh.prev == NULL) && ((delptr)->hh.next == NULL)) { \\\n uthash_free((head)->hh.tbl->buckets, (head)->hh.tbl->num_buckets * sizeof(struct UT_hash_bucket)); \\\n HASH_BLOOM_FREE((head)->hh.tbl); \\\n uthash_free((head)->hh.tbl, sizeof(UT_hash_table)); \\\n head = NULL; \\\n } else { \\\n _hd_hh_del = &((delptr)->hh); \\\n if ((delptr) == ELMT_FROM_HH((head)->hh.tbl, (head)->hh.tbl->tail)) { \\\n (head)->hh.tbl->tail = (UT_hash_handle *)((ptrdiff_t)((delptr)->hh.prev) + (head)->hh.tbl->hho); \\\n } \\\n if ((delptr)->hh.prev) { \\\n ((UT_hash_handle *)((ptrdiff_t)((delptr)->hh.prev) + (head)->hh.tbl->hho))->next = (delptr)->hh.next; \\\n } else { \\\n DECLTYPE_ASSIGN(head, (delptr)->hh.next); \\\n } \\\n if (_hd_hh_del->next) { \\\n ((UT_hash_handle *)((ptrdiff_t)_hd_hh_del->next + (head)->hh.tbl->hho))->prev = _hd_hh_del->prev; \\\n } \\\n HASH_TO_BKT(_hd_hh_del->hashv, (head)->hh.tbl->num_buckets, _hd_bkt); \\\n HASH_DEL_IN_BKT(hh, (head)->hh.tbl->buckets[_hd_bkt], _hd_hh_del); \\\n (head)->hh.tbl->num_items--; \\\n } \\\n HASH_FSCK(hh, head); \\\n } while (0)\n\n\n/* convenience forms of HASH_FIND/HASH_ADD/HASH_DEL */\n#define HASH_FIND_STR(head, findstr, out) \\\n HASH_FIND(hh, head, findstr, strlen(findstr), out)\n#define HASH_ADD_STR(head, strfield, add) \\\n HASH_ADD(hh, head, strfield[0], strlen(add->strfield), add)\n#define HASH_REPLACE_STR(head, strfield, add, replaced) \\\n HASH_REPLACE(hh, head, strfield[0], strlen(add->strfield), add, replaced)\n#define HASH_FIND_INT(head, findint, out) \\\n HASH_FIND(hh, head, findint, sizeof(int), out)\n#define HASH_ADD_INT(head, intfield, add) \\\n HASH_ADD(hh, head, intfield, sizeof(int), add)\n#define HASH_REPLACE_INT(head, intfield, add, replaced) \\\n HASH_REPLACE(hh, head, intfield, sizeof(int), add, replaced)\n#define HASH_FIND_PTR(head, findptr, out) \\\n HASH_FIND(hh, head, findptr, sizeof(void *), out)\n#define HASH_ADD_PTR(head, ptrfield, add) \\\n HASH_ADD(hh, head, ptrfield, sizeof(void *), add)\n#define HASH_REPLACE_PTR(head, ptrfield, add, replaced) \\\n HASH_REPLACE(hh, head, ptrfield, sizeof(void *), add, replaced)\n#define HASH_DEL(head, delptr) \\\n HASH_DELETE(hh, head, delptr)\n\n/* HASH_FSCK checks hash integrity on every add/delete when HASH_DEBUG is defined.\n * This is for uthash developer only; it compiles away if HASH_DEBUG isn't defined.\n */\n#ifdef HASH_DEBUG\n#define HASH_OOPS(...) \\\n do { \\\n fprintf(stderr, __VA_ARGS__); \\\n exit(-1); \\\n } while (0)\n#define HASH_FSCK(hh, head) \\\n do { \\\n unsigned _bkt_i; \\\n unsigned _count, _bkt_count; \\\n char *_prev; \\\n struct UT_hash_handle *_thh; \\\n if (head) { \\\n _count = 0; \\\n for (_bkt_i = 0; _bkt_i < (head)->hh.tbl->num_buckets; _bkt_i++) { \\\n _bkt_count = 0; \\\n _thh = (head)->hh.tbl->buckets[_bkt_i].hh_head; \\\n _prev = NULL; \\\n while (_thh) { \\\n if (_prev != (char *)(_thh->hh_prev)) { \\\n HASH_OOPS(\"invalid hh_prev %p, actual %p\\n\", _thh->hh_prev, _prev); \\\n } \\\n _bkt_count++; \\\n _prev = (char *)(_thh); \\\n _thh = _thh->hh_next; \\\n } \\\n _count += _bkt_count; \\\n if ((head)->hh.tbl->buckets[_bkt_i].count != _bkt_count) { \\\n HASH_OOPS(\"invalid bucket count %d, actual %d\\n\", (head)->hh.tbl->buckets[_bkt_i].count, _bkt_count); \\\n } \\\n } \\\n if (_count != (head)->hh.tbl->num_items) { \\\n HASH_OOPS(\"invalid hh item count %d, actual %d\\n\", (head)->hh.tbl->num_items, _count); \\\n } \\\n /* traverse hh in app order; check next/prev integrity, count */ \\\n _count = 0; \\\n _prev = NULL; \\\n _thh = &(head)->hh; \\\n while (_thh) { \\\n _count++; \\\n if (_prev != (char *)(_thh->prev)) { \\\n HASH_OOPS(\"invalid prev %p, actual %p\\n\", _thh->prev, _prev); \\\n } \\\n _prev = (char *)ELMT_FROM_HH((head)->hh.tbl, _thh); \\\n _thh = (_thh->next ? (UT_hash_handle *)((char *)(_thh->next) + (head)->hh.tbl->hho) : NULL); \\\n } \\\n if (_count != (head)->hh.tbl->num_items) { \\\n HASH_OOPS(\"invalid app item count %d, actual %d\\n\", (head)->hh.tbl->num_items, _count); \\\n } \\\n } \\\n } while (0)\n#else\n#define HASH_FSCK(hh, head)\n#endif\n\n/* When compiled with -DHASH_EMIT_KEYS, length-prefixed keys are emitted to\n * the descriptor to which this macro is defined for tuning the hash function.\n * The app can #include to get the prototype for write(2). */\n#ifdef HASH_EMIT_KEYS\n#define HASH_EMIT_KEY(hh, head, keyptr, fieldlen) \\\n do { \\\n unsigned _klen = fieldlen; \\\n write(HASH_EMIT_KEYS, &_klen, sizeof(_klen)); \\\n write(HASH_EMIT_KEYS, keyptr, fieldlen); \\\n } while (0)\n#else\n#define HASH_EMIT_KEY(hh, head, keyptr, fieldlen)\n#endif\n\n/* default to Jenkin's hash unless overridden e.g. DHASH_FUNCTION=HASH_SAX */\n#ifdef HASH_FUNCTION\n#define HASH_FCN HASH_FUNCTION\n#else\n#define HASH_FCN HASH_JEN\n#endif\n\n/* The Bernstein hash function, used in Perl prior to v5.6. Note (x<<5+x)=x*33. */\n#define HASH_BER(key, keylen, num_bkts, hashv, bkt) \\\n do { \\\n unsigned _hb_keylen = keylen; \\\n char *_hb_key = (char *)(key); \\\n (hashv) = 0; \\\n while (_hb_keylen--) { \\\n (hashv) = (((hashv) << 5) + (hashv)) + *_hb_key++; \\\n } \\\n bkt = (hashv) & (num_bkts - 1); \\\n } while (0)\n\n\n/* SAX/FNV/OAT/JEN hash functions are macro variants of those listed at\n * http://eternallyconfuzzled.com/tuts/algorithms/jsw_tut_hashing.aspx */\n#define HASH_SAX(key, keylen, num_bkts, hashv, bkt) \\\n do { \\\n unsigned _sx_i; \\\n char *_hs_key = (char *)(key); \\\n hashv = 0; \\\n for (_sx_i = 0; _sx_i < keylen; _sx_i++) \\\n hashv ^= (hashv << 5) + (hashv >> 2) + _hs_key[_sx_i]; \\\n bkt = hashv & (num_bkts - 1); \\\n } while (0)\n/* FNV-1a variation */\n#define HASH_FNV(key, keylen, num_bkts, hashv, bkt) \\\n do { \\\n unsigned _fn_i; \\\n char *_hf_key = (char *)(key); \\\n hashv = 2166136261UL; \\\n for (_fn_i = 0; _fn_i < keylen; _fn_i++) \\\n hashv = hashv ^ _hf_key[_fn_i]; \\\n hashv = hashv * 16777619; \\\n bkt = hashv & (num_bkts - 1); \\\n } while (0)\n\n#define HASH_OAT(key, keylen, num_bkts, hashv, bkt) \\\n do { \\\n unsigned _ho_i; \\\n char *_ho_key = (char *)(key); \\\n hashv = 0; \\\n for (_ho_i = 0; _ho_i < keylen; _ho_i++) { \\\n hashv += _ho_key[_ho_i]; \\\n hashv += (hashv << 10); \\\n hashv ^= (hashv >> 6); \\\n } \\\n hashv += (hashv << 3); \\\n hashv ^= (hashv >> 11); \\\n hashv += (hashv << 15); \\\n bkt = hashv & (num_bkts - 1); \\\n } while (0)\n\n#define HASH_JEN_MIX(a, b, c) \\\n do { \\\n a -= b; \\\n a -= c; \\\n a ^= (c >> 13); \\\n b -= c; \\\n b -= a; \\\n b ^= (a << 8); \\\n c -= a; \\\n c -= b; \\\n c ^= (b >> 13); \\\n a -= b; \\\n a -= c; \\\n a ^= (c >> 12); \\\n b -= c; \\\n b -= a; \\\n b ^= (a << 16); \\\n c -= a; \\\n c -= b; \\\n c ^= (b >> 5); \\\n a -= b; \\\n a -= c; \\\n a ^= (c >> 3); \\\n b -= c; \\\n b -= a; \\\n b ^= (a << 10); \\\n c -= a; \\\n c -= b; \\\n c ^= (b >> 15); \\\n } while (0)\n\n#define HASH_JEN(key, keylen, num_bkts, hashv, bkt) \\\n do { \\\n unsigned _hj_i, _hj_j, _hj_k; \\\n unsigned char *_hj_key = (unsigned char *)(key); \\\n hashv = 0xfeedbeef; \\\n _hj_i = _hj_j = 0x9e3779b9; \\\n _hj_k = (unsigned)(keylen); \\\n while (_hj_k >= 12) { \\\n _hj_i += (_hj_key[0] + ((unsigned)_hj_key[1] << 8) + ((unsigned)_hj_key[2] << 16) + ((unsigned)_hj_key[3] << 24)); \\\n _hj_j += (_hj_key[4] + ((unsigned)_hj_key[5] << 8) + ((unsigned)_hj_key[6] << 16) + ((unsigned)_hj_key[7] << 24)); \\\n hashv += (_hj_key[8] + ((unsigned)_hj_key[9] << 8) + ((unsigned)_hj_key[10] << 16) + ((unsigned)_hj_key[11] << 24)); \\\n \\\n HASH_JEN_MIX(_hj_i, _hj_j, hashv); \\\n \\\n _hj_key += 12; \\\n _hj_k -= 12; \\\n } \\\n hashv += keylen; \\\n switch (_hj_k) { \\\n case 11: \\\n hashv += ((unsigned)_hj_key[10] << 24); \\\n /* fall through */ \\\n case 10: \\\n hashv += ((unsigned)_hj_key[9] << 16); \\\n /* fall through */ \\\n case 9: \\\n hashv += ((unsigned)_hj_key[8] << 8); \\\n /* fall through */ \\\n case 8: \\\n _hj_j += ((unsigned)_hj_key[7] << 24); \\\n /* fall through */ \\\n case 7: \\\n _hj_j += ((unsigned)_hj_key[6] << 16); \\\n /* fall through */ \\\n case 6: \\\n _hj_j += ((unsigned)_hj_key[5] << 8); \\\n /* fall through */ \\\n case 5: \\\n _hj_j += _hj_key[4]; \\\n /* fall through */ \\\n case 4: \\\n _hj_i += ((unsigned)_hj_key[3] << 24); \\\n /* fall through */ \\\n case 3: \\\n _hj_i += ((unsigned)_hj_key[2] << 16); \\\n /* fall through */ \\\n case 2: \\\n _hj_i += ((unsigned)_hj_key[1] << 8); \\\n /* fall through */ \\\n case 1: \\\n _hj_i += _hj_key[0]; \\\n } \\\n HASH_JEN_MIX(_hj_i, _hj_j, hashv); \\\n bkt = hashv & (num_bkts - 1); \\\n } while (0)\n\n/* The Paul Hsieh hash function */\n#undef get16bits\n#if (defined(__GNUC__) && defined(__i386__)) || defined(__WATCOMC__) || defined(_MSC_VER) || defined(__BORLANDC__) || defined(__TURBOC__)\n#define get16bits(d) (*((const uint16_t *)(d)))\n#endif\n\n#if !defined(get16bits)\n#define get16bits(d) ((((uint32_t)(((const uint8_t *)(d))[1])) << 8) + (uint32_t)(((const uint8_t *)(d))[0]))\n#endif\n#define HASH_SFH(key, keylen, num_bkts, hashv, bkt) \\\n do { \\\n unsigned char *_sfh_key = (unsigned char *)(key); \\\n uint32_t _sfh_tmp, _sfh_len = keylen; \\\n \\\n int _sfh_rem = _sfh_len & 3; \\\n _sfh_len >>= 2; \\\n hashv = 0xcafebabe; \\\n \\\n /* Main loop */ \\\n for (; _sfh_len > 0; _sfh_len--) { \\\n hashv += get16bits(_sfh_key); \\\n _sfh_tmp = (uint32_t)(get16bits(_sfh_key + 2)) << 11 ^ hashv; \\\n hashv = (hashv << 16) ^ _sfh_tmp; \\\n _sfh_key += 2 * sizeof(uint16_t); \\\n hashv += hashv >> 11; \\\n } \\\n \\\n /* Handle end cases */ \\\n switch (_sfh_rem) { \\\n case 3: \\\n hashv += get16bits(_sfh_key); \\\n hashv ^= hashv << 16; \\\n hashv ^= (uint32_t)(_sfh_key[sizeof(uint16_t)] << 18); \\\n hashv += hashv >> 11; \\\n break; \\\n case 2: \\\n hashv += get16bits(_sfh_key); \\\n hashv ^= hashv << 11; \\\n hashv += hashv >> 17; \\\n break; \\\n case 1: \\\n hashv += *_sfh_key; \\\n hashv ^= hashv << 10; \\\n hashv += hashv >> 1; \\\n } \\\n \\\n /* Force \"avalanching\" of final 127 bits */ \\\n hashv ^= hashv << 3; \\\n hashv += hashv >> 5; \\\n hashv ^= hashv << 4; \\\n hashv += hashv >> 17; \\\n hashv ^= hashv << 25; \\\n hashv += hashv >> 6; \\\n bkt = hashv & (num_bkts - 1); \\\n } while (0)\n\n#ifdef HASH_USING_NO_STRICT_ALIASING\n/* The MurmurHash exploits some CPU's (x86,x86_64) tolerance for unaligned reads.\n * For other types of CPU's (e.g. Sparc) an unaligned read causes a bus error.\n * MurmurHash uses the faster approach only on CPU's where we know it's safe.\n *\n * Note the preprocessor built-in defines can be emitted using:\n *\n * gcc -m64 -dM -E - < /dev/null (on gcc)\n * cc -## a.c (where a.c is a simple test file) (Sun Studio)\n */\n#if (defined(__i386__) || defined(__x86_64__) || defined(_M_IX86))\n#define MUR_GETBLOCK(p, i) p[i]\n#else /* non intel */\n#define MUR_PLUS0_ALIGNED(p) (((unsigned long)p & 0x3) == 0)\n#define MUR_PLUS1_ALIGNED(p) (((unsigned long)p & 0x3) == 1)\n#define MUR_PLUS2_ALIGNED(p) (((unsigned long)p & 0x3) == 2)\n#define MUR_PLUS3_ALIGNED(p) (((unsigned long)p & 0x3) == 3)\n#define WP(p) ((uint32_t *)((unsigned long)(p) & ~3UL))\n#if (defined(__BIG_ENDIAN__) || defined(SPARC) || defined(__ppc__) || defined(__ppc64__))\n#define MUR_THREE_ONE(p) ((((*WP(p)) & 0x00ffffff) << 8) | (((*(WP(p) + 1)) & 0xff000000) >> 24))\n#define MUR_TWO_TWO(p) ((((*WP(p)) & 0x0000ffff) << 16) | (((*(WP(p) + 1)) & 0xffff0000) >> 16))\n#define MUR_ONE_THREE(p) ((((*WP(p)) & 0x000000ff) << 24) | (((*(WP(p) + 1)) & 0xffffff00) >> 8))\n#else /* assume little endian non-intel */\n#define MUR_THREE_ONE(p) ((((*WP(p)) & 0xffffff00) >> 8) | (((*(WP(p) + 1)) & 0x000000ff) << 24))\n#define MUR_TWO_TWO(p) ((((*WP(p)) & 0xffff0000) >> 16) | (((*(WP(p) + 1)) & 0x0000ffff) << 16))\n#define MUR_ONE_THREE(p) ((((*WP(p)) & 0xff000000) >> 24) | (((*(WP(p) + 1)) & 0x00ffffff) << 8))\n#endif\n#define MUR_GETBLOCK(p, i) (MUR_PLUS0_ALIGNED(p) ? ((p)[i]) : (MUR_PLUS1_ALIGNED(p) ? MUR_THREE_ONE(p) : (MUR_PLUS2_ALIGNED(p) ? MUR_TWO_TWO(p) : MUR_ONE_THREE(p))))\n#endif\n#define MUR_ROTL32(x, r) (((x) << (r)) | ((x) >> (32 - (r))))\n#define MUR_FMIX(_h) \\\n do { \\\n _h ^= _h >> 16; \\\n _h *= 0x85ebca6b; \\\n _h ^= _h >> 13; \\\n _h *= 0xc2b2ae35l; \\\n _h ^= _h >> 16; \\\n } while (0)\n\n#define HASH_MUR(key, keylen, num_bkts, hashv, bkt) \\\n do { \\\n const uint8_t *_mur_data = (const uint8_t *)(key); \\\n const int _mur_nblocks = (keylen) / 4; \\\n uint32_t _mur_h1 = 0xf88D5353; \\\n uint32_t _mur_c1 = 0xcc9e2d51; \\\n uint32_t _mur_c2 = 0x1b873593; \\\n uint32_t _mur_k1 = 0; \\\n const uint8_t *_mur_tail; \\\n const uint32_t *_mur_blocks = (const uint32_t *)(_mur_data + _mur_nblocks * 4); \\\n int _mur_i; \\\n for (_mur_i = -_mur_nblocks; _mur_i; _mur_i++) { \\\n _mur_k1 = MUR_GETBLOCK(_mur_blocks, _mur_i); \\\n _mur_k1 *= _mur_c1; \\\n _mur_k1 = MUR_ROTL32(_mur_k1, 15); \\\n _mur_k1 *= _mur_c2; \\\n \\\n _mur_h1 ^= _mur_k1; \\\n _mur_h1 = MUR_ROTL32(_mur_h1, 13); \\\n _mur_h1 = _mur_h1 * 5 + 0xe6546b64; \\\n } \\\n _mur_tail = (const uint8_t *)(_mur_data + _mur_nblocks * 4); \\\n _mur_k1 = 0; \\\n switch (keylen & 3) { \\\n case 3: \\\n _mur_k1 ^= _mur_tail[2] << 16; \\\n case 2: \\\n _mur_k1 ^= _mur_tail[1] << 8; \\\n case 1: \\\n _mur_k1 ^= _mur_tail[0]; \\\n _mur_k1 *= _mur_c1; \\\n _mur_k1 = MUR_ROTL32(_mur_k1, 15); \\\n _mur_k1 *= _mur_c2; \\\n _mur_h1 ^= _mur_k1; \\\n } \\\n _mur_h1 ^= (keylen); \\\n MUR_FMIX(_mur_h1); \\\n hashv = _mur_h1; \\\n bkt = hashv & (num_bkts - 1); \\\n } while (0)\n#endif /* HASH_USING_NO_STRICT_ALIASING */\n\n/* key comparison function; return 0 if keys equal */\n#define HASH_KEYCMP(a, b, len) memcmp(a, b, len)\n\n/* iterate over items in a known bucket to find desired item */\n#define HASH_FIND_IN_BKT(tbl, hh, head, keyptr, keylen_in, out) \\\n do { \\\n if (head.hh_head) \\\n DECLTYPE_ASSIGN(out, ELMT_FROM_HH(tbl, head.hh_head)); \\\n else \\\n out = NULL; \\\n while (out) { \\\n if ((out)->hh.keylen == keylen_in) { \\\n if ((HASH_KEYCMP((out)->hh.key, keyptr, keylen_in)) == 0) \\\n break; \\\n } \\\n if ((out)->hh.hh_next) \\\n DECLTYPE_ASSIGN(out, ELMT_FROM_HH(tbl, (out)->hh.hh_next)); \\\n else \\\n out = NULL; \\\n } \\\n } while (0)\n\n/* add an item to a bucket */\n#define HASH_ADD_TO_BKT(head, addhh) \\\n do { \\\n head.count++; \\\n (addhh)->hh_next = head.hh_head; \\\n (addhh)->hh_prev = NULL; \\\n if (head.hh_head) { \\\n (head).hh_head->hh_prev = (addhh); \\\n } \\\n (head).hh_head = addhh; \\\n if (head.count >= ((head.expand_mult + 1) * HASH_BKT_CAPACITY_THRESH) && (addhh)->tbl->noexpand != 1) { \\\n HASH_EXPAND_BUCKETS((addhh)->tbl); \\\n } \\\n } while (0)\n\n/* remove an item from a given bucket */\n#define HASH_DEL_IN_BKT(hh, head, hh_del) \\\n (head).count--; \\\n if ((head).hh_head == hh_del) { \\\n (head).hh_head = hh_del->hh_next; \\\n } \\\n if (hh_del->hh_prev) { \\\n hh_del->hh_prev->hh_next = hh_del->hh_next; \\\n } \\\n if (hh_del->hh_next) { \\\n hh_del->hh_next->hh_prev = hh_del->hh_prev; \\\n }\n\n/* Bucket expansion has the effect of doubling the number of buckets\n * and redistributing the items into the new buckets. Ideally the\n * items will distribute more or less evenly into the new buckets\n * (the extent to which this is true is a measure of the quality of\n * the hash function as it applies to the key domain).\n *\n * With the items distributed into more buckets, the chain length\n * (item count) in each bucket is reduced. Thus by expanding buckets\n * the hash keeps a bound on the chain length. This bounded chain\n * length is the essence of how a hash provides constant time lookup.\n *\n * The calculation of tbl->ideal_chain_maxlen below deserves some\n * explanation. First, keep in mind that we're calculating the ideal\n * maximum chain length based on the *new* (doubled) bucket count.\n * In fractions this is just n/b (n=number of items,b=new num buckets).\n * Since the ideal chain length is an integer, we want to calculate\n * ceil(n/b). We don't depend on floating point arithmetic in this\n * hash, so to calculate ceil(n/b) with integers we could write\n *\n * ceil(n/b) = (n/b) + ((n%b)?1:0)\n *\n * and in fact a previous version of this hash did just that.\n * But now we have improved things a bit by recognizing that b is\n * always a power of two. We keep its base 2 log handy (call it lb),\n * so now we can write this with a bit shift and logical AND:\n *\n * ceil(n/b) = (n>>lb) + ( (n & (b-1)) ? 1:0)\n *\n */\n#define HASH_EXPAND_BUCKETS(tbl) \\\n do { \\\n unsigned _he_bkt; \\\n unsigned _he_bkt_i; \\\n struct UT_hash_handle *_he_thh, *_he_hh_nxt; \\\n UT_hash_bucket *_he_new_buckets, *_he_newbkt; \\\n _he_new_buckets = (UT_hash_bucket *)uthash_malloc(2 * tbl->num_buckets * sizeof(struct UT_hash_bucket)); \\\n if (!_he_new_buckets) { \\\n uthash_fatal(\"out of memory\"); \\\n } \\\n memset(_he_new_buckets, 0, 2 * tbl->num_buckets * sizeof(struct UT_hash_bucket)); \\\n tbl->ideal_chain_maxlen = (tbl->num_items >> (tbl->log2_num_buckets + 1)) + ((tbl->num_items & ((tbl->num_buckets * 2) - 1)) ? 1 : 0); \\\n tbl->nonideal_items = 0; \\\n for (_he_bkt_i = 0; _he_bkt_i < tbl->num_buckets; _he_bkt_i++) { \\\n _he_thh = tbl->buckets[_he_bkt_i].hh_head; \\\n while (_he_thh) { \\\n _he_hh_nxt = _he_thh->hh_next; \\\n HASH_TO_BKT(_he_thh->hashv, tbl->num_buckets * 2, _he_bkt); \\\n _he_newbkt = &(_he_new_buckets[_he_bkt]); \\\n if (++(_he_newbkt->count) > tbl->ideal_chain_maxlen) { \\\n tbl->nonideal_items++; \\\n _he_newbkt->expand_mult = _he_newbkt->count / tbl->ideal_chain_maxlen; \\\n } \\\n _he_thh->hh_prev = NULL; \\\n _he_thh->hh_next = _he_newbkt->hh_head; \\\n if (_he_newbkt->hh_head) \\\n _he_newbkt->hh_head->hh_prev = _he_thh; \\\n _he_newbkt->hh_head = _he_thh; \\\n _he_thh = _he_hh_nxt; \\\n } \\\n } \\\n uthash_free(tbl->buckets, tbl->num_buckets * sizeof(struct UT_hash_bucket)); \\\n tbl->num_buckets *= 2; \\\n tbl->log2_num_buckets++; \\\n tbl->buckets = _he_new_buckets; \\\n tbl->ineff_expands = (tbl->nonideal_items > (tbl->num_items >> 1)) ? (tbl->ineff_expands + 1) : 0; \\\n if (tbl->ineff_expands > 1) { \\\n tbl->noexpand = 1; \\\n uthash_noexpand_fyi(tbl); \\\n } \\\n uthash_expand_fyi(tbl); \\\n } while (0)\n\n\n/* This is an adaptation of Simon Tatham's O(n log(n)) mergesort */\n/* Note that HASH_SORT assumes the hash handle name to be hh.\n * HASH_SRT was added to allow the hash handle name to be passed in. */\n#define HASH_SORT(head, cmpfcn) HASH_SRT(hh, head, cmpfcn)\n#define HASH_SRT(hh, head, cmpfcn) \\\n do { \\\n unsigned _hs_i; \\\n unsigned _hs_looping, _hs_nmerges, _hs_insize, _hs_psize, _hs_qsize; \\\n struct UT_hash_handle *_hs_p, *_hs_q, *_hs_e, *_hs_list, *_hs_tail; \\\n if (head) { \\\n _hs_insize = 1; \\\n _hs_looping = 1; \\\n _hs_list = &((head)->hh); \\\n while (_hs_looping) { \\\n _hs_p = _hs_list; \\\n _hs_list = NULL; \\\n _hs_tail = NULL; \\\n _hs_nmerges = 0; \\\n while (_hs_p) { \\\n _hs_nmerges++; \\\n _hs_q = _hs_p; \\\n _hs_psize = 0; \\\n for (_hs_i = 0; _hs_i < _hs_insize; _hs_i++) { \\\n _hs_psize++; \\\n _hs_q = (UT_hash_handle *)((_hs_q->next) ? ((void *)((char *)(_hs_q->next) + (head)->hh.tbl->hho)) : NULL); \\\n if (!(_hs_q)) \\\n break; \\\n } \\\n _hs_qsize = _hs_insize; \\\n while ((_hs_psize > 0) || ((_hs_qsize > 0) && _hs_q)) { \\\n if (_hs_psize == 0) { \\\n _hs_e = _hs_q; \\\n _hs_q = (UT_hash_handle *)((_hs_q->next) ? ((void *)((char *)(_hs_q->next) + (head)->hh.tbl->hho)) : NULL); \\\n _hs_qsize--; \\\n } else if ((_hs_qsize == 0) || !(_hs_q)) { \\\n _hs_e = _hs_p; \\\n if (_hs_p) { \\\n _hs_p = (UT_hash_handle *)((_hs_p->next) ? ((void *)((char *)(_hs_p->next) + (head)->hh.tbl->hho)) : NULL); \\\n } \\\n _hs_psize--; \\\n } else if ((cmpfcn(DECLTYPE(head)(ELMT_FROM_HH((head)->hh.tbl, _hs_p)), DECLTYPE(head)(ELMT_FROM_HH((head)->hh.tbl, _hs_q)))) <= 0) { \\\n _hs_e = _hs_p; \\\n if (_hs_p) { \\\n _hs_p = (UT_hash_handle *)((_hs_p->next) ? ((void *)((char *)(_hs_p->next) + (head)->hh.tbl->hho)) : NULL); \\\n } \\\n _hs_psize--; \\\n } else { \\\n _hs_e = _hs_q; \\\n _hs_q = (UT_hash_handle *)((_hs_q->next) ? ((void *)((char *)(_hs_q->next) + (head)->hh.tbl->hho)) : NULL); \\\n _hs_qsize--; \\\n } \\\n if (_hs_tail) { \\\n _hs_tail->next = ((_hs_e) ? ELMT_FROM_HH((head)->hh.tbl, _hs_e) : NULL); \\\n } else { \\\n _hs_list = _hs_e; \\\n } \\\n if (_hs_e) { \\\n _hs_e->prev = ((_hs_tail) ? ELMT_FROM_HH((head)->hh.tbl, _hs_tail) : NULL); \\\n } \\\n _hs_tail = _hs_e; \\\n } \\\n _hs_p = _hs_q; \\\n } \\\n if (_hs_tail) { \\\n _hs_tail->next = NULL; \\\n } \\\n if (_hs_nmerges <= 1) { \\\n _hs_looping = 0; \\\n (head)->hh.tbl->tail = _hs_tail; \\\n DECLTYPE_ASSIGN(head, ELMT_FROM_HH((head)->hh.tbl, _hs_list)); \\\n } \\\n _hs_insize *= 2; \\\n } \\\n HASH_FSCK(hh, head); \\\n } \\\n } while (0)\n\n/* This function selects items from one hash into another hash.\n * The end result is that the selected items have dual presence\n * in both hashes. There is no copy of the items made; rather\n * they are added into the new hash through a secondary hash\n * hash handle that must be present in the structure. */\n#define HASH_SELECT(hh_dst, dst, hh_src, src, cond) \\\n do { \\\n unsigned _src_bkt, _dst_bkt; \\\n void *_last_elt = NULL, *_elt; \\\n UT_hash_handle *_src_hh, *_dst_hh, *_last_elt_hh = NULL; \\\n ptrdiff_t _dst_hho = ((char *)(&(dst)->hh_dst) - (char *)(dst)); \\\n if (src) { \\\n for (_src_bkt = 0; _src_bkt < (src)->hh_src.tbl->num_buckets; _src_bkt++) { \\\n for (_src_hh = (src)->hh_src.tbl->buckets[_src_bkt].hh_head; _src_hh; _src_hh = _src_hh->hh_next) { \\\n _elt = ELMT_FROM_HH((src)->hh_src.tbl, _src_hh); \\\n if (cond(_elt)) { \\\n _dst_hh = (UT_hash_handle *)(((char *)_elt) + _dst_hho); \\\n _dst_hh->key = _src_hh->key; \\\n _dst_hh->keylen = _src_hh->keylen; \\\n _dst_hh->hashv = _src_hh->hashv; \\\n _dst_hh->prev = _last_elt; \\\n _dst_hh->next = NULL; \\\n if (_last_elt_hh) { \\\n _last_elt_hh->next = _elt; \\\n } \\\n if (!dst) { \\\n DECLTYPE_ASSIGN(dst, _elt); \\\n HASH_MAKE_TABLE(hh_dst, dst); \\\n } else { \\\n _dst_hh->tbl = (dst)->hh_dst.tbl; \\\n } \\\n HASH_TO_BKT(_dst_hh->hashv, _dst_hh->tbl->num_buckets, _dst_bkt); \\\n HASH_ADD_TO_BKT(_dst_hh->tbl->buckets[_dst_bkt], _dst_hh); \\\n (dst)->hh_dst.tbl->num_items++; \\\n _last_elt = _elt; \\\n _last_elt_hh = _dst_hh; \\\n } \\\n } \\\n } \\\n } \\\n HASH_FSCK(hh_dst, dst); \\\n } while (0)\n\n#define HASH_CLEAR(hh, head) \\\n do { \\\n if (head) { \\\n uthash_free((head)->hh.tbl->buckets, (head)->hh.tbl->num_buckets * sizeof(struct UT_hash_bucket)); \\\n HASH_BLOOM_FREE((head)->hh.tbl); \\\n uthash_free((head)->hh.tbl, sizeof(UT_hash_table)); \\\n (head) = NULL; \\\n } \\\n } while (0)\n\n#define HASH_OVERHEAD(hh, head) \\\n (size_t)((((head)->hh.tbl->num_items * sizeof(UT_hash_handle)) + \\\n ((head)->hh.tbl->num_buckets * sizeof(UT_hash_bucket)) + \\\n (sizeof(UT_hash_table)) + \\\n (HASH_BLOOM_BYTELEN)))\n\n#ifdef NO_DECLTYPE\n#define HASH_ITER(hh, head, el, tmp) for ((el) = (head), (*(char **)(&(tmp))) = (char *)((head) ? (head)->hh.next : NULL); \\\n el; (el) = (tmp), (*(char **)(&(tmp))) = (char *)((tmp) ? (tmp)->hh.next : NULL))\n#else\n#define HASH_ITER(hh, head, el, tmp) for ((el) = (head), (tmp) = DECLTYPE(el)((head) ? (head)->hh.next : NULL); \\\n el; (el) = (tmp), (tmp) = DECLTYPE(el)((tmp) ? (tmp)->hh.next : NULL))\n#endif\n\n/* obtain a count of items in the hash */\n#define HASH_COUNT(head) HASH_CNT(hh, head)\n#define HASH_CNT(hh, head) ((head) ? ((head)->hh.tbl->num_items) : 0)\n\ntypedef struct UT_hash_bucket {\n struct UT_hash_handle *hh_head;\n unsigned count;\n\n /* expand_mult is normally set to 0. In this situation, the max chain length\n * threshold is enforced at its default value, HASH_BKT_CAPACITY_THRESH. (If\n * the bucket's chain exceeds this length, bucket expansion is triggered).\n * However, setting expand_mult to a non-zero value delays bucket expansion\n * (that would be triggered by additions to this particular bucket)\n * until its chain length reaches a *multiple* of HASH_BKT_CAPACITY_THRESH.\n * (The multiplier is simply expand_mult+1). The whole idea of this\n * multiplier is to reduce bucket expansions, since they are expensive, in\n * situations where we know that a particular bucket tends to be overused.\n * It is better to let its chain length grow to a longer yet-still-bounded\n * value, than to do an O(n) bucket expansion too often.\n */\n unsigned expand_mult;\n\n} UT_hash_bucket;\n\n/* random signature used only to find hash tables in external analysis */\n#define HASH_SIGNATURE 0xa0111fe1\n#define HASH_BLOOM_SIGNATURE 0xb12220f2\n\ntypedef struct UT_hash_table {\n UT_hash_bucket *buckets;\n unsigned num_buckets, log2_num_buckets;\n unsigned num_items;\n struct UT_hash_handle *tail; /* tail hh in app order, for fast append */\n ptrdiff_t hho; /* hash handle offset (byte pos of hash handle in element */\n\n /* in an ideal situation (all buckets used equally), no bucket would have\n * more than ceil(#items/#buckets) items. that's the ideal chain length. */\n unsigned ideal_chain_maxlen;\n\n /* nonideal_items is the number of items in the hash whose chain position\n * exceeds the ideal chain maxlen. these items pay the penalty for an uneven\n * hash distribution; reaching them in a chain traversal takes >ideal steps */\n unsigned nonideal_items;\n\n /* ineffective expands occur when a bucket doubling was performed, but\n * afterward, more than half the items in the hash had nonideal chain\n * positions. If this happens on two consecutive expansions we inhibit any\n * further expansion, as it's not helping; this happens when the hash\n * function isn't a good fit for the key domain. When expansion is inhibited\n * the hash will still work, albeit no longer in constant time. */\n unsigned ineff_expands, noexpand;\n\n uint32_t signature; /* used only to find hash tables in external analysis */\n#ifdef HASH_BLOOM\n uint32_t bloom_sig; /* used only to test bloom exists in external analysis */\n uint8_t *bloom_bv;\n char bloom_nbits;\n#endif\n\n} UT_hash_table;\n\ntypedef struct UT_hash_handle {\n struct UT_hash_table *tbl;\n void *prev; /* prev element in app order */\n void *next; /* next element in app order */\n struct UT_hash_handle *hh_prev; /* previous hh in bucket order */\n struct UT_hash_handle *hh_next; /* next hh in bucket order */\n void *key; /* ptr to enclosing struct's key */\n unsigned keylen; /* enclosing struct's key len */\n unsigned hashv; /* result of hash-fcn(key) */\n} UT_hash_handle;\n\n#endif /* UTHASH_H */\n" }, { "path": "src/util.c", "content": "#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"config.h\"\n#include \"util.h\"\n\n#ifdef _WIN32\n#include \n#define flockfile(x)\n#define funlockfile(x)\n#define getc_unlocked(x) getc(x)\n#endif\n\n#define CHECK_AND_RETURN(ptr) \\\n if (ptr == NULL) { \\\n die(\"Memory allocation failed.\"); \\\n } \\\n return ptr;\n\nFILE *out_fd = NULL;\nag_stats stats;\nvoid *ag_malloc(size_t size) {\n void *ptr = malloc(size);\n CHECK_AND_RETURN(ptr)\n}\n\nvoid *ag_realloc(void *ptr, size_t size) {\n void *new_ptr = realloc(ptr, size);\n CHECK_AND_RETURN(new_ptr)\n}\n\nvoid *ag_calloc(size_t count, size_t size) {\n void *ptr = calloc(count, size);\n CHECK_AND_RETURN(ptr)\n}\n\nchar *ag_strdup(const char *s) {\n char *str = strdup(s);\n CHECK_AND_RETURN(str)\n}\n\nchar *ag_strndup(const char *s, size_t size) {\n char *str = NULL;\n#ifdef HAVE_STRNDUP\n str = strndup(s, size);\n CHECK_AND_RETURN(str)\n#else\n str = (char *)ag_malloc(size + 1);\n strlcpy(str, s, size + 1);\n return str;\n#endif\n}\n\nvoid free_strings(char **strs, const size_t strs_len) {\n if (strs == NULL) {\n return;\n }\n size_t i;\n for (i = 0; i < strs_len; i++) {\n free(strs[i]);\n }\n free(strs);\n}\n\nvoid generate_alpha_skip(const char *find, size_t f_len, size_t skip_lookup[], const int case_sensitive) {\n size_t i;\n\n for (i = 0; i < 256; i++) {\n skip_lookup[i] = f_len;\n }\n\n f_len--;\n\n for (i = 0; i < f_len; i++) {\n if (case_sensitive) {\n skip_lookup[(unsigned char)find[i]] = f_len - i;\n } else {\n skip_lookup[(unsigned char)tolower(find[i])] = f_len - i;\n skip_lookup[(unsigned char)toupper(find[i])] = f_len - i;\n }\n }\n}\n\nint is_prefix(const char *s, const size_t s_len, const size_t pos, const int case_sensitive) {\n size_t i;\n\n for (i = 0; pos + i < s_len; i++) {\n if (case_sensitive) {\n if (s[i] != s[i + pos]) {\n return 0;\n }\n } else {\n if (tolower(s[i]) != tolower(s[i + pos])) {\n return 0;\n }\n }\n }\n\n return 1;\n}\n\nsize_t suffix_len(const char *s, const size_t s_len, const size_t pos, const int case_sensitive) {\n size_t i;\n\n for (i = 0; i < pos; i++) {\n if (case_sensitive) {\n if (s[pos - i] != s[s_len - i - 1]) {\n break;\n }\n } else {\n if (tolower(s[pos - i]) != tolower(s[s_len - i - 1])) {\n break;\n }\n }\n }\n\n return i;\n}\n\nvoid generate_find_skip(const char *find, const size_t f_len, size_t **skip_lookup, const int case_sensitive) {\n size_t i;\n size_t s_len;\n size_t *sl = ag_malloc(f_len * sizeof(size_t));\n *skip_lookup = sl;\n size_t last_prefix = f_len;\n\n for (i = last_prefix; i > 0; i--) {\n if (is_prefix(find, f_len, i, case_sensitive)) {\n last_prefix = i;\n }\n sl[i - 1] = last_prefix + (f_len - i);\n }\n\n for (i = 0; i < f_len; i++) {\n s_len = suffix_len(find, f_len, i, case_sensitive);\n if (find[i - s_len] != find[f_len - 1 - s_len]) {\n sl[f_len - 1 - s_len] = f_len - 1 - i + s_len;\n }\n }\n}\n\nsize_t ag_max(size_t a, size_t b) {\n if (b > a) {\n return b;\n }\n return a;\n}\n\nsize_t ag_min(size_t a, size_t b) {\n if (b < a) {\n return b;\n }\n return a;\n}\n\nvoid generate_hash(const char *find, const size_t f_len, uint8_t *h_table, const int case_sensitive) {\n int i;\n for (i = f_len - sizeof(uint16_t); i >= 0; i--) {\n // Add all 2^sizeof(uint16_t) combinations of capital letters to the hash table\n int caps_set;\n for (caps_set = 0; caps_set < (1 << sizeof(uint16_t)); caps_set++) {\n word_t word;\n memcpy(&word.as_chars, find + i, sizeof(uint16_t));\n int cap_index;\n // Capitalize the letters whose corresponding bits in caps_set are 1\n for (cap_index = 0; caps_set >> cap_index; cap_index++) {\n if ((caps_set >> cap_index) & 1)\n word.as_chars[cap_index] -= 'a' - 'A';\n }\n size_t h;\n // Find next free cell\n for (h = word.as_word % H_SIZE; h_table[h]; h = (h + 1) % H_SIZE)\n ;\n h_table[h] = i + 1;\n // Don't add capital letters if case sensitive\n if (case_sensitive)\n break;\n }\n }\n}\n\n/* Boyer-Moore strstr */\nconst char *boyer_moore_strnstr(const char *s, const char *find, const size_t s_len, const size_t f_len,\n const size_t alpha_skip_lookup[], const size_t *find_skip_lookup, const int case_insensitive) {\n ssize_t i;\n size_t pos = f_len - 1;\n\n while (pos < s_len) {\n for (i = f_len - 1; i >= 0 && (case_insensitive ? tolower(s[pos]) : s[pos]) == find[i]; pos--, i--) {\n }\n if (i < 0) {\n return s + pos + 1;\n }\n pos += ag_max(alpha_skip_lookup[(unsigned char)s[pos]], find_skip_lookup[i]);\n }\n\n return NULL;\n}\n\n// Clang's -fsanitize=alignment (included in -fsanitize=undefined) will flag\n// the intentional unaligned access here, so suppress it for this function\nNO_SANITIZE_ALIGNMENT const char *hash_strnstr(const char *s, const char *find, const size_t s_len, const size_t f_len, uint8_t *h_table, const int case_sensitive) {\n if (s_len < f_len)\n return NULL;\n\n // Step through s\n const size_t step = f_len - sizeof(uint16_t) + 1;\n size_t s_i = f_len - sizeof(uint16_t);\n for (; s_i <= s_len - f_len; s_i += step) {\n size_t h;\n for (h = *(const uint16_t *)(s + s_i) % H_SIZE; h_table[h]; h = (h + 1) % H_SIZE) {\n const char *R = s + s_i - (h_table[h] - 1);\n size_t i;\n // Check putative match\n for (i = 0; i < f_len; i++) {\n if ((case_sensitive ? R[i] : tolower(R[i])) != find[i])\n goto next_hash_cell;\n }\n return R; // Found\n next_hash_cell:;\n }\n }\n // Check tail\n for (s_i = s_i - step + 1; s_i <= s_len - f_len; s_i++) {\n size_t i;\n const char *R = s + s_i;\n for (i = 0; i < f_len; i++) {\n char s_c = case_sensitive ? R[i] : tolower(R[i]);\n if (s_c != find[i])\n goto next_start;\n }\n return R;\n next_start:;\n }\n return NULL;\n}\n\nsize_t invert_matches(const char *buf, const size_t buf_len, match_t matches[], size_t matches_len) {\n size_t i;\n size_t match_read_index = 0;\n size_t inverted_match_count = 0;\n size_t inverted_match_start = 0;\n size_t last_line_end = 0;\n int in_inverted_match = TRUE;\n match_t next_match;\n\n log_debug(\"Inverting %u matches.\", matches_len);\n\n if (matches_len > 0) {\n next_match = matches[0];\n } else {\n next_match.start = buf_len + 1;\n }\n\n /* No matches, so the whole buffer is now a match. */\n if (matches_len == 0) {\n matches[0].start = 0;\n matches[0].end = buf_len - 1;\n return 1;\n }\n\n for (i = 0; i < buf_len; i++) {\n if (i == next_match.start) {\n i = next_match.end - 1;\n\n match_read_index++;\n\n if (match_read_index < matches_len) {\n next_match = matches[match_read_index];\n }\n\n if (in_inverted_match && last_line_end > inverted_match_start) {\n matches[inverted_match_count].start = inverted_match_start;\n matches[inverted_match_count].end = last_line_end - 1;\n\n inverted_match_count++;\n }\n\n in_inverted_match = FALSE;\n } else if (i == buf_len - 1 && in_inverted_match) {\n matches[inverted_match_count].start = inverted_match_start;\n matches[inverted_match_count].end = i;\n\n inverted_match_count++;\n } else if (buf[i] == '\\n') {\n last_line_end = i + 1;\n\n if (!in_inverted_match) {\n inverted_match_start = last_line_end;\n }\n\n in_inverted_match = TRUE;\n }\n }\n\n for (i = 0; i < matches_len; i++) {\n log_debug(\"Inverted match %i start %i end %i.\", i, matches[i].start, matches[i].end);\n }\n\n return inverted_match_count;\n}\n\nvoid realloc_matches(match_t **matches, size_t *matches_size, size_t matches_len) {\n if (matches_len < *matches_size) {\n return;\n }\n /* TODO: benchmark initial size of matches. 100 may be too small/big */\n *matches_size = *matches ? *matches_size * 2 : 100;\n *matches = ag_realloc(*matches, *matches_size * sizeof(match_t));\n}\n\nvoid compile_study(pcre **re, pcre_extra **re_extra, char *q, const int pcre_opts, const int study_opts) {\n const char *pcre_err = NULL;\n int pcre_err_offset = 0;\n\n *re = pcre_compile(q, pcre_opts, &pcre_err, &pcre_err_offset, NULL);\n if (*re == NULL) {\n die(\"Bad regex! pcre_compile() failed at position %i: %s\\nIf you meant to search for a literal string, run ag with -Q\",\n pcre_err_offset,\n pcre_err);\n }\n *re_extra = pcre_study(*re, study_opts, &pcre_err);\n if (*re_extra == NULL) {\n log_debug(\"pcre_study returned nothing useful. Error: %s\", pcre_err);\n }\n}\n\n/* This function is very hot. It's called on every file. */\nint is_binary(const void *buf, const size_t buf_len) {\n size_t suspicious_bytes = 0;\n size_t total_bytes = buf_len > 512 ? 512 : buf_len;\n const unsigned char *buf_c = buf;\n size_t i;\n\n if (buf_len == 0) {\n /* Is an empty file binary? Is it text? */\n return 0;\n }\n\n if (buf_len >= 3 && buf_c[0] == 0xEF && buf_c[1] == 0xBB && buf_c[2] == 0xBF) {\n /* UTF-8 BOM. This isn't binary. */\n return 0;\n }\n\n if (buf_len >= 5 && strncmp(buf, \"%PDF-\", 5) == 0) {\n /* PDF. This is binary. */\n return 1;\n }\n\n for (i = 0; i < total_bytes; i++) {\n if (buf_c[i] == '\\0') {\n /* NULL char. It's binary */\n return 1;\n } else if ((buf_c[i] < 7 || buf_c[i] > 14) && (buf_c[i] < 32 || buf_c[i] > 127)) {\n /* UTF-8 detection */\n if (buf_c[i] > 193 && buf_c[i] < 224 && i + 1 < total_bytes) {\n i++;\n if (buf_c[i] > 127 && buf_c[i] < 192) {\n continue;\n }\n } else if (buf_c[i] > 223 && buf_c[i] < 240 && i + 2 < total_bytes) {\n i++;\n if (buf_c[i] > 127 && buf_c[i] < 192 && buf_c[i + 1] > 127 && buf_c[i + 1] < 192) {\n i++;\n continue;\n }\n }\n suspicious_bytes++;\n /* Disk IO is so slow that it's worthwhile to do this calculation after every suspicious byte. */\n /* This is true even on a 1.6Ghz Atom with an Intel 320 SSD. */\n /* Read at least 32 bytes before making a decision */\n if (i >= 32 && (suspicious_bytes * 100) / total_bytes > 10) {\n return 1;\n }\n }\n }\n if ((suspicious_bytes * 100) / total_bytes > 10) {\n return 1;\n }\n\n return 0;\n}\n\nint is_regex(const char *query) {\n char regex_chars[] = {\n '$',\n '(',\n ')',\n '*',\n '+',\n '.',\n '?',\n '[',\n '\\\\',\n '^',\n '{',\n '|',\n '\\0'\n };\n\n return (strpbrk(query, regex_chars) != NULL);\n}\n\nint is_fnmatch(const char *filename) {\n char fnmatch_chars[] = {\n '!',\n '*',\n '?',\n '[',\n ']',\n '\\0'\n };\n\n return (strpbrk(filename, fnmatch_chars) != NULL);\n}\n\nint binary_search(const char *needle, char **haystack, int start, int end) {\n int mid;\n int rc;\n\n if (start == end) {\n return -1;\n }\n\n mid = start + ((end - start) / 2);\n\n rc = strcmp(needle, haystack[mid]);\n if (rc < 0) {\n return binary_search(needle, haystack, start, mid);\n } else if (rc > 0) {\n return binary_search(needle, haystack, mid + 1, end);\n }\n\n return mid;\n}\n\nstatic int wordchar_table[256];\n\nvoid init_wordchar_table(void) {\n int i;\n for (i = 0; i < 256; ++i) {\n char ch = (char)i;\n wordchar_table[i] =\n ('a' <= ch && ch <= 'z') ||\n ('A' <= ch && ch <= 'Z') ||\n ('0' <= ch && ch <= '9') ||\n ch == '_';\n }\n}\n\nint is_wordchar(char ch) {\n return wordchar_table[(unsigned char)ch];\n}\n\nint is_lowercase(const char *s) {\n int i;\n for (i = 0; s[i] != '\\0'; i++) {\n if (!isascii(s[i]) || isupper(s[i])) {\n return FALSE;\n }\n }\n return TRUE;\n}\n\nint is_directory(const char *path, const struct dirent *d) {\n#ifdef HAVE_DIRENT_DTYPE\n /* Some filesystems, e.g. ReiserFS, always return a type DT_UNKNOWN from readdir or scandir. */\n /* Call stat if we don't find DT_DIR to get the information we need. */\n /* Also works for symbolic links to directories. */\n if (d->d_type != DT_UNKNOWN && d->d_type != DT_LNK) {\n return d->d_type == DT_DIR;\n }\n#endif\n char *full_path;\n struct stat s;\n ag_asprintf(&full_path, \"%s/%s\", path, d->d_name);\n if (stat(full_path, &s) != 0) {\n free(full_path);\n return FALSE;\n }\n#ifdef _WIN32\n int is_dir = GetFileAttributesA(full_path) & FILE_ATTRIBUTE_DIRECTORY;\n#else\n int is_dir = S_ISDIR(s.st_mode);\n#endif\n free(full_path);\n return is_dir;\n}\n\nint is_symlink(const char *path, const struct dirent *d) {\n#ifdef _WIN32\n char full_path[MAX_PATH + 1] = { 0 };\n sprintf(full_path, \"%s\\\\%s\", path, d->d_name);\n return (GetFileAttributesA(full_path) & FILE_ATTRIBUTE_REPARSE_POINT);\n#else\n#ifdef HAVE_DIRENT_DTYPE\n /* Some filesystems, e.g. ReiserFS, always return a type DT_UNKNOWN from readdir or scandir. */\n /* Call lstat if we find DT_UNKNOWN to get the information we need. */\n if (d->d_type != DT_UNKNOWN) {\n return (d->d_type == DT_LNK);\n }\n#endif\n char *full_path;\n struct stat s;\n ag_asprintf(&full_path, \"%s/%s\", path, d->d_name);\n if (lstat(full_path, &s) != 0) {\n free(full_path);\n return FALSE;\n }\n free(full_path);\n return S_ISLNK(s.st_mode);\n#endif\n}\n\nint is_named_pipe(const char *path, const struct dirent *d) {\n#ifdef HAVE_DIRENT_DTYPE\n if (d->d_type != DT_UNKNOWN && d->d_type != DT_LNK) {\n return d->d_type == DT_FIFO || d->d_type == DT_SOCK;\n }\n#endif\n char *full_path;\n struct stat s;\n ag_asprintf(&full_path, \"%s/%s\", path, d->d_name);\n if (stat(full_path, &s) != 0) {\n free(full_path);\n return FALSE;\n }\n free(full_path);\n return S_ISFIFO(s.st_mode)\n#ifdef S_ISSOCK\n || S_ISSOCK(s.st_mode)\n#endif\n ;\n}\n\nvoid ag_asprintf(char **ret, const char *fmt, ...) {\n va_list args;\n va_start(args, fmt);\n if (vasprintf(ret, fmt, args) == -1) {\n die(\"vasprintf returned -1\");\n }\n va_end(args);\n}\n\nvoid die(const char *fmt, ...) {\n va_list args;\n va_start(args, fmt);\n vplog(LOG_LEVEL_ERR, fmt, args);\n va_end(args);\n exit(2);\n}\n\n#ifndef HAVE_FGETLN\nchar *fgetln(FILE *fp, size_t *lenp) {\n char *buf = NULL;\n int c, used = 0, len = 0;\n\n flockfile(fp);\n while ((c = getc_unlocked(fp)) != EOF) {\n if (!buf || len >= used) {\n size_t nsize;\n char *newbuf;\n nsize = used + BUFSIZ;\n if (!(newbuf = realloc(buf, nsize))) {\n funlockfile(fp);\n if (buf)\n free(buf);\n return NULL;\n }\n buf = newbuf;\n used = nsize;\n }\n buf[len++] = c;\n if (c == '\\n') {\n break;\n }\n }\n funlockfile(fp);\n *lenp = len;\n return buf;\n}\n#endif\n\n#ifndef HAVE_GETLINE\n/*\n * Do it yourself getline() implementation\n */\nssize_t getline(char **lineptr, size_t *n, FILE *stream) {\n size_t len = 0;\n char *srcln = NULL;\n char *newlnptr = NULL;\n\n /* get line, bail on error */\n if (!(srcln = fgetln(stream, &len))) {\n return -1;\n }\n\n if (len >= *n) {\n /* line is too big for buffer, must realloc */\n /* double the buffer, bail on error */\n if (!(newlnptr = realloc(*lineptr, len * 2))) {\n return -1;\n }\n *lineptr = newlnptr;\n *n = len * 2;\n }\n\n memcpy(*lineptr, srcln, len);\n\n#ifndef HAVE_FGETLN\n /* Our own implementation of fgetln() returns a malloc()d buffer that we\n * must free\n */\n free(srcln);\n#endif\n\n (*lineptr)[len] = '\\0';\n return len;\n}\n#endif\n\nssize_t buf_getline(const char **line, const char *buf, const size_t buf_len, const size_t buf_offset) {\n const char *cur = buf + buf_offset;\n ssize_t i;\n for (i = 0; (buf_offset + i < buf_len) && cur[i] != '\\n'; i++) {\n }\n *line = cur;\n return i;\n}\n\n#ifndef HAVE_REALPATH\n/*\n * realpath() for Windows. Turns slashes into backslashes and calls _fullpath\n */\nchar *realpath(const char *path, char *resolved_path) {\n char *p;\n char tmp[_MAX_PATH + 1];\n strlcpy(tmp, path, sizeof(tmp));\n p = tmp;\n while (*p) {\n if (*p == '/') {\n *p = '\\\\';\n }\n p++;\n }\n return _fullpath(resolved_path, tmp, _MAX_PATH);\n}\n#endif\n\n#ifndef HAVE_STRLCPY\nsize_t strlcpy(char *dst, const char *src, size_t size) {\n char *d = dst;\n const char *s = src;\n size_t n = size;\n\n /* Copy as many bytes as will fit */\n if (n != 0) {\n while (--n != 0) {\n if ((*d++ = *s++) == '\\0') {\n break;\n }\n }\n }\n\n /* Not enough room in dst, add NUL and traverse rest of src */\n if (n == 0) {\n if (size != 0) {\n *d = '\\0'; /* NUL-terminate dst */\n }\n\n while (*s++) {\n }\n }\n\n return (s - src - 1); /* count does not include NUL */\n}\n#endif\n\n#ifndef HAVE_VASPRINTF\nint vasprintf(char **ret, const char *fmt, va_list args) {\n int rv;\n *ret = NULL;\n va_list args2;\n/* vsnprintf can destroy args, so we need to copy it for the second call */\n#ifdef __va_copy\n /* non-standard macro, but usually exists */\n __va_copy(args2, args);\n#elif va_copy\n /* C99 macro. We compile with -std=c89 but you never know */\n va_copy(args2, args);\n#else\n /* Ancient compiler. This usually works but there are no guarantees. */\n memcpy(args2, args, sizeof(va_list));\n#endif\n rv = vsnprintf(NULL, 0, fmt, args);\n va_end(args);\n if (rv < 0) {\n return rv;\n }\n *ret = malloc(++rv); /* vsnprintf doesn't count \\0 */\n if (*ret == NULL) {\n return -1;\n }\n rv = vsnprintf(*ret, rv, fmt, args2);\n va_end(args2);\n if (rv < 0) {\n free(*ret);\n }\n return rv;\n}\n#endif\n" }, { "path": "src/util.h", "content": "#ifndef UTIL_H\n#define UTIL_H\n\n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"config.h\"\n#include \"log.h\"\n#include \"options.h\"\n\nextern FILE *out_fd;\n\n#ifndef TRUE\n#define TRUE 1\n#endif\n\n#ifndef FALSE\n#define FALSE 0\n#endif\n\n#define H_SIZE (64 * 1024)\n\n#ifdef __clang__\n#define NO_SANITIZE_ALIGNMENT __attribute__((no_sanitize(\"alignment\")))\n#else\n#define NO_SANITIZE_ALIGNMENT\n#endif\n\nvoid *ag_malloc(size_t size);\nvoid *ag_realloc(void *ptr, size_t size);\nvoid *ag_calloc(size_t nelem, size_t elsize);\nchar *ag_strdup(const char *s);\nchar *ag_strndup(const char *s, size_t size);\n\ntypedef struct {\n size_t start; /* Byte at which the match starts */\n size_t end; /* and where it ends */\n} match_t;\n\ntypedef struct {\n size_t total_bytes;\n size_t total_files;\n size_t total_matches;\n size_t total_file_matches;\n struct timeval time_start;\n struct timeval time_end;\n} ag_stats;\n\n\nextern ag_stats stats;\n\n/* Union to translate between chars and words without violating strict aliasing */\ntypedef union {\n char as_chars[sizeof(uint16_t)];\n uint16_t as_word;\n} word_t;\n\nvoid free_strings(char **strs, const size_t strs_len);\n\nvoid generate_alpha_skip(const char *find, size_t f_len, size_t skip_lookup[], const int case_sensitive);\nint is_prefix(const char *s, const size_t s_len, const size_t pos, const int case_sensitive);\nsize_t suffix_len(const char *s, const size_t s_len, const size_t pos, const int case_sensitive);\nvoid generate_find_skip(const char *find, const size_t f_len, size_t **skip_lookup, const int case_sensitive);\nvoid generate_hash(const char *find, const size_t f_len, uint8_t *H, const int case_sensitive);\n\n/* max is already defined on spec-violating compilers such as MinGW */\nsize_t ag_max(size_t a, size_t b);\nsize_t ag_min(size_t a, size_t b);\n\nconst char *boyer_moore_strnstr(const char *s, const char *find, const size_t s_len, const size_t f_len,\n const size_t alpha_skip_lookup[], const size_t *find_skip_lookup, const int case_insensitive);\nconst char *hash_strnstr(const char *s, const char *find, const size_t s_len, const size_t f_len, uint8_t *h_table, const int case_sensitive);\n\nsize_t invert_matches(const char *buf, const size_t buf_len, match_t matches[], size_t matches_len);\nvoid realloc_matches(match_t **matches, size_t *matches_size, size_t matches_len);\nvoid compile_study(pcre **re, pcre_extra **re_extra, char *q, const int pcre_opts, const int study_opts);\n\n\nint is_binary(const void *buf, const size_t buf_len);\nint is_regex(const char *query);\nint is_fnmatch(const char *filename);\nint binary_search(const char *needle, char **haystack, int start, int end);\n\nvoid init_wordchar_table(void);\nint is_wordchar(char ch);\n\nint is_lowercase(const char *s);\n\nint is_directory(const char *path, const struct dirent *d);\nint is_symlink(const char *path, const struct dirent *d);\nint is_named_pipe(const char *path, const struct dirent *d);\n\nvoid die(const char *fmt, ...);\n\nvoid ag_asprintf(char **ret, const char *fmt, ...);\n\nssize_t buf_getline(const char **line, const char *buf, const size_t buf_len, const size_t buf_offset);\n\n#ifndef HAVE_FGETLN\nchar *fgetln(FILE *fp, size_t *lenp);\n#endif\n#ifndef HAVE_GETLINE\nssize_t getline(char **lineptr, size_t *n, FILE *stream);\n#endif\n#ifndef HAVE_REALPATH\nchar *realpath(const char *path, char *resolved_path);\n#endif\n#ifndef HAVE_STRLCPY\nsize_t strlcpy(char *dest, const char *src, size_t size);\n#endif\n#ifndef HAVE_VASPRINTF\nint vasprintf(char **ret, const char *fmt, va_list args);\n#endif\n\n#endif\n" }, { "path": "src/win32/config.h", "content": "#define HAVE_LZMA_H\n#define HAVE_PTHREAD_H\n" }, { "path": "src/zfile.c", "content": "#ifdef __FreeBSD__\n#include \n#endif\n#include \n\n#ifdef __CYGWIN__\ntypedef _off64_t off64_t;\n#endif\n\n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n#include \n\n#include \"config.h\"\n\n#ifdef HAVE_ERR_H\n#include \n#endif\n#ifdef HAVE_ZLIB_H\n#include \n#endif\n#ifdef HAVE_LZMA_H\n#include \n#endif\n\n#include \"decompress.h\"\n\n#if HAVE_FOPENCOOKIE\n\n#define min(a, b) ({\t\t\t\t\\\n\t__typeof (a) _a = (a);\t\t\t\\\n\t__typeof (b) _b = (b);\t\t\t\\\n\t_a < _b ? _a : _b; })\n\nstatic cookie_read_function_t zfile_read;\nstatic cookie_seek_function_t zfile_seek;\nstatic cookie_close_function_t zfile_close;\n\nstatic const cookie_io_functions_t zfile_io = {\n .read = zfile_read,\n .write = NULL,\n .seek = zfile_seek,\n .close = zfile_close,\n};\n\n#define KB (1024)\nstruct zfile {\n FILE *in; // Source FILE stream\n uint64_t logic_offset, // Logical offset in output (forward seeks)\n decode_offset, // Where we've decoded to\n actual_len;\n uint32_t outbuf_start;\n\n ag_compression_type ctype;\n\n union {\n z_stream gz;\n lzma_stream lzma;\n } stream;\n\n uint8_t inbuf[32 * KB];\n uint8_t outbuf[256 * KB];\n bool eof;\n};\n\n#define CAVAIL_IN(c) ((c)->ctype == AG_GZIP ? (c)->stream.gz.avail_in : (c)->stream.lzma.avail_in)\n#define CNEXT_OUT(c) ((c)->ctype == AG_GZIP ? (c)->stream.gz.next_out : (c)->stream.lzma.next_out)\n\nstatic int\nzfile_cookie_init(struct zfile *cookie) {\n#ifdef HAVE_LZMA_H\n lzma_ret lzrc;\n#endif\n int rc;\n\n assert(cookie->logic_offset == 0);\n assert(cookie->decode_offset == 0);\n\n cookie->actual_len = 0;\n\n switch (cookie->ctype) {\n#ifdef HAVE_ZLIB_H\n case AG_GZIP:\n memset(&cookie->stream.gz, 0, sizeof cookie->stream.gz);\n rc = inflateInit2(&cookie->stream.gz, 32 + 15);\n if (rc != Z_OK) {\n log_err(\"Unable to initialize zlib: %s\", zError(rc));\n return EIO;\n }\n cookie->stream.gz.next_in = NULL;\n cookie->stream.gz.avail_in = 0;\n cookie->stream.gz.next_out = cookie->outbuf;\n cookie->stream.gz.avail_out = sizeof cookie->outbuf;\n break;\n#endif\n#ifdef HAVE_LZMA_H\n case AG_XZ:\n cookie->stream.lzma = (lzma_stream)LZMA_STREAM_INIT;\n lzrc = lzma_auto_decoder(&cookie->stream.lzma, -1, 0);\n if (lzrc != LZMA_OK) {\n log_err(\"Unable to initialize lzma_auto_decoder: %d\", lzrc);\n return EIO;\n }\n cookie->stream.lzma.next_in = NULL;\n cookie->stream.lzma.avail_in = 0;\n cookie->stream.lzma.next_out = cookie->outbuf;\n cookie->stream.lzma.avail_out = sizeof cookie->outbuf;\n break;\n#endif\n default:\n log_err(\"Unsupported compression type: %d\", cookie->ctype);\n return EINVAL;\n }\n\n\n cookie->outbuf_start = 0;\n cookie->eof = false;\n return 0;\n}\n\nstatic void\nzfile_cookie_cleanup(struct zfile *cookie) {\n switch (cookie->ctype) {\n#ifdef HAVE_ZLIB_H\n case AG_GZIP:\n inflateEnd(&cookie->stream.gz);\n break;\n#endif\n#ifdef HAVE_LZMA_H\n case AG_XZ:\n lzma_end(&cookie->stream.lzma);\n break;\n#endif\n default:\n /* Compiler false positive - unreachable. */\n break;\n }\n}\n\n/*\n * Open compressed file 'path' as a (forward-)seekable (and rewindable),\n * read-only stream.\n */\nFILE *\ndecompress_open(int fd, const char *mode, ag_compression_type ctype) {\n struct zfile *cookie;\n FILE *res, *in;\n int error;\n\n cookie = NULL;\n in = res = NULL;\n if (strstr(mode, \"w\") || strstr(mode, \"a\")) {\n errno = EINVAL;\n goto out;\n }\n\n in = fdopen(fd, mode);\n if (in == NULL)\n goto out;\n\n /*\n\t * No validation of compression type is done -- file is assumed to\n\t * match input. In Ag, the compression type is already detected, so\n\t * that's ok.\n\t */\n cookie = malloc(sizeof *cookie);\n if (cookie == NULL) {\n errno = ENOMEM;\n goto out;\n }\n\n cookie->in = in;\n cookie->logic_offset = 0;\n cookie->decode_offset = 0;\n cookie->ctype = ctype;\n\n error = zfile_cookie_init(cookie);\n if (error != 0) {\n errno = error;\n goto out;\n }\n\n res = fopencookie(cookie, mode, zfile_io);\n\nout:\n if (res == NULL) {\n if (in != NULL)\n fclose(in);\n if (cookie != NULL)\n free(cookie);\n }\n return res;\n}\n\n/*\n * Return number of bytes into buf, 0 on EOF, -1 on error. Update stream\n * offset.\n */\nstatic ssize_t\nzfile_read(void *cookie_, char *buf, size_t size) {\n struct zfile *cookie = cookie_;\n size_t nb, ignorebytes;\n ssize_t total = 0;\n lzma_ret lzret;\n int ret;\n\n assert(size <= SSIZE_MAX);\n\n if (size == 0)\n return 0;\n\n if (cookie->eof)\n return 0;\n\n ret = Z_OK;\n lzret = LZMA_OK;\n\n ignorebytes = cookie->logic_offset - cookie->decode_offset;\n assert(ignorebytes == 0);\n\n do {\n size_t inflated;\n\n /* Drain output buffer first */\n while (CNEXT_OUT(cookie) >\n &cookie->outbuf[cookie->outbuf_start]) {\n size_t left = CNEXT_OUT(cookie) -\n &cookie->outbuf[cookie->outbuf_start];\n size_t ignoreskip = min(ignorebytes, left);\n size_t toread;\n\n if (ignoreskip > 0) {\n ignorebytes -= ignoreskip;\n left -= ignoreskip;\n cookie->outbuf_start += ignoreskip;\n cookie->decode_offset += ignoreskip;\n }\n\n // Ran out of output before we seek()ed up.\n if (ignorebytes > 0)\n break;\n\n toread = min(left, size);\n memcpy(buf, &cookie->outbuf[cookie->outbuf_start],\n toread);\n\n buf += toread;\n size -= toread;\n left -= toread;\n cookie->outbuf_start += toread;\n cookie->decode_offset += toread;\n cookie->logic_offset += toread;\n total += toread;\n\n if (size == 0)\n break;\n }\n\n if (size == 0)\n break;\n\n /*\n\t\t * If we have not satisfied read, the output buffer must be\n\t\t * empty.\n\t\t */\n assert(cookie->stream.gz.next_out ==\n &cookie->outbuf[cookie->outbuf_start]);\n\n if ((cookie->ctype == AG_XZ && lzret == LZMA_STREAM_END) ||\n (cookie->ctype == AG_GZIP && ret == Z_STREAM_END)) {\n cookie->eof = true;\n break;\n }\n\n /* Read more input if empty */\n if (CAVAIL_IN(cookie) == 0) {\n nb = fread(cookie->inbuf, 1, sizeof cookie->inbuf,\n cookie->in);\n if (ferror(cookie->in)) {\n warn(\"error read core\");\n exit(1);\n }\n if (nb == 0 && feof(cookie->in)) {\n warn(\"truncated file\");\n exit(1);\n }\n if (cookie->ctype == AG_XZ) {\n cookie->stream.lzma.avail_in = nb;\n cookie->stream.lzma.next_in = cookie->inbuf;\n } else {\n cookie->stream.gz.avail_in = nb;\n cookie->stream.gz.next_in = cookie->inbuf;\n }\n }\n\n /* Reset stream state to beginning of output buffer */\n if (cookie->ctype == AG_XZ) {\n cookie->stream.lzma.next_out = cookie->outbuf;\n cookie->stream.lzma.avail_out = sizeof cookie->outbuf;\n } else {\n cookie->stream.gz.next_out = cookie->outbuf;\n cookie->stream.gz.avail_out = sizeof cookie->outbuf;\n }\n cookie->outbuf_start = 0;\n\n if (cookie->ctype == AG_GZIP) {\n ret = inflate(&cookie->stream.gz, Z_NO_FLUSH);\n if (ret != Z_OK && ret != Z_STREAM_END) {\n log_err(\"Found mem/data error while decompressing zlib stream: %s\", zError(ret));\n return -1;\n }\n } else {\n lzret = lzma_code(&cookie->stream.lzma, LZMA_RUN);\n if (lzret != LZMA_OK && lzret != LZMA_STREAM_END) {\n log_err(\"Found mem/data error while decompressing xz/lzma stream: %d\", lzret);\n return -1;\n }\n }\n inflated = CNEXT_OUT(cookie) - &cookie->outbuf[0];\n cookie->actual_len += inflated;\n } while (!ferror(cookie->in) && size > 0);\n\n assert(total <= SSIZE_MAX);\n return total;\n}\n\nstatic int\nzfile_seek(void *cookie_, off64_t *offset_, int whence) {\n struct zfile *cookie = cookie_;\n off64_t new_offset = 0, offset = *offset_;\n\n if (whence == SEEK_SET) {\n new_offset = offset;\n } else if (whence == SEEK_CUR) {\n new_offset = (off64_t)cookie->logic_offset + offset;\n } else {\n /* SEEK_END not ok */\n return -1;\n }\n\n if (new_offset < 0)\n return -1;\n\n /* Backward seeks to anywhere but 0 are not ok */\n if (new_offset < (off64_t)cookie->logic_offset && new_offset != 0) {\n return -1;\n }\n\n if (new_offset == 0) {\n /* rewind(3) */\n cookie->decode_offset = 0;\n cookie->logic_offset = 0;\n zfile_cookie_cleanup(cookie);\n zfile_cookie_init(cookie);\n } else if ((uint64_t)new_offset > cookie->logic_offset) {\n /* Emulate forward seek by skipping ... */\n char *buf;\n const size_t bsz = 32 * 1024;\n\n buf = malloc(bsz);\n while ((uint64_t)new_offset > cookie->logic_offset) {\n size_t diff = min(bsz,\n (uint64_t)new_offset - cookie->logic_offset);\n ssize_t err = zfile_read(cookie_, buf, diff);\n if (err < 0) {\n free(buf);\n return -1;\n }\n\n /* Seek past EOF gets positioned at EOF */\n if (err == 0) {\n assert(cookie->eof);\n new_offset = cookie->logic_offset;\n break;\n }\n }\n free(buf);\n }\n\n assert(cookie->logic_offset == (uint64_t)new_offset);\n\n *offset_ = new_offset;\n return 0;\n}\n\nstatic int\nzfile_close(void *cookie_) {\n struct zfile *cookie = cookie_;\n\n zfile_cookie_cleanup(cookie);\n fclose(cookie->in);\n free(cookie);\n\n return 0;\n}\n\n#endif /* HAVE_FOPENCOOKIE */\n" }, { "path": "tests/adjacent_matches.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ alias ag=\"$TESTDIR/../ag --noaffinity --workers=1 --parallel --color\"\n $ printf 'blahfoofooblah\\n' > ./fooblah.txt\n\nHighlights are adjacent:\n\n $ ag --no-numbers foo\n \\x1b[1;32mfooblah.txt\\x1b[0m\\x1b[K:blah\\x1b[30;43mfoo\\x1b[0m\\x1b[K\\x1b[30;43mfoo\\x1b[0m\\x1b[Kblah (esc)\n" }, { "path": "tests/bad_path.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n\nComplain about nonexistent path:\n\n $ ag foo doesnt_exist\n ERR: Error stat()ing: doesnt_exist\n ERR: Error opening directory doesnt_exist: No such file or directory\n [1]\n" }, { "path": "tests/big/big_file.t", "content": "Setup and create really big file:\n\n $ . $TESTDIR/../setup.sh\n $ python3 $TESTDIR/create_big_file.py $TESTDIR/big_file.txt\n\nSearch a big file:\n\n $ $TESTDIR/../../ag --nocolor --workers=1 --parallel hello $TESTDIR/big_file.txt\n 33554432:hello1073741824\n 67108864:hello2147483648\n 100663296:hello3221225472\n 134217728:hello4294967296\n 167772160:hello5368709120\n 201326592:hello6442450944\n 234881024:hello7516192768\n 268435456:hello\n\nFail to regex search a big file:\n $ $TESTDIR/../../ag --nocolor --workers=1 --parallel 'hello.*' $TESTDIR/big_file.txt\n ERR: Skipping */big_file.txt: pcre_exec() can't handle files larger than 2147483647 bytes. (glob)\n [1]\n" }, { "path": "tests/big/create_big_file.py", "content": "#!/usr/bin/env python\n\n# Create an 8GB file of mostly \"abcdefghijklmnopqrstuvwxyz01234\",\n# with a few instances of \"hello\"\n\nimport sys\n\nif len(sys.argv) != 2:\n print(\"Usage: %s big_file.txt\" % sys.argv[0])\n sys.exit(1)\n\nbig_file = sys.argv[1]\n\n\ndef create_big_file():\n with open(big_file, \"w\") as fd:\n for i in range(1, 2**28):\n byte = i * 32\n if byte % 2**30 == 0:\n fd.write(\"hello%s\\n\" % byte)\n else:\n fd.write(\"abcdefghijklmnopqrstuvwxyz01234\\n\")\n fd.write(\"hello\\n\")\n\n\ntry:\n fd = open(big_file, \"r\")\nexcept Exception as e:\n create_big_file()\n" }, { "path": "tests/case_sensitivity.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf 'Foo\\n' >> ./sample\n $ printf 'bar\\n' >> ./sample\n\nSmart case by default:\n\n $ ag foo sample\n 1:Foo\n $ ag FOO sample\n [1]\n $ ag 'f.o' sample\n 1:Foo\n $ ag Foo sample\n 1:Foo\n $ ag 'F.o' sample\n 1:Foo\n\nCase sensitive mode:\n\n $ ag -s foo sample\n [1]\n $ ag -s FOO sample\n [1]\n $ ag -s 'f.o' sample\n [1]\n $ ag -s Foo sample\n 1:Foo\n $ ag -s 'F.o' sample\n 1:Foo\n\nCase insensitive mode:\n\n $ ag fOO -i sample\n 1:Foo\n $ ag fOO --ignore-case sample\n 1:Foo\n $ ag 'f.o' -i sample\n 1:Foo\n\nCase insensitive file regex\n\n $ ag -i -g 'Samp.*'\n sample\n" }, { "path": "tests/color.t", "content": "Setup. Note that we have to turn --color on manually since ag detects that\nstdout isn't a tty when running in cram.\n\n $ . $TESTDIR/setup.sh\n $ alias ag=\"$TESTDIR/../ag --noaffinity --workers=1 --parallel --color\"\n $ printf 'foo\\n' > ./blah.txt\n $ printf 'bar\\n' >> ./blah.txt\n\nMatches should contain colors:\n\n $ ag --no-numbers foo blah.txt\n \\x1b[30;43mfoo\\x1b[0m\\x1b[K (esc)\n\n--nocolor should suppress colors:\n\n $ ag --nocolor foo blah.txt\n 1:foo\n\n--invert-match should suppress colors:\n\n $ ag --invert-match foo blah.txt\n 2:bar\n\n-v is the same as --invert-match\n\n $ ag -v foo blah.txt\n 2:bar\n" }, { "path": "tests/column.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf \"blah\\nblah2\\n\" > blah.txt\n\nEnsure column is correct:\n\n $ ag --column \"blah\\nb\"\n blah.txt:1:1:blah\n blah.txt:2:0:blah2\n\n# Test ackmate output. Not quite right, but at least offsets are in the\n# ballpark instead of being 9 quintillion\n\n $ ag --ackmate \"lah\\nb\"\n :blah.txt\n 1;blah\n 2;1 5:blah2\n" }, { "path": "tests/count.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ unalias ag\n $ alias ag=\"$TESTDIR/../ag --noaffinity --nocolor --workers=1\"\n $ printf \"blah\\n\" > blah.txt\n $ printf \"blah2\\n\" >> blah.txt\n $ printf \"blah_OTHER\\n\" > other_file.txt\n $ printf \"blah_OTHER\\n\" >> other_file.txt\n\nCount matches:\n\n $ ag --count --parallel blah | sort\n blah.txt:2\n other_file.txt:2\n\nCount stream matches:\n\n $ printf 'blah blah blah\\n' | ag --count blah\n 3\n\nCount stream matches per line (not very useful since it does not print zero):\n\n $ cat blah.txt | ag --count blah\n 1\n 1\n" }, { "path": "tests/ds_store_ignore.t", "content": "Setup.\n $ . $TESTDIR/setup.sh\n $ mkdir -p dir0/dir1/dir2\n $ printf '*.DS_Store\\n' > dir0/.ignore\n $ printf 'blah\\n' > dir0/dir1/dir2/blah.txt\n $ touch dir0/dir1/.DS_Store\n\nFind blah in blah.txt\n\n $ ag blah\n dir0/dir1/dir2/blah.txt:1:blah\n" }, { "path": "tests/empty_environment.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf \"hello world\\n\" >test.txt\n\nVerify ag runs with an empty environment:\n\n $ env -i $TESTDIR/../ag --noaffinity --nocolor --workers=1 --parallel hello\n test.txt:1:hello world\n" }, { "path": "tests/empty_match.t", "content": "Setup.\n $ . $TESTDIR/setup.sh\n $ touch empty.txt\n $ printf 'foo\\n' > nonempty.txt\n\nZero-length match on an empty file should fail silently with return code 1\n\n $ ag \"^\" empty.txt\n [1]\n\nA genuine zero-length match should succeed:\n $ ag \"^\" nonempty.txt\n 1:foo\n\nEmpty files should be listed with --unrestricted --files-with-matches (-ul)\n $ ag -lu --stats | sed '$d' | sort # Remove the last line about timing which will differ\n 2 files contained matches\n 2 files searched\n 2 matches\n 4 bytes searched\n empty.txt\n nonempty.txt\n" }, { "path": "tests/exitcodes.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf 'foo\\n' > ./exitcodes_test.txt\n $ printf 'bar\\n' >> ./exitcodes_test.txt\n\nNormal matching:\n\n $ ag foo exitcodes_test.txt\n 1:foo\n $ ag zoo exitcodes_test.txt\n [1]\n\nInverted matching:\n\n $ ag -v foo exitcodes_test.txt\n 2:bar\n $ ag -v zoo exitcodes_test.txt\n 1:foo\n 2:bar\n $ ag -v \"foo|bar\" exitcodes_test.txt\n [1]\n" }, { "path": "tests/fail/unicode_case_insensitive.t", "content": "Setup:\n\n $ . $TESTDIR/../setup.sh\n $ printf \"hello=你好\\n\" > test.txt\n $ printf \"hello=你好\\n\" >> test.txt\n\nNormal search:\n\n $ $TESTDIR/../../ag --nocolor --workers=1 --parallel 你好\n test.txt:1:hello=你好\n test.txt:2:hello=你好\n\nCase-insensitive search:\n\n $ $TESTDIR/../../ag --nocolor --workers=1 --parallel -i 你好\n test.txt:1:hello=你好\n test.txt:2:hello=你好\n" }, { "path": "tests/fail/unicode_case_insensitive.t.err", "content": "Setup:\n\n $ . $TESTDIR/../setup.sh\n $ printf \"hello=你好\\n\" > test.txt\n $ printf \"hello=你好\\n\" >> test.txt\n\nNormal search:\n\n $ $TESTDIR/../../ag --nocolor --workers=1 --parallel 你好\n test.txt:1:hello=\\xe4\\xbd\\xa0\\xe5\\xa5\\xbd (esc)\n test.txt:2:hello=\\xe4\\xbd\\xa0\\xe5\\xa5\\xbd (esc)\n\nCase-insensitive search:\n\n $ $TESTDIR/../../ag --nocolor --workers=1 --parallel -i 你好\n test.txt:1:hello=\\xe4\\xbd\\xa0\\xe5\\xa5\\xbd (esc)\n test.txt:2:hello=\\xe4\\xbd\\xa0\\xe5\\xa5\\xbd (esc)\n" }, { "path": "tests/files_with_matches.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf 'foo\\n' > ./foo.txt\n $ printf 'bar\\n' > ./bar.txt\n $ printf 'foo\\nbar\\nbaz\\n' > ./baz.txt\n $ printf 'duck\\nanother duck\\nyet another duck\\n' > ./duck.txt\n $ cp duck.txt goose.txt\n $ echo \"GOOSE!!!\" >> ./goose.txt\n\nFiles with matches:\n\n $ ag --files-with-matches foo foo.txt\n foo.txt\n $ ag --files-with-matches foo foo.txt bar.txt\n foo.txt\n $ ag --files-with-matches foo bar.txt\n [1]\n $ ag --files-with-matches foo foo.txt bar.txt baz.txt\n foo.txt\n baz.txt\n $ ag --files-with-matches bar foo.txt bar.txt baz.txt\n bar.txt\n baz.txt\n $ ag --files-with-matches foo bar.txt baz.txt\n baz.txt\n\nFiles without matches:\n(Prints names of files in which no line matches query)\n\n $ ag --files-without-matches bar foo.txt\n foo.txt\n $ ag --files-without-matches bar foo.txt bar.txt\n foo.txt\n $ ag --files-without-matches bar bar.txt\n [1]\n $ ag --files-without-matches foo foo.txt bar.txt baz.txt\n bar.txt\n $ ag --files-without-matches bar foo.txt bar.txt baz.txt\n foo.txt\n\nFiles with inverted matches:\n(Prints names of files in which some line doesn't match query)\n\n $ ag --files-with-matches --invert-match bar bar.txt\n [1]\n $ ag --files-with-matches --invert-match foo foo.txt bar.txt baz.txt\n bar.txt\n baz.txt\n $ ag --files-with-matches --invert-match bar foo.txt bar.txt baz.txt\n foo.txt\n baz.txt\n\nFiles without inverted matches:\n(Prints names of files in which no line doesn't match query,\n i.e. where every line matches query)\n\n $ ag --files-without-matches --invert-match duck duck.txt\n duck.txt\n $ ag --files-without-matches --invert-match duck goose.txt\n [1]\n $ ag --files-without-matches --invert-match duck duck.txt goose.txt\n duck.txt\n" }, { "path": "tests/filetype.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ TEST_FILETYPE_EXT1=`ag --list-file-types | grep -E '^[ \\t]+\\..+' | head -n 1 | awk '{ print $1 }'`\n $ TEST_FILETYPE_EXT2=`ag --list-file-types | grep -E '^[ \\t]+\\..+' | tail -n 1 | awk '{ print $1 }'`\n $ TEST_FILETYPE_DIR=filetype_test\n $ mkdir $TEST_FILETYPE_DIR\n $ printf \"This is filetype test1.\\n\" > $TEST_FILETYPE_DIR/test.$TEST_FILETYPE_EXT1\n $ printf \"This is filetype test2.\\n\" > $TEST_FILETYPE_DIR/test.$TEST_FILETYPE_EXT2\n\nMatch only top file type:\n\n $ TEST_FILETYPE_OPTION=`ag --list-file-types | grep -E '^[ \\t]+--.+' | head -n 1 | awk '{ print $1 }'`\n $ ag 'This is filetype test' --nofilename $TEST_FILETYPE_OPTION $TEST_FILETYPE_DIR\n This is filetype test1.\n" }, { "path": "tests/hidden_option.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir hidden_bug\n $ cd hidden_bug\n $ printf \"test\\n\" > a.txt\n $ git init --quiet\n $ if [ ! -d .git/info ] ; then mkdir .git/info ; fi\n $ printf \"a.txt\\n\" > .git/info/exclude\n\n $ ag --ignore-dir .git test\n [1]\n\n $ ag --hidden --ignore-dir .git test\n [1]\n\n $ ag -U --ignore-dir .git test\n a.txt:1:test\n\n $ ag --hidden -U --ignore-dir .git test\n a.txt:1:test\n\n $ mkdir -p ./.hidden\n $ printf 'whatever\\n' > ./.hidden/a.txt\n\n $ ag whatever\n [1]\n\n $ ag --hidden whatever\n [1]\n\n $ printf \"\\n\" > .git/info/exclude\n\n $ ag whatever\n [1]\n\n $ ag --hidden whatever\n .hidden/a.txt:1:whatever\n" }, { "path": "tests/ignore_abs_path.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir -p ./a/b/c\n $ printf 'whatever1\\n' > ./a/b/c/blah.yml\n $ printf 'whatever2\\n' > ./a/b/foo.yml\n $ printf '/a/b/foo.yml\\n' > ./.ignore\n\nIgnore foo.yml but not blah.yml:\n\n $ ag whatever .\n a/b/c/blah.yml:1:whatever1\n\nDont ignore anything (unrestricted search):\n\n $ ag -u whatever . | sort\n a/b/c/blah.yml:1:whatever1\n a/b/foo.yml:1:whatever2\n\nIgnore foo.yml given an absolute search path [#448]:\n\n $ ag whatever $(pwd)\n /.*/a/b/c/blah.yml:1:whatever1 (re)\n" }, { "path": "tests/ignore_absolute_search_path_with_glob.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir -p parent/multi-part\n $ printf 'match1\\n' > parent/multi-part/file1.txt\n $ printf 'parent/multi-*\\n' > .ignore\n\n# Ignore directory specified by glob:\n\n# $ ag match .\n# [1]\n\n# Ignore directory specified by glob with absolute search path (#448):\n\n# $ ag match $(pwd)\n# [1]\n" }, { "path": "tests/ignore_backups.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir -p ./a/b/c\n $ printf 'whatever1\\n' > ./a/b/c/foo.yml\n $ printf 'whatever2\\n' > ./a/b/c/foo.yml~\n $ printf 'whatever3\\n' > ./a/b/c/.foo.yml.swp\n $ printf 'whatever4\\n' > ./a/b/c/.foo.yml.swo\n $ printf 'whatever5\\n' > ./a/b/foo.yml\n $ printf 'whatever6\\n' > ./a/b/foo.yml~\n $ printf 'whatever7\\n' > ./a/b/.foo.yml.swp\n $ printf 'whatever8\\n' > ./a/b/.foo.yml.swo\n $ printf 'whatever9\\n' > ./a/foo.yml\n $ printf 'whatever10\\n' > ./a/foo.yml~\n $ printf 'whatever11\\n' > ./a/.foo.yml.swp\n $ printf 'whatever12\\n' > ./a/.foo.yml.swo\n $ printf 'whatever13\\n' > ./foo.yml\n $ printf 'whatever14\\n' > ./foo.yml~\n $ printf 'whatever15\\n' > ./.foo.yml.swp\n $ printf 'whatever16\\n' > ./.foo.yml.swo\n $ printf '*~\\n' > ./.ignore\n $ printf '*.sw[po]\\n' >> ./.ignore\n\nIgnore all files except foo.yml\n\n $ ag whatever . | sort\n a/b/c/foo.yml:1:whatever1\n a/b/foo.yml:1:whatever5\n a/foo.yml:1:whatever9\n foo.yml:1:whatever13\n\nDont ignore anything (unrestricted search):\n\n $ ag -u whatever . | sort\n .foo.yml.swo:1:whatever16\n .foo.yml.swp:1:whatever15\n a/.foo.yml.swo:1:whatever12\n a/.foo.yml.swp:1:whatever11\n a/b/.foo.yml.swo:1:whatever8\n a/b/.foo.yml.swp:1:whatever7\n a/b/c/.foo.yml.swo:1:whatever4\n a/b/c/.foo.yml.swp:1:whatever3\n a/b/c/foo.yml:1:whatever1\n a/b/c/foo.yml~:1:whatever2\n a/b/foo.yml:1:whatever5\n a/b/foo.yml~:1:whatever6\n a/foo.yml:1:whatever9\n a/foo.yml~:1:whatever10\n foo.yml:1:whatever13\n foo.yml~:1:whatever14\n" }, { "path": "tests/ignore_examine_parent_ignorefiles.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir -p subdir\n $ printf 'match1\\n' > subdir/file1.txt\n $ printf 'file1.txt\\n' > .ignore\n\nIgnore directory specified by name:\n\n $ ag match\n [1]\n\n# Ignore directory specified by name in parent directory when using path (#144):\n\n# $ ag match subdir\n# [1]\n\n# Ignore directory specified by name in parent directory when using current directory (#144):\n\n# $ cd subdir\n# $ ag match\n# [1]\n" }, { "path": "tests/ignore_extensions.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf '*.js\\n' > .ignore\n $ printf '*.test.txt\\n' >> .ignore\n $ printf 'targetA\\n' > something.js\n $ printf 'targetB\\n' > aFile.test.txt\n $ printf 'targetC\\n' > aFile.txt\n $ printf 'targetG\\n' > something.min.js\n $ mkdir -p subdir\n $ printf 'targetD\\n' > subdir/somethingElse.js\n $ printf 'targetE\\n' > subdir/anotherFile.test.txt\n $ printf 'targetF\\n' > subdir/anotherFile.txt\n $ printf 'targetH\\n' > subdir/somethingElse.min.js\n\nIgnore patterns with single extension in root directory:\n\n $ ag \"targetA\"\n [1]\n\nIgnore patterns with multiple extensions in root directory:\n\n $ ag \"targetB\"\n [1]\n\n*.js ignores *.min.js in root directory:\n\n $ ag \"targetG\"\n [1]\n\nDo not ignore patterns with partial extensions in root directory:\n\n $ ag \"targetC\"\n aFile.txt:1:targetC\n\nIgnore patterns with single extension in subdirectory:\n\n $ ag \"targetD\"\n [1]\n\nIgnore patterns with multiple extensions in subdirectory:\n\n $ ag \"targetE\"\n [1]\n\n*.js ignores *.min.js in subdirectory:\n\n $ ag \"targetH\"\n [1]\n\nDo not ignore patterns with partial extensions in subdirectory:\n\n $ ag \"targetF\"\n subdir/anotherFile.txt:1:targetF\n" }, { "path": "tests/ignore_gitignore.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ export HOME=$PWD\n $ printf '[core]\\nexcludesfile = ~/.gitignore.global' >> $HOME/.gitconfig\n $ printf 'PATTERN_MARKER\\n' > .gitignore.global\n\nTest that the ignore pattern got picked up:\n\n $ ag --debug . | grep PATTERN_MARKER\n DEBUG: added ignore pattern PATTERN_MARKER to root ignores\n\n" }, { "path": "tests/ignore_invert.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf 'blah1\\n' > ./printme.txt\n $ printf 'blah2\\n' > ./dontprintme.c\n $ printf '*\\n' > ./.ignore\n $ printf '!*.txt\\n' >> ./.ignore\n\nIgnore .gitignore patterns but not .ignore patterns:\n\n $ ag blah\n printme.txt:1:blah1\n" }, { "path": "tests/ignore_pattern_in_subdirectory.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir subdir\n $ printf 'first\\n' > file1.txt\n $ printf 'second\\n' > subdir/file2.txt\n $ printf '*.txt\\n' > .gitignore\n\nIgnore file based on extension match:\n\n $ ag first\n [1]\n\nIgnore file in subdirectory based on extension match (#442):\n\n $ ag second\n [1]\n" }, { "path": "tests/ignore_slash_in_subdir.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir -p subdir/ignoredir\n $ mkdir ignoredir\n $ printf 'match1\\n' > subdir/ignoredir/file1.txt\n $ printf 'match1\\n' > ignoredir/file1.txt\n $ printf '/ignoredir\\n' > subdir/.ignore\n\nIgnore file in subdir/ignoredir, but not in ignoredir:\n\n $ ag match\n ignoredir/file1.txt:1:match1\n\nFrom subdir, ignore file in subdir/ignoredir:\n\n $ cd subdir\n $ ag match\n [1]\n" }, { "path": "tests/ignore_subdir.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ mkdir -p ./a/b/c\n $ printf 'whatever1\\n' > ./a/b/c/blah.yml\n $ printf 'whatever2\\n' > ./a/b/foo.yml\n $ printf 'a/b/foo.yml\\n' > ./.gitignore\n# TODO: have this work instead of the above\n# $ printf 'a/b/*.yml\\n' > ./.gitignore\n\nIgnore foo.yml but not blah.yml:\n\n $ ag whatever .\n a/b/c/blah.yml:1:whatever1\n\nDont ignore anything (unrestricted search):\n\n $ ag -u whatever . | sort\n a/b/c/blah.yml:1:whatever1\n a/b/foo.yml:1:whatever2\n" }, { "path": "tests/ignore_vcs.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf 'whatever1\\n' > ./always.txt\n $ printf 'whatever2\\n' > ./git.txt\n $ printf 'whatever3\\n' > ./text.txt\n $ printf 'git.txt\\n' > ./.gitignore\n $ printf 'text.*\\n' > ./.ignore\n\nObey .gitignore and .ignore patterns:\n\n $ ag whatever .\n always.txt:1:whatever1\n\nIgnore .gitignore patterns but not .ignore patterns:\n\n $ ag -U whatever . | sort\n always.txt:1:whatever1\n git.txt:1:whatever2\n" }, { "path": "tests/invert_match.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf 'valid: 1\\n' > ./blah.txt\n $ printf 'some_string\\n' >> ./blah.txt\n $ printf 'valid: 654\\n' >> ./blah.txt\n $ printf 'some_other_string\\n' >> ./blah.txt\n $ printf 'valid: 0\\n' >> ./blah.txt\n $ printf 'valid: 23\\n' >> ./blah.txt\n $ printf 'valid: 0\\n' >> ./blah.txt\n\nSearch for lines not matching \"valid: 0\" in blah.txt:\n\n $ ag -v 'valid: '\n blah.txt:2:some_string\n blah.txt:4:some_other_string\n" }, { "path": "tests/is_binary_pdf.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ cp $TESTDIR/is_binary.pdf .\n\nPDF files are binary. Do not search them by default:\n\n $ ag PDF\n [1]\n\nOK, search binary files\n\n $ ag --search-binary PDF\n Binary file is_binary.pdf matches.\n" }, { "path": "tests/line_width.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf \"12345678901234567890123456789012345678901234567890\\n\" >> ./blah.txt\n\nTruncate to width inside input line length:\n\n $ ag -W 20 1 < ./blah.txt\n blah.txt:1:12345678901234567890 [...]\n\nTruncate to width inside input line length, long-form:\n\n $ ag --width 20 1 < ./blah.txt\n blah.txt:1:12345678901234567890 [...]\n\nTruncate to width outside input line length:\n\n $ ag -W 60 1 < ./blah.txt\n blah.txt:1:12345678901234567890123456789012345678901234567890\n\nTruncate to width one less than input line length:\n\n $ ag -W 49 1 < ./blah.txt\n blah.txt:1:1234567890123456789012345678901234567890123456789 [...]\n\nTruncate to width exactly input line length:\n\n $ ag -W 50 1 < ./blah.txt\n blah.txt:1:12345678901234567890123456789012345678901234567890\n\n" }, { "path": "tests/list_file_types.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n\nLanguage types are output:\n\n $ ag --list-file-types\n The following file types are supported:\n --actionscript\n .as .mxml\n \n --ada\n .ada .adb .ads\n \n --asciidoc\n .adoc .ad .asc .asciidoc\n \n --apl\n .apl\n \n --asm\n .asm .s\n \n --asp\n .asp .asa .aspx .asax .ashx .ascx .asmx\n \n --aspx\n .asp .asa .aspx .asax .ashx .ascx .asmx\n \n --batch\n .bat .cmd\n \n --bazel\n .bazel\n \n --bitbake\n .bb .bbappend .bbclass .inc\n \n --cc\n .c .h .xs\n \n --cfmx\n .cfc .cfm .cfml\n \n --chpl\n .chpl\n \n --clojure\n .clj .cljs .cljc .cljx .edn\n \n --coffee\n .coffee .cjsx\n \n --config\n .config\n \n --coq\n .coq .g .v\n \n --cpp\n .cpp .cc .C .cxx .m .hpp .hh .h .H .hxx .tpp\n \n --crystal\n .cr .ecr\n \n --csharp\n .cs\n \n --cshtml\n .cshtml\n \n --css\n .css\n \n --cython\n .pyx .pxd .pxi\n \n --delphi\n .pas .int .dfm .nfm .dof .dpk .dpr .dproj .groupproj .bdsgroup .bdsproj\n \n --dlang\n .d .di\n \n --dot\n .dot .gv\n \n --dts\n .dts .dtsi\n \n --ebuild\n .ebuild .eclass\n \n --elisp\n .el\n \n --elixir\n .ex .eex .exs\n \n --elm\n .elm\n \n --erlang\n .erl .hrl\n \n --factor\n .factor\n \n --fortran\n .f .F .f77 .f90 .F90 .f95 .f03 .for .ftn .fpp .FPP\n \n --fsharp\n .fs .fsi .fsx\n \n --gettext\n .po .pot .mo\n \n --glsl\n .vert .tesc .tese .geom .frag .comp\n \n --go\n .go\n \n --gradle\n .gradle\n \n --groovy\n .groovy .gtmpl .gpp .grunit .gradle\n \n --haml\n .haml\n \n --handlebars\n .hbs\n \n --haskell\n .hs .hsig .lhs\n \n --haxe\n .hx\n \n --hh\n .h\n \n --html\n .htm .html .shtml .xhtml\n \n --idris\n .idr .ipkg .lidr\n \n --ini\n .ini\n \n --ipython\n .ipynb\n \n --isabelle\n .thy\n \n --j\n .ijs\n \n --jade\n .jade\n \n --java\n .java .properties\n \n --jinja2\n .j2\n \n --js\n .es6 .js .jsx .vue\n \n --json\n .json\n \n --jsp\n .jsp .jspx .jhtm .jhtml .jspf .tag .tagf\n \n --julia\n .jl\n \n --kotlin\n .kt\n \n --less\n .less\n \n --liquid\n .liquid\n \n --lisp\n .lisp .lsp\n \n --log\n .log\n \n --lua\n .lua\n \n --m4\n .m4\n \n --make\n .Makefiles .mk .mak\n \n --mako\n .mako\n \n --markdown\n .markdown .mdown .mdwn .mkdn .mkd .md\n \n --mason\n .mas .mhtml .mpl .mtxt\n \n --matlab\n .m\n \n --mathematica\n .m .wl\n \n --md\n .markdown .mdown .mdwn .mkdn .mkd .md\n \n --mercury\n .m .moo\n \n --naccess\n .asa .rsa\n \n --nim\n .nim\n \n --nix\n .nix\n \n --objc\n .m .h\n \n --objcpp\n .mm .h\n \n --ocaml\n .ml .mli .mll .mly\n \n --octave\n .m\n \n --org\n .org\n \n --parrot\n .pir .pasm .pmc .ops .pod .pg .tg\n \n --pdb\n .pdb\n \n --perl\n .pl .pm .pm6 .pod .t\n \n --php\n .php .phpt .php3 .php4 .php5 .phtml\n \n --pike\n .pike .pmod\n \n --plist\n .plist\n \n --plone\n .pt .cpt .metadata .cpy .py .xml .zcml\n \n --powershell\n .ps1\n \n --proto\n .proto\n \n --ps1\n .ps1\n \n --pug\n .pug\n \n --puppet\n .pp\n \n --python\n .py\n \n --qml\n .qml\n \n --racket\n .rkt .ss .scm\n \n --rake\n .Rakefile\n \n --razor\n .cshtml\n \n --restructuredtext\n .rst\n \n --rs\n .rs\n \n --r\n .r .R .Rmd .Rnw .Rtex .Rrst\n \n --rdoc\n .rdoc\n \n --ruby\n .rb .rhtml .rjs .rxml .erb .rake .spec\n \n --rust\n .rs\n \n --salt\n .sls\n \n --sass\n .sass .scss\n \n --scala\n .scala\n \n --scheme\n .scm .ss\n \n --shell\n .sh .bash .csh .tcsh .ksh .zsh .fish\n \n --smalltalk\n .st\n \n --sml\n .sml .fun .mlb .sig\n \n --sql\n .sql .ctl\n \n --stata\n .do .ado\n \n --stylus\n .styl\n \n --swift\n .swift\n \n --tcl\n .tcl .itcl .itk\n \n --terraform\n .tf .tfvars\n \n --tex\n .tex .cls .sty\n \n --thrift\n .thrift\n \n --tla\n .tla\n \n --tt\n .tt .tt2 .ttml\n \n --toml\n .toml\n \n --ts\n .ts .tsx\n \n --twig\n .twig\n \n --vala\n .vala .vapi\n \n --vb\n .bas .cls .frm .ctl .vb .resx\n \n --velocity\n .vm .vtl .vsl\n \n --verilog\n .v .vh .sv .svh\n \n --vhdl\n .vhd .vhdl\n \n --vim\n .vim\n \n --vue\n .vue\n \n --wix\n .wxi .wxs\n \n --wsdl\n .wsdl\n \n --wadl\n .wadl\n \n --xml\n .xml .dtd .xsl .xslt .xsd .ent .tld .plist .wsdl\n \n --yaml\n .yaml .yml\n \n --zeek\n .zeek .bro .bif\n \n --zephir\n .zep\n " }, { "path": "tests/literal_word_regexp.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ echo 'blah abc def' > blah1.txt\n $ echo 'abc blah def' > blah2.txt\n $ echo 'abc def blah' > blah3.txt\n $ echo 'abcblah def' > blah4.txt\n $ echo 'abc blahdef' >> blah4.txt\n $ echo 'blahx blah' > blah5.txt\n $ echo 'abcblah blah blah' > blah6.txt\n\nMatch a word of the beginning:\n\n $ ag -wF --column 'blah' blah1.txt\n 1:1:blah abc def\n\nMatch a middle word:\n\n $ ag -wF --column 'blah' blah2.txt\n 1:5:abc blah def\n\nMatch a last word:\n\n $ ag -wF --column 'blah' blah3.txt\n 1:9:abc def blah\n\nNo match:\n\n $ ag -wF --column 'blah' blah4.txt\n [1]\n\nMatch:\n\n $ ag -wF --column 'blah' blah5.txt\n 1:7:blahx blah\n\nCase of a word repeating the same part:\n\n $ ag -wF --column 'blah blah' blah6.txt\n 1:9:abcblah blah blah\n" }, { "path": "tests/max_count.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf \"blah\\n\" > blah.txt\n $ printf \"blah2\\n\" >> blah.txt\n $ printf \"blah2\\n\" > blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt\n $ printf \"blah2\\n\" >> blah2.txt # 10 lines\n\nMax match of 1:\n\n $ ag --max-count 1 blah blah.txt\n ERR: Too many matches in blah.txt. Skipping the rest of this file.\n 1:blah\n\nMax match of 10, one file:\n\n $ ag --count --max-count 10 blah blah2.txt\n ERR: Too many matches in blah2.txt. Skipping the rest of this file.\n 10\n\nMax match of 10, multiple files:\n\n $ ag --count --max-count 10 blah blah.txt blah2.txt\n ERR: Too many matches in blah2.txt. Skipping the rest of this file.\n blah.txt:2\n blah2.txt:10\n" }, { "path": "tests/multiline.t", "content": "Setup:\n\n $ . $TESTDIR/setup.sh\n $ printf 'what\\n' > blah.txt\n $ printf 'ever\\n' >> blah.txt\n $ printf 'whatever\\n' >> blah.txt\n\nMultiline:\n\n $ ag 'wh[^w]+er' .\n blah.txt:1:what\n blah.txt:2:ever\n blah.txt:3:whatever\n\nNo multiline:\n\n $ ag --nomultiline 'wh[^w]+er' .\n blah.txt:3:whatever\n\nMultiline explicit:\n\n $ ag '^wh[^w\\n]+er$' .\n blah.txt:3:whatever\n" }, { "path": "tests/negated_options.t", "content": "Setup:\n\n $ . \"${TESTDIR}/setup.sh\"\n\nShould accept both --no-