[
  {
    "path": ".github/ISSUE_TEMPLATE/bug_report.md",
    "content": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: \"[Issue] \"\nlabels: ''\nassignees: ''\n\n---\n\n**Describe the bug**\nA clear and concise description of what the bug is.\n\n**Subfinder version**\nInclude the version of subfinder you are using, `subfinder -version`\n\n**Complete command you used to reproduce this**\n\n\n**Screenshots**\nAdd screenshots of the error for a better context.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/config.yml",
    "content": "blank_issues_enabled: false\n\ncontact_links:\n  - name: Ask an question / advise on using subfinder\n    url: https://github.com/projectdiscovery/subfinder/discussions/categories/q-a\n    about: Ask a question or request support for using subfinder\n\n  - name: Share idea / feature to discuss for subfinder\n    url: https://github.com/projectdiscovery/subfinder/discussions/categories/ideas\n    about: Share idea / feature to discuss for subfinder\n\n  - name: Connect with PD Team (Discord)\n    url: https://discord.gg/projectdiscovery\n    about: Connect with PD Team for direct communication"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature_request.md",
    "content": "---\nname: Feature request\nabout: Request feature to implement in this project\nlabels: 'Type: Enhancement'\n---\n\n<!--\n1. Please make sure to provide a detailed description with all the relevant information that might be required to start working on this feature.\n2. In case you are not sure about your request or whether the particular feature is already supported or not, please start a discussion instead.\n3. GitHub Discussion: https://github.com/projectdiscovery/subfinder/discussions/categories/ideas\n4. Join our discord server at https://discord.gg/projectdiscovery to discuss the idea on the #subfinder channel.\n-->\n\n### Please describe your feature request:\n<!-- A clear and concise description of feature to implement -->\n\n### Describe the use case of this feature:\n<!-- A clear and concise description of the feature request's motivation and the use-cases in which it could be useful. -->\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/issue-report.md",
    "content": "---\nname: Issue report\nabout: Create a report to help us to improve the project\nlabels: 'Type: Bug'\n\n---\n\n<!-- \n1. Please search to see if an issue already exists for the bug you encountered.\n2. For support requests, FAQs or \"How to\" questions, please use the GitHub Discussions section instead - https://github.com/projectdiscovery/subfinder/discussions or\n3. Join our discord server at https://discord.gg/projectdiscovery and post the question on the #subfinder channel.\n-->\n\n<!-- ISSUES MISSING IMPORTANT INFORMATION MAY BE CLOSED WITHOUT INVESTIGATION. -->\n\n### Subfinder version:\n<!-- You can find current version of subfinder with \"subfinder -version\" -->\n<!-- We only accept issues that are reproducible on the latest version of subfinder. -->\n<!-- You can find the latest version of project at https://github.com/projectdiscovery/subfinder/releases/ -->\n\n### Current Behavior:\n<!-- A concise description of what you're experiencing. -->\n\n### Expected Behavior:\n<!-- A concise description of what you expected to happen. -->\n\n### Steps To Reproduce:\n<!--\nExample: steps to reproduce the behavior:\n1. Run 'subfinder ..'\n2. See error...\n-->\n\n\n### Anything else:\n<!-- Links? References? Screnshots? Anything that will give us more context about the issue that you are encountering! -->"
  },
  {
    "path": ".github/PULL_REQUEST_TEMPLATE.md",
    "content": "## Proposed changes\n\n<!-- Describe the overall picture of your modifications to help maintainers understand the pull request. PRs are required to be associated to their related issue tickets or feature request. -->\n\n### Proof\n\n<!-- How has this been tested? Please describe the tests that you ran to verify your changes. -->\n\n## Checklist\n\n<!-- Put an \"x\" in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code. -->\n\n- [ ] Pull request is created against the [dev](https://github.com/projectdiscovery/subfinder/tree/dev) branch\n- [ ] All checks passed (lint, unit/integration/regression tests etc.) with my changes\n- [ ] I have added tests that prove my fix is effective or that my feature works\n- [ ] I have added necessary documentation (if appropriate)"
  },
  {
    "path": ".github/dependabot.yml",
    "content": "# To get started with Dependabot version updates, you'll need to specify which\n# package ecosystems to update and where the package manifests are located.\n# Please see the documentation for all configuration options:\n# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates\n\nversion: 2\nupdates:\n\n  # Maintain dependencies for go modules\n  - package-ecosystem: \"gomod\"\n    directory: \"/\"\n    schedule:\n      interval: \"weekly\"\n    target-branch: \"dev\"\n    commit-message:\n      prefix: \"chore\"\n      include: \"scope\"\n    allow:\n      - dependency-name: \"github.com/projectdiscovery/*\"\n    groups:\n      modules:\n        patterns: [\"github.com/projectdiscovery/*\"]\n    labels:\n      - \"Type: Maintenance\"\n\n#  # Maintain dependencies for GitHub Actions\n#  - package-ecosystem: \"github-actions\"\n#    directory: \"/\"\n#    schedule:\n#      interval: \"weekly\"\n#    target-branch: \"dev\"\n#    commit-message:\n#      prefix: \"chore\"\n#      include: \"scope\"\n#    labels:\n#      - \"Type: Maintenance\"\n#\n#  # Maintain dependencies for docker\n#  - package-ecosystem: \"docker\"\n#    directory: \"/\"\n#    schedule:\n#      interval: \"weekly\"\n#    target-branch: \"dev\"\n#    commit-message:\n#      prefix: \"chore\"\n#      include: \"scope\"\n#    labels:\n#      - \"Type: Maintenance\"\n"
  },
  {
    "path": ".github/release.yml",
    "content": "changelog:\n  exclude:\n    authors:\n      - dependabot\n  categories:\n    - title: 🎉 New Features\n      labels:\n        - \"Type: Enhancement\"\n    - title: 🐞 Bug Fixes\n      labels:\n        - \"Type: Bug\" \n    - title: 🔨 Maintenance\n      labels:\n        - \"Type: Maintenance\" \n    - title: Other Changes\n      labels:\n        - \"*\""
  },
  {
    "path": ".github/workflows/build-test.yml",
    "content": "name: 🔨 Build Test\n\non:\n  pull_request:\n    paths:\n      - \"**.go\"\n      - \"**.mod\"\n  workflow_dispatch:\n    inputs:\n      short:\n        description: \"Use -short flag for tests\"\n        required: false\n        type: boolean\n        default: false\n\njobs:\n  lint:\n    name: Lint Test\n    if: \"${{ !endsWith(github.actor, '[bot]') }}\"\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: projectdiscovery/actions/setup/go@v1\n        with:\n          go-version-file: go.mod\n      - name: Run golangci-lint\n        uses: golangci/golangci-lint-action@v8\n        with:\n          version: latest\n          args: --timeout 5m\n\n  build:\n    name: Test Builds\n    needs: [lint]\n    runs-on: ${{ matrix.os }}\n    strategy:\n      matrix:\n        os: [ubuntu-latest, windows-latest, macOS-latest]\n    steps:\n      - uses: actions/checkout@v4\n      - uses: projectdiscovery/actions/setup/go@v1\n        with:\n          go-version-file: go.mod\n      - run: go build ./...\n\n      - name: Run tests\n        env:\n          BEVIGIL_API_KEY: ${{secrets.BEVIGIL_API_KEY}}\n          BINARYEDGE_API_KEY: ${{secrets.BINARYEDGE_API_KEY}}\n          BUFFEROVER_API_KEY: ${{secrets.BUFFEROVER_API_KEY}}\n          C99_API_KEY: ${{secrets.C99_API_KEY}}\n          CENSYS_API_KEY: ${{secrets.CENSYS_API_KEY}}\n          CERTSPOTTER_API_KEY: ${{secrets.CERTSPOTTER_API_KEY}}\n          CHAOS_API_KEY: ${{secrets.CHAOS_API_KEY}}\n          CHINAZ_API_KEY: ${{secrets.CHINAZ_API_KEY}}\n          DNSDB_API_KEY: ${{secrets.DNSDB_API_KEY}}\n          DNSREPO_API_KEY: ${{secrets.DNSREPO_API_KEY}}\n          FOFA_API_KEY: ${{secrets.FOFA_API_KEY}}\n          FULLHUNT_API_KEY: ${{secrets.FULLHUNT_API_KEY}}\n          GITHUB_API_KEY: ${{secrets.GITHUB_API_KEY}}\n          INTELX_API_KEY: ${{secrets.INTELX_API_KEY}}\n          LEAKIX_API_KEY: ${{secrets.LEAKIX_API_KEY}}\n          QUAKE_API_KEY: ${{secrets.QUAKE_API_KEY}}\n          ROBTEX_API_KEY: ${{secrets.ROBTEX_API_KEY}}\n          SECURITYTRAILS_API_KEY: ${{secrets.SECURITYTRAILS_API_KEY}}\n          SHODAN_API_KEY: ${{secrets.SHODAN_API_KEY}}\n          THREATBOOK_API_KEY: ${{secrets.THREATBOOK_API_KEY}}\n          URLSCAN_API_KEY: ${{secrets.URLSCAN_API_KEY}}\n          VIRUSTOTAL_API_KEY: ${{secrets.VIRUSTOTAL_API_KEY}}\n          WHOISXMLAPI_API_KEY: ${{secrets.WHOISXMLAPI_API_KEY}}\n          ZOOMEYEAPI_API_KEY: ${{secrets.ZOOMEYEAPI_API_KEY}}\n        uses: nick-invision/retry@v2\n        with:\n          timeout_seconds: 360\n          max_attempts: 3\n          command: go test ./... -v ${{ github.event.inputs.short == 'true' && '-short' || '' }}\n\n      - name: Race Condition Tests\n        run: go build -race ./...\n\n      - name: Run Example\n        run: go run .\n        working-directory: examples\n"
  },
  {
    "path": ".github/workflows/codeql-analysis.yml",
    "content": "name: 🚨 CodeQL Analysis\n\non:\n  workflow_dispatch:\n  pull_request:\n    paths:\n      - '**.go'\n      - '**.mod'\n\njobs:\n  analyze:\n    name: Analyze\n    runs-on: ubuntu-latest\n    permissions:\n      actions: read\n      contents: read\n      security-events: write\n\n    strategy:\n      fail-fast: false\n      matrix:\n        language: [ 'go' ]\n        # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]\n\n    steps:\n    - name: Checkout repository\n      uses: actions/checkout@v3\n\n    # Initializes the CodeQL tools for scanning.\n    - name: Initialize CodeQL\n      uses: github/codeql-action/init@v2\n      with:\n        languages: ${{ matrix.language }}\n\n    - name: Autobuild\n      uses: github/codeql-action/autobuild@v2\n\n    - name: Perform CodeQL Analysis\n      uses: github/codeql-action/analyze@v2"
  },
  {
    "path": ".github/workflows/compat-checks.yaml",
    "content": "name: ♾️ Compatibility Checks\n\non:\n  pull_request:\n    types: [opened, synchronize]\n    branches:\n      - dev\n\njobs:\n  check:\n    if: github.actor == 'dependabot[bot]'\n    runs-on: ubuntu-latest\n    permissions:\n      contents: write\n    steps:\n      - uses: actions/checkout@v4\n      - uses: projectdiscovery/actions/setup/go/compat-checks@master\n        with:\n          go-version-file: 'go.mod'\n\n"
  },
  {
    "path": ".github/workflows/dep-auto-merge.yml",
    "content": "name: 🤖 dep auto merge\n\non:\n  pull_request:\n    branches:\n      - dev\n  workflow_dispatch:\n\npermissions:\n  pull-requests: write\n  issues: write\n  repository-projects: write\n\njobs:\n  automerge:\n    runs-on: ubuntu-latest\n    if: github.actor == 'dependabot[bot]'\n    steps:\n      - uses: actions/checkout@v3\n        with:\n          token: ${{ secrets.DEPENDABOT_PAT }}\n\n      - uses: ahmadnassri/action-dependabot-auto-merge@v2\n        with:\n          github-token: ${{ secrets.DEPENDABOT_PAT }}\n          target: all"
  },
  {
    "path": ".github/workflows/dockerhub-push.yml",
    "content": "name: 🌥 Docker Push\n\non:\n  workflow_run:\n    workflows: [\"🎉 Release Binary\"]\n    types:\n      - completed\n  workflow_dispatch:\n\njobs:\n  docker:\n    runs-on: ubuntu-latest-16-cores\n    steps:\n      - name: Checkout\n        uses: actions/checkout@v3\n\n      - name: Get Github tag\n        id: meta\n        run: |\n          curl --silent \"https://api.github.com/repos/projectdiscovery/subfinder/releases/latest\" | jq -r .tag_name | xargs -I {} echo TAG={} >> $GITHUB_OUTPUT\n\n      - name: Set up QEMU\n        uses: docker/setup-qemu-action@v2\n\n      - name: Set up Docker Buildx\n        uses: docker/setup-buildx-action@v2\n\n      - name: Login to DockerHub\n        uses: docker/login-action@v2 \n        with:\n          username: ${{ secrets.DOCKER_USERNAME }}\n          password: ${{ secrets.DOCKER_TOKEN }}\n\n      - name: Build and push\n        uses: docker/build-push-action@v4\n        with:\n          context: .\n          platforms: linux/amd64,linux/arm64,linux/arm\n          push: true\n          tags: projectdiscovery/subfinder:latest,projectdiscovery/subfinder:${{ steps.meta.outputs.TAG }}\n"
  },
  {
    "path": ".github/workflows/release-binary.yml",
    "content": "name: 🎉 Release Binary\n\non:\n  push:\n    tags:\n      - v*\n  workflow_dispatch:\n\njobs:\n  release:\n    runs-on: ubuntu-latest-16-cores\n    steps:\n      - name: \"Check out code\"\n        uses: actions/checkout@v3\n        with: \n          fetch-depth: 0\n      \n      - name: \"Set up Go\"\n        uses: actions/setup-go@v4\n        with: \n          go-version: 1.21.x\n      \n      - name: \"Create release on GitHub\"\n        uses: goreleaser/goreleaser-action@v3\n        with:\n          args: \"release --clean\"\n          version: latest\n        env:\n          GITHUB_TOKEN: \"${{ secrets.GITHUB_TOKEN }}\"\n          SLACK_WEBHOOK: \"${{ secrets.RELEASE_SLACK_WEBHOOK }}\"\n          DISCORD_WEBHOOK_ID: \"${{ secrets.DISCORD_WEBHOOK_ID }}\"\n          DISCORD_WEBHOOK_TOKEN: \"${{ secrets.DISCORD_WEBHOOK_TOKEN }}\"\n"
  },
  {
    "path": ".github/workflows/release-test.yml",
    "content": "name: 🔨 Release Test\n\non:\n  pull_request:\n    paths:\n      - '**.go'\n      - '**.mod'\n  workflow_dispatch:\n\njobs:\n  release-test:\n    runs-on: ubuntu-latest-16-cores\n    steps:\n      - name: \"Check out code\"\n        uses: actions/checkout@v3\n        with: \n          fetch-depth: 0\n\n      - name: Set up Go\n        uses: actions/setup-go@v4\n        with:\n          go-version: 1.21.x\n      \n      - name: release test\n        uses: goreleaser/goreleaser-action@v4\n        with:\n          args: \"release --clean --snapshot\"\n          version: latest\n"
  },
  {
    "path": ".gitignore",
    "content": ".DS_Store\ncmd/subfinder/subfinder\n# subfinder binary when built with `go build`\nv2/cmd/subfinder/subfinder\n# subfinder binary when built with `make`\nv2/subfinder\nvendor/\n.idea\n.devcontainer\n.vscode\ndist\n/subfinder"
  },
  {
    "path": ".goreleaser.yml",
    "content": "version: 2\n\nbefore:\n  hooks:\n    - go mod tidy\n\nbuilds:\n- env:\n  - CGO_ENABLED=0\n  goos:\n    - windows\n    - linux\n    - darwin\n  goarch:\n    - amd64\n    - '386'\n    - arm\n    - arm64\n\n  ignore:\n    - goos: darwin\n      goarch: '386'\n    - goos: windows\n      goarch: 'arm'\n\n  binary: '{{ .ProjectName }}'\n  main: cmd/subfinder/main.go\n\narchives:\n- formats:\n    - zip\n  name_template: '{{ .ProjectName }}_{{ .Version }}_{{ if eq .Os \"darwin\" }}macOS{{ else }}{{ .Os }}{{ end }}_{{ .Arch }}'\n\nchecksum:\n  algorithm: sha256\n\nannounce:\n  slack:\n    enabled: true\n    channel: '#release'\n    username: GoReleaser\n    message_template: 'New Release: {{ .ProjectName }} {{.Tag}} is published! Check it out at {{ .ReleaseURL }}'\n\n  discord:\n    enabled: true\n    message_template: '**New Release: {{ .ProjectName }} {{.Tag}}** is published! Check it out at {{ .ReleaseURL }}'\n"
  },
  {
    "path": "DISCLAIMER.md",
    "content": "## Disclaimer\n\nSubfinder leverages multiple open APIs, it is developed for individuals to help them for research or internal work. If you wish to incorporate this tool into a commercial offering or purposes, you must agree to the Terms of the leveraged services:\n\n- Bufferover:  https://tls.bufferover.run\n- CommonCrawl: https://commoncrawl.org/terms-of-use/full\n- certspotter: https://sslmate.com/terms\n- dnsdumpster: https://hackertarget.com/terms\n- Google Transparency: https://policies.google.com/terms\n- Alienvault: https://www.alienvault.com/terms/website-terms-of-use07may2018\n\n---\n\nYou expressly understand and agree that Subfinder (creators and contributors) shall not be liable for any damages or losses resulting from your use of this tool or third-party products that use it.\n\nCreators aren't in charge of any and have/has no responsibility for any kind of:\n\n- Unlawful or illegal use of the tool.\n- Legal or Law infringement (acted in any country, state, municipality, place) by third parties and users.\n- Act against ethical and / or human moral, ethic, and peoples and cultures of the world.\n- Malicious act, capable of causing damage to third parties, promoted or distributed by third parties or the user through this tool.\n\n\n### Contact\n\nPlease contact at contact@projectdiscovery.io for any questions.\n"
  },
  {
    "path": "Dockerfile",
    "content": "# Build\nFROM golang:1.24-alpine AS build-env\nRUN apk add build-base\nWORKDIR /app\nCOPY . /app\nRUN go mod download\nRUN go build ./cmd/subfinder\n\n# Release\nFROM alpine:latest\nRUN apk upgrade --no-cache \\\n    && apk add --no-cache bind-tools ca-certificates\nCOPY --from=build-env /app/subfinder /usr/local/bin/\n\nENTRYPOINT [\"subfinder\"]\n"
  },
  {
    "path": "LICENSE.md",
    "content": "MIT License\n\nCopyright (c) 2021 ProjectDiscovery, Inc.\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "Makefile",
    "content": "# Go parameters\nGOCMD=go\nGOBUILD=$(GOCMD) build\nGOMOD=$(GOCMD) mod\nGOTEST=$(GOCMD) test\nGOFLAGS := -v \nLDFLAGS := -s -w\n\nifneq ($(shell go env GOOS),darwin)\nLDFLAGS := -extldflags \"-static\"\nendif\n    \nall: build\nbuild:\n\t$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o \"subfinder\" cmd/subfinder/main.go\ntest: \n\t$(GOTEST) $(GOFLAGS) ./...\ntidy:\n\t$(GOMOD) tidy\n"
  },
  {
    "path": "README.md",
    "content": "<h1 align=\"center\">\n  <img src=\"static/subfinder-logo.png\" alt=\"subfinder\" width=\"200px\">\n  <br>\n</h1>\n\n<h4 align=\"center\">Fast passive subdomain enumeration tool.</h4>\n\n\n<p align=\"center\">\n<a href=\"https://goreportcard.com/report/github.com/projectdiscovery/subfinder/v2\"><img src=\"https://goreportcard.com/badge/github.com/projectdiscovery/subfinder\"></a>\n<a href=\"https://github.com/projectdiscovery/subfinder/issues\"><img src=\"https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat\"></a>\n<a href=\"https://github.com/projectdiscovery/subfinder/releases\"><img src=\"https://img.shields.io/github/release/projectdiscovery/subfinder\"></a>\n<a href=\"https://twitter.com/pdiscoveryio\"><img src=\"https://img.shields.io/twitter/follow/pdiscoveryio.svg?logo=twitter\"></a>\n<a href=\"https://discord.gg/projectdiscovery\"><img src=\"https://img.shields.io/discord/695645237418131507.svg?logo=discord\"></a>\n</p>\n\n<p align=\"center\">\n  <a href=\"#features\">Features</a> •\n  <a href=\"#installation\">Install</a> •\n  <a href=\"#running-subfinder\">Usage</a> •\n  <a href=\"#post-installation-instructions\">API Setup</a> •\n  <a href=\"#subfinder-go-library\">Library</a> •\n  <a href=\"https://discord.gg/projectdiscovery\">Join Discord</a>\n</p>\n\n---\n\n\n`subfinder` is a subdomain discovery tool that returns valid subdomains for websites, using passive online sources. It has a simple, modular architecture and is optimized for speed. `subfinder` is built for\ndoing one thing only - passive subdomain enumeration, and it does that very well.\n\nWe have made it to comply with all the used passive source licenses and usage restrictions. The passive model guarantees speed and stealthiness that can be leveraged by both penetration testers and bug bounty\nhunters alike.\n\n# Features\n\n<h1 align=\"left\">\n  <img src=\"static/subfinder-run.png\" alt=\"subfinder\" width=\"700px\"></a>\n  <br>\n</h1>\n\n- Fast and powerful resolution and wildcard elimination modules\n- **Curated** passive sources to maximize results\n- Multiple output formats supported (JSON, file, stdout)\n- Optimized for speed and **lightweight** on resources\n- **STDIN/OUT** support enables easy integration into workflows\n\n# Usage\n\n```sh\nsubfinder -h\n```\n\nThis will display help for the tool. Here are all the switches it supports.\n\n```yaml\nUsage:\n  ./subfinder [flags]\n\nFlags:\nINPUT:\n  -d, -domain string[]  domains to find subdomains for\n  -dL, -list string     file containing list of domains for subdomain discovery\n\nSOURCE:\n  -s, -sources string[]           specific sources to use for discovery (-s crtsh,github). Use -ls to display all available sources.\n  -recursive                      use only sources that can handle subdomains recursively (e.g. subdomain.domain.tld vs domain.tld)\n  -all                            use all sources for enumeration (slow)\n  -es, -exclude-sources string[]  sources to exclude from enumeration (-es alienvault,zoomeyeapi)\n\nFILTER:\n  -m, -match string[]   subdomain or list of subdomain to match (file or comma separated)\n  -f, -filter string[]   subdomain or list of subdomain to filter (file or comma separated)\n\nRATE-LIMIT:\n  -rl, -rate-limit int  maximum number of http requests to send per second\n  -rls value            maximum number of http requests to send per second for providers in key=value format (-rls \"hackertarget=10/s,shodan=15/s\")\n  -t int                number of concurrent goroutines for resolving (-active only) (default 10)\n\nUPDATE:\n  -up, -update                 update subfinder to latest version\n  -duc, -disable-update-check  disable automatic subfinder update check\n\nOUTPUT:\n  -o, -output string       file to write output to\n  -oJ, -json               write output in JSONL(ines) format\n  -oD, -output-dir string  directory to write output (-dL only)\n  -cs, -collect-sources    include all sources in the output (-json only)\n  -oI, -ip                 include host IP in output (-active only)\n\nCONFIGURATION:\n  -config string                flag config file (default \"$CONFIG/subfinder/config.yaml\")\n  -pc, -provider-config string  provider config file (default \"$CONFIG/subfinder/provider-config.yaml\")\n  -r string[]                   comma separated list of resolvers to use\n  -rL, -rlist string            file containing list of resolvers to use\n  -nW, -active                  display active subdomains only\n  -proxy string                 http proxy to use with subfinder\n  -ei, -exclude-ip              exclude IPs from the list of domains\n\nDEBUG:\n  -silent             show only subdomains in output\n  -version            show version of subfinder\n  -v                  show verbose output\n  -nc, -no-color      disable color in output\n  -ls, -list-sources  list all available sources\n\nOPTIMIZATION:\n  -timeout int   seconds to wait before timing out (default 30)\n  -max-time int  minutes to wait for enumeration results (default 10)\n```\n\n## Environment Variables\n\nSubfinder supports environment variables to specify custom paths for configuration files:\n\n- `SUBFINDER_CONFIG` - Path to config.yaml file (overrides default `$CONFIG/subfinder/config.yaml`)\n- `SUBFINDER_PROVIDER_CONFIG` - Path to provider-config.yaml file (overrides default `$CONFIG/subfinder/provider-config.yaml`)\n\n# Installation\n\n`subfinder` requires **go1.24** to install successfully. Run the following command to install the latest version:\n\n```sh\ngo install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest\n```\n\nLearn about more ways to install subfinder here: https://docs.projectdiscovery.io/tools/subfinder/install.\n\n## Post Installation Instructions\n\n`subfinder` can be used right after the installation, however many sources required API keys to work. Learn more here: https://docs.projectdiscovery.io/tools/subfinder/install#post-install-configuration.\n\n## Running Subfinder\n\nLearn about how to run Subfinder here: https://docs.projectdiscovery.io/tools/subfinder/running.\n\n## Subfinder Go library\n\nSubfinder can also be used as library and a minimal examples of using subfinder SDK is available [here](examples/main.go)\n\n</td>\n</tr>\n</table>\n\n### Resources\n\n- [Recon with Me !!!](https://dhiyaneshgeek.github.io/bug/bounty/2020/02/06/recon-with-me/)\n\n# License\n\n`subfinder` is made with 🖤 by the [projectdiscovery](https://projectdiscovery.io) team. Community contributions have made the project what it is. See\nthe **[THANKS.md](https://github.com/projectdiscovery/subfinder/blob/main/THANKS.md)** file for more details.\n\nRead the usage disclaimer at [DISCLAIMER.md](https://github.com/projectdiscovery/subfinder/blob/main/DISCLAIMER.md) and [contact us](mailto:contact@projectdiscovery.io) for any API removal.\n"
  },
  {
    "path": "THANKS.md",
    "content": "### Thanks\n\nMany people have contributed to subfinder making it a wonderful tool either by making a pull request fixing some stuff or giving generous donations to support the further development of this tool. Here, we recognize these persons and thank them. \n\n- All the contributors at [CONTRIBUTORS](https://github.com/projectdiscovery/subfinder/graphs/contributors) who made subfinder what it is.\n\nWe'd like to thank some additional amazing people, who contributed a lot in subfinder's journey - \n\n- [@vzamanillo](https://github.com/vzamanillo) - For adding multiple features and overall project improvements.\n- [@infosec-au](https://github.com/infosec-au) - Donating to the project.\n- [@codingo](https://github.com/codingo) - Initial work on the project, managing it, lot of work!\n- [@picatz](https://github.com/picatz) - Improving the structure of the project a lot. New ideas!"
  },
  {
    "path": "cmd/subfinder/main.go",
    "content": "package main\n\nimport (\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/runner\"\n\t// Attempts to increase the OS file descriptors - Fail silently\n\t_ \"github.com/projectdiscovery/fdmax/autofdmax\"\n\t\"github.com/projectdiscovery/gologger\"\n)\n\nfunc main() {\n\t// Parse the command line flags and read config files\n\toptions := runner.ParseOptions()\n\n\tnewRunner, err := runner.NewRunner(options)\n\tif err != nil {\n\t\tgologger.Fatal().Msgf(\"Could not create runner: %s\\n\", err)\n\t}\n\n\terr = newRunner.RunEnumeration()\n\tif err != nil {\n\t\tgologger.Fatal().Msgf(\"Could not run enumeration: %s\\n\", err)\n\t}\n}\n"
  },
  {
    "path": "examples/main.go",
    "content": "package main\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"io\"\n\t\"log\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/runner\"\n)\n\nfunc main() {\n\tsubfinderOpts := &runner.Options{\n\t\tThreads:            10, // Thread controls the number of threads to use for active enumerations\n\t\tTimeout:            30, // Timeout is the seconds to wait for sources to respond\n\t\tMaxEnumerationTime: 10, // MaxEnumerationTime is the maximum amount of time in mins to wait for enumeration\n\t\t// ResultCallback: func(s *resolve.HostEntry) {\n\t\t// callback function executed after each unique subdomain is found\n\t\t// },\n\t\t// ProviderConfig: \"your_provider_config.yaml\",\n\t\t// and other config related options\n\t}\n\n\t// disable timestamps in logs / configure logger\n\tlog.SetFlags(0)\n\n\tsubfinder, err := runner.NewRunner(subfinderOpts)\n\tif err != nil {\n\t\tlog.Fatalf(\"failed to create subfinder runner: %v\", err)\n\t}\n\n\toutput := &bytes.Buffer{}\n\tvar sourceMap map[string]map[string]struct{}\n\t// To run subdomain enumeration on a single domain\n\tif sourceMap, err = subfinder.EnumerateSingleDomainWithCtx(context.Background(), \"hackerone.com\", []io.Writer{output}); err != nil {\n\t\tlog.Fatalf(\"failed to enumerate single domain: %v\", err)\n\t}\n\n\t// To run subdomain enumeration on a list of domains from file/reader\n\t// file, err := os.Open(\"domains.txt\")\n\t// if err != nil {\n\t// \tlog.Fatalf(\"failed to open domains file: %v\", err)\n\t// }\n\t// defer file.Close()\n\t// if err = subfinder.EnumerateMultipleDomainsWithCtx(context.Background(), file, []io.Writer{output}); err != nil {\n\t// \tlog.Fatalf(\"failed to enumerate subdomains from file: %v\", err)\n\t// }\n\n\t// print the output\n\tlog.Println(output.String())\n\n\t// Or use sourceMap to access the results in your application\n\tfor subdomain, sources := range sourceMap {\n\t\tsourcesList := make([]string, 0, len(sources))\n\t\tfor source := range sources {\n\t\t\tsourcesList = append(sourcesList, source)\n\t\t}\n\t\tlog.Printf(\"%s %s (%d)\\n\", subdomain, sourcesList, len(sources))\n\t}\n}\n"
  },
  {
    "path": "go.mod",
    "content": "module github.com/projectdiscovery/subfinder/v2\n\ngo 1.24.0\n\ntoolchain go1.24.1\n\nrequire (\n\tgithub.com/corpix/uarand v0.2.0\n\tgithub.com/hako/durafmt v0.0.0-20210316092057-3a2c319c1acd\n\tgithub.com/json-iterator/go v1.1.12\n\tgithub.com/lib/pq v1.10.9\n\tgithub.com/projectdiscovery/chaos-client v0.5.2\n\tgithub.com/projectdiscovery/dnsx v1.2.3\n\tgithub.com/projectdiscovery/fdmax v0.0.4\n\tgithub.com/projectdiscovery/gologger v1.1.62\n\tgithub.com/projectdiscovery/ratelimit v0.0.82\n\tgithub.com/projectdiscovery/retryablehttp-go v1.1.0\n\tgithub.com/projectdiscovery/utils v0.7.3\n\tgithub.com/rs/xid v1.5.0\n\tgithub.com/stretchr/testify v1.11.1\n\tgithub.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80\n\tgolang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8\n\tgopkg.in/yaml.v3 v3.0.1\n)\n\nrequire (\n\taead.dev/minisign v0.2.0 // indirect\n\tgithub.com/Masterminds/semver/v3 v3.2.1 // indirect\n\tgithub.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057 // indirect\n\tgithub.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 // indirect\n\tgithub.com/STARRY-S/zip v0.2.1 // indirect\n\tgithub.com/VividCortex/ewma v1.2.0 // indirect\n\tgithub.com/akrylysov/pogreb v0.10.1 // indirect\n\tgithub.com/alecthomas/chroma/v2 v2.14.0 // indirect\n\tgithub.com/andybalholm/brotli v1.1.1 // indirect\n\tgithub.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect\n\tgithub.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect\n\tgithub.com/aymerick/douceur v0.2.0 // indirect\n\tgithub.com/bodgit/plumbing v1.3.0 // indirect\n\tgithub.com/bodgit/sevenzip v1.6.0 // indirect\n\tgithub.com/bodgit/windows v1.0.1 // indirect\n\tgithub.com/charmbracelet/glamour v0.8.0 // indirect\n\tgithub.com/charmbracelet/lipgloss v0.13.0 // indirect\n\tgithub.com/charmbracelet/x/ansi v0.3.2 // indirect\n\tgithub.com/cheggaaa/pb/v3 v3.1.4 // indirect\n\tgithub.com/cloudflare/circl v1.6.1 // indirect\n\tgithub.com/dimchansky/utfbom v1.1.1 // indirect\n\tgithub.com/dlclark/regexp2 v1.11.5 // indirect\n\tgithub.com/docker/go-units v0.5.0 // indirect\n\tgithub.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect\n\tgithub.com/fatih/color v1.15.0 // indirect\n\tgithub.com/gaissmai/bart v0.26.0 // indirect\n\tgithub.com/go-ole/go-ole v1.2.6 // indirect\n\tgithub.com/golang/snappy v0.0.4 // indirect\n\tgithub.com/google/go-github/v30 v30.1.0 // indirect\n\tgithub.com/google/go-querystring v1.1.0 // indirect\n\tgithub.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect\n\tgithub.com/google/uuid v1.3.1 // indirect\n\tgithub.com/gorilla/css v1.0.1 // indirect\n\tgithub.com/hashicorp/errwrap v1.1.0 // indirect\n\tgithub.com/hashicorp/go-multierror v1.1.1 // indirect\n\tgithub.com/hashicorp/golang-lru/v2 v2.0.7 // indirect\n\tgithub.com/klauspost/compress v1.17.11 // indirect\n\tgithub.com/klauspost/pgzip v1.2.6 // indirect\n\tgithub.com/kr/pretty v0.3.1 // indirect\n\tgithub.com/lucasb-eyer/go-colorful v1.2.0 // indirect\n\tgithub.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect\n\tgithub.com/mattn/go-colorable v0.1.13 // indirect\n\tgithub.com/mattn/go-isatty v0.0.20 // indirect\n\tgithub.com/mattn/go-runewidth v0.0.16 // indirect\n\tgithub.com/mholt/archives v0.1.0 // indirect\n\tgithub.com/microcosm-cc/bluemonday v1.0.27 // indirect\n\tgithub.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 // indirect\n\tgithub.com/muesli/reflow v0.3.0 // indirect\n\tgithub.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a // indirect\n\tgithub.com/nwaples/rardecode/v2 v2.2.0 // indirect\n\tgithub.com/pierrec/lz4/v4 v4.1.21 // indirect\n\tgithub.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect\n\tgithub.com/projectdiscovery/blackrock v0.0.1 // indirect\n\tgithub.com/projectdiscovery/cdncheck v1.2.13 // indirect\n\tgithub.com/projectdiscovery/fastdialer v0.4.19 // indirect\n\tgithub.com/projectdiscovery/hmap v0.0.98 // indirect\n\tgithub.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983 // indirect\n\tgithub.com/projectdiscovery/networkpolicy v0.1.31 // indirect\n\tgithub.com/refraction-networking/utls v1.7.1 // indirect\n\tgithub.com/rivo/uniseg v0.4.7 // indirect\n\tgithub.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d // indirect\n\tgithub.com/shirou/gopsutil/v3 v3.23.7 // indirect\n\tgithub.com/shoenig/go-m1cpu v0.1.6 // indirect\n\tgithub.com/sorairolake/lzip-go v0.3.5 // indirect\n\tgithub.com/syndtr/goleveldb v1.0.0 // indirect\n\tgithub.com/therootcompany/xz v1.0.1 // indirect\n\tgithub.com/tidwall/btree v1.6.0 // indirect\n\tgithub.com/tidwall/buntdb v1.3.0 // indirect\n\tgithub.com/tidwall/gjson v1.18.0 // indirect\n\tgithub.com/tidwall/grect v0.1.4 // indirect\n\tgithub.com/tidwall/match v1.1.1 // indirect\n\tgithub.com/tidwall/pretty v1.2.1 // indirect\n\tgithub.com/tidwall/rtred v0.1.2 // indirect\n\tgithub.com/tidwall/tinyqueue v0.1.1 // indirect\n\tgithub.com/tklauser/go-sysconf v0.3.12 // indirect\n\tgithub.com/tklauser/numcpus v0.6.1 // indirect\n\tgithub.com/ulikunitz/xz v0.5.15 // indirect\n\tgithub.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39 // indirect\n\tgithub.com/yuin/goldmark v1.7.4 // indirect\n\tgithub.com/yuin/goldmark-emoji v1.0.3 // indirect\n\tgithub.com/yusufpapurcu/wmi v1.2.4 // indirect\n\tgithub.com/zcalusic/sysinfo v1.0.2 // indirect\n\tgithub.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248 // indirect\n\tgithub.com/zmap/zcrypto v0.0.0-20230422215203-9a665e1e9968 // indirect\n\tgo.etcd.io/bbolt v1.3.7 // indirect\n\tgo.uber.org/multierr v1.11.0 // indirect\n\tgo4.org v0.0.0-20230225012048-214862532bf5 // indirect\n\tgolang.org/x/crypto v0.45.0 // indirect\n\tgolang.org/x/mod v0.29.0 // indirect\n\tgolang.org/x/oauth2 v0.27.0 // indirect\n\tgolang.org/x/sync v0.18.0 // indirect\n\tgolang.org/x/term v0.37.0 // indirect\n\tgolang.org/x/text v0.31.0 // indirect\n\tgolang.org/x/time v0.5.0 // indirect\n\tgolang.org/x/tools v0.38.0 // indirect\n\tgopkg.in/djherbis/times.v1 v1.3.0 // indirect\n)\n\nrequire (\n\tgithub.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect\n\tgithub.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect\n\tgithub.com/logrusorgru/aurora v2.0.3+incompatible // indirect\n\tgithub.com/miekg/dns v1.1.62 // indirect\n\tgithub.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect\n\tgithub.com/modern-go/reflect2 v1.0.2 // indirect\n\tgithub.com/pkg/errors v0.9.1 // indirect\n\tgithub.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect\n\tgithub.com/projectdiscovery/goflags v0.1.74\n\tgithub.com/projectdiscovery/retryabledns v1.0.111 // indirect\n\tgolang.org/x/net v0.47.0 // indirect\n\tgolang.org/x/sys v0.38.0 // indirect\n)\n"
  },
  {
    "path": "go.sum",
    "content": "aead.dev/minisign v0.2.0 h1:kAWrq/hBRu4AARY6AlciO83xhNnW9UaC8YipS2uhLPk=\naead.dev/minisign v0.2.0/go.mod h1:zdq6LdSd9TbuSxchxwhpA9zEb9YXcVGoE8JakuiGaIQ=\ncloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ncloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ncloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=\ncloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=\ncloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=\ncloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=\ncloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=\ncloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=\ncloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=\ncloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=\ncloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=\ncloud.google.com/go/compute/metadata v0.2.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k=\ncloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=\ncloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=\ncloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=\ncloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=\ncloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=\ndmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=\ngithub.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=\ngithub.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=\ngithub.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0=\ngithub.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ=\ngithub.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057 h1:KFac3SiGbId8ub47e7kd2PLZeACxc1LkiiNoDOFRClE=\ngithub.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057/go.mod h1:iLB2pivrPICvLOuROKmlqURtFIEsoJZaMidQfCG1+D4=\ngithub.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 h1:ZbFL+BDfBqegi+/Ssh7im5+aQfBRx6it+kHnC7jaDU8=\ngithub.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809/go.mod h1:upgc3Zs45jBDnBT4tVRgRcgm26ABpaP7MoTSdgysca4=\ngithub.com/STARRY-S/zip v0.2.1 h1:pWBd4tuSGm3wtpoqRZZ2EAwOmcHK6XFf7bU9qcJXyFg=\ngithub.com/STARRY-S/zip v0.2.1/go.mod h1:xNvshLODWtC4EJ702g7cTYn13G53o1+X9BWnPFpcWV4=\ngithub.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=\ngithub.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=\ngithub.com/akrylysov/pogreb v0.10.1 h1:FqlR8VR7uCbJdfUob916tPM+idpKgeESDXOA1K0DK4w=\ngithub.com/akrylysov/pogreb v0.10.1/go.mod h1:pNs6QmpQ1UlTJKDezuRWmaqkgUE2TuU0YTWyqJZ7+lI=\ngithub.com/alecthomas/assert/v2 v2.7.0 h1:QtqSACNS3tF7oasA8CU6A6sXZSBDqnm7RfpLl9bZqbE=\ngithub.com/alecthomas/assert/v2 v2.7.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=\ngithub.com/alecthomas/chroma/v2 v2.14.0 h1:R3+wzpnUArGcQz7fCETQBzO5n9IMNi13iIs46aU4V9E=\ngithub.com/alecthomas/chroma/v2 v2.14.0/go.mod h1:QolEbTfmUHIMVpBqxeDnNBj2uoeI4EbYP4i6n68SG4I=\ngithub.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=\ngithub.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=\ngithub.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=\ngithub.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=\ngithub.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=\ngithub.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=\ngithub.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=\ngithub.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=\ngithub.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWpi6yML8=\ngithub.com/aymanbagabas/go-udiff v0.2.0/go.mod h1:RE4Ex0qsGkTAJoQdQQCA0uG+nAzJO/pI/QwceO5fgrA=\ngithub.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=\ngithub.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=\ngithub.com/bits-and-blooms/bitset v1.13.0 h1:bAQ9OPNFYbGHV6Nez0tmNI0RiEu7/hxlYJRUA0wFAVE=\ngithub.com/bits-and-blooms/bitset v1.13.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=\ngithub.com/bits-and-blooms/bloom/v3 v3.5.0 h1:AKDvi1V3xJCmSR6QhcBfHbCN4Vf8FfxeWkMNQfmAGhY=\ngithub.com/bits-and-blooms/bloom/v3 v3.5.0/go.mod h1:Y8vrn7nk1tPIlmLtW2ZPV+W7StdVMor6bC1xgpjMZFs=\ngithub.com/bodgit/plumbing v1.3.0 h1:pf9Itz1JOQgn7vEOE7v7nlEfBykYqvUYioC61TwWCFU=\ngithub.com/bodgit/plumbing v1.3.0/go.mod h1:JOTb4XiRu5xfnmdnDJo6GmSbSbtSyufrsyZFByMtKEs=\ngithub.com/bodgit/sevenzip v1.6.0 h1:a4R0Wu6/P1o1pP/3VV++aEOcyeBxeO/xE2Y9NSTrr6A=\ngithub.com/bodgit/sevenzip v1.6.0/go.mod h1:zOBh9nJUof7tcrlqJFv1koWRrhz3LbDbUNngkuZxLMc=\ngithub.com/bodgit/windows v1.0.1 h1:tF7K6KOluPYygXa3Z2594zxlkbKPAOvqr97etrGNIz4=\ngithub.com/bodgit/windows v1.0.1/go.mod h1:a6JLwrB4KrTR5hBpp8FI9/9W9jJfeQ2h4XDXU74ZCdM=\ngithub.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=\ngithub.com/charmbracelet/glamour v0.8.0 h1:tPrjL3aRcQbn++7t18wOpgLyl8wrOHUEDS7IZ68QtZs=\ngithub.com/charmbracelet/glamour v0.8.0/go.mod h1:ViRgmKkf3u5S7uakt2czJ272WSg2ZenlYEZXT2x7Bjw=\ngithub.com/charmbracelet/lipgloss v0.13.0 h1:4X3PPeoWEDCMvzDvGmTajSyYPcZM4+y8sCA/SsA3cjw=\ngithub.com/charmbracelet/lipgloss v0.13.0/go.mod h1:nw4zy0SBX/F/eAO1cWdcvy6qnkDUxr8Lw7dvFrAIbbY=\ngithub.com/charmbracelet/x/ansi v0.3.2 h1:wsEwgAN+C9U06l9dCVMX0/L3x7ptvY1qmjMwyfE6USY=\ngithub.com/charmbracelet/x/ansi v0.3.2/go.mod h1:dk73KoMTT5AX5BsX0KrqhsTqAnhZZoCBjs7dGWp4Ktw=\ngithub.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a h1:G99klV19u0QnhiizODirwVksQB91TJKV/UaTnACcG30=\ngithub.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a/go.mod h1:wDlXFlCrmJ8J+swcL/MnGUuYnqgQdW9rhSD61oNMb6U=\ngithub.com/cheggaaa/pb/v3 v3.1.4 h1:DN8j4TVVdKu3WxVwcRKu0sG00IIU6FewoABZzXbRQeo=\ngithub.com/cheggaaa/pb/v3 v3.1.4/go.mod h1:6wVjILNBaXMs8c21qRiaUM8BR82erfgau1DQ4iUXmSA=\ngithub.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=\ngithub.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=\ngithub.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=\ngithub.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=\ngithub.com/cloudflare/circl v1.6.1 h1:zqIqSPIndyBh1bjLVVDHMPpVKqp8Su/V+6MeDzzQBQ0=\ngithub.com/cloudflare/circl v1.6.1/go.mod h1:uddAzsPgqdMAYatqJ0lsjX1oECcQLIlRpzZh3pJrofs=\ngithub.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 h1:ox2F0PSMlrAAiAdknSRMDrAr8mfxPCfSZolH+/qQnyQ=\ngithub.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08/go.mod h1:pCxVEbcm3AMg7ejXyorUXi6HQCzOIBf7zEDVPtw0/U4=\ngithub.com/corpix/uarand v0.2.0 h1:U98xXwud/AVuCpkpgfPF7J5TQgr7R5tqT8VZP5KWbzE=\ngithub.com/corpix/uarand v0.2.0/go.mod h1:/3Z1QIqWkDIhf6XWn/08/uMHoQ8JUoTIKc2iPchBOmM=\ngithub.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=\ngithub.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=\ngithub.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=\ngithub.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=\ngithub.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=\ngithub.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=\ngithub.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=\ngithub.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=\ngithub.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 h1:2tV76y6Q9BB+NEBasnqvs7e49aEBFI8ejC89PSnWH+4=\ngithub.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707/go.mod h1:qssHWj60/X5sZFNxpG4HBPDHVqxNm4DfnCKgrbZOT+s=\ngithub.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=\ngithub.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=\ngithub.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=\ngithub.com/fatih/color v1.15.0 h1:kOqh6YHBtK8aywxGerMG2Eq3H6Qgoqeo13Bk2Mv/nBs=\ngithub.com/fatih/color v1.15.0/go.mod h1:0h5ZqXfHYED7Bhv2ZJamyIOUej9KtShiJESRwBDUSsw=\ngithub.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=\ngithub.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=\ngithub.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=\ngithub.com/gaissmai/bart v0.26.0 h1:xOZ57E9hJLBiQaSyeZa9wgWhGuzfGACgqp4BE77OkO0=\ngithub.com/gaissmai/bart v0.26.0/go.mod h1:GREWQfTLRWz/c5FTOsIw+KkscuFkIV5t8Rp7Nd1Td5c=\ngithub.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=\ngithub.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=\ngithub.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=\ngithub.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=\ngithub.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=\ngithub.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=\ngithub.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=\ngithub.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=\ngithub.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=\ngithub.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=\ngithub.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=\ngithub.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=\ngithub.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=\ngithub.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=\ngithub.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=\ngithub.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=\ngithub.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=\ngithub.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=\ngithub.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=\ngithub.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=\ngithub.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=\ngithub.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=\ngithub.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=\ngithub.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=\ngithub.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=\ngithub.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=\ngithub.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=\ngithub.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=\ngithub.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=\ngithub.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=\ngithub.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=\ngithub.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=\ngithub.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=\ngithub.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=\ngithub.com/google/go-github/v30 v30.1.0 h1:VLDx+UolQICEOKu2m4uAoMti1SxuEBAl7RSEG16L+Oo=\ngithub.com/google/go-github/v30 v30.1.0/go.mod h1:n8jBpHl45a/rlBUtRJMOG4GhNADUQFEufcolZ95JfU8=\ngithub.com/google/go-github/v50 v50.1.0/go.mod h1:Ev4Tre8QoKiolvbpOSG3FIi4Mlon3S2Nt9W5JYqKiwA=\ngithub.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=\ngithub.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8=\ngithub.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU=\ngithub.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=\ngithub.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=\ngithub.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=\ngithub.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=\ngithub.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=\ngithub.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=\ngithub.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=\ngithub.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=\ngithub.com/google/uuid v1.3.1 h1:KjJaJ9iWZ3jOFZIf1Lqf4laDRCasjl0BCmnEGxkdLb4=\ngithub.com/google/uuid v1.3.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=\ngithub.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=\ngithub.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=\ngithub.com/gorilla/css v1.0.1 h1:ntNaBIghp6JmvWnxbZKANoLyuXTPZ4cAMlo6RyhlbO8=\ngithub.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A3b0=\ngithub.com/hako/durafmt v0.0.0-20210316092057-3a2c319c1acd h1:FsX+T6wA8spPe4c1K9vi7T0LvNCO1TTqiL8u7Wok2hw=\ngithub.com/hako/durafmt v0.0.0-20210316092057-3a2c319c1acd/go.mod h1:VzxiSdG6j1pi7rwGm/xYI5RbtpBgM8sARDXlvEvxlu0=\ngithub.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=\ngithub.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=\ngithub.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=\ngithub.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=\ngithub.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=\ngithub.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=\ngithub.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=\ngithub.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=\ngithub.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=\ngithub.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=\ngithub.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=\ngithub.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=\ngithub.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=\ngithub.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=\ngithub.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=\ngithub.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=\ngithub.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=\ngithub.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=\ngithub.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=\ngithub.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=\ngithub.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=\ngithub.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=\ngithub.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=\ngithub.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=\ngithub.com/klauspost/pgzip v1.2.6 h1:8RXeL5crjEUFnR2/Sn6GJNWtSQ3Dk8pq4CL3jvdDyjU=\ngithub.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=\ngithub.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=\ngithub.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=\ngithub.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=\ngithub.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=\ngithub.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=\ngithub.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=\ngithub.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=\ngithub.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=\ngithub.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=\ngithub.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=\ngithub.com/logrusorgru/aurora v2.0.3+incompatible h1:tOpm7WcpBTn4fjmVfgpQq0EfczGlG91VSDkswnjF5A8=\ngithub.com/logrusorgru/aurora v2.0.3+incompatible/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4=\ngithub.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY=\ngithub.com/lucasb-eyer/go-colorful v1.2.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=\ngithub.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=\ngithub.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=\ngithub.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=\ngithub.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=\ngithub.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=\ngithub.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=\ngithub.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=\ngithub.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=\ngithub.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=\ngithub.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=\ngithub.com/mholt/archives v0.1.0 h1:FacgJyrjiuyomTuNA92X5GyRBRZjE43Y/lrzKIlF35Q=\ngithub.com/mholt/archives v0.1.0/go.mod h1:j/Ire/jm42GN7h90F5kzj6hf6ZFzEH66de+hmjEKu+I=\ngithub.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=\ngithub.com/microcosm-cc/bluemonday v1.0.27/go.mod h1:jFi9vgW+H7c3V0lb6nR74Ib/DIB5OBs92Dimizgw2cA=\ngithub.com/miekg/dns v1.1.62 h1:cN8OuEF1/x5Rq6Np+h1epln8OiyPWV+lROx9LxcGgIQ=\ngithub.com/miekg/dns v1.1.62/go.mod h1:mvDlcItzm+br7MToIKqkglaGhlFMHJ9DTNNWONWXbNQ=\ngithub.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 h1:yRZGarbxsRytL6EGgbqK2mCY+Lk5MWKQYKJT2gEglhc=\ngithub.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7/go.mod h1:bO02GTIPCMQFTEvE5h4DjYB58bCoZ35XLeBf0buTDdM=\ngithub.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=\ngithub.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=\ngithub.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=\ngithub.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=\ngithub.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=\ngithub.com/mreiferson/go-httpclient v0.0.0-20160630210159-31f0106b4474/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=\ngithub.com/mreiferson/go-httpclient v0.0.0-20201222173833-5e475fde3a4d/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=\ngithub.com/muesli/reflow v0.3.0 h1:IFsN6K9NfGtjeggFP+68I4chLZV2yIKsXJFNZ+eWh6s=\ngithub.com/muesli/reflow v0.3.0/go.mod h1:pbwTDkVPibjO2kyvBQRBxTWEEGDGq0FlB1BIKtnHY/8=\ngithub.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a h1:2MaM6YC3mGu54x+RKAA6JiFFHlHDY1UbkxqppT7wYOg=\ngithub.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a/go.mod h1:hxSnBBYLK21Vtq/PHd0S2FYCxBXzBua8ov5s1RobyRQ=\ngithub.com/nwaples/rardecode/v2 v2.2.0 h1:4ufPGHiNe1rYJxYfehALLjup4Ls3ck42CWwjKiOqu0A=\ngithub.com/nwaples/rardecode/v2 v2.2.0/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=\ngithub.com/nxadm/tail v1.4.11 h1:8feyoE3OzPrcshW5/MJ4sGESc5cqmGkGCWlco4l0bqY=\ngithub.com/nxadm/tail v1.4.11/go.mod h1:OTaG3NK980DZzxbRq6lEuzgU+mug70nY11sMd4JXXHc=\ngithub.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=\ngithub.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=\ngithub.com/onsi/ginkgo v1.16.4 h1:29JGrr5oVBm5ulCWet69zQkzWipVXIol6ygQUe/EzNc=\ngithub.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=\ngithub.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=\ngithub.com/onsi/gomega v1.27.6 h1:ENqfyGeS5AX/rlXDd/ETokDz93u0YufY1Pgxuy/PvWE=\ngithub.com/onsi/gomega v1.27.6/go.mod h1:PIQNjfQwkP3aQAH7lf7j87O/5FiNr+ZR8+ipb+qQlhg=\ngithub.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=\ngithub.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=\ngithub.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=\ngithub.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=\ngithub.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=\ngithub.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=\ngithub.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=\ngithub.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=\ngithub.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=\ngithub.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=\ngithub.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=\ngithub.com/projectdiscovery/blackrock v0.0.1 h1:lHQqhaaEFjgf5WkuItbpeCZv2DUIE45k0VbGJyft6LQ=\ngithub.com/projectdiscovery/blackrock v0.0.1/go.mod h1:ANUtjDfaVrqB453bzToU+YB4cUbvBRpLvEwoWIwlTss=\ngithub.com/projectdiscovery/cdncheck v1.2.13 h1:6zs4Mn8JV3yKyMoAr857Hf2NLvyOMpOfqCCT2V2OI1Q=\ngithub.com/projectdiscovery/cdncheck v1.2.13/go.mod h1:/OhuZ9T25yXSqU6+oWvmVQ3QFvtew/Tp03u0jM+NJBE=\ngithub.com/projectdiscovery/chaos-client v0.5.2 h1:dN+7GXEypsJAbCD//dBcUxzAEAEH1fjc/7Rf4F/RiNU=\ngithub.com/projectdiscovery/chaos-client v0.5.2/go.mod h1:KnoJ/NJPhll42uaqlDga6oafFfNw5l2XI2ajRijtDuU=\ngithub.com/projectdiscovery/dnsx v1.2.3 h1:S87U9kYuuqqvMFyen8mZQy1FMuR5EGCsXHqfHPQAeuc=\ngithub.com/projectdiscovery/dnsx v1.2.3/go.mod h1:NjAEyJt6+meNqZqnYHL4ZPxXfysuva+et56Eq/e1cVE=\ngithub.com/projectdiscovery/fastdialer v0.4.19 h1:MLHwEGM0x0pyltJaNvAVvwc27bnXdZ5mYr50S/2kMEE=\ngithub.com/projectdiscovery/fastdialer v0.4.19/go.mod h1:HGdVsez+JgJ9/ljXjHRplOqkB7og+nqi0nrNWVNi03o=\ngithub.com/projectdiscovery/fdmax v0.0.4 h1:K9tIl5MUZrEMzjvwn/G4drsHms2aufTn1xUdeVcmhmc=\ngithub.com/projectdiscovery/fdmax v0.0.4/go.mod h1:oZLqbhMuJ5FmcoaalOm31B1P4Vka/CqP50nWjgtSz+I=\ngithub.com/projectdiscovery/goflags v0.1.74 h1:n85uTRj5qMosm0PFBfsvOL24I7TdWRcWq/1GynhXS7c=\ngithub.com/projectdiscovery/goflags v0.1.74/go.mod h1:UMc9/7dFz2oln+10tv6cy+7WZKTHf9UGhaNkF95emh4=\ngithub.com/projectdiscovery/gologger v1.1.62 h1:wzKqvL6HQRzf0/PpBEhInZqqL1q4mKe2gFGJeDG3FqE=\ngithub.com/projectdiscovery/gologger v1.1.62/go.mod h1:YWvMSxlHybU3SkFCcWn+driSJ8yY+3CR3g/textnp+Y=\ngithub.com/projectdiscovery/hmap v0.0.98 h1:XxYIi7yJCNiDAKCJXvuY9IBM5O6OgDgx4XHgKxkR4eg=\ngithub.com/projectdiscovery/hmap v0.0.98/go.mod h1:bgN5fuZPJMj2YnAGEEnCypoifCnALJixHEVQszktQIU=\ngithub.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983 h1:ZScLodGSezQVwsQDtBSMFp72WDq0nNN+KE/5DHKY5QE=\ngithub.com/projectdiscovery/machineid v0.0.0-20240226150047-2e2c51e35983/go.mod h1:3G3BRKui7nMuDFAZKR/M2hiOLtaOmyukT20g88qRQjI=\ngithub.com/projectdiscovery/networkpolicy v0.1.31 h1:mE6iJeYOSql8gps/91vwiztE/kEHe5Im8oUO5Mkj9Zg=\ngithub.com/projectdiscovery/networkpolicy v0.1.31/go.mod h1:5x4rGh4XhnoYl9wACnZyrjDGKIB/bQqxw2KrIM5V+XU=\ngithub.com/projectdiscovery/ratelimit v0.0.82 h1:rtO5SQf5uQFu5zTahTaTcO06OxmG8EIF1qhdFPIyTak=\ngithub.com/projectdiscovery/ratelimit v0.0.82/go.mod h1:z076BrLkBb5yS7uhHNoCTf8X/BvFSGRxwQ8EzEL9afM=\ngithub.com/projectdiscovery/retryabledns v1.0.111 h1:iyMdCDgNmaSRJYcGqB+SLlvlw9WijlbJ6Q9OEpRAWsQ=\ngithub.com/projectdiscovery/retryabledns v1.0.111/go.mod h1:6TOPJ3QAE4reBu6bvsGsTcyEb+OypcKYFQH7yVsjyIM=\ngithub.com/projectdiscovery/retryablehttp-go v1.1.0 h1:uYp3EnuhhamTwvG41X6q6TAc/SHEO9pw9CBWbRASIQk=\ngithub.com/projectdiscovery/retryablehttp-go v1.1.0/go.mod h1:9DU57ezv5cfZSWw/m5XFDTMjy1yKeMyn1kj35lPlcfM=\ngithub.com/projectdiscovery/utils v0.7.3 h1:kX+77AA58yK6EZgkTRJEnK9V/7AZYzlXdcu/o/kJhFs=\ngithub.com/projectdiscovery/utils v0.7.3/go.mod h1:uDdQ3/VWomai98l+a3Ye/srDXdJ4xUIar/mSXlQ9gBM=\ngithub.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=\ngithub.com/refraction-networking/utls v1.7.1 h1:dxg+jla3uocgN8HtX+ccwDr68uCBBO3qLrkZUbqkcw0=\ngithub.com/refraction-networking/utls v1.7.1/go.mod h1:TUhh27RHMGtQvjQq+RyO11P6ZNQNBb3N0v7wsEjKAIQ=\ngithub.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=\ngithub.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=\ngithub.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=\ngithub.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=\ngithub.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=\ngithub.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=\ngithub.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=\ngithub.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=\ngithub.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=\ngithub.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=\ngithub.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=\ngithub.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d h1:hrujxIzL1woJ7AwssoOcM/tq5JjjG2yYOc8odClEiXA=\ngithub.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d/go.mod h1:uugorj2VCxiV1x+LzaIdVa9b4S4qGAcH6cbhh4qVxOU=\ngithub.com/shirou/gopsutil/v3 v3.23.7 h1:C+fHO8hfIppoJ1WdsVm1RoI0RwXoNdfTK7yWXV0wVj4=\ngithub.com/shirou/gopsutil/v3 v3.23.7/go.mod h1:c4gnmoRC0hQuaLqvxnx1//VXQ0Ms/X9UnJF8pddY5z4=\ngithub.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=\ngithub.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=\ngithub.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=\ngithub.com/shoenig/test v0.6.4/go.mod h1:byHiCGXqrVaflBLAMq/srcZIHynQPQgeyvkvXnjqq0k=\ngithub.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=\ngithub.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=\ngithub.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=\ngithub.com/sorairolake/lzip-go v0.3.5 h1:ms5Xri9o1JBIWvOFAorYtUNik6HI3HgBTkISiqu0Cwg=\ngithub.com/sorairolake/lzip-go v0.3.5/go.mod h1:N0KYq5iWrMXI0ZEXKXaS9hCyOjZUQdBDEIbXfoUwbdk=\ngithub.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=\ngithub.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=\ngithub.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=\ngithub.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=\ngithub.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=\ngithub.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=\ngithub.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=\ngithub.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=\ngithub.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=\ngithub.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=\ngithub.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=\ngithub.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=\ngithub.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=\ngithub.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=\ngithub.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=\ngithub.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=\ngithub.com/therootcompany/xz v1.0.1 h1:CmOtsn1CbtmyYiusbfmhmkpAAETj0wBIH6kCYaX+xzw=\ngithub.com/therootcompany/xz v1.0.1/go.mod h1:3K3UH1yCKgBneZYhuQUvJ9HPD19UEXEI0BWbMn8qNMY=\ngithub.com/tidwall/assert v0.1.0 h1:aWcKyRBUAdLoVebxo95N7+YZVTFF/ASTr7BN4sLP6XI=\ngithub.com/tidwall/assert v0.1.0/go.mod h1:QLYtGyeqse53vuELQheYl9dngGCJQ+mTtlxcktb+Kj8=\ngithub.com/tidwall/btree v1.6.0 h1:LDZfKfQIBHGHWSwckhXI0RPSXzlo+KYdjK7FWSqOzzg=\ngithub.com/tidwall/btree v1.6.0/go.mod h1:twD9XRA5jj9VUQGELzDO4HPQTNJsoWWfYEL+EUQ2cKY=\ngithub.com/tidwall/buntdb v1.3.0 h1:gdhWO+/YwoB2qZMeAU9JcWWsHSYU3OvcieYgFRS0zwA=\ngithub.com/tidwall/buntdb v1.3.0/go.mod h1:lZZrZUWzlyDJKlLQ6DKAy53LnG7m5kHyrEHvvcDmBpU=\ngithub.com/tidwall/gjson v1.12.1/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=\ngithub.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=\ngithub.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=\ngithub.com/tidwall/grect v0.1.4 h1:dA3oIgNgWdSspFzn1kS4S/RDpZFLrIxAZOdJKjYapOg=\ngithub.com/tidwall/grect v0.1.4/go.mod h1:9FBsaYRaR0Tcy4UwefBX/UDcDcDy9V5jUcxHzv2jd5Q=\ngithub.com/tidwall/lotsa v1.0.2 h1:dNVBH5MErdaQ/xd9s769R31/n2dXavsQ0Yf4TMEHHw8=\ngithub.com/tidwall/lotsa v1.0.2/go.mod h1:X6NiU+4yHA3fE3Puvpnn1XMDrFZrE9JO2/w+UMuqgR8=\ngithub.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=\ngithub.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=\ngithub.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=\ngithub.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=\ngithub.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=\ngithub.com/tidwall/rtred v0.1.2 h1:exmoQtOLvDoO8ud++6LwVsAMTu0KPzLTUrMln8u1yu8=\ngithub.com/tidwall/rtred v0.1.2/go.mod h1:hd69WNXQ5RP9vHd7dqekAz+RIdtfBogmglkZSRxCHFQ=\ngithub.com/tidwall/tinyqueue v0.1.1 h1:SpNEvEggbpyN5DIReaJ2/1ndroY8iyEGxPYxoSaymYE=\ngithub.com/tidwall/tinyqueue v0.1.1/go.mod h1:O/QNHwrnjqr6IHItYrzoHAKYhBkLI67Q096fQP5zMYw=\ngithub.com/tklauser/go-sysconf v0.3.11/go.mod h1:GqXfhXY3kiPa0nAXPDIQIWzJbMCB7AmcWpGR8lSZfqI=\ngithub.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU=\ngithub.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI=\ngithub.com/tklauser/numcpus v0.6.0/go.mod h1:FEZLMke0lhOUG6w2JadTzp0a+Nl8PF/GFkQ5UVIcaL4=\ngithub.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=\ngithub.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY=\ngithub.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80 h1:nrZ3ySNYwJbSpD6ce9duiP+QkD3JuLCcWkdaehUS/3Y=\ngithub.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80/go.mod h1:iFyPdL66DjUD96XmzVL3ZntbzcflLnznH0fr99w5VqE=\ngithub.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=\ngithub.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY=\ngithub.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=\ngithub.com/weppos/publicsuffix-go v0.13.0/go.mod h1:z3LCPQ38eedDQSwmsSRW4Y7t2L8Ln16JPQ02lHAdn5k=\ngithub.com/weppos/publicsuffix-go v0.30.1-0.20230422193905-8fecedd899db/go.mod h1:aiQaH1XpzIfgrJq3S1iw7w+3EDbRP7mF5fmwUhWyRUs=\ngithub.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39 h1:Bz/zVM/LoGZ9IztGBHrq2zlFQQbEG8dBYnxb4hamIHM=\ngithub.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39/go.mod h1:2oFzEwGYI7lhiqG0YkkcKa6VcpjVinQbWxaPzytDmLA=\ngithub.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=\ngithub.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=\ngithub.com/yl2chen/cidranger v1.0.2 h1:lbOWZVCG1tCRX4u24kuM1Tb4nHqWkDxwLdoS+SevawU=\ngithub.com/yl2chen/cidranger v1.0.2/go.mod h1:9U1yz7WPYDwf0vpNWFaeRh0bjwz5RVgRy/9UEQfHl0g=\ngithub.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=\ngithub.com/yuin/goldmark v1.7.1/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=\ngithub.com/yuin/goldmark v1.7.4 h1:BDXOHExt+A7gwPCJgPIIq7ENvceR7we7rOS9TNoLZeg=\ngithub.com/yuin/goldmark v1.7.4/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=\ngithub.com/yuin/goldmark-emoji v1.0.3 h1:aLRkLHOuBR2czCY4R8olwMjID+tENfhyFDMCRhbIQY4=\ngithub.com/yuin/goldmark-emoji v1.0.3/go.mod h1:tTkZEbwu5wkPmgTcitqddVxY9osFZiavD+r4AzQrh1U=\ngithub.com/yusufpapurcu/wmi v1.2.3/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=\ngithub.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=\ngithub.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=\ngithub.com/zcalusic/sysinfo v1.0.2 h1:nwTTo2a+WQ0NXwo0BGRojOJvJ/5XKvQih+2RrtWqfxc=\ngithub.com/zcalusic/sysinfo v1.0.2/go.mod h1:kluzTYflRWo6/tXVMJPdEjShsbPpsFRyy+p1mBQPC30=\ngithub.com/zmap/rc2 v0.0.0-20131011165748-24b9757f5521/go.mod h1:3YZ9o3WnatTIZhuOtot4IcUfzoKVjUHqu6WALIyI0nE=\ngithub.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248 h1:Nzukz5fNOBIHOsnP+6I79kPx3QhLv8nBy2mfFhBRq30=\ngithub.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248/go.mod h1:3YZ9o3WnatTIZhuOtot4IcUfzoKVjUHqu6WALIyI0nE=\ngithub.com/zmap/zcertificate v0.0.0-20180516150559-0e3d58b1bac4/go.mod h1:5iU54tB79AMBcySS0R2XIyZBAVmeHranShAFELYx7is=\ngithub.com/zmap/zcertificate v0.0.1/go.mod h1:q0dlN54Jm4NVSSuzisusQY0hqDWvu92C+TWveAxiVWk=\ngithub.com/zmap/zcrypto v0.0.0-20201128221613-3719af1573cf/go.mod h1:aPM7r+JOkfL+9qSB4KbYjtoEzJqUK50EXkkJabeNJDQ=\ngithub.com/zmap/zcrypto v0.0.0-20201211161100-e54a5822fb7e/go.mod h1:aPM7r+JOkfL+9qSB4KbYjtoEzJqUK50EXkkJabeNJDQ=\ngithub.com/zmap/zcrypto v0.0.0-20230422215203-9a665e1e9968 h1:YOQ1vXEwE4Rnj+uQ/3oCuJk5wgVsvUyW+glsndwYuyA=\ngithub.com/zmap/zcrypto v0.0.0-20230422215203-9a665e1e9968/go.mod h1:xIuOvYCZX21S5Z9bK1BMrertTGX/F8hgAPw7ERJRNS0=\ngithub.com/zmap/zlint/v3 v3.0.0/go.mod h1:paGwFySdHIBEMJ61YjoqT4h7Ge+fdYG4sUQhnTb1lJ8=\ngo.etcd.io/bbolt v1.3.7 h1:j+zJOnnEjF/kyHlDDgGnVL/AIqIJPq8UoB2GSNfkUfQ=\ngo.etcd.io/bbolt v1.3.7/go.mod h1:N9Mkw9X8x5fupy0IKsmuqVtoGDyxsaDlbk4Rd05IAQw=\ngo.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=\ngo.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=\ngo.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=\ngo.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=\ngo.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=\ngo.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=\ngo4.org v0.0.0-20230225012048-214862532bf5 h1:nifaUDeh+rPaBCMPMQHZmvJf+QdpLFnuQPwx+LxVmtc=\ngo4.org v0.0.0-20230225012048-214862532bf5/go.mod h1:F57wTi5Lrj6WLyswp5EYV1ncrEbFGHD4hhz6S1ZYeaU=\ngolang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=\ngolang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=\ngolang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=\ngolang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=\ngolang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=\ngolang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=\ngolang.org/x/crypto v0.0.0-20201124201722-c8d3bf9c5392/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=\ngolang.org/x/crypto v0.0.0-20201208171446-5f87f3452ae9/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=\ngolang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=\ngolang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=\ngolang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=\ngolang.org/x/crypto v0.0.0-20211209193657-4570a0811e8b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=\ngolang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=\ngolang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=\ngolang.org/x/crypto v0.45.0 h1:jMBrvKuj23MTlT0bQEOBcAE0mjg8mK9RXFhRH6nyF3Q=\ngolang.org/x/crypto v0.45.0/go.mod h1:XTGrrkGJve7CYK7J8PEww4aY7gM3qMCElcJQ8n8JdX4=\ngolang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=\ngolang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=\ngolang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=\ngolang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=\ngolang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=\ngolang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=\ngolang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=\ngolang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=\ngolang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8 h1:yqrTHse8TCMW1M1ZCP+VAR/l0kKxwaAIqN/il7x4voA=\ngolang.org/x/exp v0.0.0-20250106191152-7588d65b2ba8/go.mod h1:tujkw807nyEEAamNbDrEGzRav+ilXA7PCRAd6xsmwiU=\ngolang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=\ngolang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=\ngolang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=\ngolang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=\ngolang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=\ngolang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=\ngolang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=\ngolang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=\ngolang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=\ngolang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=\ngolang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=\ngolang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=\ngolang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=\ngolang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=\ngolang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=\ngolang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=\ngolang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=\ngolang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=\ngolang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=\ngolang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA=\ngolang.org/x/mod v0.29.0/go.mod h1:NyhrlYXJ2H4eJiRy/WDBO6HMqZQ6q9nk4JzS3NuCK+w=\ngolang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=\ngolang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=\ngolang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=\ngolang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=\ngolang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=\ngolang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=\ngolang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=\ngolang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=\ngolang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=\ngolang.org/x/net v0.47.0 h1:Mx+4dIFzqraBXUugkia1OOvlD6LemFo1ALMHjrXDOhY=\ngolang.org/x/net v0.47.0/go.mod h1:/jNxtkgq5yWUGYkaZGqo27cfGZ1c5Nen03aYrrKpVRU=\ngolang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=\ngolang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=\ngolang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=\ngolang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=\ngolang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=\ngolang.org/x/oauth2 v0.5.0/go.mod h1:9/XBHVqLaWO3/BRHs5jbpYCnOZVjj5V0ndyaAM7KB4I=\ngolang.org/x/oauth2 v0.27.0 h1:da9Vo7/tDv5RH/7nZDz1eMGS/q1Vv1N/7FCrBhI9I3M=\ngolang.org/x/oauth2 v0.27.0/go.mod h1:onh5ek6nERTohokkhCD/y2cV4Do3fxFHFuAejCkRWT8=\ngolang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=\ngolang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=\ngolang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20201126233918-771906719818/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20210228012217-479acdf4ea46/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=\ngolang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=\ngolang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=\ngolang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=\ngolang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=\ngolang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=\ngolang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=\ngolang.org/x/term v0.37.0 h1:8EGAD0qCmHYZg6J17DvsMy9/wJ7/D/4pV/wfnld5lTU=\ngolang.org/x/term v0.37.0/go.mod h1:5pB4lxRNYYVZuTLmy8oR2BH8dflOR+IbTYFD8fi3254=\ngolang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=\ngolang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=\ngolang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=\ngolang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=\ngolang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=\ngolang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=\ngolang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=\ngolang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=\ngolang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=\ngolang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=\ngolang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=\ngolang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=\ngolang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=\ngolang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=\ngolang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk=\ngolang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM=\ngolang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=\ngolang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=\ngolang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=\ngolang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=\ngolang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=\ngolang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=\ngolang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=\ngolang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=\ngolang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=\ngolang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=\ngolang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=\ngolang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=\ngolang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=\ngolang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=\ngolang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=\ngolang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=\ngolang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=\ngolang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=\ngolang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=\ngolang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ=\ngolang.org/x/tools v0.38.0/go.mod h1:yEsQ/d/YK8cjh0L6rZlY8tgtlKiBNTL14pGDJPJpYQs=\ngolang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngolang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngolang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngoogle.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=\ngoogle.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=\ngoogle.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=\ngoogle.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=\ngoogle.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=\ngoogle.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=\ngoogle.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=\ngoogle.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=\ngoogle.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=\ngoogle.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=\ngoogle.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=\ngoogle.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=\ngoogle.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=\ngoogle.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=\ngoogle.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=\ngoogle.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=\ngoogle.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=\ngoogle.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=\ngoogle.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=\ngoogle.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=\ngoogle.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=\ngoogle.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=\ngoogle.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=\ngoogle.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=\ngoogle.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=\ngoogle.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=\ngoogle.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=\ngoogle.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=\ngoogle.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=\ngoogle.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=\ngoogle.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=\ngoogle.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=\ngoogle.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=\ngoogle.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=\ngoogle.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=\ngoogle.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=\ngoogle.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=\ngoogle.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=\ngopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=\ngopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=\ngopkg.in/djherbis/times.v1 v1.3.0 h1:uxMS4iMtH6Pwsxog094W0FYldiNnfY/xba00vq6C2+o=\ngopkg.in/djherbis/times.v1 v1.3.0/go.mod h1:AQlg6unIsrsCEdQYhTzERy542dz6SFdQFZFv6mUY0P8=\ngopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=\ngopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=\ngopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=\ngopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=\ngopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=\ngopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=\ngopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\ngopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=\ngopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\nhonnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=\nhonnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=\nhonnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=\nhonnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=\nhonnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=\nrsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=\nrsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=\nrsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=\n"
  },
  {
    "path": "pkg/passive/doc.go",
    "content": "// Package passive provides capability for doing passive subdomain\n// enumeration on targets.\npackage passive\n"
  },
  {
    "path": "pkg/passive/passive.go",
    "content": "package passive\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"math\"\n\t\"sort\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/ratelimit\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype EnumerationOptions struct {\n\tcustomRateLimiter *subscraping.CustomRateLimit\n}\n\ntype EnumerateOption func(opts *EnumerationOptions)\n\nfunc WithCustomRateLimit(crl *subscraping.CustomRateLimit) EnumerateOption {\n\treturn func(opts *EnumerationOptions) {\n\t\topts.customRateLimiter = crl\n\t}\n}\n\n// EnumerateSubdomains wraps EnumerateSubdomainsWithCtx with an empty context\nfunc (a *Agent) EnumerateSubdomains(domain string, proxy string, rateLimit int, timeout int, maxEnumTime time.Duration, options ...EnumerateOption) chan subscraping.Result {\n\treturn a.EnumerateSubdomainsWithCtx(context.Background(), domain, proxy, rateLimit, timeout, maxEnumTime, options...)\n}\n\n// EnumerateSubdomainsWithCtx enumerates all the subdomains for a given domain\nfunc (a *Agent) EnumerateSubdomainsWithCtx(ctx context.Context, domain string, proxy string, rateLimit int, timeout int, maxEnumTime time.Duration, options ...EnumerateOption) chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\n\tgo func() {\n\t\tdefer close(results)\n\n\t\tvar enumerateOptions EnumerationOptions\n\t\tfor _, enumerateOption := range options {\n\t\t\tenumerateOption(&enumerateOptions)\n\t\t}\n\n\t\tmultiRateLimiter, err := a.buildMultiRateLimiter(ctx, rateLimit, enumerateOptions.customRateLimiter)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tType: subscraping.Error, Error: fmt.Errorf(\"could not init multi rate limiter for %s: %s\", domain, err),\n\t\t\t}\n\t\t\treturn\n\t\t}\n\t\tsession, err := subscraping.NewSession(domain, proxy, multiRateLimiter, timeout)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tType: subscraping.Error, Error: fmt.Errorf(\"could not init passive session for %s: %s\", domain, err),\n\t\t\t}\n\t\t\treturn\n\t\t}\n\t\tdefer session.Close()\n\n\t\tctx, cancel := context.WithTimeout(ctx, maxEnumTime)\n\n\t\twg := &sync.WaitGroup{}\n\t\t// Run each source in parallel on the target domain\n\t\tfor _, runner := range a.sources {\n\t\t\twg.Add(1)\n\t\t\tgo func(source subscraping.Source) {\n\t\t\t\tdefer wg.Done()\n\t\t\t\tctxWithValue := context.WithValue(ctx, subscraping.CtxSourceArg, source.Name())\n\t\t\t\tfor resp := range source.Run(ctxWithValue, domain, session) {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- resp:\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}(runner)\n\t\t}\n\t\twg.Wait()\n\t\tcancel()\n\t}()\n\treturn results\n}\n\nfunc (a *Agent) buildMultiRateLimiter(ctx context.Context, globalRateLimit int, rateLimit *subscraping.CustomRateLimit) (*ratelimit.MultiLimiter, error) {\n\tvar multiRateLimiter *ratelimit.MultiLimiter\n\tvar err error\n\tfor _, source := range a.sources {\n\t\tvar rl uint\n\t\tif sourceRateLimit, ok := rateLimit.Custom.Get(strings.ToLower(source.Name())); ok {\n\t\t\trl = sourceRateLimitOrDefault(uint(globalRateLimit), sourceRateLimit)\n\t\t}\n\n\t\tif rl > 0 {\n\t\t\tmultiRateLimiter, err = addRateLimiter(ctx, multiRateLimiter, source.Name(), rl, time.Second)\n\t\t} else {\n\t\t\tmultiRateLimiter, err = addRateLimiter(ctx, multiRateLimiter, source.Name(), math.MaxUint32, time.Millisecond)\n\t\t}\n\n\t\tif err != nil {\n\t\t\tbreak\n\t\t}\n\t}\n\treturn multiRateLimiter, err\n}\n\nfunc sourceRateLimitOrDefault(defaultRateLimit uint, sourceRateLimit uint) uint {\n\tif sourceRateLimit > 0 {\n\t\treturn sourceRateLimit\n\t}\n\treturn defaultRateLimit\n}\n\nfunc addRateLimiter(ctx context.Context, multiRateLimiter *ratelimit.MultiLimiter, key string, maxCount uint, duration time.Duration) (*ratelimit.MultiLimiter, error) {\n\tif multiRateLimiter == nil {\n\t\tmrl, err := ratelimit.NewMultiLimiter(ctx, &ratelimit.Options{\n\t\t\tKey:         key,\n\t\t\tIsUnlimited: maxCount == math.MaxUint32,\n\t\t\tMaxCount:    maxCount,\n\t\t\tDuration:    duration,\n\t\t})\n\t\treturn mrl, err\n\t}\n\terr := multiRateLimiter.Add(&ratelimit.Options{\n\t\tKey:         key,\n\t\tIsUnlimited: maxCount == math.MaxUint32,\n\t\tMaxCount:    maxCount,\n\t\tDuration:    duration,\n\t})\n\treturn multiRateLimiter, err\n}\n\nfunc (a *Agent) GetStatistics() map[string]subscraping.Statistics {\n\tstats := make(map[string]subscraping.Statistics)\n\tsort.Slice(a.sources, func(i, j int) bool {\n\t\treturn a.sources[i].Name() > a.sources[j].Name()\n\t})\n\n\tfor _, source := range a.sources {\n\t\tstats[source.Name()] = source.Statistics()\n\t}\n\treturn stats\n}\n"
  },
  {
    "path": "pkg/passive/sources.go",
    "content": "package passive\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"strings\"\n\n\t\"golang.org/x/exp/maps\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/alienvault\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/anubis\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/bevigil\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/bufferover\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/builtwith\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/c99\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/censys\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/certspotter\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/chaos\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/chinaz\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/commoncrawl\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/crtsh\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/digitalyama\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/digitorus\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/dnsdb\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/dnsdumpster\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/dnsrepo\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/domainsproject\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/driftnet\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/facebook\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/fofa\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/fullhunt\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/github\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/hackertarget\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/hudsonrock\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/intelx\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/leakix\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/merklemap\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/netlas\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/onyphe\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/profundis\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/pugrecon\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/quake\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/rapiddns\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/reconeer\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/redhuntlabs\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/robtex\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/rsecloud\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/securitytrails\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/shodan\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/sitedossier\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/thc\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/threatbook\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/threatcrowd\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/urlscan\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/virustotal\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/waybackarchive\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/whoisxmlapi\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/windvane\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping/sources/zoomeyeapi\"\n\tmapsutil \"github.com/projectdiscovery/utils/maps\"\n)\n\nvar AllSources = [...]subscraping.Source{\n\t&alienvault.Source{},\n\t&anubis.Source{},\n\t&bevigil.Source{},\n\t&bufferover.Source{},\n\t&builtwith.Source{},\n\t&c99.Source{},\n\t&censys.Source{},\n\t&certspotter.Source{},\n\t&chaos.Source{},\n\t&chinaz.Source{},\n\t&commoncrawl.Source{},\n\t&crtsh.Source{},\n\t&digitalyama.Source{},\n\t&digitorus.Source{},\n\t&dnsdb.Source{},\n\t&dnsdumpster.Source{},\n\t&dnsrepo.Source{},\n\t&domainsproject.Source{},\n\t&driftnet.Source{},\n\t&facebook.Source{},\n\t&fofa.Source{},\n\t&fullhunt.Source{},\n\t&github.Source{},\n\t&hackertarget.Source{},\n\t&hudsonrock.Source{},\n\t&intelx.Source{},\n\t&leakix.Source{},\n\t&merklemap.Source{},\n\t&netlas.Source{},\n\t&onyphe.Source{},\n\t&profundis.Source{},\n\t&pugrecon.Source{},\n\t&quake.Source{},\n\t&rapiddns.Source{},\n\t// &reconcloud.Source{}, // failing due to cloudflare bot protection\n\t&reconeer.Source{},\n\t&redhuntlabs.Source{},\n\t// &riddler.Source{}, // failing due to cloudfront protection\n\t&robtex.Source{},\n\t&rsecloud.Source{},\n\t&securitytrails.Source{},\n\t&shodan.Source{},\n\t&sitedossier.Source{},\n\t&thc.Source{},\n\t&threatbook.Source{},\n\t&threatcrowd.Source{},\n\t// &threatminer.Source{}, // failing  api\n\t&urlscan.Source{},\n\t&virustotal.Source{},\n\t&waybackarchive.Source{},\n\t&whoisxmlapi.Source{},\n\t&windvane.Source{},\n\t&zoomeyeapi.Source{},\n}\n\nvar sourceWarnings = mapsutil.NewSyncLockMap[string, string](\n\tmapsutil.WithMap(mapsutil.Map[string, string]{}))\n\nvar NameSourceMap = make(map[string]subscraping.Source, len(AllSources))\n\nfunc init() {\n\tfor _, currentSource := range AllSources {\n\t\tNameSourceMap[strings.ToLower(currentSource.Name())] = currentSource\n\t}\n}\n\n// Agent is a struct for running passive subdomain enumeration\n// against a given host. It wraps subscraping package and provides\n// a layer to build upon.\ntype Agent struct {\n\tsources []subscraping.Source\n}\n\n// New creates a new agent for passive subdomain discovery\nfunc New(sourceNames, excludedSourceNames []string, useAllSources, useSourcesSupportingRecurse bool) *Agent {\n\tsources := make(map[string]subscraping.Source, len(AllSources))\n\n\tif useAllSources {\n\t\tmaps.Copy(sources, NameSourceMap)\n\t} else {\n\t\tif len(sourceNames) > 0 {\n\t\t\tfor _, source := range sourceNames {\n\t\t\t\tif NameSourceMap[source] == nil {\n\t\t\t\t\tgologger.Warning().Msgf(\"There is no source with the name: %s\", source)\n\t\t\t\t} else {\n\t\t\t\t\tsources[source] = NameSourceMap[source]\n\t\t\t\t}\n\t\t\t}\n\t\t} else {\n\t\t\tfor _, currentSource := range AllSources {\n\t\t\t\tif currentSource.IsDefault() {\n\t\t\t\t\tsources[currentSource.Name()] = currentSource\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tif len(excludedSourceNames) > 0 {\n\t\tfor _, sourceName := range excludedSourceNames {\n\t\t\tdelete(sources, sourceName)\n\t\t}\n\t}\n\n\tif useSourcesSupportingRecurse {\n\t\tfor sourceName, source := range sources {\n\t\t\tif !source.HasRecursiveSupport() {\n\t\t\t\tdelete(sources, sourceName)\n\t\t\t}\n\t\t}\n\t}\n\n\tif len(sources) == 0 {\n\t\tgologger.Fatal().Msg(\"No sources selected for this search\")\n\t}\n\n\tgologger.Debug().Msgf(\"Selected source(s) for this search: %s\", strings.Join(maps.Keys(sources), \", \"))\n\n\tfor _, currentSource := range sources {\n\t\tif warning, ok := sourceWarnings.Get(strings.ToLower(currentSource.Name())); ok {\n\t\t\tgologger.Warning().Msg(warning)\n\t\t}\n\t}\n\n\tfor _, source := range sources {\n\t\tkeyReq := source.KeyRequirement()\n\t\tif keyReq == subscraping.RequiredKey || keyReq == subscraping.OptionalKey {\n\t\t\tif apiKey := os.Getenv(fmt.Sprintf(\"%s_API_KEY\", strings.ToUpper(source.Name()))); apiKey != \"\" {\n\t\t\t\tsource.AddApiKeys([]string{apiKey})\n\t\t\t}\n\t\t}\n\t}\n\n\t// Create the agent, insert the sources and remove the excluded sources\n\tagent := &Agent{sources: maps.Values(sources)}\n\n\treturn agent\n}\n"
  },
  {
    "path": "pkg/passive/sources_test.go",
    "content": "package passive\n\nimport (\n\t\"fmt\"\n\t\"strconv\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"golang.org/x/exp/maps\"\n)\n\nvar (\n\texpectedAllSources = []string{\n\t\t\"alienvault\",\n\t\t\"anubis\",\n\t\t\"bevigil\",\n\t\t\"bufferover\",\n\t\t\"c99\",\n\t\t\"censys\",\n\t\t\"certspotter\",\n\t\t\"chaos\",\n\t\t\"chinaz\",\n\t\t\"commoncrawl\",\n\t\t\"crtsh\",\n\t\t\"digitorus\",\n\t\t\"dnsdumpster\",\n\t\t\"dnsdb\",\n\t\t\"dnsrepo\",\n\t\t\"domainsproject\",\n\t\t\"driftnet\",\n\t\t\"fofa\",\n\t\t\"fullhunt\",\n\t\t\"github\",\n\t\t\"hackertarget\",\n\t\t\"intelx\",\n\t\t\"netlas\",\n\t\t\"onyphe\",\n\t\t\"quake\",\n\t\t\"pugrecon\",\n\t\t\"rapiddns\",\n\t\t\"redhuntlabs\",\n\t\t// \"riddler\", // failing due to cloudfront protection\n\t\t\"robtex\",\n\t\t\"rsecloud\",\n\t\t\"securitytrails\",\n\t\t\"profundis\",\n\t\t\"shodan\",\n\t\t\"sitedossier\",\n\t\t\"threatbook\",\n\t\t\"threatcrowd\",\n\t\t\"virustotal\",\n\t\t\"waybackarchive\",\n\t\t\"whoisxmlapi\",\n\t\t\"windvane\",\n\t\t\"zoomeyeapi\",\n\t\t\"leakix\",\n\t\t\"facebook\",\n\t\t// \"threatminer\",\n\t\t// \"reconcloud\",\n\t\t\"reconeer\",\n\t\t\"builtwith\",\n\t\t\"hudsonrock\",\n\t\t\"digitalyama\",\n\t\t\"merklemap\",\n\t\t\"thc\",\n\t\t\"urlscan\",\n\t}\n\n\texpectedDefaultSources = []string{\n\t\t\"alienvault\",\n\t\t\"anubis\",\n\t\t\"bevigil\",\n\t\t\"bufferover\",\n\t\t\"c99\",\n\t\t\"certspotter\",\n\t\t\"censys\",\n\t\t\"chaos\",\n\t\t\"chinaz\",\n\t\t\"crtsh\",\n\t\t\"digitorus\",\n\t\t\"dnsdumpster\",\n\t\t\"domainsproject\",\n\t\t\"dnsrepo\",\n\t\t\"driftnet\",\n\t\t\"fofa\",\n\t\t\"fullhunt\",\n\t\t\"hackertarget\",\n\t\t\"intelx\",\n\t\t\"onyphe\",\n\t\t\"quake\",\n\t\t\"redhuntlabs\",\n\t\t\"robtex\",\n\t\t// \"riddler\", // failing due to cloudfront protection\n\t\t\"rsecloud\",\n\t\t\"securitytrails\",\n\t\t\"profundis\",\n\t\t\"shodan\",\n\t\t\"windvane\",\n\t\t\"virustotal\",\n\t\t\"whoisxmlapi\",\n\t\t\"leakix\",\n\t\t\"facebook\",\n\t\t// \"threatminer\",\n\t\t// \"reconcloud\",\n\t\t\"reconeer\",\n\t\t\"builtwith\",\n\t\t\"digitalyama\",\n\t\t\"thc\",\n\t\t\"urlscan\",\n\t}\n\n\texpectedDefaultRecursiveSources = []string{\n\t\t\"alienvault\",\n\t\t\"bufferover\",\n\t\t\"certspotter\",\n\t\t\"crtsh\",\n\t\t\"dnsdb\",\n\t\t\"digitorus\",\n\t\t\"driftnet\",\n\t\t\"hackertarget\",\n\t\t\"securitytrails\",\n\t\t\"virustotal\",\n\t\t\"leakix\",\n\t\t\"facebook\",\n\t\t\"merklemap\",\n\t\t\"urlscan\",\n\t\t// \"reconcloud\",\n\t}\n)\n\nfunc TestSourceCategorization(t *testing.T) {\n\tdefaultSources := make([]string, 0, len(AllSources))\n\trecursiveSources := make([]string, 0, len(AllSources))\n\tfor _, source := range AllSources {\n\t\tsourceName := source.Name()\n\t\tif source.IsDefault() {\n\t\t\tdefaultSources = append(defaultSources, sourceName)\n\t\t}\n\n\t\tif source.HasRecursiveSupport() {\n\t\t\trecursiveSources = append(recursiveSources, sourceName)\n\t\t}\n\t}\n\n\tassert.ElementsMatch(t, expectedDefaultSources, defaultSources)\n\tassert.ElementsMatch(t, expectedDefaultRecursiveSources, recursiveSources)\n\tassert.ElementsMatch(t, expectedAllSources, maps.Keys(NameSourceMap))\n}\n\n// Review: not sure if this test is necessary/useful\n// implementation is straightforward where sources are stored in maps and filtered based on options\n// the test is just checking if the filtering works as expected using count of sources\nfunc TestSourceFiltering(t *testing.T) {\n\tsomeSources := []string{\n\t\t\"alienvault\",\n\t\t\"chaos\",\n\t\t\"crtsh\",\n\t\t\"virustotal\",\n\t}\n\n\tsomeExclusions := []string{\n\t\t\"alienvault\",\n\t\t\"virustotal\",\n\t}\n\n\ttests := []struct {\n\t\tsources        []string\n\t\texclusions     []string\n\t\twithAllSources bool\n\t\twithRecursion  bool\n\t\texpectedLength int\n\t}{\n\t\t{someSources, someExclusions, false, false, len(someSources) - len(someExclusions)},\n\t\t{someSources, someExclusions, false, true, 1},\n\t\t{someSources, someExclusions, true, false, len(AllSources) - len(someExclusions)},\n\n\t\t{someSources, []string{}, false, false, len(someSources)},\n\t\t{someSources, []string{}, true, false, len(AllSources)},\n\n\t\t{[]string{}, []string{}, false, false, len(expectedDefaultSources)},\n\t\t{[]string{}, []string{}, true, false, len(AllSources)},\n\t\t{[]string{}, []string{}, true, true, len(expectedDefaultRecursiveSources)},\n\t}\n\tfor index, test := range tests {\n\t\tt.Run(strconv.Itoa(index+1), func(t *testing.T) {\n\t\t\tagent := New(test.sources, test.exclusions, test.withAllSources, test.withRecursion)\n\n\t\t\tfor _, v := range agent.sources {\n\t\t\t\tfmt.Println(v.Name())\n\t\t\t}\n\n\t\t\tassert.Equal(t, test.expectedLength, len(agent.sources))\n\t\t\tagent = nil\n\t\t})\n\t}\n}\n"
  },
  {
    "path": "pkg/passive/sources_w_auth_test.go",
    "content": "package passive\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"math\"\n\t\"os\"\n\t\"reflect\"\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/gologger/levels\"\n\t\"github.com/projectdiscovery/ratelimit\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nfunc TestSourcesWithKeys(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"skipping test in short mode.\")\n\t}\n\n\tdomain := \"hackerone.com\"\n\ttimeout := 60\n\n\tgologger.DefaultLogger.SetMaxLevel(levels.LevelDebug)\n\n\tctxParent := context.Background()\n\tvar multiRateLimiter *ratelimit.MultiLimiter\n\tfor _, source := range AllSources {\n\t\tif !source.NeedsKey() {\n\t\t\tcontinue\n\t\t}\n\t\tmultiRateLimiter, _ = addRateLimiter(ctxParent, multiRateLimiter, source.Name(), math.MaxInt32, time.Millisecond)\n\t}\n\n\tsession, err := subscraping.NewSession(domain, \"\", multiRateLimiter, timeout)\n\tassert.Nil(t, err)\n\n\tvar expected = subscraping.Result{Type: subscraping.Subdomain, Value: domain, Error: nil}\n\n\tfor _, source := range AllSources {\n\t\tif !source.NeedsKey() {\n\t\t\tcontinue\n\t\t}\n\n\t\tvar apiKey string\n\t\tif source.Name() == \"chaos\" {\n\t\t\tapiKey = os.Getenv(\"PDCP_API_KEY\")\n\t\t} else {\n\t\t\tapiKey = os.Getenv(fmt.Sprintf(\"%s_API_KEY\", strings.ToUpper(source.Name())))\n\t\t}\n\t\tif apiKey == \"\" {\n\t\t\tfmt.Printf(\"Skipping %s as no API key is provided\\n\", source.Name())\n\t\t\tcontinue\n\t\t}\n\t\tsource.AddApiKeys([]string{apiKey})\n\n\t\tt.Run(source.Name(), func(t *testing.T) {\n\t\t\tvar results []subscraping.Result\n\n\t\t\tctxWithValue := context.WithValue(ctxParent, subscraping.CtxSourceArg, source.Name())\n\t\t\tfor result := range source.Run(ctxWithValue, domain, session) {\n\t\t\t\tresults = append(results, result)\n\n\t\t\t\tassert.Equal(t, source.Name(), result.Source, \"wrong source name\")\n\n\t\t\t\tif result.Type != subscraping.Error {\n\t\t\t\t\tassert.True(t, strings.HasSuffix(strings.ToLower(result.Value), strings.ToLower(expected.Value)),\n\t\t\t\t\t\tfmt.Sprintf(\"result(%s) is not subdomain of %s\", strings.ToLower(result.Value), expected.Value))\n\t\t\t\t} else {\n\t\t\t\t\tassert.Equal(t, reflect.TypeOf(expected.Error), reflect.TypeOf(result.Error), fmt.Sprintf(\"%s: %s\", result.Source, result.Error))\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tassert.GreaterOrEqual(t, len(results), 1, fmt.Sprintf(\"No result found for %s\", source.Name()))\n\t\t})\n\t}\n}\n"
  },
  {
    "path": "pkg/passive/sources_wo_auth_test.go",
    "content": "package passive\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"math\"\n\t\"reflect\"\n\t\"strings\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"golang.org/x/exp/slices\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/gologger/levels\"\n\t\"github.com/projectdiscovery/ratelimit\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nfunc TestSourcesWithoutKeys(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"skipping test in short mode.\")\n\t}\n\n\tignoredSources := []string{\n\t\t\"commoncrawl\",    // commoncrawl is under resourced and will likely time-out so step over it for this test https://groups.google.com/u/2/g/common-crawl/c/3QmQjFA_3y4/m/vTbhGqIBBQAJ\n\t\t\"riddler\",        // failing due to cloudfront protection\n\t\t\"crtsh\",          // Fails in GH Action (possibly IP-based ban) causing a timeout.\n\t\t\"hackertarget\",   // Fails in GH Action (possibly IP-based ban) but works locally\n\t\t\"waybackarchive\", // Fails randomly\n\t\t\"alienvault\",     // 503 Service Temporarily Unavailable\n\t\t\"digitorus\",      // failing with \"Failed to retrieve certificate\"\n\t\t\"dnsdumpster\",    // failing with \"unexpected status code 403 received\"\n\t\t\"anubis\",         // failing with \"too many redirects\"\n\t\t\"threatcrowd\",    // failing with \"randomly failing with unmarshal error when hit multiple times\"\n\t\t\"leakix\",         // now requires API key (returns 401)\n\t\t\"reconeer\",       // now requires API key (returns 401)\n\t\t\"sitedossier\",    // flaky - returns no results in CI\n\t}\n\n\tdomain := \"hackerone.com\"\n\ttimeout := 60\n\n\tgologger.DefaultLogger.SetMaxLevel(levels.LevelDebug)\n\n\tctxParent := context.Background()\n\n\tvar multiRateLimiter *ratelimit.MultiLimiter\n\tfor _, source := range AllSources {\n\t\tif source.NeedsKey() || slices.Contains(ignoredSources, source.Name()) {\n\t\t\tcontinue\n\t\t}\n\t\tmultiRateLimiter, _ = addRateLimiter(ctxParent, multiRateLimiter, source.Name(), math.MaxInt32, time.Millisecond)\n\t}\n\n\tsession, err := subscraping.NewSession(domain, \"\", multiRateLimiter, timeout)\n\tassert.Nil(t, err)\n\n\tvar expected = subscraping.Result{Type: subscraping.Subdomain, Value: domain, Error: nil}\n\n\tfor _, source := range AllSources {\n\t\tif source.NeedsKey() || slices.Contains(ignoredSources, source.Name()) {\n\t\t\tcontinue\n\t\t}\n\n\t\tt.Run(source.Name(), func(t *testing.T) {\n\t\t\tvar results []subscraping.Result\n\n\t\t\tctxWithValue := context.WithValue(ctxParent, subscraping.CtxSourceArg, source.Name())\n\t\t\tfor result := range source.Run(ctxWithValue, domain, session) {\n\t\t\t\tresults = append(results, result)\n\n\t\t\t\tassert.Equal(t, source.Name(), result.Source, \"wrong source name\")\n\n\t\t\t\tif result.Type != subscraping.Error {\n\t\t\t\t\tassert.True(t, strings.HasSuffix(strings.ToLower(result.Value), strings.ToLower(expected.Value)),\n\t\t\t\t\t\tfmt.Sprintf(\"result(%s) is not subdomain of %s\", strings.ToLower(result.Value), expected.Value))\n\t\t\t\t} else {\n\t\t\t\t\tassert.Equal(t, reflect.TypeOf(expected.Error), reflect.TypeOf(result.Error), fmt.Sprintf(\"%s: %s\", result.Source, result.Error))\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tassert.GreaterOrEqual(t, len(results), 1, fmt.Sprintf(\"No result found for %s\", source.Name()))\n\t\t})\n\t}\n}\n"
  },
  {
    "path": "pkg/resolve/client.go",
    "content": "package resolve\n\nimport (\n\t\"github.com/projectdiscovery/dnsx/libs/dnsx\"\n)\n\n// DefaultResolvers contains the default list of resolvers known to be good\nvar DefaultResolvers = []string{\n\t\"1.1.1.1:53\",        // Cloudflare primary\n\t\"1.0.0.1:53\",        // Cloudflare secondary\n\t\"8.8.8.8:53\",        // Google primary\n\t\"8.8.4.4:53\",        // Google secondary\n\t\"9.9.9.9:53\",        // Quad9 Primary\n\t\"9.9.9.10:53\",       // Quad9 Secondary\n\t\"77.88.8.8:53\",      // Yandex Primary\n\t\"77.88.8.1:53\",      // Yandex Secondary\n\t\"208.67.222.222:53\", // OpenDNS Primary\n\t\"208.67.220.220:53\", // OpenDNS Secondary\n}\n\n// Resolver is a struct for resolving DNS names\ntype Resolver struct {\n\tDNSClient *dnsx.DNSX\n\tResolvers []string\n}\n\n// New creates a new resolver struct with the default resolvers\nfunc New() *Resolver {\n\treturn &Resolver{\n\t\tResolvers: []string{},\n\t}\n}\n"
  },
  {
    "path": "pkg/resolve/doc.go",
    "content": "// Package resolve is used to handle resolving records\n// It also handles wildcard subdomains and rotating resolvers.\npackage resolve\n"
  },
  {
    "path": "pkg/resolve/resolve.go",
    "content": "package resolve\n\nimport (\n\t\"fmt\"\n\t\"sync\"\n\n\t\"github.com/rs/xid\"\n)\n\nconst (\n\tmaxWildcardChecks = 3\n)\n\n// ResolutionPool is a pool of resolvers created for resolving subdomains\n// for a given host.\ntype ResolutionPool struct {\n\t*Resolver\n\tTasks          chan HostEntry\n\tResults        chan Result\n\twg             *sync.WaitGroup\n\tremoveWildcard bool\n\n\twildcardIPs map[string]struct{}\n}\n\n// HostEntry defines a host with the source\ntype HostEntry struct {\n\tDomain              string\n\tHost                string\n\tSource              string\n\tWildcardCertificate bool\n}\n\n// Result contains the result for a host resolution\ntype Result struct {\n\tType                ResultType\n\tHost                string\n\tIP                  string\n\tError               error\n\tSource              string\n\tWildcardCertificate bool\n}\n\n// ResultType is the type of result found\ntype ResultType int\n\n// Types of data result can return\nconst (\n\tSubdomain ResultType = iota\n\tError\n)\n\n// NewResolutionPool creates a pool of resolvers for resolving subdomains of a given domain\nfunc (r *Resolver) NewResolutionPool(workers int, removeWildcard bool) *ResolutionPool {\n\tresolutionPool := &ResolutionPool{\n\t\tResolver:       r,\n\t\tTasks:          make(chan HostEntry),\n\t\tResults:        make(chan Result),\n\t\twg:             &sync.WaitGroup{},\n\t\tremoveWildcard: removeWildcard,\n\t\twildcardIPs:    make(map[string]struct{}),\n\t}\n\n\tgo func() {\n\t\tfor range workers {\n\t\t\tresolutionPool.wg.Add(1)\n\t\t\tgo resolutionPool.resolveWorker()\n\t\t}\n\t\tresolutionPool.wg.Wait()\n\t\tclose(resolutionPool.Results)\n\t}()\n\n\treturn resolutionPool\n}\n\n// InitWildcards inits the wildcard ips array\nfunc (r *ResolutionPool) InitWildcards(domain string) error {\n\tfor range maxWildcardChecks {\n\t\tuid := xid.New().String()\n\n\t\thosts, _ := r.DNSClient.Lookup(uid + \".\" + domain)\n\t\tif len(hosts) == 0 {\n\t\t\treturn fmt.Errorf(\"%s is not a wildcard domain\", domain)\n\t\t}\n\n\t\t// Append all wildcard ips found for domains\n\t\tfor _, host := range hosts {\n\t\t\tr.wildcardIPs[host] = struct{}{}\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc (r *ResolutionPool) resolveWorker() {\n\tfor task := range r.Tasks {\n\t\tif !r.removeWildcard {\n\t\t\tr.Results <- Result{Type: Subdomain, Host: task.Host, IP: \"\", Source: task.Source, WildcardCertificate: task.WildcardCertificate}\n\t\t\tcontinue\n\t\t}\n\n\t\thosts, err := r.DNSClient.Lookup(task.Host)\n\t\tif err != nil {\n\t\t\tr.Results <- Result{Type: Error, Host: task.Host, Source: task.Source, Error: err, WildcardCertificate: task.WildcardCertificate}\n\t\t\tcontinue\n\t\t}\n\n\t\tif len(hosts) == 0 {\n\t\t\tcontinue\n\t\t}\n\n\t\tvar skip bool\n\t\tfor _, host := range hosts {\n\t\t\t// Ignore the host if it exists in wildcard ips map\n\t\t\tif _, ok := r.wildcardIPs[host]; ok {\n\t\t\t\tskip = true\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\n\t\tif !skip {\n\t\t\tr.Results <- Result{Type: Subdomain, Host: task.Host, IP: hosts[0], Source: task.Source, WildcardCertificate: task.WildcardCertificate}\n\t\t}\n\t}\n\tr.wg.Done()\n}\n"
  },
  {
    "path": "pkg/runner/banners.go",
    "content": "package runner\n\nimport (\n\t\"github.com/projectdiscovery/gologger\"\n\tupdateutils \"github.com/projectdiscovery/utils/update\"\n)\n\nconst banner = `\n               __    _____           __         \n   _______  __/ /_  / __(_)___  ____/ /__  _____\n  / ___/ / / / __ \\/ /_/ / __ \\/ __  / _ \\/ ___/\n (__  ) /_/ / /_/ / __/ / / / / /_/ /  __/ /    \n/____/\\__,_/_.___/_/ /_/_/ /_/\\__,_/\\___/_/\n`\n\n// Name\nconst ToolName = `subfinder`\n\n// Version is the current version of subfinder\nconst version = `v2.13.0`\n\n// showBanner is used to show the banner to the user\nfunc showBanner() {\n\tgologger.Print().Msgf(\"%s\\n\", banner)\n\tgologger.Print().Msgf(\"\\t\\tprojectdiscovery.io\\n\\n\")\n}\n\n// GetUpdateCallback returns a callback function that updates subfinder\nfunc GetUpdateCallback() func() {\n\treturn func() {\n\t\tshowBanner()\n\t\tupdateutils.GetUpdateToolCallback(\"subfinder\", version)()\n\t}\n}\n"
  },
  {
    "path": "pkg/runner/config.go",
    "content": "package runner\n\nimport (\n\t\"os\"\n\t\"strings\"\n\n\t\"gopkg.in/yaml.v3\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/passive\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\tfileutil \"github.com/projectdiscovery/utils/file\"\n)\n\n// createProviderConfigYAML marshals the input map to the given location on the disk\nfunc createProviderConfigYAML(configFilePath string) error {\n\tconfigFile, err := os.Create(configFilePath)\n\tif err != nil {\n\t\treturn err\n\t}\n\tdefer func() {\n\t\tif err := configFile.Close(); err != nil {\n\t\t\tgologger.Error().Msgf(\"Error closing config file: %s\", err)\n\t\t}\n\t}()\n\n\tsourcesRequiringApiKeysMap := make(map[string][]string)\n\tfor _, source := range passive.AllSources {\n\t\tkeyReq := source.KeyRequirement()\n\t\tif keyReq == subscraping.RequiredKey || keyReq == subscraping.OptionalKey {\n\t\t\tsourceName := strings.ToLower(source.Name())\n\t\t\tsourcesRequiringApiKeysMap[sourceName] = []string{}\n\t\t}\n\t}\n\n\treturn yaml.NewEncoder(configFile).Encode(sourcesRequiringApiKeysMap)\n}\n\n// UnmarshalFrom writes the marshaled yaml config to disk\nfunc UnmarshalFrom(file string) error {\n\treader, err := fileutil.SubstituteConfigFromEnvVars(file)\n\tif err != nil {\n\t\treturn err\n\t}\n\n\tsourceApiKeysMap := map[string][]string{}\n\terr = yaml.NewDecoder(reader).Decode(sourceApiKeysMap)\n\tfor _, source := range passive.AllSources {\n\t\tsourceName := strings.ToLower(source.Name())\n\t\tapiKeys := sourceApiKeysMap[sourceName]\n\t\tif len(apiKeys) > 0 {\n\t\t\tgologger.Debug().Msgf(\"API key(s) found for %s.\", sourceName)\n\t\t\tsource.AddApiKeys(apiKeys)\n\t\t}\n\t}\n\treturn err\n}\n"
  },
  {
    "path": "pkg/runner/doc.go",
    "content": "// Package runner implements the mechanism to drive the\n// subdomain enumeration process\npackage runner\n"
  },
  {
    "path": "pkg/runner/enumerate.go",
    "content": "package runner\n\nimport (\n\t\"context\"\n\t\"io\"\n\t\"strings\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/hako/durafmt\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/passive\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/resolve\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nconst maxNumCount = 2\n\nvar replacer = strings.NewReplacer(\n\t\"/\", \"\",\n\t\"•.\", \"\",\n\t\"•\", \"\",\n\t\"*.\", \"\",\n\t\"http://\", \"\",\n\t\"https://\", \"\",\n)\n\n// EnumerateSingleDomain wraps EnumerateSingleDomainWithCtx with an empty context\nfunc (r *Runner) EnumerateSingleDomain(domain string, writers []io.Writer) (map[string]map[string]struct{}, error) {\n\treturn r.EnumerateSingleDomainWithCtx(context.Background(), domain, writers)\n}\n\n// EnumerateSingleDomainWithCtx performs subdomain enumeration against a single domain\nfunc (r *Runner) EnumerateSingleDomainWithCtx(ctx context.Context, domain string, writers []io.Writer) (map[string]map[string]struct{}, error) {\n\tgologger.Info().Msgf(\"Enumerating subdomains for %s\\n\", domain)\n\n\t// Check if the user has asked to remove wildcards explicitly.\n\t// If yes, create the resolution pool and get the wildcards for the current domain\n\tvar resolutionPool *resolve.ResolutionPool\n\tif r.options.RemoveWildcard {\n\t\tresolutionPool = r.resolverClient.NewResolutionPool(r.options.Threads, r.options.RemoveWildcard)\n\t\terr := resolutionPool.InitWildcards(domain)\n\t\tif err != nil {\n\t\t\t// Log the error but don't quit.\n\t\t\tgologger.Warning().Msgf(\"Could not get wildcards for domain %s: %s\\n\", domain, err)\n\t\t}\n\t}\n\n\t// Run the passive subdomain enumeration\n\tnow := time.Now()\n\tpassiveResults := r.passiveAgent.EnumerateSubdomainsWithCtx(ctx, domain, r.options.Proxy, r.options.RateLimit, r.options.Timeout, time.Duration(r.options.MaxEnumerationTime)*time.Minute, passive.WithCustomRateLimit(r.rateLimit))\n\n\twg := &sync.WaitGroup{}\n\twg.Add(1)\n\t// Create a unique map for filtering duplicate subdomains out\n\tuniqueMap := make(map[string]resolve.HostEntry)\n\t// Create a map to track sources for each host\n\tsourceMap := make(map[string]map[string]struct{})\n\tskippedCounts := make(map[string]int)\n\t// Process the results in a separate goroutine\n\tgo func() {\n\t\tfor result := range passiveResults {\n\t\t\tswitch result.Type {\n\t\t\tcase subscraping.Error:\n\t\t\t\tgologger.Warning().Msgf(\"Encountered an error with source %s: %s\\n\", result.Source, result.Error)\n\t\t\tcase subscraping.Subdomain:\n\t\t\t\tsubdomain := replacer.Replace(result.Value)\n\t\t\t\t// check if this subdomain is actually a wildcard subdomain\n\t\t\t\t// that may have furthur subdomains associated with it\n\t\t\t\tisWildcard := strings.Contains(result.Value, \"*.\"+subdomain)\n\n\t\t\t\t// Validate the subdomain found and remove wildcards from\n\t\t\t\tif !strings.HasSuffix(subdomain, \".\"+domain) {\n\t\t\t\t\tskippedCounts[result.Source]++\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tif matchSubdomain := r.filterAndMatchSubdomain(subdomain); matchSubdomain {\n\t\t\t\t\tif _, ok := uniqueMap[subdomain]; !ok {\n\t\t\t\t\t\tsourceMap[subdomain] = make(map[string]struct{})\n\t\t\t\t\t}\n\n\t\t\t\t\t// Log the verbose message about the found subdomain per source\n\t\t\t\t\tif _, ok := sourceMap[subdomain][result.Source]; !ok {\n\t\t\t\t\t\tgologger.Verbose().Label(result.Source).Msg(subdomain)\n\t\t\t\t\t}\n\n\t\t\t\t\tsourceMap[subdomain][result.Source] = struct{}{}\n\n\t\t\t\t\t// Check if the subdomain is a duplicate. If not,\n\t\t\t\t\t// send the subdomain for resolution.\n\t\t\t\t\tif _, ok := uniqueMap[subdomain]; ok {\n\t\t\t\t\t\tskippedCounts[result.Source]++\n\t\t\t\t\t\t// even if it is duplicate if it was not marked as wildcard before but this source says it is wildcard\n\t\t\t\t\t\t// then we should mark it as wildcard\n\t\t\t\t\t\tif !uniqueMap[subdomain].WildcardCertificate && isWildcard {\n\t\t\t\t\t\t\tval := uniqueMap[subdomain]\n\t\t\t\t\t\t\tval.WildcardCertificate = true\n\t\t\t\t\t\t\tuniqueMap[subdomain] = val\n\t\t\t\t\t\t}\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\n\t\t\t\t\thostEntry := resolve.HostEntry{Domain: domain, Host: subdomain, Source: result.Source, WildcardCertificate: isWildcard}\n\t\t\t\t\tif r.options.ResultCallback != nil && !r.options.RemoveWildcard {\n\t\t\t\t\t\tr.options.ResultCallback(&hostEntry)\n\t\t\t\t\t}\n\n\t\t\t\t\tuniqueMap[subdomain] = hostEntry\n\t\t\t\t\t// If the user asked to remove wildcard then send on the resolve\n\t\t\t\t\t// queue. Otherwise, if mode is not verbose print the results on\n\t\t\t\t\t// the screen as they are discovered.\n\t\t\t\t\tif r.options.RemoveWildcard {\n\t\t\t\t\t\tresolutionPool.Tasks <- hostEntry\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\t// Close the task channel only if wildcards are asked to be removed\n\t\tif r.options.RemoveWildcard {\n\t\t\tclose(resolutionPool.Tasks)\n\t\t}\n\n\t\twg.Done()\n\t}()\n\n\t// If the user asked to remove wildcards, listen from the results\n\t// queue and write to the map. At the end, print the found results to the screen\n\tfoundResults := make(map[string]resolve.Result)\n\tif r.options.RemoveWildcard {\n\t\t// Process the results coming from the resolutions pool\n\t\tfor result := range resolutionPool.Results {\n\t\t\tswitch result.Type {\n\t\t\tcase resolve.Error:\n\t\t\t\tgologger.Warning().Msgf(\"Could not resolve host: %s\\n\", result.Error)\n\t\t\tcase resolve.Subdomain:\n\t\t\t\t// Add the found subdomain to a map.\n\t\t\t\tif _, ok := foundResults[result.Host]; !ok {\n\t\t\t\t\tfoundResults[result.Host] = result\n\t\t\t\t\tif r.options.ResultCallback != nil {\n\t\t\t\t\t\tr.options.ResultCallback(&resolve.HostEntry{Domain: domain, Host: result.Host, Source: result.Source, WildcardCertificate: result.WildcardCertificate})\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\t// Merge wildcard certificate information from uniqueMap into foundResults\n\t\t// This handles cases where a later source marked a subdomain as wildcard\n\t\t// after it was already sent to the resolution pool\n\t\tfor host, result := range foundResults {\n\t\t\tif entry, ok := uniqueMap[host]; ok && entry.WildcardCertificate && !result.WildcardCertificate {\n\t\t\t\tresult.WildcardCertificate = true\n\t\t\t\tfoundResults[host] = result\n\t\t\t}\n\t\t}\n\t}\n\twg.Wait()\n\toutputWriter := NewOutputWriter(r.options.JSON)\n\t// Now output all results in output writers\n\tvar err error\n\tfor _, writer := range writers {\n\t\tif r.options.HostIP {\n\t\t\terr = outputWriter.WriteHostIP(domain, foundResults, writer)\n\t\t} else {\n\t\t\tif r.options.RemoveWildcard {\n\t\t\t\terr = outputWriter.WriteHostNoWildcard(domain, foundResults, writer)\n\t\t\t} else {\n\t\t\t\tif r.options.CaptureSources {\n\t\t\t\t\terr = outputWriter.WriteSourceHost(domain, sourceMap, writer)\n\t\t\t\t} else {\n\t\t\t\t\terr = outputWriter.WriteHost(domain, uniqueMap, writer)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tif err != nil {\n\t\t\tgologger.Error().Msgf(\"Could not write results for %s: %s\\n\", domain, err)\n\t\t\treturn nil, err\n\t\t}\n\t}\n\n\t// Show found subdomain count in any case.\n\tduration := durafmt.Parse(time.Since(now)).LimitFirstN(maxNumCount).String()\n\tvar numberOfSubDomains int\n\tif r.options.RemoveWildcard {\n\t\tnumberOfSubDomains = len(foundResults)\n\t} else {\n\t\tnumberOfSubDomains = len(uniqueMap)\n\t}\n\n\tgologger.Info().Msgf(\"Found %d subdomains for %s in %s\\n\", numberOfSubDomains, domain, duration)\n\n\tif r.options.Statistics {\n\t\tgologger.Info().Msgf(\"Printing source statistics for %s\", domain)\n\t\tstatistics := r.passiveAgent.GetStatistics()\n\t\t// This is a hack to remove the skipped count from the statistics\n\t\t// as we don't want to show it in the statistics.\n\t\t// TODO: Design a better way to do this.\n\t\tfor source, count := range skippedCounts {\n\t\t\tif stat, ok := statistics[source]; ok {\n\t\t\t\tstat.Results -= count\n\t\t\t\tstatistics[source] = stat\n\t\t\t}\n\t\t}\n\t\tprintStatistics(statistics)\n\t}\n\n\treturn sourceMap, nil\n}\n\nfunc (r *Runner) filterAndMatchSubdomain(subdomain string) bool {\n\tif r.options.filterRegexes != nil {\n\t\tfor _, filter := range r.options.filterRegexes {\n\t\t\tif m := filter.MatchString(subdomain); m {\n\t\t\t\treturn false\n\t\t\t}\n\t\t}\n\t}\n\tif r.options.matchRegexes != nil {\n\t\tfor _, match := range r.options.matchRegexes {\n\t\t\tif m := match.MatchString(subdomain); m {\n\t\t\t\treturn true\n\t\t\t}\n\t\t}\n\t\treturn false\n\t}\n\treturn true\n}\n"
  },
  {
    "path": "pkg/runner/enumerate_test.go",
    "content": "package runner\n\nimport (\n\t\"os\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/require\"\n)\n\nfunc TestFilterAndMatchSubdomain(t *testing.T) {\n\toptions := &Options{}\n\toptions.Domain = []string{\"example.com\"}\n\toptions.Threads = 10\n\toptions.Timeout = 10\n\toptions.Output = os.Stdout\n\tt.Run(\"Literal Match\", func(t *testing.T) {\n\t\toptions.Match = []string{\"req.example.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tmatch := runner.filterAndMatchSubdomain(\"req.example.com\")\n\t\trequire.True(t, match, \"Expecting a boolean True value \")\n\t})\n\tt.Run(\"Multiple Wildcards Match\", func(t *testing.T) {\n\t\toptions.Match = []string{\"*.ns.*.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tsubdomain := []string{\"a.ns.example.com\", \"b.ns.hackerone.com\"}\n\t\tfor _, sub := range subdomain {\n\t\t\tmatch := runner.filterAndMatchSubdomain(sub)\n\t\t\trequire.True(t, match, \"Expecting a boolean True value \")\n\t\t}\n\t})\n\tt.Run(\"Sequential Match\", func(t *testing.T) {\n\t\toptions.Match = []string{\"*.ns.example.com\", \"*.hackerone.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tsubdomain := []string{\"a.ns.example.com\", \"b.hackerone.com\"}\n\t\tfor _, sub := range subdomain {\n\t\t\tmatch := runner.filterAndMatchSubdomain(sub)\n\t\t\trequire.True(t, match, \"Expecting a boolean True value \")\n\t\t}\n\t})\n\tt.Run(\"Literal Filter\", func(t *testing.T) {\n\t\toptions.Filter = []string{\"req.example.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tmatch := runner.filterAndMatchSubdomain(\"req.example.com\")\n\t\trequire.False(t, match, \"Expecting a boolean False value \")\n\t})\n\tt.Run(\"Multiple Wildcards Filter\", func(t *testing.T) {\n\t\toptions.Filter = []string{\"*.ns.*.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tsubdomain := []string{\"a.ns.example.com\", \"b.ns.hackerone.com\"}\n\t\tfor _, sub := range subdomain {\n\t\t\tmatch := runner.filterAndMatchSubdomain(sub)\n\t\t\trequire.False(t, match, \"Expecting a boolean False value \")\n\t\t}\n\t})\n\tt.Run(\"Sequential Filter\", func(t *testing.T) {\n\t\toptions.Filter = []string{\"*.ns.example.com\", \"*.hackerone.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tsubdomain := []string{\"a.ns.example.com\", \"b.hackerone.com\"}\n\t\tfor _, sub := range subdomain {\n\t\t\tmatch := runner.filterAndMatchSubdomain(sub)\n\t\t\trequire.False(t, match, \"Expecting a boolean False value \")\n\t\t}\n\t})\n\tt.Run(\"Filter and Match\", func(t *testing.T) {\n\t\toptions.Filter = []string{\"example.com\"}\n\t\toptions.Match = []string{\"hackerone.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tsubdomain := []string{\"example.com\", \"example.com\"}\n\t\tfor _, sub := range subdomain {\n\t\t\tmatch := runner.filterAndMatchSubdomain(sub)\n\t\t\trequire.False(t, match, \"Expecting a boolean False value \")\n\t\t}\n\t})\n\n\tt.Run(\"Filter and Match - Same Root Domain\", func(t *testing.T) {\n\t\toptions.Filter = []string{\"example.com\"}\n\t\toptions.Match = []string{\"www.example.com\"}\n\t\terr := options.validateOptions()\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while validation\\n\", err)\n\t\t}\n\t\trunner, err := NewRunner(options)\n\t\tif err != nil {\n\t\t\tt.Fatalf(\"Expected nil got %v while creating runner\\n\", err)\n\t\t}\n\t\tsubdomain := map[string]string{\"filter\": \"example.com\", \"match\": \"www.example.com\"}\n\t\tfor key, sub := range subdomain {\n\t\t\tresult := runner.filterAndMatchSubdomain(sub)\n\t\t\tif key == \"filter\" {\n\t\t\t\trequire.False(t, result, \"Expecting a boolean False value \")\n\t\t\t} else {\n\t\t\t\trequire.True(t, result, \"Expecting a boolean True value \")\n\t\t\t}\n\t\t}\n\t})\n}\n"
  },
  {
    "path": "pkg/runner/initialize.go",
    "content": "package runner\n\nimport (\n\t\"net\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/dnsx/libs/dnsx\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/passive\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/resolve\"\n)\n\n// initializePassiveEngine creates the passive engine and loads sources etc\nfunc (r *Runner) initializePassiveEngine() {\n\tr.passiveAgent = passive.New(r.options.Sources, r.options.ExcludeSources, r.options.All, r.options.OnlyRecursive)\n}\n\n// initializeResolver creates the resolver used to resolve the found subdomains\nfunc (r *Runner) initializeResolver() error {\n\tvar resolvers []string\n\n\t// If the file has been provided, read resolvers from the file\n\tif r.options.ResolverList != \"\" {\n\t\tvar err error\n\t\tresolvers, err = loadFromFile(r.options.ResolverList)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\tif len(r.options.Resolvers) > 0 {\n\t\tresolvers = append(resolvers, r.options.Resolvers...)\n\t} else {\n\t\tresolvers = append(resolvers, resolve.DefaultResolvers...)\n\t}\n\n\t// Add default 53 UDP port if missing\n\tfor i, resolver := range resolvers {\n\t\tif !strings.Contains(resolver, \":\") {\n\t\t\tresolvers[i] = net.JoinHostPort(resolver, \"53\")\n\t\t}\n\t}\n\n\tr.resolverClient = resolve.New()\n\tvar err error\n\tr.resolverClient.DNSClient, err = dnsx.New(dnsx.Options{BaseResolvers: resolvers, MaxRetries: 5})\n\tif err != nil {\n\t\treturn nil\n\t}\n\n\treturn nil\n}\n"
  },
  {
    "path": "pkg/runner/options.go",
    "content": "package runner\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"math\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/chaos-client/pkg/chaos\"\n\t\"github.com/projectdiscovery/goflags\"\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/passive\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/resolve\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\tenvutil \"github.com/projectdiscovery/utils/env\"\n\tfileutil \"github.com/projectdiscovery/utils/file\"\n\tfolderutil \"github.com/projectdiscovery/utils/folder\"\n\tlogutil \"github.com/projectdiscovery/utils/log\"\n\tupdateutils \"github.com/projectdiscovery/utils/update\"\n)\n\nvar (\n\tconfigDir                     = folderutil.AppConfigDirOrDefault(\".\", \"subfinder\")\n\tdefaultConfigLocation         = envutil.GetEnvOrDefault(\"SUBFINDER_CONFIG\", filepath.Join(configDir, \"config.yaml\"))\n\tdefaultProviderConfigLocation = envutil.GetEnvOrDefault(\"SUBFINDER_PROVIDER_CONFIG\", filepath.Join(configDir, \"provider-config.yaml\"))\n)\n\n// Options contains the configuration options for tuning\n// the subdomain enumeration process.\ntype Options struct {\n\tVerbose            bool                // Verbose flag indicates whether to show verbose output or not\n\tNoColor            bool                // NoColor disables the colored output\n\tJSON               bool                // JSON specifies whether to use json for output format or text file\n\tHostIP             bool                // HostIP specifies whether to write subdomains in host:ip format\n\tSilent             bool                // Silent suppresses any extra text and only writes subdomains to screen\n\tListSources        bool                // ListSources specifies whether to list all available sources\n\tRemoveWildcard     bool                // RemoveWildcard specifies whether to remove potential wildcard or dead subdomains from the results.\n\tCaptureSources     bool                // CaptureSources specifies whether to save all sources that returned a specific domains or just the first source\n\tStdin              bool                // Stdin specifies whether stdin input was given to the process\n\tVersion            bool                // Version specifies if we should just show version and exit\n\tOnlyRecursive      bool                // Recursive specifies whether to use only recursive subdomain enumeration sources\n\tAll                bool                // All specifies whether to use all (slow) sources.\n\tStatistics         bool                // Statistics specifies whether to report source statistics\n\tThreads            int                 // Threads controls the number of threads to use for active enumerations\n\tTimeout            int                 // Timeout is the seconds to wait for sources to respond\n\tMaxEnumerationTime int                 // MaxEnumerationTime is the maximum amount of time in minutes to wait for enumeration\n\tDomain             goflags.StringSlice // Domain is the domain to find subdomains for\n\tDomainsFile        string              // DomainsFile is the file containing list of domains to find subdomains for\n\tOutput             io.Writer\n\tOutputFile         string               // Output is the file to write found subdomains to.\n\tOutputDirectory    string               // OutputDirectory is the directory to write results to in case list of domains is given\n\tSources            goflags.StringSlice  `yaml:\"sources,omitempty\"`         // Sources contains a comma-separated list of sources to use for enumeration\n\tExcludeSources     goflags.StringSlice  `yaml:\"exclude-sources,omitempty\"` // ExcludeSources contains the comma-separated sources to not include in the enumeration process\n\tResolvers          goflags.StringSlice  `yaml:\"resolvers,omitempty\"`       // Resolvers is the comma-separated resolvers to use for enumeration\n\tResolverList       string               // ResolverList is a text file containing list of resolvers to use for enumeration\n\tConfig             string               // Config contains the location of the config file\n\tProviderConfig     string               // ProviderConfig contains the location of the provider config file\n\tProxy              string               // HTTP proxy\n\tRateLimit          int                  // Global maximum number of HTTP requests to send per second\n\tRateLimits         goflags.RateLimitMap // Maximum number of HTTP requests to send per second\n\tExcludeIps         bool\n\tMatch              goflags.StringSlice\n\tFilter             goflags.StringSlice\n\tmatchRegexes       []*regexp.Regexp\n\tfilterRegexes      []*regexp.Regexp\n\tResultCallback     OnResultCallback // OnResult callback\n\tDisableUpdateCheck bool             // DisableUpdateCheck disable update checking\n}\n\n// OnResultCallback (hostResult)\ntype OnResultCallback func(result *resolve.HostEntry)\n\n// ParseOptions parses the command line flags provided by a user\nfunc ParseOptions() *Options {\n\tlogutil.DisableDefaultLogger()\n\n\toptions := &Options{}\n\n\tvar err error\n\tflagSet := goflags.NewFlagSet()\n\tflagSet.SetDescription(`Subfinder is a subdomain discovery tool that discovers subdomains for websites by using passive online sources.`)\n\n\tflagSet.CreateGroup(\"input\", \"Input\",\n\t\tflagSet.StringSliceVarP(&options.Domain, \"domain\", \"d\", nil, \"domains to find subdomains for\", goflags.NormalizedStringSliceOptions),\n\t\tflagSet.StringVarP(&options.DomainsFile, \"list\", \"dL\", \"\", \"file containing list of domains for subdomain discovery\"),\n\t)\n\n\tflagSet.CreateGroup(\"source\", \"Source\",\n\t\tflagSet.StringSliceVarP(&options.Sources, \"sources\", \"s\", nil, \"specific sources to use for discovery (-s crtsh,github). Use -ls to display all available sources.\", goflags.NormalizedStringSliceOptions),\n\t\tflagSet.BoolVar(&options.OnlyRecursive, \"recursive\", false, \"use only sources that can handle subdomains recursively rather than both recursive and non-recursive sources\"),\n\t\tflagSet.BoolVar(&options.All, \"all\", false, \"use all sources for enumeration (slow)\"),\n\t\tflagSet.StringSliceVarP(&options.ExcludeSources, \"exclude-sources\", \"es\", nil, \"sources to exclude from enumeration (-es alienvault,zoomeyeapi)\", goflags.NormalizedStringSliceOptions),\n\t)\n\n\tflagSet.CreateGroup(\"filter\", \"Filter\",\n\t\tflagSet.StringSliceVarP(&options.Match, \"match\", \"m\", nil, \"subdomain or list of subdomain to match (file or comma separated)\", goflags.FileNormalizedStringSliceOptions),\n\t\tflagSet.StringSliceVarP(&options.Filter, \"filter\", \"f\", nil, \" subdomain or list of subdomain to filter (file or comma separated)\", goflags.FileNormalizedStringSliceOptions),\n\t)\n\n\tflagSet.CreateGroup(\"rate-limit\", \"Rate-limit\",\n\t\tflagSet.IntVarP(&options.RateLimit, \"rate-limit\", \"rl\", 0, \"maximum number of http requests to send per second (global)\"),\n\t\tflagSet.RateLimitMapVarP(&options.RateLimits, \"rate-limits\", \"rls\", defaultRateLimits, \"maximum number of http requests to send per second for providers in key=value format (-rls hackertarget=10/m)\", goflags.NormalizedStringSliceOptions),\n\t\tflagSet.IntVar(&options.Threads, \"t\", 10, \"number of concurrent goroutines for resolving (-active only)\"),\n\t)\n\n\tflagSet.CreateGroup(\"update\", \"Update\",\n\t\tflagSet.CallbackVarP(GetUpdateCallback(), \"update\", \"up\", \"update subfinder to latest version\"),\n\t\tflagSet.BoolVarP(&options.DisableUpdateCheck, \"disable-update-check\", \"duc\", false, \"disable automatic subfinder update check\"),\n\t)\n\n\tflagSet.CreateGroup(\"output\", \"Output\",\n\t\tflagSet.StringVarP(&options.OutputFile, \"output\", \"o\", \"\", \"file to write output to\"),\n\t\tflagSet.BoolVarP(&options.JSON, \"json\", \"oJ\", false, \"write output in JSONL(ines) format\"),\n\t\tflagSet.StringVarP(&options.OutputDirectory, \"output-dir\", \"oD\", \"\", \"directory to write output (-dL only)\"),\n\t\tflagSet.BoolVarP(&options.CaptureSources, \"collect-sources\", \"cs\", false, \"include all sources in the output (-json only)\"),\n\t\tflagSet.BoolVarP(&options.HostIP, \"ip\", \"oI\", false, \"include host IP in output (-active only)\"),\n\t)\n\n\tflagSet.CreateGroup(\"configuration\", \"Configuration\",\n\t\tflagSet.StringVar(&options.Config, \"config\", defaultConfigLocation, \"flag config file\"),\n\t\tflagSet.StringVarP(&options.ProviderConfig, \"provider-config\", \"pc\", defaultProviderConfigLocation, \"provider config file\"),\n\t\tflagSet.StringSliceVar(&options.Resolvers, \"r\", nil, \"comma separated list of resolvers to use\", goflags.NormalizedStringSliceOptions),\n\t\tflagSet.StringVarP(&options.ResolverList, \"rlist\", \"rL\", \"\", \"file containing list of resolvers to use\"),\n\t\tflagSet.BoolVarP(&options.RemoveWildcard, \"active\", \"nW\", false, \"display active subdomains only\"),\n\t\tflagSet.StringVar(&options.Proxy, \"proxy\", \"\", \"http proxy to use with subfinder\"),\n\t\tflagSet.BoolVarP(&options.ExcludeIps, \"exclude-ip\", \"ei\", false, \"exclude IPs from the list of domains\"),\n\t)\n\n\tflagSet.CreateGroup(\"debug\", \"Debug\",\n\t\tflagSet.BoolVar(&options.Silent, \"silent\", false, \"show only subdomains in output\"),\n\t\tflagSet.BoolVar(&options.Version, \"version\", false, \"show version of subfinder\"),\n\t\tflagSet.BoolVar(&options.Verbose, \"v\", false, \"show verbose output\"),\n\t\tflagSet.BoolVarP(&options.NoColor, \"no-color\", \"nc\", false, \"disable color in output\"),\n\t\tflagSet.BoolVarP(&options.ListSources, \"list-sources\", \"ls\", false, \"list all available sources\"),\n\t\tflagSet.BoolVar(&options.Statistics, \"stats\", false, \"report source statistics\"),\n\t)\n\n\tflagSet.CreateGroup(\"optimization\", \"Optimization\",\n\t\tflagSet.IntVar(&options.Timeout, \"timeout\", 30, \"seconds to wait before timing out\"),\n\t\tflagSet.IntVar(&options.MaxEnumerationTime, \"max-time\", 10, \"minutes to wait for enumeration results\"),\n\t)\n\n\tif err := flagSet.Parse(); err != nil {\n\t\tfmt.Println(err.Error())\n\t\tos.Exit(1)\n\t}\n\n\t// set chaos mode\n\tchaos.IsSDK = false\n\n\tif exists := fileutil.FileExists(defaultProviderConfigLocation); !exists {\n\t\tif err := createProviderConfigYAML(defaultProviderConfigLocation); err != nil {\n\t\t\tgologger.Error().Msgf(\"Could not create provider config file: %s\\n\", err)\n\t\t}\n\t}\n\n\tif options.Config != defaultConfigLocation {\n\t\t// An empty source file is not a fatal error\n\t\tif err := flagSet.MergeConfigFile(options.Config); err != nil && !errors.Is(err, io.EOF) {\n\t\t\tgologger.Fatal().Msgf(\"Could not read config: %s\\n\", err)\n\t\t}\n\t}\n\n\t// Default output is stdout\n\toptions.Output = os.Stdout\n\n\t// Check if stdin pipe was given\n\toptions.Stdin = fileutil.HasStdin()\n\n\tif options.Version {\n\t\tgologger.Info().Msgf(\"Current Version: %s\\n\", version)\n\t\tgologger.Info().Msgf(\"Subfinder Config Directory: %s\", configDir)\n\t\tos.Exit(0)\n\t}\n\n\toptions.preProcessDomains()\n\n\toptions.ConfigureOutput()\n\tshowBanner()\n\n\tif !options.DisableUpdateCheck {\n\t\tlatestVersion, err := updateutils.GetToolVersionCallback(\"subfinder\", version)()\n\t\tif err != nil {\n\t\t\tif options.Verbose {\n\t\t\t\tgologger.Error().Msgf(\"subfinder version check failed: %v\", err.Error())\n\t\t\t}\n\t\t} else {\n\t\t\tgologger.Info().Msgf(\"Current subfinder version %v %v\", version, updateutils.GetVersionDescription(version, latestVersion))\n\t\t}\n\t}\n\n\tif options.ListSources {\n\t\tlistSources(options)\n\t\tos.Exit(0)\n\t}\n\n\t// Validate the options passed by the user and if any\n\t// invalid options have been used, exit.\n\terr = options.validateOptions()\n\tif err != nil {\n\t\tgologger.Fatal().Msgf(\"Program exiting: %s\\n\", err)\n\t}\n\n\treturn options\n}\n\n// loadProvidersFrom runs the app with source config\nfunc (options *Options) loadProvidersFrom(location string) {\n\t// todo: move elsewhere\n\tif len(options.Resolvers) == 0 {\n\t\toptions.Resolvers = resolve.DefaultResolvers\n\t}\n\n\t// We skip bailing out if file doesn't exist because we'll create it\n\t// at the end of options parsing from default via goflags.\n\tif err := UnmarshalFrom(location); err != nil && (!strings.Contains(err.Error(), \"file doesn't exist\") || errors.Is(err, os.ErrNotExist)) {\n\t\tgologger.Error().Msgf(\"Could not read providers from %s: %s\\n\", location, err)\n\t}\n}\n\nfunc listSources(options *Options) {\n\tgologger.Info().Msgf(\"Current list of available sources. [%d]\\n\", len(passive.AllSources))\n\tgologger.Info().Msgf(\"Sources marked with an * require key(s) or token(s) to work.\\n\")\n\tgologger.Info().Msgf(\"Sources marked with a ~ optionally support key(s) for better results.\\n\")\n\tgologger.Info().Msgf(\"You can modify %s to configure your keys/tokens.\\n\\n\", options.ProviderConfig)\n\n\tfor _, source := range passive.AllSources {\n\t\tsourceName := source.Name()\n\t\tswitch source.KeyRequirement() {\n\t\tcase subscraping.RequiredKey:\n\t\t\tgologger.Silent().Msgf(\"%s *\\n\", sourceName)\n\t\tcase subscraping.OptionalKey:\n\t\t\tgologger.Silent().Msgf(\"%s ~\\n\", sourceName)\n\t\tdefault:\n\t\t\tgologger.Silent().Msgf(\"%s\\n\", sourceName)\n\t\t}\n\t}\n}\n\nfunc (options *Options) preProcessDomains() {\n\tfor i, domain := range options.Domain {\n\t\toptions.Domain[i] = preprocessDomain(domain)\n\t}\n}\n\nvar defaultRateLimits = []string{\n\t\"github=30/m\",\n\t\"fullhunt=60/m\",\n\t\"pugrecon=10/s\",\n\tfmt.Sprintf(\"robtex=%d/ms\", uint(math.MaxUint)),\n\t\"securitytrails=1/s\",\n\t\"shodan=1/s\",\n\t\"virustotal=4/m\",\n\t\"hackertarget=2/s\",\n\t// \"threatminer=10/m\",\n\t\"waybackarchive=15/m\",\n\t\"whoisxmlapi=50/s\",\n\t\"securitytrails=2/s\",\n\t\"sitedossier=8/m\",\n\t\"netlas=1/s\",\n\t// \"gitlab=2/s\",\n\t\"github=83/m\",\n\t\"hudsonrock=5/s\",\n\t\"urlscan=1/s\",\n}\n"
  },
  {
    "path": "pkg/runner/outputter.go",
    "content": "package runner\n\nimport (\n\t\"bufio\"\n\t\"errors\"\n\t\"io\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/resolve\"\n)\n\n// OutputWriter outputs content to writers.\ntype OutputWriter struct {\n\tJSON bool\n}\n\ntype jsonSourceResult struct {\n\tHost                string `json:\"host\"`\n\tInput               string `json:\"input\"`\n\tSource              string `json:\"source\"`\n\tWildcardCertificate bool   `json:\"wildcard_certificate,omitempty\"`\n}\n\ntype jsonSourceIPResult struct {\n\tHost                string `json:\"host\"`\n\tIP                  string `json:\"ip\"`\n\tInput               string `json:\"input\"`\n\tSource              string `json:\"source\"`\n\tWildcardCertificate bool   `json:\"wildcard_certificate,omitempty\"`\n}\n\ntype jsonSourcesResult struct {\n\tHost                string   `json:\"host\"`\n\tInput               string   `json:\"input\"`\n\tSources             []string `json:\"sources\"`\n\tWildcardCertificate bool     `json:\"wildcard_certificate,omitempty\"`\n}\n\n// NewOutputWriter creates a new OutputWriter\nfunc NewOutputWriter(json bool) *OutputWriter {\n\treturn &OutputWriter{JSON: json}\n}\n\nfunc (o *OutputWriter) createFile(filename string, appendToFile bool) (*os.File, error) {\n\tif filename == \"\" {\n\t\treturn nil, errors.New(\"empty filename\")\n\t}\n\n\tdir := filepath.Dir(filename)\n\n\tif dir != \"\" {\n\t\tif _, err := os.Stat(dir); os.IsNotExist(err) {\n\t\t\terr := os.MkdirAll(dir, os.ModePerm)\n\t\t\tif err != nil {\n\t\t\t\treturn nil, err\n\t\t\t}\n\t\t}\n\t}\n\n\tvar file *os.File\n\tvar err error\n\tif appendToFile {\n\t\tfile, err = os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)\n\t} else {\n\t\tfile, err = os.Create(filename)\n\t}\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn file, nil\n}\n\n// WriteHostIP writes the output list of subdomain to an io.Writer\nfunc (o *OutputWriter) WriteHostIP(input string, results map[string]resolve.Result, writer io.Writer) error {\n\tvar err error\n\tif o.JSON {\n\t\terr = writeJSONHostIP(input, results, writer)\n\t} else {\n\t\terr = writePlainHostIP(input, results, writer)\n\t}\n\treturn err\n}\n\nfunc writePlainHostIP(_ string, results map[string]resolve.Result, writer io.Writer) error {\n\tbufwriter := bufio.NewWriter(writer)\n\tsb := &strings.Builder{}\n\n\tfor _, result := range results {\n\t\tsb.WriteString(result.Host)\n\t\tsb.WriteString(\",\")\n\t\tsb.WriteString(result.IP)\n\t\tsb.WriteString(\",\")\n\t\tsb.WriteString(result.Source)\n\t\tsb.WriteString(\"\\n\")\n\n\t\t_, err := bufwriter.WriteString(sb.String())\n\t\tif err != nil {\n\t\t\tif flushErr := bufwriter.Flush(); flushErr != nil {\n\t\t\t\treturn errors.Join(err, flushErr)\n\t\t\t}\n\t\t\treturn err\n\t\t}\n\t\tsb.Reset()\n\t}\n\treturn bufwriter.Flush()\n}\n\nfunc writeJSONHostIP(input string, results map[string]resolve.Result, writer io.Writer) error {\n\tencoder := jsoniter.NewEncoder(writer)\n\n\tvar data jsonSourceIPResult\n\n\tfor _, result := range results {\n\t\tdata.Host = result.Host\n\t\tdata.IP = result.IP\n\t\tdata.Input = input\n\t\tdata.Source = result.Source\n\t\tdata.WildcardCertificate = result.WildcardCertificate\n\t\terr := encoder.Encode(&data)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil\n}\n\n// WriteHostNoWildcard writes the output list of subdomain with nW flag to an io.Writer\nfunc (o *OutputWriter) WriteHostNoWildcard(input string, results map[string]resolve.Result, writer io.Writer) error {\n\thosts := make(map[string]resolve.HostEntry)\n\tfor host, result := range results {\n\t\thosts[host] = resolve.HostEntry{Domain: host, Host: result.Host, Source: result.Source, WildcardCertificate: result.WildcardCertificate}\n\t}\n\n\treturn o.WriteHost(input, hosts, writer)\n}\n\n// WriteHost writes the output list of subdomain to an io.Writer\nfunc (o *OutputWriter) WriteHost(input string, results map[string]resolve.HostEntry, writer io.Writer) error {\n\tvar err error\n\tif o.JSON {\n\t\terr = writeJSONHost(input, results, writer)\n\t} else {\n\t\terr = writePlainHost(input, results, writer)\n\t}\n\treturn err\n}\n\nfunc writePlainHost(_ string, results map[string]resolve.HostEntry, writer io.Writer) error {\n\tbufwriter := bufio.NewWriter(writer)\n\tsb := &strings.Builder{}\n\n\tfor _, result := range results {\n\t\tsb.WriteString(result.Host)\n\t\tsb.WriteString(\"\\n\")\n\n\t\t_, err := bufwriter.WriteString(sb.String())\n\t\tif err != nil {\n\t\t\tif flushErr := bufwriter.Flush(); flushErr != nil {\n\t\t\t\treturn errors.Join(err, flushErr)\n\t\t\t}\n\t\t\treturn err\n\t\t}\n\t\tsb.Reset()\n\t}\n\treturn bufwriter.Flush()\n}\n\nfunc writeJSONHost(input string, results map[string]resolve.HostEntry, writer io.Writer) error {\n\tencoder := jsoniter.NewEncoder(writer)\n\n\tvar data jsonSourceResult\n\tfor _, result := range results {\n\t\tdata.Host = result.Host\n\t\tdata.Input = input\n\t\tdata.Source = result.Source\n\t\tdata.WildcardCertificate = result.WildcardCertificate\n\t\terr := encoder.Encode(data)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil\n}\n\n// WriteSourceHost writes the output list of subdomain to an io.Writer\nfunc (o *OutputWriter) WriteSourceHost(input string, sourceMap map[string]map[string]struct{}, writer io.Writer) error {\n\tvar err error\n\tif o.JSON {\n\t\terr = writeSourceJSONHost(input, sourceMap, writer)\n\t} else {\n\t\terr = writeSourcePlainHost(input, sourceMap, writer)\n\t}\n\treturn err\n}\n\nfunc writeSourceJSONHost(input string, sourceMap map[string]map[string]struct{}, writer io.Writer) error {\n\tencoder := jsoniter.NewEncoder(writer)\n\n\tvar data jsonSourcesResult\n\n\tfor host, sources := range sourceMap {\n\t\tdata.Host = host\n\t\tdata.Input = input\n\t\tkeys := make([]string, 0, len(sources))\n\t\tfor source := range sources {\n\t\t\tkeys = append(keys, source)\n\t\t}\n\t\tdata.Sources = keys\n\n\t\terr := encoder.Encode(&data)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil\n}\n\nfunc writeSourcePlainHost(_ string, sourceMap map[string]map[string]struct{}, writer io.Writer) error {\n\tbufwriter := bufio.NewWriter(writer)\n\tsb := &strings.Builder{}\n\n\tfor host, sources := range sourceMap {\n\t\tsb.WriteString(host)\n\t\tsb.WriteString(\",[\")\n\t\tvar sourcesString strings.Builder\n\t\tfor source := range sources {\n\t\t\tsourcesString.WriteString(source)\n\t\t\tsourcesString.WriteRune(',')\n\t\t}\n\t\tsb.WriteString(strings.TrimSuffix(sourcesString.String(), \",\"))\n\t\tsb.WriteString(\"]\\n\")\n\n\t\t_, err := bufwriter.WriteString(sb.String())\n\t\tif err != nil {\n\t\t\tif flushErr := bufwriter.Flush(); flushErr != nil {\n\t\t\t\treturn errors.Join(err, flushErr)\n\t\t\t}\n\t\t\treturn err\n\t\t}\n\t\tsb.Reset()\n\t}\n\treturn bufwriter.Flush()\n}\n"
  },
  {
    "path": "pkg/runner/runner.go",
    "content": "package runner\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"io\"\n\t\"math\"\n\t\"os\"\n\t\"path\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\tcontextutil \"github.com/projectdiscovery/utils/context\"\n\tfileutil \"github.com/projectdiscovery/utils/file\"\n\tmapsutil \"github.com/projectdiscovery/utils/maps\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/passive\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/resolve\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Runner is an instance of the subdomain enumeration\n// client used to orchestrate the whole process.\ntype Runner struct {\n\toptions        *Options\n\tpassiveAgent   *passive.Agent\n\tresolverClient *resolve.Resolver\n\trateLimit      *subscraping.CustomRateLimit\n}\n\n// NewRunner creates a new runner struct instance by parsing\n// the configuration options, configuring sources, reading lists\n// and setting up loggers, etc.\nfunc NewRunner(options *Options) (*Runner, error) {\n\toptions.ConfigureOutput()\n\trunner := &Runner{options: options}\n\n\t// Check if the application loading with any provider configuration, then take it\n\t// Otherwise load the default provider config\n\tif fileutil.FileExists(options.ProviderConfig) {\n\t\tgologger.Info().Msgf(\"Loading provider config from %s\", options.ProviderConfig)\n\t\toptions.loadProvidersFrom(options.ProviderConfig)\n\t} else {\n\t\tgologger.Info().Msgf(\"Loading provider config from the default location: %s\", defaultProviderConfigLocation)\n\t\toptions.loadProvidersFrom(defaultProviderConfigLocation)\n\t}\n\n\t// Initialize the passive subdomain enumeration engine\n\trunner.initializePassiveEngine()\n\n\t// Initialize the subdomain resolver\n\terr := runner.initializeResolver()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// Initialize the custom rate limit\n\trunner.rateLimit = &subscraping.CustomRateLimit{\n\t\tCustom: mapsutil.SyncLockMap[string, uint]{\n\t\t\tMap: make(map[string]uint),\n\t\t},\n\t}\n\n\tfor source, sourceRateLimit := range options.RateLimits.AsMap() {\n\t\tif sourceRateLimit.MaxCount > 0 && sourceRateLimit.MaxCount <= math.MaxUint {\n\t\t\t_ = runner.rateLimit.Custom.Set(source, sourceRateLimit.MaxCount)\n\t\t}\n\t}\n\n\treturn runner, nil\n}\n\n// RunEnumeration wraps RunEnumerationWithCtx with an empty context\nfunc (r *Runner) RunEnumeration() error {\n\tctx, _ := contextutil.WithValues(context.Background(), contextutil.ContextArg(\"All\"), contextutil.ContextArg(strconv.FormatBool(r.options.All)))\n\treturn r.RunEnumerationWithCtx(ctx)\n}\n\n// RunEnumerationWithCtx runs the subdomain enumeration flow on the targets specified\nfunc (r *Runner) RunEnumerationWithCtx(ctx context.Context) error {\n\toutputs := []io.Writer{r.options.Output}\n\n\tif len(r.options.Domain) > 0 {\n\t\tdomainsReader := strings.NewReader(strings.Join(r.options.Domain, \"\\n\"))\n\t\treturn r.EnumerateMultipleDomainsWithCtx(ctx, domainsReader, outputs)\n\t}\n\n\t// If we have multiple domains as input,\n\tif r.options.DomainsFile != \"\" {\n\t\tf, err := os.Open(r.options.DomainsFile)\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t\terr = r.EnumerateMultipleDomainsWithCtx(ctx, f, outputs)\n\t\tif closeErr := f.Close(); closeErr != nil {\n\t\t\tgologger.Error().Msgf(\"Error closing file %s: %s\", r.options.DomainsFile, closeErr)\n\t\t}\n\t\treturn err\n\t}\n\n\t// If we have STDIN input, treat it as multiple domains\n\tif r.options.Stdin {\n\t\treturn r.EnumerateMultipleDomainsWithCtx(ctx, os.Stdin, outputs)\n\t}\n\treturn nil\n}\n\n// EnumerateMultipleDomains wraps EnumerateMultipleDomainsWithCtx with an empty context\nfunc (r *Runner) EnumerateMultipleDomains(reader io.Reader, writers []io.Writer) error {\n\tctx, _ := contextutil.WithValues(context.Background(), contextutil.ContextArg(\"All\"), contextutil.ContextArg(strconv.FormatBool(r.options.All)))\n\treturn r.EnumerateMultipleDomainsWithCtx(ctx, reader, writers)\n}\n\n// EnumerateMultipleDomainsWithCtx enumerates subdomains for multiple domains\n// We keep enumerating subdomains for a given domain until we reach an error\nfunc (r *Runner) EnumerateMultipleDomainsWithCtx(ctx context.Context, reader io.Reader, writers []io.Writer) error {\n\tvar err error\n\tscanner := bufio.NewScanner(reader)\n\tip, _ := regexp.Compile(`^([0-9\\.]+$)`)\n\tfor scanner.Scan() {\n\t\tdomain := preprocessDomain(scanner.Text())\n\t\tdomain = replacer.Replace(domain)\n\n\t\tif domain == \"\" || (r.options.ExcludeIps && ip.MatchString(domain)) {\n\t\t\tcontinue\n\t\t}\n\n\t\tvar file *os.File\n\t\t// If the user has specified an output file, use that output file instead\n\t\t// of creating a new output file for each domain. Else create a new file\n\t\t// for each domain in the directory.\n\t\tif r.options.OutputFile != \"\" {\n\t\t\toutputWriter := NewOutputWriter(r.options.JSON)\n\t\t\tfile, err = outputWriter.createFile(r.options.OutputFile, true)\n\t\t\tif err != nil {\n\t\t\t\tgologger.Error().Msgf(\"Could not create file %s for %s: %s\\n\", r.options.OutputFile, r.options.Domain, err)\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\t_, err = r.EnumerateSingleDomainWithCtx(ctx, domain, append(writers, file))\n\n\t\t\tif closeErr := file.Close(); closeErr != nil {\n\t\t\t\tgologger.Error().Msgf(\"Error closing file %s: %s\", r.options.OutputFile, closeErr)\n\t\t\t}\n\t\t} else if r.options.OutputDirectory != \"\" {\n\t\t\toutputFile := path.Join(r.options.OutputDirectory, domain)\n\t\t\tif r.options.JSON {\n\t\t\t\toutputFile += \".json\"\n\t\t\t} else {\n\t\t\t\toutputFile += \".txt\"\n\t\t\t}\n\n\t\t\toutputWriter := NewOutputWriter(r.options.JSON)\n\t\t\tfile, err = outputWriter.createFile(outputFile, false)\n\t\t\tif err != nil {\n\t\t\t\tgologger.Error().Msgf(\"Could not create file %s for %s: %s\\n\", r.options.OutputFile, r.options.Domain, err)\n\t\t\t\treturn err\n\t\t\t}\n\n\t\t\t_, err = r.EnumerateSingleDomainWithCtx(ctx, domain, append(writers, file))\n\n\t\t\tif closeErr := file.Close(); closeErr != nil {\n\t\t\t\tgologger.Error().Msgf(\"Error closing file %s: %s\", outputFile, closeErr)\n\t\t\t}\n\t\t} else {\n\t\t\t_, err = r.EnumerateSingleDomainWithCtx(ctx, domain, writers)\n\t\t}\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\treturn nil\n}\n"
  },
  {
    "path": "pkg/runner/stats.go",
    "content": "package runner\n\nimport (\n\t\"fmt\"\n\t\"sort\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\t\"golang.org/x/exp/maps\"\n)\n\nfunc printStatistics(stats map[string]subscraping.Statistics) {\n\n\tsources := maps.Keys(stats)\n\tsort.Strings(sources)\n\n\tvar lines []string\n\tvar skipped []string\n\n\tfor _, source := range sources {\n\t\tsourceStats := stats[source]\n\t\tif sourceStats.Skipped {\n\t\t\tskipped = append(skipped, fmt.Sprintf(\" %s\", source))\n\t\t} else {\n\t\t\tlines = append(lines, fmt.Sprintf(\" %-20s %-10s %10d %10d %10d\", source, sourceStats.TimeTaken.Round(time.Millisecond).String(), sourceStats.Results, sourceStats.Requests, sourceStats.Errors))\n\t\t}\n\t}\n\n\tif len(lines) > 0 {\n\t\tgologger.Print().Msgf(\"\\n Source               Duration      Results   Requests     Errors\\n%s\\n\", strings.Repeat(\"─\", 68))\n\t\tgologger.Print().Msg(strings.Join(lines, \"\\n\"))\n\t\tgologger.Print().Msgf(\"\\n\")\n\t}\n\n\tif len(skipped) > 0 {\n\t\tgologger.Print().Msgf(\"\\n The following sources were included but skipped...\\n\\n\")\n\t\tgologger.Print().Msg(strings.Join(skipped, \"\\n\"))\n\t\tgologger.Print().Msgf(\"\\n\\n\")\n\t}\n}\n\nfunc (r *Runner) GetStatistics() map[string]subscraping.Statistics {\n\treturn r.passiveAgent.GetStatistics()\n}\n"
  },
  {
    "path": "pkg/runner/util.go",
    "content": "package runner\n\nimport (\n\tfileutil \"github.com/projectdiscovery/utils/file\"\n\tstringsutil \"github.com/projectdiscovery/utils/strings\"\n)\n\nfunc loadFromFile(file string) ([]string, error) {\n\tchanItems, err := fileutil.ReadFile(file)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tvar items []string\n\tfor item := range chanItems {\n\t\titem = preprocessDomain(item)\n\t\tif item == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\titems = append(items, item)\n\t}\n\treturn items, nil\n}\n\nfunc preprocessDomain(s string) string {\n\treturn stringsutil.NormalizeWithOptions(s,\n\t\tstringsutil.NormalizeOptions{\n\t\t\tStripComments: true,\n\t\t\tTrimCutset:    \"\\n\\t\\\"'` \",\n\t\t\tLowercase:     true,\n\t\t},\n\t)\n}\n"
  },
  {
    "path": "pkg/runner/validate.go",
    "content": "package runner\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/gologger/formatter\"\n\t\"github.com/projectdiscovery/gologger/levels\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/passive\"\n\tmapsutil \"github.com/projectdiscovery/utils/maps\"\n\tsliceutil \"github.com/projectdiscovery/utils/slice\"\n)\n\n// validateOptions validates the configuration options passed\nfunc (options *Options) validateOptions() error {\n\t// Check if domain, list of domains, or stdin info was provided.\n\t// If none was provided, then return.\n\tif len(options.Domain) == 0 && options.DomainsFile == \"\" && !options.Stdin {\n\t\treturn errors.New(\"no input list provided\")\n\t}\n\n\t// Both verbose and silent flags were used\n\tif options.Verbose && options.Silent {\n\t\treturn errors.New(\"both verbose and silent mode specified\")\n\t}\n\n\t// Validate threads and options\n\tif options.Threads == 0 {\n\t\treturn errors.New(\"threads cannot be zero\")\n\t}\n\tif options.Timeout == 0 {\n\t\treturn errors.New(\"timeout cannot be zero\")\n\t}\n\n\t// Always remove wildcard with hostip\n\tif options.HostIP && !options.RemoveWildcard {\n\t\treturn errors.New(\"hostip flag must be used with RemoveWildcard option\")\n\t}\n\n\tif options.Match != nil {\n\t\toptions.matchRegexes = make([]*regexp.Regexp, len(options.Match))\n\t\tvar err error\n\t\tfor i, re := range options.Match {\n\t\t\tif options.matchRegexes[i], err = regexp.Compile(stripRegexString(re)); err != nil {\n\t\t\t\treturn errors.New(\"invalid value for match regex option\")\n\t\t\t}\n\t\t}\n\t}\n\tif options.Filter != nil {\n\t\toptions.filterRegexes = make([]*regexp.Regexp, len(options.Filter))\n\t\tvar err error\n\t\tfor i, re := range options.Filter {\n\t\t\tif options.filterRegexes[i], err = regexp.Compile(stripRegexString(re)); err != nil {\n\t\t\t\treturn errors.New(\"invalid value for filter regex option\")\n\t\t\t}\n\t\t}\n\t}\n\n\tsources := mapsutil.GetKeys(passive.NameSourceMap)\n\tfor source := range options.RateLimits.AsMap() {\n\t\tif !sliceutil.Contains(sources, source) {\n\t\t\treturn fmt.Errorf(\"invalid source %s specified in -rls flag\", source)\n\t\t}\n\t}\n\treturn nil\n}\nfunc stripRegexString(val string) string {\n\tval = strings.ReplaceAll(val, \".\", \"\\\\.\")\n\tval = strings.ReplaceAll(val, \"*\", \".*\")\n\treturn fmt.Sprint(\"^\", val, \"$\")\n}\n\n// ConfigureOutput configures the output on the screen\nfunc (options *Options) ConfigureOutput() {\n\t// If the user desires verbose output, show verbose output\n\tif options.Verbose {\n\t\tgologger.DefaultLogger.SetMaxLevel(levels.LevelVerbose)\n\t}\n\tif options.NoColor {\n\t\tgologger.DefaultLogger.SetFormatter(formatter.NewCLI(true))\n\t}\n\tif options.Silent {\n\t\tgologger.DefaultLogger.SetMaxLevel(levels.LevelSilent)\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/agent.go",
    "content": "package subscraping\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"crypto/tls\"\n\t\"fmt\"\n\t\"io\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"time\"\n\n\t\"github.com/corpix/uarand\"\n\t\"github.com/projectdiscovery/ratelimit\"\n\n\t\"github.com/projectdiscovery/gologger\"\n)\n\n// NewSession creates a new session object for a domain\nfunc NewSession(domain string, proxy string, multiRateLimiter *ratelimit.MultiLimiter, timeout int) (*Session, error) {\n\tTransport := &http.Transport{\n\t\tMaxIdleConns:        100,\n\t\tMaxIdleConnsPerHost: 100,\n\t\tTLSClientConfig: &tls.Config{\n\t\t\tInsecureSkipVerify: true,\n\t\t},\n\t\tDial: (&net.Dialer{\n\t\t\tTimeout: time.Duration(timeout) * time.Second,\n\t\t}).Dial,\n\t}\n\n\t// Add proxy\n\tif proxy != \"\" {\n\t\tproxyURL, _ := url.Parse(proxy)\n\t\tif proxyURL == nil {\n\t\t\t// Log warning but continue anyway\n\t\t\tgologger.Warning().Msgf(\"Invalid proxy provided: %s\", proxy)\n\t\t} else {\n\t\t\tTransport.Proxy = http.ProxyURL(proxyURL)\n\t\t}\n\t}\n\n\tclient := &http.Client{\n\t\tTransport: Transport,\n\t\tTimeout:   time.Duration(timeout) * time.Second,\n\t}\n\n\tsession := &Session{Client: client, Timeout: timeout}\n\n\t// Initiate rate limit instance\n\tsession.MultiRateLimiter = multiRateLimiter\n\n\t// Create a new extractor object for the current domain\n\textractor, err := NewSubdomainExtractor(domain)\n\tsession.Extractor = extractor\n\n\treturn session, err\n}\n\n// Get makes a GET request to a URL with extended parameters\nfunc (s *Session) Get(ctx context.Context, getURL, cookies string, headers map[string]string) (*http.Response, error) {\n\treturn s.HTTPRequest(ctx, http.MethodGet, getURL, cookies, headers, nil, BasicAuth{})\n}\n\n// SimpleGet makes a simple GET request to a URL\nfunc (s *Session) SimpleGet(ctx context.Context, getURL string) (*http.Response, error) {\n\treturn s.HTTPRequest(ctx, http.MethodGet, getURL, \"\", map[string]string{}, nil, BasicAuth{})\n}\n\n// Post makes a POST request to a URL with extended parameters\nfunc (s *Session) Post(ctx context.Context, postURL, cookies string, headers map[string]string, body io.Reader) (*http.Response, error) {\n\treturn s.HTTPRequest(ctx, http.MethodPost, postURL, cookies, headers, body, BasicAuth{})\n}\n\n// SimplePost makes a simple POST request to a URL\nfunc (s *Session) SimplePost(ctx context.Context, postURL, contentType string, body io.Reader) (*http.Response, error) {\n\treturn s.HTTPRequest(ctx, http.MethodPost, postURL, \"\", map[string]string{\"Content-Type\": contentType}, body, BasicAuth{})\n}\n\n// HTTPRequest makes any HTTP request to a URL with extended parameters\nfunc (s *Session) HTTPRequest(ctx context.Context, method, requestURL, cookies string, headers map[string]string, body io.Reader, basicAuth BasicAuth) (*http.Response, error) {\n\treq, err := http.NewRequestWithContext(ctx, method, requestURL, body)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treq.Header.Set(\"User-Agent\", uarand.GetRandom())\n\treq.Header.Set(\"Accept\", \"*/*\")\n\treq.Header.Set(\"Accept-Language\", \"en\")\n\treq.Header.Set(\"Connection\", \"close\")\n\n\tif basicAuth.Username != \"\" || basicAuth.Password != \"\" {\n\t\treq.SetBasicAuth(basicAuth.Username, basicAuth.Password)\n\t}\n\n\tif cookies != \"\" {\n\t\treq.Header.Set(\"Cookie\", cookies)\n\t}\n\n\tfor key, value := range headers {\n\t\treq.Header.Set(key, value)\n\t}\n\n\tsourceName := ctx.Value(CtxSourceArg).(string)\n\tmrlErr := s.MultiRateLimiter.Take(sourceName)\n\tif mrlErr != nil {\n\t\treturn nil, mrlErr\n\t}\n\n\treturn httpRequestWrapper(s.Client, req)\n}\n\n// DiscardHTTPResponse discards the response content by demand\nfunc (s *Session) DiscardHTTPResponse(response *http.Response) {\n\tif response != nil {\n\t\t_, err := io.Copy(io.Discard, response.Body)\n\t\tif err != nil {\n\t\t\tgologger.Warning().Msgf(\"Could not discard response body: %s\\n\", err)\n\t\t\treturn\n\t\t}\n\t\tif closeErr := response.Body.Close(); closeErr != nil {\n\t\t\tgologger.Warning().Msgf(\"Could not close response body: %s\\n\", closeErr)\n\t\t}\n\t}\n}\n\n// Close the session\nfunc (s *Session) Close() {\n\ts.MultiRateLimiter.Stop()\n\ts.Client.CloseIdleConnections()\n}\n\nfunc httpRequestWrapper(client *http.Client, request *http.Request) (*http.Response, error) {\n\tresponse, err := client.Do(request)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif response.StatusCode != http.StatusOK {\n\t\trequestURL, _ := url.QueryUnescape(request.URL.String())\n\n\t\tgologger.Debug().MsgFunc(func() string {\n\t\t\tbuffer := new(bytes.Buffer)\n\t\t\t_, _ = buffer.ReadFrom(response.Body)\n\t\t\treturn fmt.Sprintf(\"Response for failed request against %s:\\n%s\", requestURL, buffer.String())\n\t\t})\n\t\treturn response, fmt.Errorf(\"unexpected status code %d received from %s\", response.StatusCode, requestURL)\n\t}\n\treturn response, nil\n}\n"
  },
  {
    "path": "pkg/subscraping/doc.go",
    "content": "// Package subscraping contains the logic of scraping agents\npackage subscraping\n"
  },
  {
    "path": "pkg/subscraping/extractor.go",
    "content": "package subscraping\n\nimport (\n\t\"regexp\"\n\t\"strings\"\n)\n\n// RegexSubdomainExtractor is a concrete implementation of the SubdomainExtractor interface, using regex for extraction.\ntype RegexSubdomainExtractor struct {\n\textractor *regexp.Regexp\n}\n\n// NewSubdomainExtractor creates a new regular expression to extract\n// subdomains from text based on the given domain.\nfunc NewSubdomainExtractor(domain string) (*RegexSubdomainExtractor, error) {\n\textractor, err := regexp.Compile(`(?i)[a-zA-Z0-9\\*_.-]+\\.` + domain)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn &RegexSubdomainExtractor{extractor: extractor}, nil\n}\n\n// Extract implements the SubdomainExtractor interface, using the regex to find subdomains in the given text.\nfunc (re *RegexSubdomainExtractor) Extract(text string) []string {\n\tmatches := re.extractor.FindAllString(text, -1)\n\tfor i, match := range matches {\n\t\tmatches[i] = strings.ToLower(match)\n\t}\n\treturn matches\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/alienvault/alienvault.go",
    "content": "// Package alienvault logic\npackage alienvault\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype alienvaultResponse struct {\n\tDetail     string `json:\"detail\"`\n\tError      string `json:\"error\"`\n\tPassiveDNS []struct {\n\t\tHostname string `json:\"hostname\"`\n\t} `json:\"passive_dns\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\tresults   int\n\terrors    int\n\trequests  int\n\tapiKeys   []string\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, fmt.Sprintf(\"https://otx.alienvault.com/api/v1/indicators/domain/%s/passive_dns\", domain), \"\",\n\t\t\tmap[string]string{\"Authorization\": \"Bearer \" + randomApiKey})\n\t\tif err != nil && resp == nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar response alienvaultResponse\n\t\t// Get the response body and decode\n\t\terr = json.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tif response.Error != \"\" {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tSource: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%s, %s\", response.Detail, response.Error),\n\t\t\t}\n\t\t\treturn\n\t\t}\n\n\t\tfor _, record := range response.PassiveDNS {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Hostname}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"alienvault\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/anubis/anubis.go",
    "content": "// Package anubis logic\npackage anubis\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://jonlu.ca/anubis/subdomains/%s\", domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tif resp.StatusCode != http.StatusOK {\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar subdomains []string\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&subdomains)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tfor _, record := range subdomains {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"anubis\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/bevigil/bevigil.go",
    "content": "// Package bevigil logic\npackage bevigil\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype Response struct {\n\tDomain     string   `json:\"domain\"`\n\tSubdomains []string `json:\"subdomains\"`\n}\n\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tgetUrl := fmt.Sprintf(\"https://osint.bevigil.com/api/%s/subdomains/\", domain)\n\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, getUrl, \"\", map[string]string{\n\t\t\t\"X-Access-Token\": randomApiKey, \"User-Agent\": \"subfinder\",\n\t\t})\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar subdomains []string\n\t\tvar response Response\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tif len(response.Subdomains) > 0 {\n\t\t\tsubdomains = response.Subdomains\n\t\t}\n\n\t\tfor _, subdomain := range subdomains {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\n\t}()\n\treturn results\n}\n\nfunc (s *Source) Name() string {\n\treturn \"bevigil\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/bufferover/bufferover.go",
    "content": "// Package bufferover is a bufferover Scraping Engine in Golang\npackage bufferover\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tMeta struct {\n\t\tErrors []string `json:\"Errors\"`\n\t} `json:\"Meta\"`\n\tFDNSA   []string `json:\"FDNS_A\"`\n\tRDNS    []string `json:\"RDNS\"`\n\tResults []string `json:\"Results\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ts.getData(ctx, fmt.Sprintf(\"https://tls.bufferover.run/dns?q=.%s\", domain), randomApiKey, session, results)\n\t}()\n\n\treturn results\n}\n\nfunc (s *Source) getData(ctx context.Context, sourceURL string, apiKey string, session *subscraping.Session, results chan subscraping.Result) {\n\ts.requests++\n\tresp, err := session.Get(ctx, sourceURL, \"\", map[string]string{\"x-api-key\": apiKey})\n\n\tif err != nil && resp == nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn\n\t}\n\n\tvar bufforesponse response\n\terr = jsoniter.NewDecoder(resp.Body).Decode(&bufforesponse)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn\n\t}\n\n\tsession.DiscardHTTPResponse(resp)\n\n\tmetaErrors := bufforesponse.Meta.Errors\n\n\tif len(metaErrors) > 0 {\n\t\tresults <- subscraping.Result{\n\t\t\tSource: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%s\", strings.Join(metaErrors, \", \")),\n\t\t}\n\t\ts.errors++\n\t\treturn\n\t}\n\n\tvar subdomains []string\n\n\tif len(bufforesponse.FDNSA) > 0 {\n\t\tsubdomains = bufforesponse.FDNSA\n\t\tsubdomains = append(subdomains, bufforesponse.RDNS...)\n\t} else if len(bufforesponse.Results) > 0 {\n\t\tsubdomains = bufforesponse.Results\n\t}\n\n\tfor _, subdomain := range subdomains {\n\t\tfor _, value := range session.Extractor.Extract(subdomain) {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: value}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"bufferover\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/builtwith/builtwith.go",
    "content": "// Package builtwith logic\npackage builtwith\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tResults []resultItem `json:\"Results\"`\n}\n\ntype resultItem struct {\n\tResult result `json:\"Result\"`\n}\n\ntype result struct {\n\tPaths []path `json:\"Paths\"`\n}\n\ntype path struct {\n\tDomain    string `json:\"Domain\"`\n\tUrl       string `json:\"Url\"`\n\tSubDomain string `json:\"SubDomain\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://api.builtwith.com/v21/api.json?KEY=%s&HIDETEXT=yes&HIDEDL=yes&NOLIVE=yes&NOMETA=yes&NOPII=yes&NOATTR=yes&LOOKUP=%s\", randomApiKey, domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar data response\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&data)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\t\tfor _, result := range data.Results {\n\t\t\tfor _, path := range result.Result.Paths {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: fmt.Sprintf(\"%s.%s\", path.SubDomain, path.Domain)}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"builtwith\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/c99/c99.go",
    "content": "// Package c99 logic\npackage c99\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype dnsdbLookupResponse struct {\n\tSuccess    bool `json:\"success\"`\n\tSubdomains []struct {\n\t\tSubdomain  string `json:\"subdomain\"`\n\t\tIP         string `json:\"ip\"`\n\t\tCloudflare bool   `json:\"cloudflare\"`\n\t} `json:\"subdomains\"`\n\tError string `json:\"error\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tsearchURL := fmt.Sprintf(\"https://api.c99.nl/subdomainfinder?key=%s&domain=%s&json\", randomApiKey, domain)\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, searchURL)\n\t\tif err != nil {\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tvar response dnsdbLookupResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tif response.Error != \"\" {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tSource: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%v\", response.Error),\n\t\t\t}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, data := range response.Subdomains {\n\t\t\tif !strings.HasPrefix(data.Subdomain, \".\") {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: data.Subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"c99\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/censys/censys.go",
    "content": "// Package censys logic\npackage censys\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"net/http\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nconst (\n\t// maxCensysPages is the maximum number of pages to fetch from the API\n\tmaxCensysPages = 10\n\t// maxPerPage is the maximum number of results per page\n\tmaxPerPage = 100\n\t// baseURL is the Censys Platform API base URL\n\tbaseURL = \"https://api.platform.censys.io\"\n\t// searchEndpoint is the global data search query endpoint\n\tsearchEndpoint = \"/v3/global/search/query\"\n\t// queryPrefix is the Censys query language prefix for certificate name search\n\tqueryPrefix = \"cert.names: \"\n\t// authHeaderPrefix is the Bearer token prefix for Authorization header\n\tauthHeaderPrefix = \"Bearer \"\n\t// contentTypeJSON is the Content-Type header value for JSON\n\tcontentTypeJSON = \"application/json\"\n\t// orgIDHeader is the header name for organization ID\n\torgIDHeader = \"X-Organization-ID\"\n)\n\n// apiKey holds the Personal Access Token and optional Organization ID\ntype apiKey struct {\n\tpat   string\n\torgID string\n}\n\n// Platform API request body\ntype searchRequest struct {\n\tQuery    string   `json:\"query\"`\n\tFields   []string `json:\"fields,omitempty\"`\n\tPageSize int      `json:\"page_size,omitempty\"`\n\tCursor   string   `json:\"cursor,omitempty\"`\n}\n\n// Platform API response structures\ntype response struct {\n\tResult result `json:\"result\"`\n}\n\ntype result struct {\n\tHits          []hit  `json:\"hits\"`\n\tTotalHits     int64  `json:\"total_hits\"`\n\tNextPageToken string `json:\"next_page_token\"`\n}\n\ntype hit struct {\n\tCertificateV1 certificateV1 `json:\"certificate_v1\"`\n}\n\ntype certificateV1 struct {\n\tResource resource `json:\"resource\"`\n}\n\ntype resource struct {\n\tNames []string `json:\"names\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []apiKey\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\t// PickRandom selects a random API key from configured keys.\n\t\t// This enables load balancing when users configure multiple PATs\n\t\t// (e.g., CENSYS_API_KEY=pat1:org1,pat2:org2) to distribute requests\n\t\t// and avoid hitting rate limits on a single key.\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey.pat == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tapiURL := baseURL + searchEndpoint\n\t\tcursor := \"\"\n\t\tcurrentPage := 1\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\n\t\t\treqBody := searchRequest{\n\t\t\t\tQuery:    queryPrefix + domain,\n\t\t\t\tFields:   []string{\"cert.names\"},\n\t\t\t\tPageSize: maxPerPage,\n\t\t\t}\n\t\t\tif cursor != \"\" {\n\t\t\t\treqBody.Cursor = cursor\n\t\t\t}\n\n\t\t\tbodyBytes, err := jsoniter.Marshal(reqBody)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\theaders := map[string]string{\n\t\t\t\t\"Content-Type\":  contentTypeJSON,\n\t\t\t\t\"Authorization\": authHeaderPrefix + randomApiKey.pat,\n\t\t\t}\n\t\t\t// Add Organization ID header if provided\n\t\t\tif randomApiKey.orgID != \"\" {\n\t\t\t\theaders[orgIDHeader] = randomApiKey.orgID\n\t\t\t}\n\n\t\t\ts.requests++\n\t\t\tresp, err := session.HTTPRequest(\n\t\t\t\tctx,\n\t\t\t\thttp.MethodPost,\n\t\t\t\tapiURL,\n\t\t\t\t\"\",\n\t\t\t\theaders,\n\t\t\t\tbytes.NewReader(bodyBytes),\n\t\t\t\tsubscraping.BasicAuth{},\n\t\t\t)\n\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar censysResponse response\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&censysResponse)\n\t\t\t_ = resp.Body.Close()\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tfor _, hit := range censysResponse.Result.Hits {\n\t\t\t\tfor _, name := range hit.CertificateV1.Resource.Names {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: name}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tcursor = censysResponse.Result.NextPageToken\n\t\t\tif cursor == \"\" || currentPage >= maxCensysPages {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tcurrentPage++\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"censys\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\n// AddApiKeys parses and adds API keys.\n// Format: \"PAT:ORG_ID\" where ORG_ID is required for paid accounts.\n// Example: \"censys_xxx_token:12345678-91011-1213\"\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = subscraping.CreateApiKeys(keys, func(pat, orgID string) apiKey {\n\t\treturn apiKey{pat: pat, orgID: orgID}\n\t})\n\t// Also support single PAT without org ID for free users\n\tfor _, key := range keys {\n\t\tif !strings.Contains(key, \":\") && key != \"\" {\n\t\t\ts.apiKeys = append(s.apiKeys, apiKey{pat: key, orgID: \"\"})\n\t\t}\n\t}\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/censys/censys_test.go",
    "content": "package censys\n\nimport (\n\t\"context\"\n\t\"math\"\n\t\"net/http\"\n\t\"testing\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/ratelimit\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/require\"\n)\n\n// createTestMultiRateLimiter creates a MultiLimiter for testing\nfunc createTestMultiRateLimiter(ctx context.Context) *ratelimit.MultiLimiter {\n\tmrl, _ := ratelimit.NewMultiLimiter(ctx, &ratelimit.Options{\n\t\tKey:         \"censys\",\n\t\tIsUnlimited: false,\n\t\tMaxCount:    math.MaxInt32,\n\t\tDuration:    time.Millisecond,\n\t})\n\treturn mrl\n}\n\nfunc TestCensysSource_NoApiKey(t *testing.T) {\n\tsource := &Source{}\n\t// Don't add any API keys\n\n\tctx := context.Background()\n\tmultiRateLimiter := createTestMultiRateLimiter(ctx)\n\tsession := &subscraping.Session{\n\t\tClient:           http.DefaultClient,\n\t\tMultiRateLimiter: multiRateLimiter,\n\t}\n\n\tctxWithValue := context.WithValue(ctx, subscraping.CtxSourceArg, \"censys\")\n\tresults := source.Run(ctxWithValue, \"example.com\", session)\n\n\t// Collect all results\n\tvar resultCount int\n\tfor range results {\n\t\tresultCount++\n\t}\n\n\t// Should be skipped when no API key\n\tstats := source.Statistics()\n\tassert.True(t, stats.Skipped, \"expected source to be skipped without API key\")\n\tassert.Equal(t, 0, resultCount, \"expected no results when skipped\")\n}\n\nfunc TestCensysSource_ContextCancellation(t *testing.T) {\n\tsource := &Source{}\n\t// Add a key with PAT:ORG_ID format\n\tsource.AddApiKeys([]string{\"test_pat:test_org_id\"})\n\n\tctx := context.Background()\n\tmultiRateLimiter := createTestMultiRateLimiter(ctx)\n\tsession := &subscraping.Session{\n\t\tClient:           http.DefaultClient,\n\t\tMultiRateLimiter: multiRateLimiter,\n\t}\n\n\t// Create a context that will be cancelled\n\tctxCancellable, cancel := context.WithCancel(ctx)\n\tctxWithValue := context.WithValue(ctxCancellable, subscraping.CtxSourceArg, \"censys\")\n\n\tresults := source.Run(ctxWithValue, \"example.com\", session)\n\n\t// Cancel immediately\n\tcancel()\n\n\t// Should exit quickly without blocking\n\tdone := make(chan struct{})\n\tgo func() {\n\t\tfor range results {\n\t\t\t// drain\n\t\t}\n\t\tclose(done)\n\t}()\n\n\tselect {\n\tcase <-done:\n\t\t// Good - completed quickly\n\tcase <-time.After(2 * time.Second):\n\t\tt.Fatal(\"context cancellation did not stop the source in time\")\n\t}\n}\n\nfunc TestCensysSource_Metadata(t *testing.T) {\n\tsource := &Source{}\n\n\tassert.Equal(t, \"censys\", source.Name())\n\tassert.True(t, source.IsDefault())\n\tassert.False(t, source.HasRecursiveSupport())\n\tassert.True(t, source.NeedsKey())\n}\n\nfunc TestCensysSource_AddApiKeys(t *testing.T) {\n\tt.Run(\"PAT with OrgID\", func(t *testing.T) {\n\t\tsource := &Source{}\n\t\tkeys := []string{\"pat_token_1:org_id_1\", \"pat_token_2:org_id_2\"}\n\t\tsource.AddApiKeys(keys)\n\n\t\trequire.Len(t, source.apiKeys, 2)\n\t\tassert.Equal(t, \"pat_token_1\", source.apiKeys[0].pat)\n\t\tassert.Equal(t, \"org_id_1\", source.apiKeys[0].orgID)\n\t\tassert.Equal(t, \"pat_token_2\", source.apiKeys[1].pat)\n\t\tassert.Equal(t, \"org_id_2\", source.apiKeys[1].orgID)\n\t})\n\n\tt.Run(\"PAT without OrgID (free user)\", func(t *testing.T) {\n\t\tsource := &Source{}\n\t\tkeys := []string{\"pat_token_only\"}\n\t\tsource.AddApiKeys(keys)\n\n\t\trequire.Len(t, source.apiKeys, 1)\n\t\tassert.Equal(t, \"pat_token_only\", source.apiKeys[0].pat)\n\t\tassert.Equal(t, \"\", source.apiKeys[0].orgID)\n\t})\n}\n\nfunc TestCensysSource_Statistics(t *testing.T) {\n\tsource := &Source{\n\t\terrors:    2,\n\t\tresults:   10,\n\t\ttimeTaken: 5 * time.Second,\n\t\tskipped:   false,\n\t}\n\n\tstats := source.Statistics()\n\tassert.Equal(t, 2, stats.Errors)\n\tassert.Equal(t, 10, stats.Results)\n\tassert.Equal(t, 5*time.Second, stats.TimeTaken)\n\tassert.False(t, stats.Skipped)\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/certspotter/certspotter.go",
    "content": "// Package certspotter logic\npackage certspotter\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype certspotterObject struct {\n\tID       string   `json:\"id\"`\n\tDNSNames []string `json:\"dns_names\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\"Authorization\": \"Bearer \" + randomApiKey}\n\t\tcookies := \"\"\n\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, fmt.Sprintf(\"https://api.certspotter.com/v1/issuances?domain=%s&include_subdomains=true&expand=dns_names\", domain), cookies, headers)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar response []certspotterObject\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tfor _, cert := range response {\n\t\t\tfor _, subdomain := range cert.DNSNames {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\tif len(response) == 0 {\n\t\t\treturn\n\t\t}\n\n\t\tid := response[len(response)-1].ID\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\treqURL := fmt.Sprintf(\"https://api.certspotter.com/v1/issuances?domain=%s&include_subdomains=true&expand=dns_names&after=%s\", domain, id)\n\n\t\t\ts.requests++\n\t\t\tresp, err := session.Get(ctx, reqURL, cookies, headers)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar response []certspotterObject\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\tif len(response) == 0 {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\tfor _, cert := range response {\n\t\t\t\tfor _, subdomain := range cert.DNSNames {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tid = response[len(response)-1].ID\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"certspotter\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/chaos/chaos.go",
    "content": "// Package chaos logic\npackage chaos\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/chaos-client/pkg/chaos\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, _ *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tchaosClient := chaos.New(randomApiKey)\n\t\ts.requests++\n\t\tfor result := range chaosClient.GetSubdomains(&chaos.SubdomainsRequest{\n\t\t\tDomain: domain,\n\t\t}) {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tif result.Error != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: result.Error}\n\t\t\t\ts.errors++\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tSource: s.Name(), Type: subscraping.Subdomain, Value: fmt.Sprintf(\"%s.%s\", result.Subdomain, domain),\n\t\t\t}\n\t\t\ts.results++\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"chaos\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/chinaz/chinaz.go",
    "content": "package chinaz\n\n// chinaz  http://my.chinaz.com/ChinazAPI/DataCenter/MyDataApi\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"io\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://apidatav2.chinaz.com/single/alexa?key=%s&domain=%s\", randomApiKey, domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tbody, err := io.ReadAll(resp.Body)\n\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tSubdomainList := jsoniter.Get(body, \"Result\").Get(\"ContributingSubdomainList\")\n\n\t\tif SubdomainList.ToBool() {\n\t\t\t_data := []byte(SubdomainList.ToString())\n\t\t\tfor i := 0; i < SubdomainList.Size(); i++ {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tsubdomain := jsoniter.Get(_data, i, \"DataUrl\").ToString()\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t} else {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"chinaz\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/commoncrawl/commoncrawl.go",
    "content": "// Package commoncrawl logic\npackage commoncrawl\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"fmt\"\n\t\"net/url\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nconst (\n\tindexURL     = \"https://index.commoncrawl.org/collinfo.json\"\n\tmaxYearsBack = 5\n)\n\nvar year = time.Now().Year()\n\ntype indexResponse struct {\n\tID     string `json:\"id\"`\n\tAPIURL string `json:\"cdx-api\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, indexURL)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar indexes []indexResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&indexes)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tyears := make([]string, 0)\n\t\tfor i := range maxYearsBack {\n\t\t\tyears = append(years, strconv.Itoa(year-i))\n\t\t}\n\n\t\tsearchIndexes := make(map[string]string)\n\t\tfor _, year := range years {\n\t\t\tfor _, index := range indexes {\n\t\t\t\tif strings.Contains(index.ID, year) {\n\t\t\t\t\tif _, ok := searchIndexes[year]; !ok {\n\t\t\t\t\t\tsearchIndexes[year] = index.APIURL\n\t\t\t\t\t\tbreak\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\tfor _, apiURL := range searchIndexes {\n\t\t\tfurther := s.getSubdomains(ctx, apiURL, domain, session, results)\n\t\t\tif !further {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"commoncrawl\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t}\n}\n\nfunc (s *Source) getSubdomains(ctx context.Context, searchURL, domain string, session *subscraping.Session, results chan subscraping.Result) bool {\n\tfor {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn false\n\t\tdefault:\n\t\t\tvar headers = map[string]string{\"Host\": \"index.commoncrawl.org\"}\n\t\t\ts.requests++\n\t\t\tresp, err := session.Get(ctx, fmt.Sprintf(\"%s?url=*.%s\", searchURL, domain), \"\", headers)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn false\n\t\t\t}\n\n\t\t\tscanner := bufio.NewScanner(resp.Body)\n\t\t\tfor scanner.Scan() {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\treturn false\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tline := scanner.Text()\n\t\t\t\tif line == \"\" {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tline, _ = url.QueryUnescape(line)\n\t\t\t\tfor _, subdomain := range session.Extractor.Extract(line) {\n\t\t\t\t\tif subdomain != \"\" {\n\t\t\t\t\t\tsubdomain = strings.ToLower(subdomain)\n\t\t\t\t\t\tsubdomain = strings.TrimPrefix(subdomain, \"25\")\n\t\t\t\t\t\tsubdomain = strings.TrimPrefix(subdomain, \"2f\")\n\n\t\t\t\t\t\tselect {\n\t\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\t\t\treturn false\n\t\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\t\t\ts.results++\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn true\n\t\t}\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/crtsh/crtsh.go",
    "content": "// Package crtsh logic\npackage crtsh\n\nimport (\n\t\"context\"\n\t\"database/sql\"\n\t\"fmt\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t// postgres driver\n\t_ \"github.com/lib/pq\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\tcontextutil \"github.com/projectdiscovery/utils/context\"\n)\n\ntype subdomain struct {\n\tID        int    `json:\"id\"`\n\tNameValue string `json:\"name_value\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\tcount := s.getSubdomainsFromSQL(ctx, domain, session, results)\n\t\tif count > 0 {\n\t\t\treturn\n\t\t}\n\t\t_ = s.getSubdomainsFromHTTP(ctx, domain, session, results)\n\t}()\n\n\treturn results\n}\n\nfunc (s *Source) getSubdomainsFromSQL(ctx context.Context, domain string, session *subscraping.Session, results chan subscraping.Result) int {\n\t// connect_timeout: limits connection establishment time (in seconds)\n\t// statement_timeout: limits query execution time (in milliseconds)\n\tconnStr := fmt.Sprintf(\"host=crt.sh user=guest dbname=certwatch sslmode=disable binary_parameters=yes connect_timeout=%d statement_timeout=%d\", session.Timeout, session.Timeout*1000)\n\tdb, err := sql.Open(\"postgres\", connStr)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\treturn 0\n\t}\n\n\tdefer func() {\n\t\tif closeErr := db.Close(); closeErr != nil {\n\t\t\tgologger.Warning().Msgf(\"Could not close database connection: %s\\n\", closeErr)\n\t\t}\n\t}()\n\n\tlimitClause := \"\"\n\tif all, ok := ctx.Value(contextutil.ContextArg(\"All\")).(contextutil.ContextArg); ok {\n\t\tif allBool, err := strconv.ParseBool(string(all)); err == nil && !allBool {\n\t\t\tlimitClause = \"LIMIT 10000\"\n\t\t}\n\t}\n\n\tquery := fmt.Sprintf(`WITH ci AS (\n\t\t\t\tSELECT min(sub.CERTIFICATE_ID) ID,\n\t\t\t\t\tmin(sub.ISSUER_CA_ID) ISSUER_CA_ID,\n\t\t\t\t\tarray_agg(DISTINCT sub.NAME_VALUE) NAME_VALUES,\n\t\t\t\t\tx509_commonName(sub.CERTIFICATE) COMMON_NAME,\n\t\t\t\t\tx509_notBefore(sub.CERTIFICATE) NOT_BEFORE,\n\t\t\t\t\tx509_notAfter(sub.CERTIFICATE) NOT_AFTER,\n\t\t\t\t\tencode(x509_serialNumber(sub.CERTIFICATE), 'hex') SERIAL_NUMBER\n\t\t\t\t\tFROM (SELECT *\n\t\t\t\t\t\t\tFROM certificate_and_identities cai\n\t\t\t\t\t\t\tWHERE plainto_tsquery('certwatch', $1) @@ identities(cai.CERTIFICATE)\n\t\t\t\t\t\t\t\tAND cai.NAME_VALUE ILIKE ('%%' || $1 || '%%')\n\t\t\t\t\t\t\t\t%s\n\t\t\t\t\t\t) sub\n\t\t\t\t\tGROUP BY sub.CERTIFICATE\n\t\t\t)\n\t\t\tSELECT array_to_string(ci.NAME_VALUES, chr(10)) NAME_VALUE\n\t\t\t\tFROM ci\n\t\t\t\t\t\tLEFT JOIN LATERAL (\n\t\t\t\t\t\t\tSELECT min(ctle.ENTRY_TIMESTAMP) ENTRY_TIMESTAMP\n\t\t\t\t\t\t\t\tFROM ct_log_entry ctle\n\t\t\t\t\t\t\t\tWHERE ctle.CERTIFICATE_ID = ci.ID\n\t\t\t\t\t\t) le ON TRUE,\n\t\t\t\t\tca\n\t\t\t\tWHERE ci.ISSUER_CA_ID = ca.ID\n\t\t\t\tORDER BY le.ENTRY_TIMESTAMP DESC NULLS LAST;`, limitClause)\n\trows, err := db.QueryContext(ctx, query, domain)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\treturn 0\n\t}\n\tif err := rows.Err(); err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\treturn 0\n\t}\n\n\tvar count int\n\tvar data string\n\tfor rows.Next() {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn count\n\t\tdefault:\n\t\t}\n\t\terr := rows.Scan(&data)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn count\n\t\t}\n\n\t\tcount++\n\t\tfor subdomain := range strings.SplitSeq(data, \"\\n\") {\n\t\t\tfor _, value := range session.Extractor.Extract(subdomain) {\n\t\t\t\tif value != \"\" {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn count\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: value}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\treturn count\n}\n\nfunc (s *Source) getSubdomainsFromHTTP(ctx context.Context, domain string, session *subscraping.Session, results chan subscraping.Result) bool {\n\ts.requests++\n\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://crt.sh/?q=%%25.%s&output=json\", domain))\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn false\n\t}\n\n\tvar subdomains []subdomain\n\terr = jsoniter.NewDecoder(resp.Body).Decode(&subdomains)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn false\n\t}\n\n\tsession.DiscardHTTPResponse(resp)\n\n\tfor _, subdomain := range subdomains {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn true\n\t\tdefault:\n\t\t}\n\t\tfor sub := range strings.SplitSeq(subdomain.NameValue, \"\\n\") {\n\t\t\tfor _, value := range session.Extractor.Extract(sub) {\n\t\t\t\tif value != \"\" {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn true\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: value}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\treturn true\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"crtsh\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/digitalyama/digitalyama.go",
    "content": "package digitalyama\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype digitalYamaResponse struct {\n\tQuery        string   `json:\"query\"`\n\tCount        int      `json:\"count\"`\n\tSubdomains   []string `json:\"subdomains\"`\n\tUsageSummary struct {\n\t\tQueryCost        float64 `json:\"query_cost\"`\n\t\tCreditsRemaining float64 `json:\"credits_remaining\"`\n\t} `json:\"usage_summary\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tsearchURL := fmt.Sprintf(\"https://api.digitalyama.com/subdomain_finder?domain=%s\", domain)\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, searchURL, \"\", map[string]string{\"x-api-key\": randomApiKey})\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tif resp.StatusCode != 200 {\n\t\t\tvar errResponse struct {\n\t\t\t\tDetail []struct {\n\t\t\t\t\tLoc  []string `json:\"loc\"`\n\t\t\t\t\tMsg  string   `json:\"msg\"`\n\t\t\t\t\tType string   `json:\"type\"`\n\t\t\t\t} `json:\"detail\"`\n\t\t\t}\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&errResponse)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"unexpected status code %d\", resp.StatusCode)}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\t\t\tif len(errResponse.Detail) > 0 {\n\t\t\t\terrMsg := errResponse.Detail[0].Msg\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%s (code %d)\", errMsg, resp.StatusCode)}\n\t\t\t} else {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"unexpected status code %d\", resp.StatusCode)}\n\t\t\t}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tvar response digitalYamaResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, subdomain := range response.Subdomains {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"digitalyama\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/digitorus/digitorus.go",
    "content": "// Package waybackarchive logic\npackage digitorus\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\t\"github.com/projectdiscovery/utils/ptr\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://certificatedetails.com/%s\", domain))\n\t\t// the 404 page still contains around 100 subdomains - https://github.com/projectdiscovery/subfinder/issues/774\n\t\tif err != nil && ptr.Safe(resp).StatusCode != http.StatusNotFound {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tscanner := bufio.NewScanner(resp.Body)\n\t\tfor scanner.Scan() {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tline := scanner.Text()\n\t\t\tif line == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tsubdomains := session.Extractor.Extract(line)\n\t\t\tfor _, subdomain := range subdomains {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: strings.TrimPrefix(subdomain, \".\")}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"digitorus\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/dnsdb/dnsdb.go",
    "content": "// Package dnsdb logic\npackage dnsdb\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/url\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nconst urlBase string = \"https://api.dnsdb.info/dnsdb/v2\"\n\ntype rateResponse struct {\n\tRate rate\n}\n\ntype rate struct {\n\tOffsetMax json.Number `json:\"offset_max\"`\n}\n\ntype safResponse struct {\n\tCondition string   `json:\"cond\"`\n\tObj       dnsdbObj `json:\"obj\"`\n\tMsg       string   `json:\"msg\"`\n}\n\ntype dnsdbObj struct {\n\tName string `json:\"rrname\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   uint64\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\tsourceName := s.Name()\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, sourceName)\n\t\tif randomApiKey == \"\" {\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\n\t\t\t\"X-API-KEY\": randomApiKey,\n\t\t\t\"Accept\":    \"application/x-ndjson\",\n\t\t}\n\n\t\ts.requests++\n\t\toffsetMax, err := getMaxOffset(ctx, session, headers)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: sourceName, Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tpath := fmt.Sprintf(\"lookup/rrset/name/*.%s\", domain)\n\t\turlTemplate := fmt.Sprintf(\"%s/%s?\", urlBase, path)\n\t\tqueryParams := url.Values{}\n\t\t// ?limit=0 means DNSDB will return the maximum number of results allowed.\n\t\tqueryParams.Add(\"limit\", \"0\")\n\t\tqueryParams.Add(\"swclient\", \"subfinder\")\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\turl := urlTemplate + queryParams.Encode()\n\n\t\t\ts.requests++\n\t\t\tresp, err := session.Get(ctx, url, \"\", headers)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: sourceName, Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar respCond string\n\t\t\treader := bufio.NewReader(resp.Body)\n\t\t\tfor {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tn, err := reader.ReadBytes('\\n')\n\t\t\t\tif err == io.EOF {\n\t\t\t\t\tbreak\n\t\t\t\t} else if err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: sourceName, Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors++\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tvar response safResponse\n\t\t\t\terr = jsoniter.Unmarshal(n, &response)\n\t\t\t\tif err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: sourceName, Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors++\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\trespCond = response.Condition\n\t\t\t\tif respCond == \"\" || respCond == \"ongoing\" {\n\t\t\t\t\tif response.Obj.Name != \"\" {\n\t\t\t\t\t\tselect {\n\t\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\t\t\treturn\n\t\t\t\t\t\tcase results <- subscraping.Result{Source: sourceName, Type: subscraping.Subdomain, Value: strings.TrimSuffix(response.Obj.Name, \".\")}:\n\t\t\t\t\t\t\ts.results++\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t} else if respCond != \"begin\" {\n\t\t\t\t\tbreak\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// Check the terminating jsonl object's condition. There are 3 possible scenarios:\n\t\t\t// 1. \"limited\" - There are more results available, make another query with an offset\n\t\t\t// 2. \"succeeded\" - The query completed successfully and all results were sent.\n\t\t\t// 3. anything else - This is an error and should be reported to the user. The user can then decide to use the results up to this\n\t\t\t// point or discard and retry.\n\t\t\tif respCond == \"limited\" {\n\t\t\t\tif offsetMax != 0 && s.results <= offsetMax {\n\t\t\t\t\t// Reset done to false to get more results with an offset query parameter set to s.results\n\t\t\t\t\tqueryParams.Set(\"offset\", strconv.FormatUint(s.results, 10))\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t} else if respCond != \"succeeded\" {\n\t\t\t\t// DNSDB's terminating jsonl object's cond is not \"limited\" or succeeded\" (#3), this is an error, notify the user.\n\t\t\t\terr = fmt.Errorf(\"%s terminated with condition: %s\", sourceName, respCond)\n\t\t\t\tresults <- subscraping.Result{Source: sourceName, Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\tbreak\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"dnsdb\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   int(s.results),\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n\nfunc getMaxOffset(ctx context.Context, session *subscraping.Session, headers map[string]string) (uint64, error) {\n\tvar offsetMax uint64\n\turl := fmt.Sprintf(\"%s/rate_limit\", urlBase)\n\tresp, err := session.Get(ctx, url, \"\", headers)\n\tdefer session.DiscardHTTPResponse(resp)\n\tif err != nil {\n\t\treturn offsetMax, err\n\t}\n\tdata, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\treturn offsetMax, err\n\t}\n\tvar rateResp rateResponse\n\terr = jsoniter.Unmarshal(data, &rateResp)\n\tif err != nil {\n\t\treturn offsetMax, err\n\t}\n\t// if the OffsetMax is \"n/a\" then the ?offset= query parameter is not allowed\n\tif rateResp.Rate.OffsetMax.String() != \"n/a\" {\n\t\toffsetMax, err = strconv.ParseUint(rateResp.Rate.OffsetMax.String(), 10, 64)\n\t\tif err != nil {\n\t\t\treturn offsetMax, err\n\t\t}\n\t}\n\n\treturn offsetMax, nil\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/dnsdumpster/dnsdumpster.go",
    "content": "// Package dnsdumpster logic\npackage dnsdumpster\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tA []struct {\n\t\tHost string `json:\"host\"`\n\t} `json:\"a\"`\n\tNs []struct {\n\t\tHost string `json:\"host\"`\n\t} `json:\"ns\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, fmt.Sprintf(\"https://api.dnsdumpster.com/domain/%s\", domain), \"\", map[string]string{\"X-API-Key\": randomApiKey})\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\tvar response response\n\t\terr = json.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tfor _, record := range append(response.A, response.Ns...) {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Host}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"dnsdumpster\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/dnsrepo/dnsrepo.go",
    "content": "package dnsrepo\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype DnsRepoResponse []struct {\n\tDomain string `json:\"domain\"`\n}\n\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\trandomApiInfo := strings.Split(randomApiKey, \":\")\n\t\tif len(randomApiInfo) != 2 {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ttoken := randomApiInfo[0]\n\t\tapiKey := randomApiInfo[1]\n\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, fmt.Sprintf(\"https://dnsarchive.net/api/?apikey=%s&search=%s\", apiKey, domain), \"\", map[string]string{\"X-API-Access\": token})\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tresponseData, err := io.ReadAll(resp.Body)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\t\tvar result DnsRepoResponse\n\t\terr = json.Unmarshal(responseData, &result)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tfor _, sub := range result {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: strings.TrimSuffix(sub.Domain, \".\")}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"dnsrepo\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/domainsproject/domainsproject.go",
    "content": "// Package domainsproject logic\npackage domainsproject\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []apiKey\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype apiKey struct {\n\tusername string\n\tpassword string\n}\n\ntype domainsProjectResponse struct {\n\tDomains []string `json:\"domains\"`\n\tError   string   `json:\"error\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey.username == \"\" || randomApiKey.password == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tsearchURL := fmt.Sprintf(\"https://api.domainsproject.org/api/tld/search?domain=%s\", domain)\n\t\ts.requests++\n\t\tresp, err := session.HTTPRequest(\n\t\t\tctx,\n\t\t\t\"GET\",\n\t\t\tsearchURL,\n\t\t\t\"\",\n\t\t\tnil,\n\t\t\tnil,\n\t\t\tsubscraping.BasicAuth{Username: randomApiKey.username, Password: randomApiKey.password},\n\t\t)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tvar response domainsProjectResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tif response.Error != \"\" {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tSource: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%v\", response.Error),\n\t\t\t}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, subdomain := range response.Domains {\n\t\t\tif !strings.HasPrefix(subdomain, \".\") {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"domainsproject\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = subscraping.CreateApiKeys(keys, func(k, v string) apiKey {\n\t\treturn apiKey{k, v}\n\t})\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/driftnet/driftnet.go",
    "content": "package driftnet\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nconst (\n\t// baseURL is the base URL for the driftnet API\n\tbaseURL = \"https://api.driftnet.io/v1/\"\n\n\t// summaryLimit is the size of the summary limit that we send to the API\n\tsummaryLimit = 10000\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    atomic.Int32\n\tresults   atomic.Int32\n\trequests  atomic.Int32\n\tskipped   bool\n}\n\n// endpointConfig describes a driftnet endpoint that can used\ntype endpointConfig struct {\n\t// The API endpoint to be touched\n\tendpoint string\n\n\t// The API parameter used for query\n\tparam string\n\n\t// The context that we should restrict to in results from this endpoint\n\tcontext string\n}\n\n// endpoints is a set of endpoint configs\nvar endpoints = []endpointConfig{\n\t{\"ct/log\", \"field=host:\", \"cert-dns-name\"},\n\t{\"scan/protocols\", \"field=host:\", \"cert-dns-name\"},\n\t{\"scan/domains\", \"field=host:\", \"cert-dns-name\"},\n\t{\"domain/rdns\", \"host=\", \"dns-ptr\"},\n}\n\n// summaryResponse is an API response\ntype summaryResponse struct {\n\tSummary struct {\n\t\tOther  int            `json:\"other\"`\n\t\tValues map[string]int `json:\"values\"`\n\t} `json:\"summary\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\t// Final results channel\n\tresults := make(chan subscraping.Result)\n\ts.errors.Store(0)\n\ts.results.Store(0)\n\ts.requests.Store(0)\n\n\t// Waitgroup for subsources\n\tvar wg sync.WaitGroup\n\twg.Add(len(endpoints))\n\n\t// Map for dedupe between subsources\n\tdedupe := sync.Map{}\n\n\t// Close down results when all subsources finished\n\tgo func(startTime time.Time) {\n\t\twg.Wait()\n\t\ts.timeTaken = time.Since(startTime)\n\t\tclose(results)\n\t}(time.Now())\n\n\t// Start up requests for all subsources\n\tfor i := range endpoints {\n\t\tgo s.runSubsource(ctx, domain, session, results, &wg, &dedupe, endpoints[i])\n\t}\n\n\t// Return the results channel\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"driftnet\"\n}\n\n// IsDefault indicates that this source should used as part of the default execution.\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\n// HasRecursiveSupport indicates that we accept subdomains in addition to apex domains\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\n// KeyRequirement indicates that we need an API key\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\n// NeedsKey indicates that we need an API key\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\n// AddApiKeys provides us with the API key(s)\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\n// Statistics returns statistics about the scraping process\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    int(s.errors.Load()),\n\t\tResults:   int(s.results.Load()),\n\t\tRequests:  int(s.requests.Load()),\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n\n// runSubsource queries a specific driftnet endpoint for subdomains and sends results to the channel\nfunc (s *Source) runSubsource(ctx context.Context, domain string, session *subscraping.Session, results chan subscraping.Result, wg *sync.WaitGroup, dedupe *sync.Map, epConfig endpointConfig) {\n\t// Default headers\n\theaders := map[string]string{\n\t\t\"accept\": \"application/json\",\n\t}\n\n\t// Pick an API key\n\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\tif randomApiKey != \"\" {\n\t\theaders[\"authorization\"] = \"Bearer \" + randomApiKey\n\t}\n\n\t// Request\n\trequestURL := fmt.Sprintf(\"%s%s?%s%s&summarize=host&summary_context=%s&summary_limit=%d\", baseURL, epConfig.endpoint, epConfig.param, url.QueryEscape(domain), epConfig.context, summaryLimit)\n\ts.requests.Add(1)\n\tresp, err := session.Get(ctx, requestURL, \"\", headers)\n\tif err != nil {\n\t\t// HTTP 204 is not an error from the Driftnet API\n\t\tif resp == nil || resp.StatusCode != http.StatusNoContent {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors.Add(1)\n\t\t}\n\n\t\twg.Done()\n\t\treturn\n\t}\n\n\tdefer session.DiscardHTTPResponse(resp)\n\n\t// 204 means no results, any other response code is an error\n\tif resp.StatusCode != 200 {\n\t\tif resp.StatusCode != 204 {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"request failed with status %d\", resp.StatusCode)}\n\t\t\ts.errors.Add(1)\n\t\t}\n\n\t\twg.Done()\n\t\treturn\n\t}\n\n\t// Parse and return results\n\tvar summary summaryResponse\n\tdecoder := json.NewDecoder(resp.Body)\n\terr = decoder.Decode(&summary)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors.Add(1)\n\t\twg.Done()\n\t\treturn\n\t}\n\n\tfor subdomain := range summary.Summary.Values {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\twg.Done()\n\t\t\treturn\n\t\tdefault:\n\t\t}\n\t\tif !strings.HasSuffix(subdomain, \".\"+domain) {\n\t\t\tcontinue\n\t\t}\n\n\t\tif _, present := dedupe.LoadOrStore(strings.ToLower(subdomain), true); !present {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\twg.Done()\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\ts.results.Add(1)\n\t\t\t}\n\t\t}\n\t}\n\n\t// Complete!\n\twg.Done()\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/facebook/ctlogs.go",
    "content": "package facebook\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/retryablehttp-go\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\t\"github.com/projectdiscovery/utils/generic\"\n\turlutil \"github.com/projectdiscovery/utils/url\"\n)\n\n// source: https://developers.facebook.com/tools/ct\n// api-docs: https://developers.facebook.com/docs/certificate-transparency-api\n// ratelimit: ~20,000 req/hour per appID https://developers.facebook.com/docs/graph-api/overview/rate-limiting/\n\nvar (\n\tdomainsPerPage = \"1000\"\n\tauthUrl        = \"https://graph.facebook.com/oauth/access_token?client_id=%s&client_secret=%s&grant_type=client_credentials\"\n\tdomainsUrl     = \"https://graph.facebook.com/certificates?fields=domains&access_token=%s&query=%s&limit=\" + domainsPerPage\n)\n\ntype apiKey struct {\n\tAppID       string\n\tSecret      string\n\tAccessToken string // obtained by calling\n\t// https://graph.facebook.com/oauth/access_token?client_id=APP_ID&client_secret=APP_SECRET&grant_type=client_credentials\n\tError error // error while fetching access token\n}\n\n// FetchAccessToken fetches the access token for the api key\n// using app id and secret\nfunc (k *apiKey) FetchAccessToken() {\n\tif generic.EqualsAny(\"\", k.AppID, k.Secret) {\n\t\tk.Error = fmt.Errorf(\"invalid app id or secret\")\n\t\treturn\n\t}\n\tresp, err := retryablehttp.Get(fmt.Sprintf(authUrl, k.AppID, k.Secret))\n\tif err != nil {\n\t\tk.Error = err\n\t\treturn\n\t}\n\tdefer func() {\n\t\tif err := resp.Body.Close(); err != nil {\n\t\t\tgologger.Error().Msgf(\"error closing response body: %s\", err)\n\t\t}\n\t}()\n\tbin, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\tk.Error = err\n\t\treturn\n\t}\n\tauth := &authResponse{}\n\tif err := json.Unmarshal(bin, auth); err != nil {\n\t\tk.Error = err\n\t\treturn\n\t}\n\tif auth.AccessToken == \"\" {\n\t\tk.Error = fmt.Errorf(\"invalid response from facebook got %v\", string(bin))\n\t\treturn\n\t}\n\tk.AccessToken = auth.AccessToken\n}\n\n// IsValid returns true if the api key is valid\nfunc (k *apiKey) IsValid() bool {\n\treturn k.AccessToken != \"\"\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []apiKey\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tif len(s.apiKeys) == 0 {\n\t\ts.skipped = true\n\t\tclose(results)\n\t\treturn results\n\t}\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\tkey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tdomainsURL := fmt.Sprintf(domainsUrl, key.AccessToken, domain)\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\ts.requests++\n\t\t\tresp, err := session.Get(ctx, domainsURL, \"\", nil)\n\t\t\tif err != nil {\n\t\t\t\ts.errors++\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\treturn\n\t\t\t}\n\t\t\tbin, err := io.ReadAll(resp.Body)\n\t\t\tif err != nil {\n\t\t\t\ts.errors++\n\t\t\t\tgologger.Verbose().Msgf(\"failed to read response body: %s\\n\", err)\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\treturn\n\t\t\t}\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\tresponse := &response{}\n\t\t\tif err := json.Unmarshal(bin, response); err != nil {\n\t\t\t\ts.errors++\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"failed to unmarshal response: %s: %w\", string(bin), err)}\n\t\t\t\treturn\n\t\t\t}\n\t\t\tfor _, v := range response.Data {\n\t\t\t\tfor _, domain := range v.Domains {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: domain}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t\tif response.Paging.Next == \"\" {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tdomainsURL = updateParamInURL(response.Paging.Next, \"limit\", domainsPerPage)\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"facebook\"\n}\n\n// IsDefault returns true if the source should be queried by default\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\n// accepts subdomains (e.g. subdomain.domain.tld)\n// but also returns all SANs for a certificate which may not match the domain\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\n// KeyRequirement returns the API key requirement level for this source\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\n// NeedsKey returns true if the source requires an API key\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\n// AddApiKeys adds api keys to the source\nfunc (s *Source) AddApiKeys(keys []string) {\n\tallapikeys := subscraping.CreateApiKeys(keys, func(k, v string) apiKey {\n\t\tapiKey := apiKey{AppID: k, Secret: v}\n\t\tapiKey.FetchAccessToken()\n\t\tif apiKey.Error != nil {\n\t\t\tgologger.Warning().Msgf(\"Could not fetch access token for %s: %s\\n\", k, apiKey.Error)\n\t\t}\n\t\treturn apiKey\n\t})\n\t// filter out invalid keys\n\tfor _, key := range allapikeys {\n\t\tif key.IsValid() {\n\t\t\ts.apiKeys = append(s.apiKeys, key)\n\t\t}\n\t}\n}\n\n// Statistics returns the statistics for the source\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n\nfunc updateParamInURL(url, param, value string) string {\n\turlx, err := urlutil.Parse(url)\n\tif err != nil {\n\t\treturn url\n\t}\n\turlx.Params.Set(param, value)\n\treturn urlx.String()\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/facebook/ctlogs_test.go",
    "content": "package facebook\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"os\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/retryablehttp-go\"\n\t\"github.com/projectdiscovery/utils/generic\"\n)\n\nvar (\n\tfb_API_ID     = \"$FB_APP_ID\"\n\tfb_API_SECRET = \"$FB_APP_SECRET\"\n)\n\nfunc TestFacebookSource(t *testing.T) {\n\tif testing.Short() {\n\t\tt.Skip(\"skipping test in short mode.\")\n\t}\n\n\tupdateWithEnv(&fb_API_ID)\n\tupdateWithEnv(&fb_API_SECRET)\n\tif generic.EqualsAny(\"\", fb_API_ID, fb_API_SECRET) {\n\t\tt.SkipNow()\n\t}\n\tk := apiKey{\n\t\tAppID:  fb_API_ID,\n\t\tSecret: fb_API_SECRET,\n\t}\n\tk.FetchAccessToken()\n\tif k.Error != nil {\n\t\tt.Fatal(k.Error)\n\t}\n\n\tfetchURL := fmt.Sprintf(\"https://graph.facebook.com/certificates?fields=domains&access_token=%s&query=hackerone.com&limit=5\", k.AccessToken)\n\tresp, err := retryablehttp.Get(fetchURL)\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tdefer func() {\n\t\tif err := resp.Body.Close(); err != nil {\n\t\t\tgologger.Error().Msgf(\"error closing response body: %s\", err)\n\t\t}\n\t}()\n\tbin, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\tt.Fatal(err)\n\t}\n\tresponse := &response{}\n\tif err := json.Unmarshal(bin, response); err != nil {\n\t\tt.Fatal(err)\n\t}\n\tif len(response.Data) == 0 {\n\t\tt.Fatal(\"no data found\")\n\t}\n}\n\nfunc updateWithEnv(key *string) {\n\tif key == nil {\n\t\treturn\n\t}\n\tvalue := *key\n\tif strings.HasPrefix(value, \"$\") {\n\t\t*key = os.Getenv(value[1:])\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/facebook/types.go",
    "content": "package facebook\n\ntype authResponse struct {\n\tAccessToken string `json:\"access_token\"`\n}\n\n/*\n{\n  \"data\": [\n    {\n      \"domains\": [\n        \"docs.hackerone.com\"\n      ],\n      \"id\": \"10056051421102939\"\n    },\n   ...\n  ],\n  \"paging\": {\n    \"cursors\": {\n      \"before\": \"MTAwNTYwNTE0MjExMDI5MzkZD\",\n      \"after\": \"Njc0OTczNTA5NTA1MzUxNwZDZD\"\n    },\n    \"next\": \"https://graph.facebook.com/v17.0/certificates?fields=domains&access_token=6161176097324222|fzhUp9I0eXa456Ye21zAhyYVozk&query=hackerone.com&limit=25&after=Njc0OTczNTA5NTA1MzUxNwZDZD\"\n  }\n}\n*/\n// example response\n\ntype response struct {\n\tData []struct {\n\t\tDomains []string `json:\"domains\"`\n\t} `json:\"data\"`\n\tPaging struct {\n\t\tNext string `json:\"next\"`\n\t} `json:\"paging\"`\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/fofa/fofa.go",
    "content": "// Package fofa logic\npackage fofa\n\nimport (\n\t\"context\"\n\t\"encoding/base64\"\n\t\"fmt\"\n\t\"regexp\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype fofaResponse struct {\n\tError   bool     `json:\"error\"`\n\tErrMsg  string   `json:\"errmsg\"`\n\tSize    int      `json:\"size\"`\n\tResults []string `json:\"results\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []apiKey\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype apiKey struct {\n\tusername string\n\tsecret   string\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey.username == \"\" || randomApiKey.secret == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\t// fofa api doc https://fofa.info/static_pages/api_help\n\t\tqbase64 := base64.StdEncoding.EncodeToString(fmt.Appendf(nil, \"domain=\\\"%s\\\"\", domain))\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://fofa.info/api/v1/search/all?full=true&fields=host&page=1&size=10000&email=%s&key=%s&qbase64=%s\", randomApiKey.username, randomApiKey.secret, qbase64))\n\t\tif err != nil && resp == nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar response fofaResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tif response.Error {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tSource: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%s\", response.ErrMsg),\n\t\t\t}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tif response.Size > 0 {\n\t\t\tfor _, subdomain := range response.Results {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tif strings.HasPrefix(strings.ToLower(subdomain), \"http://\") || strings.HasPrefix(strings.ToLower(subdomain), \"https://\") {\n\t\t\t\t\tsubdomain = subdomain[strings.Index(subdomain, \"//\")+2:]\n\t\t\t\t}\n\t\t\t\tre := regexp.MustCompile(`:\\d+$`)\n\t\t\t\tif re.MatchString(subdomain) {\n\t\t\t\t\tsubdomain = re.ReplaceAllString(subdomain, \"\")\n\t\t\t\t}\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"fofa\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = subscraping.CreateApiKeys(keys, func(k, v string) apiKey {\n\t\treturn apiKey{k, v}\n\t})\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/fullhunt/fullhunt.go",
    "content": "package fullhunt\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// fullhunt response\ntype fullHuntResponse struct {\n\tHosts   []string `json:\"hosts\"`\n\tMessage string   `json:\"message\"`\n\tStatus  int      `json:\"status\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, fmt.Sprintf(\"https://fullhunt.io/api/v1/domain/%s/subdomains\", domain), \"\", map[string]string{\"X-API-KEY\": randomApiKey})\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar response fullHuntResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\t\tfor _, record := range response.Hosts {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"fullhunt\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/github/github.go",
    "content": "// Package github GitHub search package\n// Based on gwen001's https://github.com/gwen001/github-search github-subdomains\npackage github\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/tomnomnom/linkheader\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype textMatch struct {\n\tFragment string `json:\"fragment\"`\n}\n\ntype item struct {\n\tName        string      `json:\"name\"`\n\tHTMLURL     string      `json:\"html_url\"`\n\tTextMatches []textMatch `json:\"text_matches\"`\n}\n\ntype response struct {\n\tTotalCount int    `json:\"total_count\"`\n\tItems      []item `json:\"items\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    atomic.Int32\n\tresults   atomic.Int32\n\trequests  atomic.Int32\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors.Store(0)\n\ts.results.Store(0)\n\ts.requests.Store(0)\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\tif len(s.apiKeys) == 0 {\n\t\t\tgologger.Debug().Msgf(\"Cannot use the %s source because there was no key defined for it.\", s.Name())\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ttokens := NewTokenManager(s.apiKeys)\n\n\t\tsearchURL := fmt.Sprintf(\"https://api.github.com/search/code?per_page=100&q=%s&sort=created&order=asc\", domain)\n\t\ts.enumerate(ctx, searchURL, domainRegexp(domain), tokens, session, results)\n\t}()\n\n\treturn results\n}\n\nfunc (s *Source) enumerate(ctx context.Context, searchURL string, domainRegexp *regexp.Regexp, tokens *Tokens, session *subscraping.Session, results chan subscraping.Result) {\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn\n\tdefault:\n\t}\n\n\ttoken := tokens.Get()\n\theaders := map[string]string{\n\t\t\"Accept\": \"application/vnd.github.v3.text-match+json\", \"Authorization\": \"token \" + token.Hash,\n\t}\n\n\t// Initial request to GitHub search\n\ts.requests.Add(1)\n\tresp, err := session.Get(ctx, searchURL, \"\", headers)\n\tisForbidden := resp != nil && resp.StatusCode == http.StatusForbidden\n\tif err != nil && !isForbidden {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors.Add(1)\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn\n\t}\n\n\t// Retry enumerarion after Retry-After seconds on rate limit abuse detected\n\tratelimitRemaining, _ := strconv.ParseInt(resp.Header.Get(\"X-Ratelimit-Remaining\"), 10, 64)\n\tif isForbidden && ratelimitRemaining == 0 {\n\t\tretryAfterSeconds, _ := strconv.ParseInt(resp.Header.Get(\"Retry-After\"), 10, 64)\n\t\ttokens.setCurrentTokenExceeded(retryAfterSeconds)\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\ts.enumerate(ctx, searchURL, domainRegexp, tokens, session, results)\n\t}\n\n\tvar data response\n\n\t// Marshall json response\n\terr = jsoniter.NewDecoder(resp.Body).Decode(&data)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors.Add(1)\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn\n\t}\n\n\tsession.DiscardHTTPResponse(resp)\n\n\terr = s.proccesItems(ctx, data.Items, domainRegexp, s.Name(), session, results)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors.Add(1)\n\t\treturn\n\t}\n\n\t// Links header, first, next, last...\n\tlinksHeader := linkheader.Parse(resp.Header.Get(\"Link\"))\n\t// Process the next link recursively\n\tfor _, link := range linksHeader {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn\n\t\tdefault:\n\t\t}\n\t\tif link.Rel == \"next\" {\n\t\t\tnextURL, err := url.QueryUnescape(link.URL)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors.Add(1)\n\t\t\t\treturn\n\t\t\t}\n\t\t\ts.enumerate(ctx, nextURL, domainRegexp, tokens, session, results)\n\t\t}\n\t}\n}\n\n// proccesItems process github response items\nfunc (s *Source) proccesItems(ctx context.Context, items []item, domainRegexp *regexp.Regexp, name string, session *subscraping.Session, results chan subscraping.Result) error {\n\tvar wg sync.WaitGroup\n\terrChan := make(chan error, len(items))\n\n\tfor _, responseItem := range items {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\tdefault:\n\t\t}\n\t\twg.Add(1)\n\t\tgo func(responseItem item) {\n\t\t\tdefer wg.Done()\n\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\n\t\t\ts.requests.Add(1)\n\t\t\tresp, err := session.SimpleGet(ctx, rawURL(responseItem.HTMLURL))\n\t\t\tif err != nil {\n\t\t\t\tif resp != nil && resp.StatusCode != http.StatusNotFound {\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t}\n\t\t\t\terrChan <- err\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tif resp.StatusCode == http.StatusOK {\n\t\t\t\tscanner := bufio.NewScanner(resp.Body)\n\t\t\t\tfor scanner.Scan() {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\t\treturn\n\t\t\t\t\tdefault:\n\t\t\t\t\t}\n\t\t\t\t\tline := scanner.Text()\n\t\t\t\t\tif line == \"\" {\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\t\t\t\t\tfor _, subdomain := range domainRegexp.FindAllString(normalizeContent(line), -1) {\n\t\t\t\t\t\tselect {\n\t\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\t\t\treturn\n\t\t\t\t\t\tcase results <- subscraping.Result{Source: name, Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\t\t\ts.results.Add(1)\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t}\n\n\t\t\tfor _, textMatch := range responseItem.TextMatches {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tfor _, subdomain := range domainRegexp.FindAllString(normalizeContent(textMatch.Fragment), -1) {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: name, Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\t\ts.results.Add(1)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}(responseItem)\n\t}\n\n\twg.Wait()\n\tclose(errChan)\n\n\tfor err := range errChan {\n\t\tif err != nil {\n\t\t\treturn err\n\t\t}\n\t}\n\n\treturn nil\n}\n\n// Normalize content before matching, query unescape, remove tabs and new line chars\nfunc normalizeContent(content string) string {\n\tnormalizedContent, _ := url.QueryUnescape(content)\n\tnormalizedContent = strings.ReplaceAll(normalizedContent, \"\\\\t\", \"\")\n\tnormalizedContent = strings.ReplaceAll(normalizedContent, \"\\\\n\", \"\")\n\treturn normalizedContent\n}\n\n// Raw URL to get the files code and match for subdomains\nfunc rawURL(htmlURL string) string {\n\tdomain := strings.ReplaceAll(htmlURL, \"https://github.com/\", \"https://raw.githubusercontent.com/\")\n\treturn strings.ReplaceAll(domain, \"/blob/\", \"/\")\n}\n\n// DomainRegexp regular expression to match subdomains in github files code\nfunc domainRegexp(domain string) *regexp.Regexp {\n\trdomain := strings.ReplaceAll(domain, \".\", \"\\\\.\")\n\treturn regexp.MustCompile(\"(\\\\w[a-zA-Z0-9][a-zA-Z0-9-\\\\.]*)\" + rdomain)\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"github\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    int(s.errors.Load()),\n\t\tResults:   int(s.results.Load()),\n\t\tRequests:  int(s.requests.Load()),\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/github/tokenmanager.go",
    "content": "package github\n\nimport \"time\"\n\n// Token struct\ntype Token struct {\n\tHash         string\n\tRetryAfter   int64\n\tExceededTime time.Time\n}\n\n// Tokens is the internal struct to manage the current token\n// and the pool\ntype Tokens struct {\n\tcurrent int\n\tpool    []Token\n}\n\n// NewTokenManager initialize the tokens pool\nfunc NewTokenManager(keys []string) *Tokens {\n\tpool := []Token{}\n\tfor _, key := range keys {\n\t\tt := Token{Hash: key, ExceededTime: time.Time{}, RetryAfter: 0}\n\t\tpool = append(pool, t)\n\t}\n\n\treturn &Tokens{\n\t\tcurrent: 0,\n\t\tpool:    pool,\n\t}\n}\n\nfunc (r *Tokens) setCurrentTokenExceeded(retryAfter int64) {\n\tif r.current >= len(r.pool) {\n\t\tr.current %= len(r.pool)\n\t}\n\tif r.pool[r.current].RetryAfter == 0 {\n\t\tr.pool[r.current].ExceededTime = time.Now()\n\t\tr.pool[r.current].RetryAfter = retryAfter\n\t}\n}\n\n// Get returns a new token from the token pool\nfunc (r *Tokens) Get() *Token {\n\tresetExceededTokens(r)\n\n\tif r.current >= len(r.pool) {\n\t\tr.current %= len(r.pool)\n\t}\n\n\tresult := &r.pool[r.current]\n\tr.current++\n\n\treturn result\n}\n\nfunc resetExceededTokens(r *Tokens) {\n\tfor i, token := range r.pool {\n\t\tif token.RetryAfter > 0 {\n\t\t\tif int64(time.Since(token.ExceededTime)/time.Second) > token.RetryAfter {\n\t\t\t\tr.pool[i].ExceededTime = time.Time{}\n\t\t\t\tr.pool[i].RetryAfter = 0\n\t\t\t}\n\t\t}\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/gitlab/gitlab.go",
    "content": "package gitlab\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"regexp\"\n\t\"strings\"\n\t\"sync\"\n\t\"sync/atomic\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\t\"github.com/tomnomnom/linkheader\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    atomic.Int32\n\tresults   atomic.Int32\n\trequests  atomic.Int32\n\tskipped   bool\n}\n\ntype item struct {\n\tData      string `json:\"data\"`\n\tProjectId int    `json:\"project_id\"`\n\tPath      string `json:\"path\"`\n\tRef       string `json:\"ref\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors.Store(0)\n\ts.results.Store(0)\n\ts.requests.Store(0)\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\"PRIVATE-TOKEN\": randomApiKey}\n\n\t\tsearchURL := fmt.Sprintf(\"https://gitlab.com/api/v4/search?scope=blobs&search=%s&per_page=100\", domain)\n\t\ts.enumerate(ctx, searchURL, domainRegexp(domain), headers, session, results)\n\n\t}()\n\n\treturn results\n}\n\nfunc (s *Source) enumerate(ctx context.Context, searchURL string, domainRegexp *regexp.Regexp, headers map[string]string, session *subscraping.Session, results chan subscraping.Result) {\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn\n\tdefault:\n\t}\n\n\ts.requests.Add(1)\n\tresp, err := session.Get(ctx, searchURL, \"\", headers)\n\tif err != nil && resp == nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors.Add(1)\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn\n\t}\n\n\tdefer session.DiscardHTTPResponse(resp)\n\n\tvar items []item\n\terr = jsoniter.NewDecoder(resp.Body).Decode(&items)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\ts.errors.Add(1)\n\t\treturn\n\t}\n\n\tvar wg sync.WaitGroup\n\twg.Add(len(items))\n\n\tfor _, it := range items {\n\t\tgo func(item item) {\n\t\t\t// The original item.Path causes 404 error because the Gitlab API is expecting the url encoded path\n\t\t\tfileUrl := fmt.Sprintf(\"https://gitlab.com/api/v4/projects/%d/repository/files/%s/raw?ref=%s\", item.ProjectId, url.QueryEscape(item.Path), item.Ref)\n\t\t\ts.requests.Add(1)\n\t\t\tresp, err := session.Get(ctx, fileUrl, \"\", headers)\n\t\t\tif err != nil {\n\t\t\t\tif resp == nil || (resp != nil && resp.StatusCode != http.StatusNotFound) {\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors.Add(1)\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tif resp.StatusCode == http.StatusOK {\n\t\t\t\tscanner := bufio.NewScanner(resp.Body)\n\t\t\t\tfor scanner.Scan() {\n\t\t\t\t\tline := scanner.Text()\n\t\t\t\t\tif line == \"\" {\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\t\t\t\t\tfor _, subdomain := range domainRegexp.FindAllString(line, -1) {\n\t\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}\n\t\t\t\t\t\ts.results.Add(1)\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t}\n\t\t\tdefer wg.Done()\n\t\t}(it)\n\t}\n\n\tlinksHeader := linkheader.Parse(resp.Header.Get(\"Link\"))\n\tfor _, link := range linksHeader {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn\n\t\tdefault:\n\t\t}\n\t\tif link.Rel == \"next\" {\n\t\t\tnextURL, err := url.QueryUnescape(link.URL)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors.Add(1)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\ts.enumerate(ctx, nextURL, domainRegexp, headers, session, results)\n\t\t}\n\t}\n\n\twg.Wait()\n}\n\nfunc domainRegexp(domain string) *regexp.Regexp {\n\trdomain := strings.ReplaceAll(domain, \".\", \"\\\\.\")\n\treturn regexp.MustCompile(\"(\\\\w[a-zA-Z0-9][a-zA-Z0-9-\\\\.]*)\" + rdomain)\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"gitlab\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\n// Statistics returns the statistics for the source\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    int(s.errors.Load()),\n\t\tResults:   int(s.results.Load()),\n\t\tRequests:  int(s.requests.Load()),\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/hackertarget/hackertarget.go",
    "content": "// Package hackertarget logic\npackage hackertarget\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\thtSearchUrl := fmt.Sprintf(\"https://api.hackertarget.com/hostsearch/?q=%s\", domain)\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey != \"\" {\n\t\t\thtSearchUrl = fmt.Sprintf(\"%s&apikey=%s\", htSearchUrl, randomApiKey)\n\t\t}\n\n\t\thtSearchUrl = fmt.Sprintf(\"%s&apikey=%s\", htSearchUrl, randomApiKey)\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, htSearchUrl)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tscanner := bufio.NewScanner(resp.Body)\n\t\tfor scanner.Scan() {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tline := scanner.Text()\n\t\t\tif line == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tmatch := session.Extractor.Extract(line)\n\t\t\tfor _, subdomain := range match {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"hackertarget\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.OptionalKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/hudsonrock/hudsonrock.go",
    "content": "// Package hudsonrock logic\npackage hudsonrock\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype hudsonrockResponse struct {\n\tData struct {\n\t\tEmployeesUrls []struct {\n\t\t\tURL string `json:\"url\"`\n\t\t} `json:\"employees_urls\"`\n\t\tClientsUrls []struct {\n\t\t\tURL string `json:\"url\"`\n\t\t} `json:\"clients_urls\"`\n\t} `json:\"data\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://cavalier.hudsonrock.com/api/json/v2/osint-tools/urls-by-domain?domain=%s\", domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\tvar response hudsonrockResponse\n\t\terr = json.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tfor _, record := range append(response.Data.EmployeesUrls, response.Data.ClientsUrls...) {\n\t\t\tfor _, subdomain := range session.Extractor.Extract(record.URL) {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"hudsonrock\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/intelx/intelx.go",
    "content": "// Package intelx logic\npackage intelx\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype searchResponseType struct {\n\tID     string `json:\"id\"`\n\tStatus int    `json:\"status\"`\n}\n\ntype selectorType struct {\n\tSelectvalue string `json:\"selectorvalue\"`\n}\n\ntype searchResultType struct {\n\tSelectors []selectorType `json:\"selectors\"`\n\tStatus    int            `json:\"status\"`\n}\n\ntype requestBody struct {\n\tTerm       string\n\tMaxresults int\n\tMedia      int\n\tTarget     int\n\tTerminate  []int\n\tTimeout    int\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []apiKey\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype apiKey struct {\n\thost string\n\tkey  string\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey.host == \"\" || randomApiKey.key == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tsearchURL := fmt.Sprintf(\"https://%s/phonebook/search?k=%s\", randomApiKey.host, randomApiKey.key)\n\t\treqBody := requestBody{\n\t\t\tTerm:       domain,\n\t\t\tMaxresults: 100000,\n\t\t\tMedia:      0,\n\t\t\tTarget:     1,\n\t\t\tTimeout:    20,\n\t\t}\n\n\t\tbody, err := json.Marshal(reqBody)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.SimplePost(ctx, searchURL, \"application/json\", bytes.NewBuffer(body))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar response searchResponseType\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tresultsURL := fmt.Sprintf(\"https://%s/phonebook/search/result?k=%s&id=%s&limit=10000\", randomApiKey.host, randomApiKey.key, response.ID)\n\t\tstatus := 0\n\t\tfor status == 0 || status == 3 {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\ts.requests++\n\t\t\tresp, err = session.Get(ctx, resultsURL, \"\", nil)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tvar response searchResultType\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\t_, err = io.ReadAll(resp.Body)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\tstatus = response.Status\n\t\t\tfor _, hostname := range response.Selectors {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: hostname.Selectvalue}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"intelx\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = subscraping.CreateApiKeys(keys, func(k, v string) apiKey {\n\t\treturn apiKey{k, v}\n\t})\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/leakix/leakix.go",
    "content": "// Package leakix logic\npackage leakix\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\t\t// Default headers\n\t\theaders := map[string]string{\n\t\t\t\"accept\": \"application/json\",\n\t\t}\n\t\t// Pick an API key\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey != \"\" {\n\t\t\theaders[\"api-key\"] = randomApiKey\n\t\t}\n\t\t// Request\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, \"https://leakix.net/api/subdomains/\"+domain, \"\", headers)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\tif resp.StatusCode != 200 {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"request failed with status %d\", resp.StatusCode)}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\t// Parse and return results\n\t\tvar subdomains []subResponse\n\t\tdecoder := json.NewDecoder(resp.Body)\n\t\terr = decoder.Decode(&subdomains)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tfor _, result := range subdomains {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: result.Subdomain}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"leakix\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n\ntype subResponse struct {\n\tSubdomain   string    `json:\"subdomain\"`\n\tDistinctIps int       `json:\"distinct_ips\"`\n\tLastSeen    time.Time `json:\"last_seen\"`\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/merklemap/merklemap.go",
    "content": "// Package merklemap logic\npackage merklemap\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"math\"\n\t\"net/url\"\n\t\"strconv\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\t\t// Pick an API key, skip if no key is found\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\t// Default headers\n\t\theaders := map[string]string{\n\t\t\t\"accept\": \"application/json\",\n\t\t\t// Set a user agent to prevent random one from pkg/subscraping/agent.go, it triggers the cloudflare protection of the api\n\t\t\t\"User-Agent\":    \"subfinder\",\n\t\t\t\"Authorization\": \"Bearer \" + randomApiKey,\n\t\t}\n\n\t\t// Fetch all pages with pagination\n\t\t// https://www.merklemap.com/documentation/search\n\t\ts.fetchAllPages(ctx, domain, headers, session, results)\n\t}()\n\treturn results\n}\n\n// fetchAllPages fetches all pages of results using pagination\nfunc (s *Source) fetchAllPages(ctx context.Context, domain string, headers map[string]string, session *subscraping.Session, results chan subscraping.Result) {\n\tbaseURL := \"https://api.merklemap.com/v1/search?query=\" + url.QueryEscape(\"*.\"+domain)\n\ttotalCount := math.MaxInt\n\tprocessedResults := 0\n\n\t// Iterate through all pages\n\tfor page := 0; processedResults < totalCount; page++ {\n\t\tpageResp, err := s.fetchPage(ctx, baseURL, page, headers, session)\n\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tif page == 0 {\n\t\t\ttotalCount = pageResp.Count\n\t\t}\n\n\t\t// Stop if this page returned no results\n\t\tif len(pageResp.Results) == 0 {\n\t\t\tbreak\n\t\t}\n\n\t\tfor _, result := range pageResp.Results {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tSource: s.Name(), Type: subscraping.Subdomain, Value: result.Hostname,\n\t\t\t}\n\t\t\ts.results++\n\t\t\tprocessedResults++\n\t\t}\n\n\t}\n}\n\n// fetchPage fetches a single page of results\nfunc (s *Source) fetchPage(ctx context.Context, baseURL string, page int, headers map[string]string, session *subscraping.Session) (*response, error) {\n\turl := baseURL + \"&page=\" + strconv.Itoa(page)\n\n\ts.requests++\n\tresp, err := session.Get(ctx, url, \"\", headers)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tdefer session.DiscardHTTPResponse(resp)\n\n\tif resp.StatusCode != 200 {\n\t\trespBody, err := io.ReadAll(resp.Body)\n\t\tif err != nil {\n\t\t\treturn nil, fmt.Errorf(\"request failed with status %d: %s\", resp.StatusCode, err)\n\t\t}\n\t\treturn nil, fmt.Errorf(\"request failed with status %d: %s\", resp.StatusCode, string(respBody))\n\t}\n\n\tvar pageResponse response\n\trespBody, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tdecoder := json.NewDecoder(bytes.NewReader(respBody))\n\tif err := decoder.Decode(&pageResponse); err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &pageResponse, nil\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"merklemap\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\n// HasRecursiveSupport indicates that we accept subdomains in addition to apex domains\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n\ntype response struct {\n\tCount   int `json:\"count\"`\n\tResults []struct {\n\t\tHostname          string `json:\"hostname\"`\n\t\tSubjectCommonName string `json:\"subject_common_name\"`\n\t\tFirstSeen         string `json:\"first_seen\"`\n\t} `json:\"results\"`\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/netlas/netlas.go",
    "content": "// Package netlas logic\npackage netlas\n\nimport (\n\t\"context\"\n\t\"io\"\n\t\"strings\"\n\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype Item struct {\n\tData struct {\n\t\tA           []string `json:\"a,omitempty\"`\n\t\tTxt         []string `json:\"txt,omitempty\"`\n\t\tLastUpdated string   `json:\"last_updated,omitempty\"`\n\t\tTimestamp   string   `json:\"@timestamp,omitempty\"`\n\t\tNs          []string `json:\"ns,omitempty\"`\n\t\tLevel       int      `json:\"level,omitempty\"`\n\t\tZone        string   `json:\"zone,omitempty\"`\n\t\tDomain      string   `json:\"domain,omitempty\"`\n\t\tCname       []string `json:\"cname,omitempty\"`\n\t\tMx          []string `json:\"mx,omitempty\"`\n\t} `json:\"data\"`\n}\n\ntype DomainsCountResponse struct {\n\tCount int `json:\"count\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\t// To get count of domains\n\t\tendpoint := \"https://app.netlas.io/api/domains_count/\"\n\t\tparams := url.Values{}\n\t\tcountQuery := fmt.Sprintf(\"domain:*.%s AND NOT domain:%s\", domain, domain)\n\t\tparams.Set(\"q\", countQuery)\n\t\tcountUrl := endpoint + \"?\" + params.Encode()\n\n\t\t// Pick an API key\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\ts.requests++\n\t\tresp1, err := session.HTTPRequest(ctx, http.MethodGet, countUrl, \"\", map[string]string{\n\t\t\t\"accept\":    \"application/json\",\n\t\t\t\"X-API-Key\": randomApiKey,\n\t\t}, nil, subscraping.BasicAuth{})\n\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t} else if resp1.StatusCode != 200 {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"request rate limited with status code %d\", resp1.StatusCode)}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tdefer func() {\n\t\t\tif err := resp1.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tbody, err := io.ReadAll(resp1.Body)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"error reading ressponse body\")}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\t// Parse the JSON response\n\t\tvar domainsCount DomainsCountResponse\n\t\terr = json.Unmarshal(body, &domainsCount)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\t// Make a single POST request to get all domains via download method\n\n\t\tapiUrl := \"https://app.netlas.io/api/domains/download/\"\n\t\tquery := fmt.Sprintf(\"domain:*.%s AND NOT domain:%s\", domain, domain)\n\t\trequestBody := map[string]any{\n\t\t\t\"q\":           query,\n\t\t\t\"fields\":      []string{\"*\"},\n\t\t\t\"source_type\": \"include\",\n\t\t\t\"size\":        domainsCount.Count,\n\t\t}\n\t\tjsonRequestBody, err := json.Marshal(requestBody)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"error marshaling request body\")}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\t// Pick an API key\n\t\trandomApiKey = subscraping.PickRandom(s.apiKeys, s.Name())\n\n\t\ts.requests++\n\t\tresp2, err := session.HTTPRequest(ctx, http.MethodPost, apiUrl, \"\", map[string]string{\n\t\t\t\"accept\":       \"application/json\",\n\t\t\t\"X-API-Key\":    randomApiKey,\n\t\t\t\"Content-Type\": \"application/json\"}, strings.NewReader(string(jsonRequestBody)), subscraping.BasicAuth{})\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tdefer func() {\n\t\t\tif err := resp2.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\t\tbody, err = io.ReadAll(resp2.Body)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"error reading ressponse body\")}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tif resp2.StatusCode == 429 {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"request rate limited with status code %d\", resp2.StatusCode)}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\t// Parse the response body and extract the domain values\n\t\tvar data []Item\n\t\terr = json.Unmarshal(body, &data)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, item := range data {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: item.Data.Domain}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"netlas\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/onyphe/onyphe.go",
    "content": "// Package onyphe logic\npackage onyphe\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"strconv\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype OnypheResponse struct {\n\tError    int      `json:\"error\"`\n\tResults  []Result `json:\"results\"`\n\tPage     int      `json:\"page\"`\n\tPageSize int      `json:\"page_size\"`\n\tTotal    int      `json:\"total\"`\n\tMaxPage  int      `json:\"max_page\"`\n}\n\ntype Result struct {\n\tSubdomains []string `json:\"subdomains\"`\n\tHostname   string   `json:\"hostname\"`\n\tForward    string   `json:\"forward\"`\n\tReverse    string   `json:\"reverse\"`\n\tHost       string   `json:\"host\"`\n\tDomain     string   `json:\"domain\"`\n}\n\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\"Content-Type\": \"application/json\", \"Authorization\": \"bearer \" + randomApiKey}\n\n\t\tpage := 1\n\t\tpageSize := 1000\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tvar resp *http.Response\n\t\t\tvar err error\n\n\t\t\turlWithQuery := fmt.Sprintf(\"https://www.onyphe.io/api/v2/search/?q=%s&page=%d&size=%d\",\n\t\t\t\turl.QueryEscape(\"category:resolver domain:\"+domain), page, pageSize)\n\t\t\ts.requests++\n\t\t\tresp, err = session.Get(ctx, urlWithQuery, \"\", headers)\n\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar respOnyphe OnypheResponse\n\t\t\terr = json.NewDecoder(resp.Body).Decode(&respOnyphe)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\tfor _, record := range respOnyphe.Results {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tfor _, subdomain := range record.Subdomains {\n\t\t\t\t\tif subdomain != \"\" {\n\t\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\tif record.Hostname != \"\" {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Hostname}\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\n\t\t\t\tif record.Forward != \"\" {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Forward}\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\n\t\t\t\tif record.Reverse != \"\" {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Reverse}\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tif len(respOnyphe.Results) == 0 || page >= respOnyphe.MaxPage {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\tpage++\n\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"onyphe\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n\ntype OnypheResponseRaw struct {\n\tError    int             `json:\"error\"`\n\tResults  []Result        `json:\"results\"`\n\tPage     json.RawMessage `json:\"page\"`\n\tPageSize json.RawMessage `json:\"page_size\"`\n\tTotal    json.RawMessage `json:\"total\"`\n\tMaxPage  json.RawMessage `json:\"max_page\"`\n}\n\nfunc (o *OnypheResponse) UnmarshalJSON(data []byte) error {\n\tvar raw OnypheResponseRaw\n\tif err := json.Unmarshal(data, &raw); err != nil {\n\t\treturn err\n\t}\n\n\to.Error = raw.Error\n\to.Results = raw.Results\n\n\tif pageStr := string(raw.Page); pageStr != \"\" {\n\t\tif page, err := strconv.Atoi(pageStr); err == nil {\n\t\t\to.Page = page\n\t\t} else {\n\t\t\tvar pageStrQuoted string\n\t\t\tif err := json.Unmarshal(raw.Page, &pageStrQuoted); err == nil {\n\t\t\t\tif page, err := strconv.Atoi(pageStrQuoted); err == nil {\n\t\t\t\t\to.Page = page\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tif pageSizeStr := string(raw.PageSize); pageSizeStr != \"\" {\n\t\tif pageSize, err := strconv.Atoi(pageSizeStr); err == nil {\n\t\t\to.PageSize = pageSize\n\t\t} else {\n\t\t\tvar pageSizeStrQuoted string\n\t\t\tif err := json.Unmarshal(raw.PageSize, &pageSizeStrQuoted); err == nil {\n\t\t\t\tif pageSize, err := strconv.Atoi(pageSizeStrQuoted); err == nil {\n\t\t\t\t\to.PageSize = pageSize\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tif totalStr := string(raw.Total); totalStr != \"\" {\n\t\tif total, err := strconv.Atoi(totalStr); err == nil {\n\t\t\to.Total = total\n\t\t} else {\n\t\t\tvar totalStrQuoted string\n\t\t\tif err := json.Unmarshal(raw.Total, &totalStrQuoted); err == nil {\n\t\t\t\tif total, err := strconv.Atoi(totalStrQuoted); err == nil {\n\t\t\t\t\to.Total = total\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tif maxPageStr := string(raw.MaxPage); maxPageStr != \"\" {\n\t\tif maxPage, err := strconv.Atoi(maxPageStr); err == nil {\n\t\t\to.MaxPage = maxPage\n\t\t} else {\n\t\t\tvar maxPageStrQuoted string\n\t\t\tif err := json.Unmarshal(raw.MaxPage, &maxPageStrQuoted); err == nil {\n\t\t\t\tif maxPage, err := strconv.Atoi(maxPageStrQuoted); err == nil {\n\t\t\t\t\to.MaxPage = maxPage\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\treturn nil\n}\n\ntype ResultRaw struct {\n\tSubdomains json.RawMessage `json:\"subdomains\"`\n\tHostname   json.RawMessage `json:\"hostname\"`\n\tForward    json.RawMessage `json:\"forward\"`\n\tReverse    json.RawMessage `json:\"reverse\"`\n\tHost       json.RawMessage `json:\"host\"`\n\tDomain     json.RawMessage `json:\"domain\"`\n}\n\nfunc (r *Result) UnmarshalJSON(data []byte) error {\n\tvar raw ResultRaw\n\tif err := json.Unmarshal(data, &raw); err != nil {\n\t\treturn err\n\t}\n\n\tvar subdomains []string\n\tif err := json.Unmarshal(raw.Subdomains, &subdomains); err == nil {\n\t\tr.Subdomains = subdomains\n\t} else {\n\t\tvar subdomainStr string\n\t\tif err := json.Unmarshal(raw.Subdomains, &subdomainStr); err == nil {\n\t\t\tr.Subdomains = []string{subdomainStr}\n\t\t}\n\t}\n\n\tif len(raw.Hostname) > 0 {\n\t\tvar hostnameStr string\n\t\tif err := json.Unmarshal(raw.Hostname, &hostnameStr); err == nil {\n\t\t\tr.Hostname = hostnameStr\n\t\t} else {\n\t\t\tvar hostnameArr []string\n\t\t\tif err := json.Unmarshal(raw.Hostname, &hostnameArr); err == nil && len(hostnameArr) > 0 {\n\t\t\t\tr.Hostname = hostnameArr[0]\n\t\t\t}\n\t\t}\n\t}\n\n\tif len(raw.Forward) > 0 {\n\t\t_ = json.Unmarshal(raw.Forward, &r.Forward)\n\t}\n\n\tif len(raw.Reverse) > 0 {\n\t\t_ = json.Unmarshal(raw.Reverse, &r.Reverse)\n\t}\n\n\tif len(raw.Host) > 0 {\n\t\tvar hostStr string\n\t\tif err := json.Unmarshal(raw.Host, &hostStr); err == nil {\n\t\t\tr.Host = hostStr\n\t\t} else {\n\t\t\tvar hostArr []string\n\t\t\tif err := json.Unmarshal(raw.Host, &hostArr); err == nil && len(hostArr) > 0 {\n\t\t\t\tr.Host = hostArr[0]\n\t\t\t}\n\t\t}\n\t}\n\n\tif len(raw.Domain) > 0 {\n\t\tvar domainStr string\n\t\tif err := json.Unmarshal(raw.Domain, &domainStr); err == nil {\n\t\t\tr.Domain = domainStr\n\t\t} else {\n\t\t\tvar domainArr []string\n\t\t\tif err := json.Unmarshal(raw.Domain, &domainArr); err == nil && len(domainArr) > 0 {\n\t\t\t\tr.Domain = domainArr[0]\n\t\t\t}\n\t\t}\n\t}\n\n\treturn nil\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/profundis/profundis.go",
    "content": "// Package profundis logic\npackage profundis\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\trequestBody, err := json.Marshal(map[string]string{\"domain\": domain})\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t\t\"X-API-KEY\":    randomApiKey,\n\t\t\t\"Accept\":       \"text/event-stream\",\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.Post(ctx, \"https://api.profundis.io/api/v2/common/data/subdomains\", \"\",\n\t\t\theaders, bytes.NewReader(requestBody))\n\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tscanner := bufio.NewScanner(resp.Body)\n\t\tfor scanner.Scan() {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tline := strings.TrimSpace(scanner.Text())\n\t\t\tif line == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: line}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\n\t\tif err := scanner.Err(); err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t}\n\t}()\n\n\treturn results\n}\n\nfunc (s *Source) Name() string {\n\treturn \"profundis\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/pugrecon/pugrecon.go",
    "content": "// Package pugrecon logic\npackage pugrecon\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// pugreconResult stores a single result from the pugrecon API\ntype pugreconResult struct {\n\tName string `json:\"name\"`\n}\n\n// pugreconAPIResponse stores the response from the pugrecon API\ntype pugreconAPIResponse struct {\n\tResults        []pugreconResult `json:\"results\"`\n\tQuotaRemaining int              `json:\"quota_remaining\"`\n\tLimited        bool             `json:\"limited\"`\n\tTotalResults   int              `json:\"total_results\"`\n\tMessage        string           `json:\"message\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\t// Prepare POST request data\n\t\tpostData := map[string]string{\"domain_name\": domain}\n\t\tbodyBytes, err := json.Marshal(postData)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"failed to marshal request body: %w\", err)}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tbodyReader := bytes.NewReader(bodyBytes)\n\n\t\t// Prepare headers\n\t\theaders := map[string]string{\n\t\t\t\"Authorization\": \"Bearer \" + randomApiKey,\n\t\t\t\"Content-Type\":  \"application/json\",\n\t\t\t\"Accept\":        \"application/json\",\n\t\t}\n\n\t\tapiURL := \"https://pugrecon.com/api/v1/domains\"\n\t\ts.requests++\n\t\tresp, err := session.HTTPRequest(ctx, http.MethodPost, apiURL, \"\", headers, bodyReader, subscraping.BasicAuth{}) // Use HTTPRequest for full header control\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"failed to close response body: %w\", err)}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tif resp.StatusCode != http.StatusOK {\n\t\t\terrorMsg := fmt.Sprintf(\"received status code %d\", resp.StatusCode)\n\t\t\t// Attempt to read error message from body if possible\n\t\t\tvar apiResp pugreconAPIResponse\n\t\t\tif json.NewDecoder(resp.Body).Decode(&apiResp) == nil && apiResp.Message != \"\" {\n\t\t\t\terrorMsg = fmt.Sprintf(\"%s: %s\", errorMsg, apiResp.Message)\n\t\t\t}\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%s\", errorMsg)}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tvar response pugreconAPIResponse\n\t\terr = json.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, subdomain := range response.Results {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain.Name}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"pugrecon\"\n}\n\n// IsDefault returns false as this is not a default source.\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\n// HasRecursiveSupport returns false as this source does not support recursive searches.\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\n// KeyRequirement returns the API key requirement level for this source.\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\n// NeedsKey returns true as this source requires an API key.\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\n// AddApiKeys adds the API keys for the source.\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\n// Statistics returns the statistics for the source.\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/quake/quake.go",
    "content": "// Package quake logic\npackage quake\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype quakeResults struct {\n\tCode    int    `json:\"code\"`\n\tMessage string `json:\"message\"`\n\tData    []struct {\n\t\tService struct {\n\t\t\tHTTP struct {\n\t\t\t\tHost string `json:\"host\"`\n\t\t\t} `json:\"http\"`\n\t\t}\n\t} `json:\"data\"`\n\tMeta struct {\n\t\tPagination struct {\n\t\t\tTotal int `json:\"total\"`\n\t\t} `json:\"pagination\"`\n\t} `json:\"meta\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\t// quake api doc https://quake.360.cn/quake/#/help\n\t\tvar pageSize = 500\n\t\tvar start = 0\n\t\tvar totalResults = -1\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tvar requestBody = fmt.Appendf(nil, `{\"query\":\"domain: %s\", \"include\":[\"service.http.host\"], \"latest\": true, \"size\":%d, \"start\":%d}`, domain, pageSize, start)\n\t\t\ts.requests++\n\t\t\tresp, err := session.Post(ctx, \"https://quake.360.net/api/v3/search/quake_service\", \"\", map[string]string{\n\t\t\t\t\"Content-Type\": \"application/json\", \"X-QuakeToken\": randomApiKey,\n\t\t\t}, bytes.NewReader(requestBody))\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar response quakeResults\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\tif response.Code != 0 {\n\t\t\t\tresults <- subscraping.Result{\n\t\t\t\t\tSource: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%s\", response.Message),\n\t\t\t\t}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tif totalResults == -1 {\n\t\t\t\ttotalResults = response.Meta.Pagination.Total\n\t\t\t}\n\n\t\t\tfor _, quakeDomain := range response.Data {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tsubdomain := quakeDomain.Service.HTTP.Host\n\t\t\t\tif strings.ContainsAny(subdomain, \"暂无权限\") {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}\n\t\t\t\ts.results++\n\t\t\t}\n\n\t\t\tif len(response.Data) == 0 || start+pageSize >= totalResults {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\tstart += pageSize\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"quake\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/rapiddns/rapiddns.go",
    "content": "// Package rapiddns is a RapidDNS Scraping Engine in Golang\npackage rapiddns\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"io\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nvar pagePattern = regexp.MustCompile(`class=\"page-link\" href=\"/subdomain/[^\"]+\\?page=(\\d+)\">`)\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\tpage := 1\n\t\tmaxPages := 1\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\ts.requests++\n\t\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://rapiddns.io/subdomain/%s?page=%d&full=1\", domain, page))\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tbody, err := io.ReadAll(resp.Body)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\tsrc := string(body)\n\t\t\tfor _, subdomain := range session.Extractor.Extract(src) {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tif maxPages == 1 {\n\t\t\t\tmatches := pagePattern.FindAllStringSubmatch(src, -1)\n\t\t\t\tif len(matches) > 0 {\n\t\t\t\t\tlastMatch := matches[len(matches)-1]\n\t\t\t\t\tif len(lastMatch) > 1 {\n\t\t\t\t\t\tmaxPages, _ = strconv.Atoi(lastMatch[1])\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tif page >= maxPages {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tpage++\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"rapiddns\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/reconcloud/reconcloud.go",
    "content": "// Package reconcloud logic\npackage reconcloud\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype reconCloudResponse struct {\n\tMsgType         string            `json:\"msg_type\"`\n\tRequestID       string            `json:\"request_id\"`\n\tOnCache         bool              `json:\"on_cache\"`\n\tStep            string            `json:\"step\"`\n\tCloudAssetsList []cloudAssetsList `json:\"cloud_assets_list\"`\n}\n\ntype cloudAssetsList struct {\n\tKey           string `json:\"key\"`\n\tDomain        string `json:\"domain\"`\n\tCloudProvider string `json:\"cloud_provider\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://recon.cloud/api/search?domain=%s\", domain))\n\t\tif err != nil && resp == nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar response reconCloudResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tif len(response.CloudAssetsList) > 0 {\n\t\t\tfor _, cloudAsset := range response.CloudAssetsList {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: cloudAsset.Domain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"reconcloud\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/reconeer/reconeer.go",
    "content": "package reconeer\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tSubdomains []subdomain `json:\"subdomains\"`\n}\n\ntype subdomain struct {\n\tSubdomain string `json:\"subdomain\"`\n}\n\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\theaders := map[string]string{\n\t\t\t\"Accept\": \"application/json\",\n\t\t}\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey != \"\" {\n\t\t\theaders[\"X-API-KEY\"] = randomApiKey\n\t\t}\n\t\tapiURL := fmt.Sprintf(\"https://www.reconeer.com/api/domain/%s\", domain)\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, apiURL, \"\", headers)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\tif resp.StatusCode != 200 {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"request failed with status %d\", resp.StatusCode)}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tvar responseData response\n\t\tdecoder := json.NewDecoder(resp.Body)\n\t\terr = decoder.Decode(&responseData)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tfor _, result := range responseData.Subdomains {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: result.Subdomain}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\treturn results\n}\n\nfunc (s *Source) Name() string {\n\treturn \"reconeer\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.OptionalKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/redhuntlabs/redhuntlabs.go",
    "content": "// Package redhuntlabs logic\npackage redhuntlabs\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype Response struct {\n\tSubdomains []string         `json:\"subdomains\"`\n\tMetadata   ResponseMetadata `json:\"metadata\"`\n}\n\ntype ResponseMetadata struct {\n\tResultCount int `json:\"result_count\"`\n\tPageSize    int `json:\"page_size\"`\n\tPageNumber  int `json:\"page_number\"`\n}\n\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\tpageSize := 1000\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" || !strings.Contains(randomApiKey, \":\") {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\trandomApiInfo := strings.Split(randomApiKey, \":\")\n\t\tif len(randomApiInfo) != 3 {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\t\tbaseUrl := randomApiInfo[0] + \":\" + randomApiInfo[1]\n\t\trequestHeaders := map[string]string{\"X-BLOBR-KEY\": randomApiInfo[2], \"User-Agent\": \"subfinder\"}\n\t\tgetUrl := fmt.Sprintf(\"%s?domain=%s&page=1&page_size=%d\", baseUrl, domain, pageSize)\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, getUrl, \"\", requestHeaders)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"encountered error: %v; note: if you get a 'limit has been reached' error, head over to https://devportal.redhuntlabs.com\", err)}\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tvar response Response\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tsession.DiscardHTTPResponse(resp)\n\t\tif response.Metadata.ResultCount > pageSize {\n\t\t\ttotalPages := (response.Metadata.ResultCount + pageSize - 1) / pageSize\n\t\t\tfor page := 1; page <= totalPages; page++ {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tgetUrl = fmt.Sprintf(\"%s?domain=%s&page=%d&page_size=%d\", baseUrl, domain, page, pageSize)\n\t\t\t\ts.requests++\n\t\t\t\tresp, err := session.Get(ctx, getUrl, \"\", requestHeaders)\n\t\t\t\tif err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"encountered error: %v; note: if you get a 'limit has been reached' error, head over to https://devportal.redhuntlabs.com\", err)}\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\ts.errors++\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\t\t\tif err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\ts.errors++\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\t\tfor _, subdomain := range response.Subdomains {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t} else {\n\t\t\tfor _, subdomain := range response.Subdomains {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t}()\n\treturn results\n}\n\nfunc (s *Source) Name() string {\n\treturn \"redhuntlabs\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/riddler/riddler.go",
    "content": "// Package riddler logic\npackage riddler\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://riddler.io/search?q=pld:%s&view_type=data_table\", domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tscanner := bufio.NewScanner(resp.Body)\n\t\tfor scanner.Scan() {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tline := scanner.Text()\n\t\t\tif line == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tfor _, subdomain := range session.Extractor.Extract(line) {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"riddler\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/robtex/robtext.go",
    "content": "// Package robtex logic\npackage robtex\n\nimport (\n\t\"bufio\"\n\t\"bytes\"\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nconst (\n\taddrRecord     = \"A\"\n\tiPv6AddrRecord = \"AAAA\"\n\tbaseURL        = \"https://proapi.robtex.com/pdns\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\tskipped   bool\n}\n\ntype result struct {\n\tRrname string `json:\"rrname\"`\n\tRrdata string `json:\"rrdata\"`\n\tRrtype string `json:\"rrtype\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\"Content-Type\": \"application/x-ndjson\"}\n\n\t\tips, err := enumerate(ctx, session, fmt.Sprintf(\"%s/forward/%s?key=%s\", baseURL, domain, randomApiKey), headers)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, result := range ips {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tif result.Rrtype == addrRecord || result.Rrtype == iPv6AddrRecord {\n\t\t\t\tdomains, err := enumerate(ctx, session, fmt.Sprintf(\"%s/reverse/%s?key=%s\", baseURL, result.Rrdata, randomApiKey), headers)\n\t\t\t\tif err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors++\n\t\t\t\t\treturn\n\t\t\t\t}\n\t\t\t\tfor _, result := range domains {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: result.Rrdata}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\nfunc enumerate(ctx context.Context, session *subscraping.Session, targetURL string, headers map[string]string) ([]result, error) {\n\tvar results []result\n\n\tresp, err := session.Get(ctx, targetURL, \"\", headers)\n\tif err != nil {\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn results, err\n\t}\n\n\tdefer session.DiscardHTTPResponse(resp)\n\n\tscanner := bufio.NewScanner(resp.Body)\n\tfor scanner.Scan() {\n\t\tline := scanner.Text()\n\t\tif line == \"\" {\n\t\t\tcontinue\n\t\t}\n\t\tvar response result\n\t\terr = jsoniter.NewDecoder(bytes.NewBufferString(line)).Decode(&response)\n\t\tif err != nil {\n\t\t\treturn results, err\n\t\t}\n\n\t\tresults = append(results, response)\n\t}\n\n\treturn results, nil\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"robtex\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/rsecloud/rsecloud.go",
    "content": "package rsecloud\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tCount      int      `json:\"count\"`\n\tData       []string `json:\"data\"`\n\tPage       int      `json:\"page\"`\n\tPageSize   int      `json:\"pagesize\"`\n\tTotalPages int      `json:\"total_pages\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\"Content-Type\": \"application/json\", \"X-API-Key\": randomApiKey}\n\n\t\tfetchSubdomains := func(endpoint string) {\n\t\t\tpage := 1\n\t\t\tfor {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\ts.requests++\n\t\t\t\tresp, err := session.Get(ctx, fmt.Sprintf(\"https://api.rsecloud.com/api/v2/subdomains/%s/%s?page=%d\", endpoint, domain, page), \"\", headers)\n\t\t\t\tif err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors++\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tvar rseCloudResponse response\n\t\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&rseCloudResponse)\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\tif err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors++\n\t\t\t\t\treturn\n\t\t\t\t}\n\n\t\t\t\tfor _, subdomain := range rseCloudResponse.Data {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\tif page >= rseCloudResponse.TotalPages {\n\t\t\t\t\tbreak\n\t\t\t\t}\n\t\t\t\tpage++\n\t\t\t}\n\t\t}\n\n\t\tfetchSubdomains(\"active\")\n\t\tfetchSubdomains(\"passive\")\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"rsecloud\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/securitytrails/securitytrails.go",
    "content": "// Package securitytrails logic\npackage securitytrails\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n\t\"github.com/projectdiscovery/utils/ptr\"\n)\n\ntype response struct {\n\tMeta struct {\n\t\tScrollID string `json:\"scroll_id\"`\n\t} `json:\"meta\"`\n\tRecords []struct {\n\t\tHostname string `json:\"hostname\"`\n\t} `json:\"records\"`\n\tSubdomains []string `json:\"subdomains\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tvar scrollId string\n\t\theaders := map[string]string{\"Content-Type\": \"application/json\", \"APIKEY\": randomApiKey}\n\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tvar resp *http.Response\n\t\t\tvar err error\n\n\t\t\tif scrollId == \"\" {\n\t\t\t\tvar requestBody = fmt.Appendf(nil, `{\"query\":\"apex_domain='%s'\"}`, domain)\n\t\t\t\ts.requests++\n\t\t\t\tresp, err = session.Post(ctx, \"https://api.securitytrails.com/v1/domains/list?include_ips=false&scroll=true\", \"\",\n\t\t\t\t\theaders, bytes.NewReader(requestBody))\n\t\t\t} else {\n\t\t\t\ts.requests++\n\t\t\t\tresp, err = session.Get(ctx, fmt.Sprintf(\"https://api.securitytrails.com/v1/scroll/%s\", scrollId), \"\", headers)\n\t\t\t}\n\n\t\t\tif err != nil && ptr.Safe(resp).StatusCode == 403 {\n\t\t\t\ts.requests++\n\t\t\t\tresp, err = session.Get(ctx, fmt.Sprintf(\"https://api.securitytrails.com/v1/domain/%s/subdomains\", domain), \"\", headers)\n\t\t\t}\n\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar securityTrailsResponse response\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&securityTrailsResponse)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t\tfor _, record := range securityTrailsResponse.Records {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Hostname}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tfor _, subdomain := range securityTrailsResponse.Subdomains {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tif strings.HasSuffix(subdomain, \".\") {\n\t\t\t\t\tsubdomain += domain\n\t\t\t\t} else {\n\t\t\t\t\tsubdomain = subdomain + \".\" + domain\n\t\t\t\t}\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}\n\t\t\t\ts.results++\n\t\t\t}\n\n\t\t\tscrollId = securityTrailsResponse.Meta.ScrollID\n\n\t\t\tif scrollId == \"\" {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"securitytrails\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/shodan/shodan.go",
    "content": "// Package shodan logic\npackage shodan\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype dnsdbLookupResponse struct {\n\tDomain     string   `json:\"domain\"`\n\tSubdomains []string `json:\"subdomains\"`\n\tResult     int      `json:\"result\"`\n\tError      string   `json:\"error\"`\n\tMore       bool     `json:\"more\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\tpage := 1\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\n\t\t\tsearchURL := fmt.Sprintf(\"https://api.shodan.io/dns/domain/%s?key=%s&page=%d\", domain, randomApiKey, page)\n\t\t\ts.requests++\n\t\t\tresp, err := session.SimpleGet(ctx, searchURL)\n\t\t\tif err != nil {\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\t\tvar response dnsdbLookupResponse\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tif response.Error != \"\" {\n\t\t\t\tresults <- subscraping.Result{\n\t\t\t\t\tSource: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"%v\", response.Error),\n\t\t\t\t}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tfor _, data := range response.Subdomains {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tdefault:\n\t\t\t\t}\n\t\t\t\tvalue := fmt.Sprintf(\"%s.%s\", data, response.Domain)\n\t\t\t\tresults <- subscraping.Result{\n\t\t\t\t\tSource: s.Name(), Type: subscraping.Subdomain, Value: value,\n\t\t\t\t}\n\t\t\t\ts.results++\n\t\t\t}\n\n\t\t\tif !response.More {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tpage++\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"shodan\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/sitedossier/sitedossier.go",
    "content": "// Package sitedossier logic\npackage sitedossier\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"regexp\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// SleepRandIntn is the integer value to get the pseudo-random number\n// to sleep before find the next match\nconst SleepRandIntn = 5\n\nvar reNext = regexp.MustCompile(`<a href=\"([A-Za-z0-9/.]+)\"><b>`)\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.enumerate(ctx, session, fmt.Sprintf(\"http://www.sitedossier.com/parentdomain/%s\", domain), results)\n\t}()\n\n\treturn results\n}\n\nfunc (s *Source) enumerate(ctx context.Context, session *subscraping.Session, baseURL string, results chan subscraping.Result) {\n\tselect {\n\tcase <-ctx.Done():\n\t\treturn\n\tdefault:\n\t}\n\n\ts.requests++\n\tresp, err := session.SimpleGet(ctx, baseURL)\n\tisnotfound := resp != nil && resp.StatusCode == http.StatusNotFound\n\tif err != nil && !isnotfound {\n\t\tresults <- subscraping.Result{Source: \"sitedossier\", Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn\n\t}\n\n\tbody, err := io.ReadAll(resp.Body)\n\tif err != nil {\n\t\tresults <- subscraping.Result{Source: \"sitedossier\", Type: subscraping.Error, Error: err}\n\t\ts.errors++\n\t\tsession.DiscardHTTPResponse(resp)\n\t\treturn\n\t}\n\tsession.DiscardHTTPResponse(resp)\n\n\tsrc := string(body)\n\tfor _, subdomain := range session.Extractor.Extract(src) {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn\n\t\tcase results <- subscraping.Result{Source: \"sitedossier\", Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\ts.results++\n\t\t}\n\t}\n\n\tmatch := reNext.FindStringSubmatch(src)\n\tif len(match) > 0 {\n\t\ts.enumerate(ctx, session, fmt.Sprintf(\"http://www.sitedossier.com%s\", match[1]), results)\n\t}\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"sitedossier\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/thc/thc.go",
    "content": "// Package thc logic\npackage thc\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tDomains []struct {\n\t\tDomain string `json:\"domain\"`\n\t} `json:\"domains\"`\n\tNextPageState string `json:\"next_page_state\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\ntype requestBody struct {\n\tDomain    string `json:\"domain\"`\n\tPageState string `json:\"page_state\"`\n\tLimit     int    `json:\"limit\"`\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\tvar pageState string\n\t\theaders := map[string]string{\"Content-Type\": \"application/json\"}\n\t\tapiURL := \"https://ip.thc.org/api/v1/lookup/subdomains\"\n\n\t\tfor {\n\t\t\treqBody := requestBody{\n\t\t\t\tDomain:    domain,\n\t\t\t\tPageState: pageState,\n\t\t\t\tLimit:     1000,\n\t\t\t}\n\n\t\t\tbodyBytes, err := json.Marshal(reqBody)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\ts.requests++\n\t\t\tresp, err := session.Post(ctx, apiURL, \"\", headers, bytes.NewReader(bodyBytes))\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar thcResponse response\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&thcResponse)\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tfor _, domainRecord := range thcResponse.Domains {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: domainRecord.Domain}\n\t\t\t\ts.results++\n\t\t\t}\n\n\t\t\tpageState = thcResponse.NextPageState\n\n\t\t\tif pageState == \"\" {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"thc\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// No API keys needed for THC\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/threatbook/threatbook.go",
    "content": "// Package threatbook logic\npackage threatbook\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strconv\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype threatBookResponse struct {\n\tResponseCode int64  `json:\"response_code\"`\n\tVerboseMsg   string `json:\"verbose_msg\"`\n\tData         struct {\n\t\tDomain     string `json:\"domain\"`\n\t\tSubDomains struct {\n\t\t\tTotal string   `json:\"total\"`\n\t\t\tData  []string `json:\"data\"`\n\t\t} `json:\"sub_domains\"`\n\t} `json:\"data\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://api.threatbook.cn/v3/domain/sub_domains?apikey=%s&resource=%s\", randomApiKey, domain))\n\t\tif err != nil && resp == nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar response threatBookResponse\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&response)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tif response.ResponseCode != 0 {\n\t\t\tresults <- subscraping.Result{\n\t\t\t\tSource: s.Name(), Type: subscraping.Error,\n\t\t\t\tError: fmt.Errorf(\"code %d, %s\", response.ResponseCode, response.VerboseMsg),\n\t\t\t}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\ttotal, err := strconv.ParseInt(response.Data.SubDomains.Total, 10, 64)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tif total > 0 {\n\t\t\tfor _, subdomain := range response.Data.SubDomains.Data {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"threatbook\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/threatcrowd/threatcrowd.go",
    "content": "package threatcrowd\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// threatCrowdResponse represents the JSON response from the ThreatCrowd API.\ntype threatCrowdResponse struct {\n\tResponseCode string   `json:\"response_code\"`\n\tSubdomains   []string `json:\"subdomains\"`\n\tUndercount   string   `json:\"undercount\"`\n}\n\n// Source implements the subscraping.Source interface for ThreatCrowd.\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run queries the ThreatCrowd API for the given domain and returns found subdomains.\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func(startTime time.Time) {\n\t\tdefer func() {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}()\n\n\t\turl := fmt.Sprintf(\"http://ci-www.threatcrowd.org/searchApi/v2/domain/report/?domain=%s\", domain)\n\t\treq, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.Client.Do(req)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\t\tdefer func() {\n\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t}\n\t\t}()\n\n\t\tif resp.StatusCode != http.StatusOK {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: fmt.Errorf(\"unexpected status code: %d\", resp.StatusCode)}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tbody, err := io.ReadAll(resp.Body)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tvar tcResponse threatCrowdResponse\n\t\tif err := json.Unmarshal(body, &tcResponse); err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, subdomain := range tcResponse.Subdomains {\n\t\t\tif subdomain != \"\" {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}(time.Now())\n\n\treturn results\n}\n\n// Name returns the name of the source.\nfunc (s *Source) Name() string {\n\treturn \"threatcrowd\"\n}\n\n// IsDefault indicates whether this source is enabled by default.\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\n// HasRecursiveSupport indicates if the source supports recursive searches.\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\n// KeyRequirement returns the API key requirement level for this source.\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\n// NeedsKey indicates if the source requires an API key.\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\n// AddApiKeys is a no-op since ThreatCrowd does not require an API key.\nfunc (s *Source) AddApiKeys(_ []string) {}\n\n// Statistics returns usage statistics.\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/threatminer/threatminer.go",
    "content": "// Package threatminer logic\npackage threatminer\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tStatusCode    string   `json:\"status_code\"`\n\tStatusMessage string   `json:\"status_message\"`\n\tResults       []string `json:\"results\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://api.threatminer.org/v2/domain.php?q=%s&rt=5\", domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\tvar data response\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&data)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\treturn\n\t\t}\n\n\t\tfor _, subdomain := range data.Results {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"threatminer\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/urlscan/urlscan.go",
    "content": "// Package urlscan logic\npackage urlscan\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"net/url\"\n\t\"strings\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\nconst (\n\t// baseURL is the URLScan API base URL\n\tbaseURL = \"https://urlscan.io/api/v1/search/\"\n\t// maxPages is the maximum number of pages to fetch\n\tmaxPages = 5\n\t// maxPerPage is the maximum results per page (URLScan max is 10000, but 100 is safer)\n\tmaxPerPage = 100\n)\n\n// response represents the URLScan API response structure\ntype response struct {\n\tResults []struct {\n\t\tTask struct {\n\t\t\tDomain string `json:\"domain\"`\n\t\t\tURL    string `json:\"url\"`\n\t\t} `json:\"task\"`\n\t\tPage struct {\n\t\t\tDomain string `json:\"domain\"`\n\t\t\tURL    string `json:\"url\"`\n\t\t} `json:\"page\"`\n\t\tSort []interface{} `json:\"sort\"`\n\t} `json:\"results\"`\n\tHasMore bool `json:\"has_more\"`\n\tTotal   int  `json:\"total\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\ts.skipped = false\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\"api-key\": randomApiKey}\n\n\t\t// Search with wildcard to get more subdomain results\n\t\ts.enumerate(ctx, domain, headers, session, results)\n\t}()\n\n\treturn results\n}\n\n// enumerate performs the actual enumeration with pagination\nfunc (s *Source) enumerate(ctx context.Context, domain string, headers map[string]string, session *subscraping.Session, results chan subscraping.Result) {\n\tvar searchAfter string\n\tcurrentPage := 0\n\n\tfor {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn\n\t\tdefault:\n\t\t}\n\n\t\tif currentPage >= maxPages {\n\t\t\tbreak\n\t\t}\n\n\t\t// Build search URL\n\t\tsearchURL := fmt.Sprintf(\"%s?q=domain:%s&size=%d\", baseURL, url.QueryEscape(domain), maxPerPage)\n\t\tif searchAfter != \"\" {\n\t\t\tsearchURL += \"&search_after=\" + url.QueryEscape(searchAfter)\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.Get(ctx, searchURL, \"\", headers)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar data response\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&data)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\t// Process results - extract subdomains from multiple fields\n\t\tfor _, result := range data.Results {\n\t\t\tcandidates := []string{\n\t\t\t\tresult.Task.Domain,\n\t\t\t\tresult.Page.Domain,\n\t\t\t}\n\n\t\t\t// Also extract from URLs if present\n\t\t\tif result.Task.URL != \"\" {\n\t\t\t\tif u, err := url.Parse(result.Task.URL); err == nil {\n\t\t\t\t\tcandidates = append(candidates, u.Hostname())\n\t\t\t\t}\n\t\t\t}\n\t\t\tif result.Page.URL != \"\" {\n\t\t\t\tif u, err := url.Parse(result.Page.URL); err == nil {\n\t\t\t\t\tcandidates = append(candidates, u.Hostname())\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tfor _, candidate := range candidates {\n\t\t\t\tif candidate == \"\" {\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\t\t\t\tfor _, subdomain := range session.Extractor.Extract(candidate) {\n\t\t\t\t\tselect {\n\t\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\t\treturn\n\t\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\t\ts.results++\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\n\t\t// Check pagination conditions\n\t\tif !data.HasMore || len(data.Results) == 0 {\n\t\t\tbreak\n\t\t}\n\n\t\t// Get sort value for next page\n\t\tlastResult := data.Results[len(data.Results)-1]\n\t\tif len(lastResult.Sort) == 0 {\n\t\t\tbreak\n\t\t}\n\n\t\t// Build search_after parameter\n\t\tsortValues := make([]string, len(lastResult.Sort))\n\t\tfor i, v := range lastResult.Sort {\n\t\t\tswitch val := v.(type) {\n\t\t\tcase float64:\n\t\t\t\tsortValues[i] = fmt.Sprintf(\"%.0f\", val)\n\t\t\tdefault:\n\t\t\t\tsortValues[i] = fmt.Sprintf(\"%v\", v)\n\t\t\t}\n\t\t}\n\t\tsearchAfter = strings.Join(sortValues, \",\")\n\t\tcurrentPage++\n\t}\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"urlscan\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/virustotal/virustotal.go",
    "content": "// Package virustotal logic\npackage virustotal\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tData []Object `json:\"data\"`\n\tMeta Meta     `json:\"meta\"`\n}\n\ntype Object struct {\n\tId string `json:\"id\"`\n}\n\ntype Meta struct {\n\tCursor string `json:\"cursor\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\treturn\n\t\t}\n\t\tvar cursor = \"\"\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tvar url = fmt.Sprintf(\"https://www.virustotal.com/api/v3/domains/%s/subdomains?limit=40\", domain)\n\t\t\tif cursor != \"\" {\n\t\t\t\turl = fmt.Sprintf(\"%s&cursor=%s\", url, cursor)\n\t\t\t}\n\t\t\ts.requests++\n\t\t\tresp, err := session.Get(ctx, url, \"\", map[string]string{\"x-apikey\": randomApiKey})\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\t\t\tdefer func() {\n\t\t\t\tif err := resp.Body.Close(); err != nil {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors++\n\t\t\t\t}\n\t\t\t}()\n\n\t\t\tvar data response\n\t\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&data)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tfor _, subdomain := range data.Data {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain.Id}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t\tcursor = data.Meta.Cursor\n\t\t\tif cursor == \"\" {\n\t\t\t\tbreak\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"virustotal\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn true\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tRequests:  s.requests,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/waybackarchive/waybackarchive.go",
    "content": "// Package waybackarchive logic\npackage waybackarchive\n\nimport (\n\t\"bufio\"\n\t\"context\"\n\t\"fmt\"\n\t\"net/url\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// Source is the passive scraping agent\ntype Source struct {\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"http://web.archive.org/cdx/search/cdx?url=*.%s/*&output=txt&fl=original&collapse=urlkey\", domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\tscanner := bufio.NewScanner(resp.Body)\n\t\tfor scanner.Scan() {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tline := scanner.Text()\n\t\t\tif line == \"\" {\n\t\t\t\tcontinue\n\t\t\t}\n\t\t\tline, _ = url.QueryUnescape(line)\n\t\t\tfor _, subdomain := range session.Extractor.Extract(line) {\n\t\t\t\tsubdomain = strings.ToLower(subdomain)\n\t\t\t\tsubdomain = strings.TrimPrefix(subdomain, \"25\")\n\t\t\t\tsubdomain = strings.TrimPrefix(subdomain, \"2f\")\n\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: subdomain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"waybackarchive\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.NoKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(_ []string) {\n\t// no key needed\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/whoisxmlapi/whoisxmlapi.go",
    "content": "// Package whoisxmlapi logic\npackage whoisxmlapi\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"time\"\n\n\tjsoniter \"github.com/json-iterator/go\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tSearch string `json:\"search\"`\n\tResult Result `json:\"result\"`\n}\n\ntype Result struct {\n\tCount   int      `json:\"count\"`\n\tRecords []Record `json:\"records\"`\n}\n\ntype Record struct {\n\tDomain    string `json:\"domain\"`\n\tFirstSeen int    `json:\"firstSeen\"`\n\tLastSeen  int    `json:\"lastSeen\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\ts.requests++\n\t\tresp, err := session.SimpleGet(ctx, fmt.Sprintf(\"https://subdomains.whoisxmlapi.com/api/v1?apiKey=%s&domainName=%s\", randomApiKey, domain))\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tvar data response\n\t\terr = jsoniter.NewDecoder(resp.Body).Decode(&data)\n\t\tif err != nil {\n\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\ts.errors++\n\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\treturn\n\t\t}\n\n\t\tsession.DiscardHTTPResponse(resp)\n\n\t\tfor _, record := range data.Result.Records {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Domain}:\n\t\t\t\ts.results++\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"whoisxmlapi\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/windvane/windvane.go",
    "content": "// Package windvane logic\npackage windvane\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"net/http\"\n\t\"strconv\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\ntype response struct {\n\tCode int          `json:\"code\"`\n\tMsg  string       `json:\"msg\"`\n\tData responseData `json:\"data\"`\n}\n\ntype responseData struct {\n\tList         []domainEntry `json:\"list\"`\n\tPageResponse pageInfo      `json:\"page_response\"`\n}\n\ntype domainEntry struct {\n\tDomain string `json:\"domain\"`\n}\n\ntype pageInfo struct {\n\tTotal     string `json:\"total\"`\n\tCount     string `json:\"count\"`\n\tTotalPage string `json:\"total_page\"`\n}\n\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\theaders := map[string]string{\"Content-Type\": \"application/json\", \"X-Api-Key\": randomApiKey}\n\n\t\tpage := 1\n\t\tcount := 1000\n\t\tfor {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tvar resp *http.Response\n\t\t\tvar err error\n\n\t\t\trequestBody, _ := json.Marshal(map[string]interface{}{\"domain\": domain, \"page_request\": map[string]int{\"page\": page, \"count\": count}})\n\t\t\ts.requests++\n\t\t\tresp, err = session.Post(ctx, \"https://windvane.lichoin.com/trpc.backendhub.public.WindvaneService/ListSubDomain\",\n\t\t\t\t\"\", headers, bytes.NewReader(requestBody))\n\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tdefer session.DiscardHTTPResponse(resp)\n\n\t\t\tvar windvaneResponse response\n\t\t\terr = json.NewDecoder(resp.Body).Decode(&windvaneResponse)\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tfor _, record := range windvaneResponse.Data.List {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: record.Domain}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tpageInfo := windvaneResponse.Data.PageResponse\n\t\t\tvar totalRecords, recordsPerPage int\n\n\t\t\tif totalRecords, err = strconv.Atoi(pageInfo.Total); err != nil {\n\t\t\t\tbreak\n\t\t\t}\n\t\t\tif recordsPerPage, err = strconv.Atoi(pageInfo.Count); err != nil {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\tif (page-1)*recordsPerPage >= totalRecords {\n\t\t\t\tbreak\n\t\t\t}\n\n\t\t\tpage++\n\t\t}\n\n\t}()\n\n\treturn results\n}\n\nfunc (s *Source) Name() string {\n\treturn \"windvane\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn true\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/sources/zoomeyeapi/zoomeyeapi.go",
    "content": "package zoomeyeapi\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/subfinder/v2/pkg/subscraping\"\n)\n\n// search results\ntype zoomeyeResults struct {\n\tStatus int `json:\"status\"`\n\tTotal  int `json:\"total\"`\n\tList   []struct {\n\t\tName string   `json:\"name\"`\n\t\tIp   []string `json:\"ip\"`\n\t} `json:\"list\"`\n}\n\n// Source is the passive scraping agent\ntype Source struct {\n\tapiKeys   []string\n\ttimeTaken time.Duration\n\terrors    int\n\tresults   int\n\trequests  int\n\tskipped   bool\n}\n\n// Run function returns all subdomains found with the service\nfunc (s *Source) Run(ctx context.Context, domain string, session *subscraping.Session) <-chan subscraping.Result {\n\tresults := make(chan subscraping.Result)\n\ts.errors = 0\n\ts.results = 0\n\ts.requests = 0\n\n\tgo func() {\n\t\tdefer func(startTime time.Time) {\n\t\t\ts.timeTaken = time.Since(startTime)\n\t\t\tclose(results)\n\t\t}(time.Now())\n\n\t\trandomApiKey := subscraping.PickRandom(s.apiKeys, s.Name())\n\t\tif randomApiKey == \"\" {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\n\t\trandomApiInfo := strings.Split(randomApiKey, \":\")\n\t\tif len(randomApiInfo) != 2 {\n\t\t\ts.skipped = true\n\t\t\treturn\n\t\t}\n\t\thost := randomApiInfo[0]\n\t\tapiKey := randomApiInfo[1]\n\n\t\theaders := map[string]string{\n\t\t\t\"API-KEY\":      apiKey,\n\t\t\t\"Accept\":       \"application/json\",\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t}\n\t\tvar pages = 1\n\t\tfor currentPage := 1; currentPage <= pages; currentPage++ {\n\t\t\tselect {\n\t\t\tcase <-ctx.Done():\n\t\t\t\treturn\n\t\t\tdefault:\n\t\t\t}\n\t\t\tapi := fmt.Sprintf(\"https://api.%s/domain/search?q=%s&type=1&s=1000&page=%d\", host, domain, currentPage)\n\t\t\ts.requests++\n\t\t\tresp, err := session.Get(ctx, api, \"\", headers)\n\t\t\tisForbidden := resp != nil && resp.StatusCode == http.StatusForbidden\n\t\t\tif err != nil {\n\t\t\t\tif !isForbidden {\n\t\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\t\ts.errors++\n\t\t\t\t\tsession.DiscardHTTPResponse(resp)\n\t\t\t\t}\n\t\t\t\treturn\n\t\t\t}\n\n\t\t\tvar res zoomeyeResults\n\t\t\terr = json.NewDecoder(resp.Body).Decode(&res)\n\n\t\t\tif err != nil {\n\t\t\t\tresults <- subscraping.Result{Source: s.Name(), Type: subscraping.Error, Error: err}\n\t\t\t\ts.errors++\n\t\t\t\t_ = resp.Body.Close()\n\t\t\t\treturn\n\t\t\t}\n\t\t\t_ = resp.Body.Close()\n\t\t\tpages = int(res.Total/1000) + 1\n\t\t\tfor _, r := range res.List {\n\t\t\t\tselect {\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn\n\t\t\t\tcase results <- subscraping.Result{Source: s.Name(), Type: subscraping.Subdomain, Value: r.Name}:\n\t\t\t\t\ts.results++\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}()\n\n\treturn results\n}\n\n// Name returns the name of the source\nfunc (s *Source) Name() string {\n\treturn \"zoomeyeapi\"\n}\n\nfunc (s *Source) IsDefault() bool {\n\treturn false\n}\n\nfunc (s *Source) HasRecursiveSupport() bool {\n\treturn false\n}\n\nfunc (s *Source) KeyRequirement() subscraping.KeyRequirement {\n\treturn subscraping.RequiredKey\n}\n\nfunc (s *Source) NeedsKey() bool {\n\treturn s.KeyRequirement() == subscraping.RequiredKey\n}\n\nfunc (s *Source) AddApiKeys(keys []string) {\n\ts.apiKeys = keys\n}\n\nfunc (s *Source) Statistics() subscraping.Statistics {\n\treturn subscraping.Statistics{\n\t\tErrors:    s.errors,\n\t\tResults:   s.results,\n\t\tTimeTaken: s.timeTaken,\n\t\tSkipped:   s.skipped,\n\t\tRequests:  s.requests,\n\t}\n}\n"
  },
  {
    "path": "pkg/subscraping/types.go",
    "content": "package subscraping\n\nimport (\n\t\"context\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/ratelimit\"\n\tmapsutil \"github.com/projectdiscovery/utils/maps\"\n)\n\ntype CtxArg string\n\nconst (\n\tCtxSourceArg CtxArg = \"source\"\n)\n\ntype CustomRateLimit struct {\n\tCustom mapsutil.SyncLockMap[string, uint]\n}\n\n// BasicAuth request's Authorization header\ntype BasicAuth struct {\n\tUsername string\n\tPassword string\n}\n\n// Statistics contains statistics about the scraping process\ntype Statistics struct {\n\tTimeTaken time.Duration\n\tRequests  int\n\tErrors    int\n\tResults   int\n\tSkipped   bool\n}\n\n// KeyRequirement represents the API key requirement level for a source\ntype KeyRequirement int\n\nconst (\n\tNoKey KeyRequirement = iota\n\tOptionalKey\n\tRequiredKey\n)\n\n// Source is an interface inherited by each passive source\ntype Source interface {\n\t// Run takes a domain as argument and a session object\n\t// which contains the extractor for subdomains, http client\n\t// and other stuff.\n\tRun(context.Context, string, *Session) <-chan Result\n\n\t// Name returns the name of the source. It is preferred to use lower case names.\n\tName() string\n\n\t// IsDefault returns true if the current source should be\n\t// used as part of the default execution.\n\tIsDefault() bool\n\n\t// HasRecursiveSupport returns true if the current source\n\t// accepts subdomains (e.g. subdomain.domain.tld),\n\t// not just root domains.\n\tHasRecursiveSupport() bool\n\n\t// KeyRequirement returns the API key requirement level for this source\n\tKeyRequirement() KeyRequirement\n\n\t// NeedsKey returns true if the source requires an API key.\n\t// Deprecated: Use KeyRequirement() instead for more granular control.\n\tNeedsKey() bool\n\n\tAddApiKeys([]string)\n\n\t// Statistics returns the scrapping statistics for the source\n\tStatistics() Statistics\n}\n\n// SubdomainExtractor is an interface that defines the contract for subdomain extraction.\ntype SubdomainExtractor interface {\n\tExtract(text string) []string\n}\n\n// Session is the option passed to the source, an option is created\n// uniquely for each source.\ntype Session struct {\n\t//SubdomainExtractor\n\tExtractor SubdomainExtractor\n\t// Client is the current http client\n\tClient *http.Client\n\t// Rate limit instance\n\tMultiRateLimiter *ratelimit.MultiLimiter\n\t// Timeout is the timeout in seconds for requests\n\tTimeout int\n}\n\n// Result is a result structure returned by a source\ntype Result struct {\n\tType   ResultType\n\tSource string\n\tValue  string\n\tError  error\n}\n\n// ResultType is the type of result returned by the source\ntype ResultType int\n\n// Types of results returned by the source\nconst (\n\tSubdomain ResultType = iota\n\tError\n)\n"
  },
  {
    "path": "pkg/subscraping/utils.go",
    "content": "package subscraping\n\nimport (\n\t\"math/rand\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n)\n\nconst MultipleKeyPartsLength = 2\n\nfunc PickRandom[T any](v []T, sourceName string) T {\n\tvar result T\n\tlength := len(v)\n\tif length == 0 {\n\t\tgologger.Debug().Msgf(\"Cannot use the %s source because there was no API key/secret defined for it.\", sourceName)\n\t\treturn result\n\t}\n\treturn v[rand.Intn(length)]\n}\n\nfunc CreateApiKeys[T any](keys []string, provider func(k, v string) T) []T {\n\tvar result []T\n\tfor _, key := range keys {\n\t\tif keyPartA, keyPartB, ok := createMultiPartKey(key); ok {\n\t\t\tresult = append(result, provider(keyPartA, keyPartB))\n\t\t}\n\t}\n\treturn result\n}\n\nfunc createMultiPartKey(key string) (keyPartA, keyPartB string, ok bool) {\n\tparts := strings.Split(key, \":\")\n\tok = len(parts) == MultipleKeyPartsLength\n\n\tif ok {\n\t\tkeyPartA = parts[0]\n\t\tkeyPartB = parts[1]\n\t}\n\n\treturn\n}\n"
  },
  {
    "path": "pkg/testutils/integration.go",
    "content": "package testutils\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"os/exec\"\n\t\"strings\"\n)\n\nfunc RunSubfinderAndGetResults(debug bool, domain string, extra ...string) ([]string, error) {\n\tcmd := exec.Command(\"bash\", \"-c\")\n\tcmdLine := fmt.Sprintf(\"echo %s | %s\", domain, \"./subfinder \")\n\tcmdLine += strings.Join(extra, \" \")\n\tcmd.Args = append(cmd.Args, cmdLine)\n\tif debug {\n\t\tcmd.Args = append(cmd.Args, \"-v\")\n\t\tcmd.Stderr = os.Stderr\n\t\tfmt.Println(cmd.String())\n\t} else {\n\t\tcmd.Args = append(cmd.Args, \"-silent\")\n\t}\n\tdata, err := cmd.Output()\n\tif debug {\n\t\tfmt.Println(string(data))\n\t}\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\tvar parts []string\n\titems := strings.SplitSeq(string(data), \"\\n\")\n\tfor i := range items {\n\t\tif i != \"\" {\n\t\t\tparts = append(parts, i)\n\t\t}\n\t}\n\treturn parts, nil\n}\n\n// TestCase is a single integration test case\ntype TestCase interface {\n\tExecute() error\n}\n"
  }
]