Showing preview only (1,286K chars total). Download the full file or copy to clipboard to get everything.
Repository: projectdiscovery/katana
Branch: dev
Commit: 2f9ecf4211b1
Files: 156
Total size: 1.2 MB
Directory structure:
gitextract_rfj92yh6/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── README.md
│ │ ├── config.yml
│ │ ├── feature_request.md.disabled
│ │ └── issue-report.md.disabled
│ ├── MAINTAINER_GUIDE.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── dependabot.yml
│ ├── release.yml
│ └── workflows/
│ ├── build-test.yml
│ ├── codeql-analysis.yml
│ ├── compat-checks.yaml
│ ├── dep-auto-merge.yml
│ ├── discussion-triage.yml
│ ├── dockerhub-push.yml
│ ├── functional-test.yml
│ ├── release-binary.yml
│ ├── release-test.yml
│ ├── security-crawl-maze-score.yaml
│ ├── stale.yml
│ └── workflow-monitor.yml
├── .gitignore
├── .goreleaser/
│ ├── linux.yml
│ ├── mac.yml
│ └── windows.yml
├── Dockerfile
├── LICENSE.md
├── Makefile
├── README.md
├── SECURITY.md
├── cmd/
│ ├── functional-test/
│ │ ├── main.go
│ │ └── run.sh
│ ├── integration-test/
│ │ ├── filters.go
│ │ ├── integration-test.go
│ │ └── library.go
│ └── tools/
│ └── crawl-maze-score/
│ └── main.go
├── go.mod
├── go.sum
├── integration_tests/
│ └── run.sh
├── internal/
│ ├── runner/
│ │ ├── banner.go
│ │ ├── executer.go
│ │ ├── healthcheck.go
│ │ ├── options.go
│ │ └── runner.go
│ └── testutils/
│ ├── helper.go
│ ├── integration.go
│ └── testutils.go
└── pkg/
├── engine/
│ ├── common/
│ │ ├── base.go
│ │ ├── error.go
│ │ └── http.go
│ ├── engine.go
│ ├── headless/
│ │ ├── TODOS.md
│ │ ├── browser/
│ │ │ ├── browser.go
│ │ │ ├── cookie/
│ │ │ │ ├── cookie.go
│ │ │ │ ├── cookie_test.go
│ │ │ │ └── rules.json
│ │ │ ├── element.go
│ │ │ └── stealth/
│ │ │ └── assets.go
│ │ ├── captcha/
│ │ │ ├── capsolver/
│ │ │ │ ├── capsolver.go
│ │ │ │ └── capsolver_test.go
│ │ │ ├── captcha.go
│ │ │ ├── helpers_test.go
│ │ │ ├── identify.go
│ │ │ ├── identify_test.go
│ │ │ ├── inject_test.go
│ │ │ ├── injection_test.go
│ │ │ ├── integration_test.go
│ │ │ ├── js/
│ │ │ │ ├── identify.js
│ │ │ │ ├── inject-hcaptcha.js
│ │ │ │ ├── inject-recaptcha.js
│ │ │ │ ├── inject-turnstile.js
│ │ │ │ └── js.go
│ │ │ ├── solver.go
│ │ │ └── solver_test.go
│ │ ├── crawler/
│ │ │ ├── crawler.go
│ │ │ ├── diagnostics/
│ │ │ │ └── diagnostics.go
│ │ │ ├── formfill.go
│ │ │ ├── normalizer/
│ │ │ │ ├── dom_utils.go
│ │ │ │ ├── dom_utils_test.go
│ │ │ │ ├── helpers.go
│ │ │ │ ├── normalizer.go
│ │ │ │ ├── simhash/
│ │ │ │ │ ├── simhash.go
│ │ │ │ │ └── simhash_test.go
│ │ │ │ ├── text_utils.go
│ │ │ │ └── text_utils_test.go
│ │ │ ├── state.go
│ │ │ └── state_test.go
│ │ ├── debugger.go
│ │ ├── graph/
│ │ │ └── graph.go
│ │ ├── headless.go
│ │ ├── js/
│ │ │ ├── js.go
│ │ │ ├── page-init.js
│ │ │ └── utils.js
│ │ └── types/
│ │ └── types.go
│ ├── hybrid/
│ │ ├── crawl.go
│ │ ├── doc.go
│ │ ├── hijack.go
│ │ └── hybrid.go
│ ├── parser/
│ │ ├── files/
│ │ │ ├── request.go
│ │ │ ├── robotstxt.go
│ │ │ ├── robotstxt_test.go
│ │ │ ├── sitemapxml.go
│ │ │ └── sitemapxml_test.go
│ │ ├── parser.go
│ │ ├── parser_generic.go
│ │ ├── parser_nojs.go
│ │ └── parser_test.go
│ └── standard/
│ ├── crawl.go
│ ├── doc.go
│ └── standard.go
├── navigation/
│ ├── request.go
│ └── response.go
├── output/
│ ├── custom_field.go
│ ├── error.go
│ ├── fields.go
│ ├── fields_test.go
│ ├── file_writer.go
│ ├── format_json.go
│ ├── format_screen.go
│ ├── format_template.go
│ ├── options.go
│ ├── output.go
│ ├── responses.go
│ └── result.go
├── types/
│ ├── crawler_options.go
│ ├── default.go
│ ├── options.go
│ └── options_test.go
└── utils/
├── extensions/
│ ├── extensions.go
│ └── extensions_test.go
├── filters/
│ ├── filters.go
│ ├── filters_test.go
│ └── simple.go
├── formfields.go
├── formfields_test.go
├── formfill.go
├── formfill_test.go
├── jsluice.go
├── jsluice_test.go
├── maps.go
├── maps_test.go
├── pathtrie.go
├── pathtrie_test.go
├── queue/
│ ├── priority_queue.go
│ ├── priority_queue_test.go
│ ├── queue.go
│ ├── stack.go
│ ├── stack_test.go
│ └── strategy.go
├── regex.go
├── regex_test.go
├── scope/
│ ├── scope.go
│ └── scope_test.go
├── urlfingerprint.go
├── urlfingerprint_test.go
├── utils.go
└── utils_test.go
================================================
FILE CONTENTS
================================================
================================================
FILE: .github/ISSUE_TEMPLATE/README.md
================================================
# Issue Template Management
## New Issue Workflow (Discussion-First)
To improve issue triage and reduce noise from questions being filed as bugs, all issue creation now goes through GitHub Discussions first.
### For Users:
**❌ Direct issue creation is disabled**
**✅ All reports must start as discussions**
Users will be redirected to:
- 🐛 **Bug Reports** → [Q&A Discussions](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)
- 💡 **Feature Requests** → [Ideas Discussions](https://github.com/projectdiscovery/katana/discussions/new?category=ideas)
- ❓ **Questions** → [Q&A Discussions](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)
### For Maintainers:
#### Converting Discussions to Issues:
1. **Review the discussion** thoroughly
2. **Determine if it's a valid bug/feature** (not just a question)
3. **Convert to issue** using GitHub's "Convert to Issue" feature:
- Go to the discussion
- Click "⋯" menu → "Convert to issue"
- Add appropriate labels and assignees
#### Triage Guidelines:
**Convert to Issue:**
- ✅ Confirmed bugs with reproduction steps
- ✅ Well-defined feature requests with clear use cases
- ✅ Security vulnerabilities (after initial assessment)
**Keep as Discussion:**
- ❌ Usage questions ("How do I...?")
- ❌ Configuration help
- ❌ Unclear or incomplete bug reports
- ❌ Feature ideas that need more discussion/refinement
### Benefits:
- 📊 **Better issue quality** - Only confirmed bugs/features become issues
- 🎯 **Easier triage** - Questions don't clutter the issue tracker
- 💬 **Community involvement** - Discussion encourages collaboration before formal issues
- 🧹 **Cleaner issue tracker** - Focus on actionable items only
## Re-enabling Templates (If Needed)
If you need to temporarily re-enable direct issue creation:
```bash
# Re-enable templates
mv issue-report.md.disabled issue-report.md
mv feature_request.md.disabled feature_request.md
# Update config.yml to add them back
```
## Template Files:
- `config.yml` - Main configuration (redirects all to discussions)
- `issue-report.md.disabled` - Bug report template (disabled)
- `feature_request.md.disabled` - Feature request template (disabled)
================================================
FILE: .github/ISSUE_TEMPLATE/config.yml
================================================
blank_issues_enabled: false
contact_links:
- name: Report a Bug (Discussion First)
url: https://github.com/projectdiscovery/katana/discussions/new?category=q-a
about: Report bugs or issues via discussion for proper triage before issue creation
- name: Feature Request (Discussion First)
url: https://github.com/projectdiscovery/katana/discussions/new?category=ideas
about: Share feature ideas via discussion for evaluation before implementation
- name: Ask a Question
url: https://github.com/projectdiscovery/katana/discussions/new?category=q-a
about: Ask questions about usage, configuration, or troubleshooting
- name: General Discussion
url: https://github.com/projectdiscovery/katana/discussions/new?category=general
about: General discussion about the project
- name: Security Issues
url: mailto:security@projectdiscovery.io
about: Report security vulnerabilities privately via email
- name: Discord Community
url: https://discord.gg/projectdiscovery
about: Connect with PD Team and community for real-time discussion
================================================
FILE: .github/ISSUE_TEMPLATE/feature_request.md.disabled
================================================
---
name: Feature request
about: Request feature to implement in this project
labels: 'Type: Enhancement'
---
<!--
1. Please make sure to provide a detailed description with all the relevant information that might be required to start working on this feature.
2. In case you are not sure about your request or whether the particular feature is already supported or not, please start a discussion instead.
3. GitHub Discussion: https://github.com/projectdiscovery/katana/discussions/categories/ideas
4. Join our discord server at https://discord.gg/projectdiscovery to discuss the idea on the #katana channel.
-->
### Please describe your feature request:
<!-- A clear and concise description of feature to implement -->
### Describe the use case of this feature:
<!-- A clear and concise description of the feature request's motivation and the use-cases in which it could be useful. -->
================================================
FILE: .github/ISSUE_TEMPLATE/issue-report.md.disabled
================================================
---
name: Issue report
about: Create a report to help us to improve the project
labels: 'Type: Bug'
---
<!--
1. Please search to see if an issue already exists for the bug you encountered.
2. For support requests, FAQs or "How to" questions, please use the GitHub Discussions section instead - https://github.com/projectdiscovery/katana/discussions or
3. Join our discord server at https://discord.gg/projectdiscovery and post the question on the #katana channel.
-->
<!-- ISSUES MISSING IMPORTANT INFORMATION MAY BE CLOSED WITHOUT INVESTIGATION. -->
### katana version:
<!-- You can find current version of katana with "katana -version" -->
<!-- We only accept issues that are reproducible on the latest version of katana. -->
<!-- You can find the latest version of project at https://github.com/projectdiscovery/katana/releases/ -->
### Current Behavior:
<!-- A concise description of what you're experiencing. -->
### Expected Behavior:
<!-- A concise description of what you expected to happen. -->
### Steps To Reproduce:
<!--
Example: steps to reproduce the behavior:
1. Run 'katana ..'
2. See error...
-->
### Anything else:
<!-- Links? References? Screnshots? Anything that will give us more context about the issue that you are encountering! -->
================================================
FILE: .github/MAINTAINER_GUIDE.md
================================================
# Maintainer Guide: Discussion-First Issue Management
## Overview
Katana now uses a **discussion-first approach** for issue management to improve triage quality and reduce noise from questions being filed as bugs.
## How It Works
### 1. **All Reports Start as Discussions**
- Users cannot create issues directly
- All bug reports → **Q&A Discussions**
- All feature requests → **Ideas Discussions**
- All questions → **Q&A Discussions**
### 2. **Automated Triage Helper**
- Auto-responds to discussions with helpful guidance
- Auto-flags potential bugs with keywords detection
- Provides checklists for proper bug reporting
### 3. **Maintainer Conversion Process**
- Review discussions for completeness
- Convert valid issues using GitHub's built-in feature
- Apply appropriate labels during conversion
## Conversion Guidelines
### 🐛 **Bug Reports** → Convert to Issue When:
**Well-Defined Problems:**
- Clear reproduction steps provided
- Katana version specified
- Expected vs actual behavior described
- Environment details included
- Error messages/logs included
**Confirmed Bugs:**
- Issue reproduced by maintainer or community
- Not a configuration/usage question
- Not working as designed
**Keep as Discussion:**
- Incomplete information
- Usage questions ("How do I...?")
- Configuration problems
- Working as intended
### 💡 **Feature Requests** → Convert to Issue When:
**Solid Proposals:**
- Clear use case defined
- Benefits to community explained
- Implementation approach considered
- Not easily achievable with existing features
**Community Support:**
- Multiple users expressing interest
- Maintainer approval for implementation
- Fits project roadmap
**Keep as Discussion:**
- Vague ideas needing refinement
- Better suited as external tools/plugins
- Conflicts with project goals
- Needs more community input
## Conversion Process
### Using GitHub's Convert Feature:
1. **Open the discussion**
2. **Click the "⋯" menu** (top right)
3. **Select "Convert to issue"**
4. **Choose repository** (same repo)
5. **Review title/body** - edit if needed
6. **Add labels:**
- `Type: Bug` for confirmed bugs
- `Type: Enhancement` for approved features
- `Priority: High/Medium/Low` as appropriate
- `Component: Engine/Parser/Output` etc.
### Template for Converted Issues:
When converting, consider adding this note:
```markdown
**Converted from Discussion:** #[discussion_number]
<!-- Original discussion provided community input and initial triage -->
[Original discussion content here]
---
**Maintainer Notes:**
- [ ] Issue confirmed through discussion
- [ ] Reproduction steps verified
- [ ] Ready for implementation/investigation
```
## Workflow Benefits
### **For Project Health:**
- **Cleaner issue tracker** - Only actionable items
- **Better metrics** - Issues vs discussions clearly separated
- **Faster resolution** - Less time sorting questions from bugs
### **For Community:**
- **Inclusive discussions** - Everyone can participate in triage
- **Better help** - Community can answer questions quickly
- **Learning opportunity** - Users see resolution process
### **For Maintainers:**
- **Pre-filtered issues** - Only valid bugs/features reach issue tracker
- **Rich context** - Discussion history provides background
- **Community input** - Others help validate before conversion
## Examples
### **Good Bug Discussion → Issue Conversion**
**Discussion Title:** "Katana crashes when using -hl with custom headers"
**Discussion Body:**
- Katana version: v1.2.1
- Command: `katana -u example.com -hl -H "Custom: value"`
- Error: panic in hybrid engine
- Platform: macOS 14.1
- Reproduction: consistent crash
**→ Convert to Issue:** Clear bug with reproduction steps
### **Question → Keep as Discussion**
**Discussion Title:** "How to crawl only PDF files?"
**Discussion Body:**
- New user question
- Asking for usage help
- Not a bug or feature request
**→ Keep as Discussion:** Usage question, answer in discussion
### **Needs More Info → Keep as Discussion**
**Discussion Title:** "Katana doesn't work"
**Discussion Body:**
- Vague description
- No version, command, or error details
- No reproduction steps
**→ Keep as Discussion:** Request more information first
## Quick Reference
| Type | Action | Labels for Conversion |
|------|---------|-------------------|
| **Confirmed Bug** | Convert → Issue | `Type: Bug`, `Priority: [level]` |
| **Approved Feature** | Convert → Issue | `Type: Enhancement`, `Priority: [level]` |
| **Usage Question** | Keep → Discussion | N/A |
| **Needs Info** | Keep → Discussion | N/A |
| **Security Issue** | Email → security@projectdiscovery.io | N/A |
This workflow ensures high-quality issues while maintaining an inclusive, helpful community environment!
================================================
FILE: .github/PULL_REQUEST_TEMPLATE.md
================================================
## Proposed changes
<!-- Describe the overall picture of your modifications to help maintainers understand the pull request. PRs are required to be associated to their related issue tickets or feature request. -->
### Proof
<!-- How has this been tested? Please describe the tests that you ran to verify your changes. -->
## Checklist
<!-- Put an "x" in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code. -->
- [ ] Pull request is created against the [dev](https://github.com/projectdiscovery/katana/tree/dev) branch
- [ ] All checks passed (lint, unit/integration/regression tests etc.) with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added necessary documentation (if appropriate)
================================================
FILE: .github/dependabot.yml
================================================
# To get started with Dependabot version updates, you'll need to specify which
# package ecosystems to update and where the package manifests are located.
# Please see the documentation for all configuration options:
# https://help.github.com/github/administering-a-repository/configuration-options-for-dependency-updates
version: 2
updates:
# Maintain dependencies for go modules
- package-ecosystem: "gomod"
directory: "/"
schedule:
interval: "weekly"
target-branch: "dev"
commit-message:
prefix: "chore"
include: "scope"
labels:
- "Type: Maintenance"
allow:
- dependency-name: "github.com/projectdiscovery/*"
# Maintain dependencies for GitHub Actions
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
target-branch: "dev"
commit-message:
prefix: "chore"
include: "scope"
labels:
- "Type: Maintenance"
# Maintain dependencies for docker
- package-ecosystem: "docker"
directory: "/"
schedule:
interval: "weekly"
target-branch: "dev"
commit-message:
prefix: "chore"
include: "scope"
labels:
- "Type: Maintenance"
================================================
FILE: .github/release.yml
================================================
changelog:
exclude:
authors:
- app/dependabot
- dependabot
categories:
- title: 🎉 New Features
labels:
- "Type: Enhancement"
- title: 🐞 Bug Fixes
labels:
- "Type: Bug"
- title: 🔨 Maintenance
labels:
- "Type: Maintenance"
- title: Other Changes
labels:
- "*"
================================================
FILE: .github/workflows/build-test.yml
================================================
name: 🔨 Build Test
on:
workflow_dispatch:
pull_request:
branches:
- dev
paths:
- '**.go'
- '**.mod'
jobs:
lint:
name: "Lint"
if: "${{ !endsWith(github.actor, '[bot]') }}"
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/go@v1
- uses: projectdiscovery/actions/golangci-lint/v2@v1
build:
name: Test Builds
needs: [lint]
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
steps:
- name: Check out code
uses: actions/checkout@v6
- name: Set up Go
uses: projectdiscovery/actions/setup/go@v1
- name: Test
run: go test ./...
working-directory: .
- name: Build
run: go build .
working-directory: cmd/katana/
- name: Integration Tests
env:
GH_ACTION: true
run: bash run.sh
working-directory: integration_tests/
- name: Install
run: go install
working-directory: cmd/katana/
- name: Race Condition Tests
run: go build -race .
working-directory: cmd/katana/
================================================
FILE: .github/workflows/codeql-analysis.yml
================================================
name: 🚨 CodeQL Analysis
on:
workflow_dispatch:
pull_request:
branches:
- dev
paths:
- '**.go'
- '**.mod'
jobs:
analyze:
name: Analyze
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
security-events: write
strategy:
fail-fast: false
matrix:
language: [ 'go' ]
# CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', 'python' ]
steps:
- name: Checkout repository
uses: actions/checkout@v6
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
with:
languages: ${{ matrix.language }}
- name: Autobuild
uses: github/codeql-action/autobuild@v4
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
================================================
FILE: .github/workflows/compat-checks.yaml
================================================
name: ♾️ Compatibility Checks
on:
pull_request:
types: [opened, synchronize]
branches:
- dev
jobs:
check:
if: github.actor == 'dependabot[bot]'
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@v6
- uses: projectdiscovery/actions/setup/go/compat-checks@v1
================================================
FILE: .github/workflows/dep-auto-merge.yml
================================================
name: 🤖 Auto Merge
on:
pull_request_review:
types: [submitted]
workflow_run:
workflows: ["♾️ Compatibility Check"]
types:
- completed
permissions:
pull-requests: write
issues: write
repository-projects: write
jobs:
auto-merge:
runs-on: ubuntu-latest
if: github.actor == 'dependabot[bot]'
steps:
- uses: actions/checkout@v6
with:
token: ${{ secrets.DEPENDABOT_PAT }}
- uses: ahmadnassri/action-dependabot-auto-merge@v2
with:
github-token: ${{ secrets.DEPENDABOT_PAT }}
target: all
================================================
FILE: .github/workflows/discussion-triage.yml
================================================
name: Discussion Triage Helper
on:
discussion:
types: [created]
permissions:
discussions: write
jobs:
triage:
runs-on: ubuntu-latest
steps:
- name: Add initial response to discussions
uses: actions/github-script@v8
with:
script: |
const { discussion } = context.payload;
// Add helpful response based on discussion category
let response = "";
if (discussion.category.slug === "q-a") {
if (discussion.title.toLowerCase().includes("bug") ||
discussion.title.toLowerCase().includes("error") ||
discussion.title.toLowerCase().includes("crash") ||
discussion.title.toLowerCase().includes("fail")) {
response = "Thanks for the discussion! This looks like it might be a bug report.\n\n" +
"**For Maintainers:** If this is confirmed to be a bug with proper reproduction steps, consider converting to an issue using \"Convert to issue\" in the discussion menu.\n\n" +
"**For the Reporter:** To help us triage this effectively, please provide:\n" +
"- [ ] Katana version (`katana -version`)\n" +
"- [ ] Complete command that reproduces the issue\n" +
"- [ ] Expected vs actual behavior\n" +
"- [ ] Operating system and environment details\n" +
"- [ ] Any error messages or logs\n\n" +
"This helps us determine if this should become a tracked issue. Thank you!";
} else {
response = "Thanks for your question!\n\n" +
"The community and maintainers will help answer this. If this turns out to be a bug or feature request after discussion, a maintainer can convert it to a tracked issue.\n\n" +
"**Tip:** For faster responses, you can also join our [Discord server](https://discord.gg/projectdiscovery) for real-time help!";
}
}
else if (discussion.category.slug === "ideas") {
response = "Thanks for sharing this idea!\n\n" +
"**For Maintainers:** If this feature request is well-defined and approved for implementation, consider converting to an issue using \"Convert to issue\" in the discussion menu.\n\n" +
"**For the Reporter:** To help evaluate this feature:\n" +
"- [ ] Describe the specific use case this would solve\n" +
"- [ ] Explain how this would benefit the katana user community\n" +
"- [ ] Consider if this could be implemented as a plugin or external tool\n" +
"- [ ] Check if similar functionality already exists\n\n" +
"Well-defined features with clear benefits may be converted to tracked issues for implementation. Thanks for contributing!";
}
if (response) {
await github.rest.discussions.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
discussion_number: discussion.number,
body: response
});
}
- name: Auto-label potential bug reports
uses: actions/github-script@v8
with:
script: |
const { discussion } = context.payload;
const title = discussion.title.toLowerCase();
const body = discussion.body.toLowerCase();
// Check for keywords that suggest this might be a bug
const bugKeywords = [
"bug", "error", "crash", "fail", "broken", "issue", "problem",
"exception", "panic", "segfault", "hang", "freeze", "incorrect"
];
const hasBugKeywords = bugKeywords.some(keyword =>
title.includes(keyword) || body.includes(keyword)
);
if (hasBugKeywords && discussion.category.slug === "q-a") {
// Add a comment suggesting this might need to be converted to an issue
const detectedKeywords = bugKeywords.filter(k => title.includes(k) || body.includes(k));
await github.rest.discussions.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
discussion_number: discussion.number,
body: "**Maintainer Note:** This discussion contains potential bug keywords and may need to be converted to a tracked issue after confirmation.\n\n" +
"**Keywords detected:** " + detectedKeywords.join(', ')
});
}
================================================
FILE: .github/workflows/dockerhub-push.yml
================================================
name: 🌥 Docker Push
on:
workflow_run:
workflows: ["🎉 Release Binary"]
types:
- completed
workflow_dispatch:
jobs:
docker:
runs-on: ubuntu-latest-16-cores
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Get Github tag
id: meta
run: |
curl --silent "https://api.github.com/repos/projectdiscovery/katana/releases/latest" | jq -r .tag_name | xargs -I {} echo TAG={} >> $GITHUB_OUTPUT
- name: Set up QEMU
uses: docker/setup-qemu-action@v3
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Login to DockerHub
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
- name: Build and push
uses: docker/build-push-action@v6
with:
context: .
platforms: linux/amd64,linux/arm64
push: true
tags: projectdiscovery/katana:latest,projectdiscovery/katana:${{ steps.meta.outputs.TAG }}
================================================
FILE: .github/workflows/functional-test.yml
================================================
name: 🧪 Functional Test
on:
pull_request:
paths:
- '**.go'
- '**.mod'
workflow_dispatch:
jobs:
functional:
name: Functional Test
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macOS-latest]
steps:
- name: Check out code
uses: actions/checkout@v6
- name: Set up Go
uses: projectdiscovery/actions/setup/go@v1
- name: Functional Tests
run: |
chmod +x run.sh
bash run.sh ${{ matrix.os }}
working-directory: cmd/functional-test
================================================
FILE: .github/workflows/release-binary.yml
================================================
name: 🎉 Release Binary
on:
push:
tags:
- v*
workflow_dispatch:
jobs:
build-mac:
runs-on: macos-latest
steps:
- name: "Check out code"
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: "Set up Go"
uses: projectdiscovery/actions/setup/go@v1
- name: "Create release on GitHub"
uses: goreleaser/goreleaser-action@v7
with:
args: "release -f .goreleaser/mac.yml --clean"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
build-windows:
runs-on: windows-latest-8-cores
steps:
- name: "Check out code"
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: "Set up Go"
uses: projectdiscovery/actions/setup/go@v1
- name: "Create release on GitHub"
uses: goreleaser/goreleaser-action@v7
with:
args: "release -f .goreleaser/windows.yml --clean"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
build-linux:
runs-on: ubuntu-latest-16-cores
steps:
- name: "Check out code"
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: "Set up Go"
uses: projectdiscovery/actions/setup/go@v1
# todo: musl compatible?
- name: Install Dependences
run: sudo apt install gcc-aarch64-linux-gnu
- name: "Create release on GitHub"
uses: goreleaser/goreleaser-action@v7
with:
args: "release -f .goreleaser/linux.yml --clean"
version: latest
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
SLACK_WEBHOOK: "${{ secrets.RELEASE_SLACK_WEBHOOK }}"
DISCORD_WEBHOOK_ID: "${{ secrets.DISCORD_WEBHOOK_ID }}"
DISCORD_WEBHOOK_TOKEN: "${{ secrets.DISCORD_WEBHOOK_TOKEN }}"
================================================
FILE: .github/workflows/release-test.yml
================================================
name: 🔨 Release Test
on:
pull_request:
paths:
- "**.yml"
- "**.go"
- "**.mod"
workflow_dispatch:
jobs:
release-test-mac:
runs-on: macos-latest
steps:
- name: "Check out code"
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Go
uses: projectdiscovery/actions/setup/go@v1
- name: release test
uses: projectdiscovery/actions/goreleaser@v1
with:
args: --config=.goreleaser/mac.yml
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
release-test-linux:
runs-on: ubuntu-latest-16-cores
steps:
- name: "Check out code"
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Go
uses: projectdiscovery/actions/setup/go@v1
# todo: musl compatible?
- name: Install Dependences
run: sudo apt update && sudo apt install gcc-aarch64-linux-gnu
- name: release test
uses: projectdiscovery/actions/goreleaser@v1
with:
args: --config=.goreleaser/linux.yml
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
release-test-windows:
runs-on: windows-latest-8-cores
steps:
- name: "Check out code"
uses: actions/checkout@v6
with:
fetch-depth: 0
- name: Set up Go
uses: projectdiscovery/actions/setup/go@v1
- name: release test
uses: projectdiscovery/actions/goreleaser@v1
with:
args: --config=.goreleaser/windows.yml
workdir: .
env:
GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
================================================
FILE: .github/workflows/security-crawl-maze-score.yaml
================================================
name: 🙏🏻 Security Crawl Maze Score
on:
workflow_dispatch:
jobs:
build:
name: Run Scoring
runs-on: ubuntu-latest-16-cores
steps:
- name: Check out code
uses: actions/checkout@v6
- name: Set up Go
uses: projectdiscovery/actions/setup/go@v1
- name: Build
run: go build .
working-directory: cmd/katana/
- name: Run Katana Standard
run: ./katana -u https://security-crawl-maze.app/ -kf all -jc -jsluice -d 10 -o output_standard.txt -cos node_modules
working-directory: cmd/katana
- name: Run Katana Headless
run: ./katana -u https://security-crawl-maze.app/ -kf all -jc -jsluice -d 10 -headless -o output_headless.txt -cos node_modules
working-directory: cmd/katana
- name: Run Score
run: go build .; ./crawl-maze-score ../../katana/output_standard.txt ../../katana/output_headless.txt
working-directory: cmd/tools/crawl-maze-score
================================================
FILE: .github/workflows/stale.yml
================================================
name: 💤 Stale
on:
schedule:
- cron: '0 0 * * 0' # Weekly
jobs:
stale:
runs-on: ubuntu-latest
permissions:
actions: write
contents: write # only for delete-branch option
issues: write
pull-requests: write
steps:
- uses: actions/stale@v10
with:
days-before-stale: 90
days-before-close: 7
stale-issue-label: "Status: Stale"
stale-pr-label: "Status: Stale"
stale-issue-message: >
This issue has been automatically marked as stale because it has not
had recent activity. It will be closed in 7 days if no further
activity occurs. Thank you for your contributions!
stale-pr-message: >
This pull request has been automatically marked as stale due to
inactivity. It will be closed in 7 days if no further activity
occurs. Please update if you wish to keep it open.
close-issue-message: >
This issue has been automatically closed due to inactivity. If you
think this is a mistake or would like to continue the discussion,
please comment or feel free to reopen it.
close-pr-message: >
This pull request has been automatically closed due to inactivity.
If you think this is a mistake or would like to continue working on
it, please comment or feel free to reopen it.
close-issue-label: "Status: Abandoned"
close-pr-label: "Status: Abandoned"
exempt-issue-labels: "Status: Abandoned"
exempt-pr-labels: "Status: Abandoned"
================================================
FILE: .github/workflows/workflow-monitor.yml
================================================
name: Workflow Monitor
on:
schedule:
# Run weekly on Mondays at 9 AM UTC
- cron: '0 9 * * 1'
workflow_dispatch: # Allow manual triggering
permissions:
issues: read
discussions: read
jobs:
monitor-workflow:
runs-on: ubuntu-latest
steps:
- name: Check Discussion vs Issue Activity
uses: actions/github-script@v8
with:
script: |
const oneWeekAgo = new Date();
oneWeekAgo.setDate(oneWeekAgo.getDate() - 7);
const since = oneWeekAgo.toISOString();
// Get discussions created in the last week
const discussions = await github.graphql(`
query($owner: String!, $repo: String!, $since: DateTime!) {
repository(owner: $owner, name: $repo) {
discussions(first: 100, orderBy: {field: CREATED_AT, direction: DESC}) {
nodes {
title
createdAt
category {
name
slug
}
comments(first: 1) {
totalCount
}
}
}
}
}
`, {
owner: context.repo.owner,
repo: context.repo.repo,
since: since
});
// Get issues created in the last week
const issues = await github.rest.issues.listForRepo({
owner: context.repo.owner,
repo: context.repo.repo,
since: since,
state: 'all'
});
// Filter discussions created in the last week
const recentDiscussions = discussions.repository.discussions.nodes.filter(
d => new Date(d.createdAt) >= oneWeekAgo
);
// Generate and log simple report
console.log("=== Discussion-First Workflow Report ===");
console.log(`Period: ${oneWeekAgo.toDateString()} - ${new Date().toDateString()}`);
console.log(`Discussions Created: ${recentDiscussions.length}`);
console.log(`Issues Created: ${issues.data.length}`);
console.log(`Conversion Ratio: ${issues.data.length}/${recentDiscussions.length}`);
if (recentDiscussions.length > 0) {
console.log("\nDiscussion Categories:");
recentDiscussions.forEach(d => {
console.log(`- ${d.category.name}: ${d.title} (${d.comments.totalCount} comments)`);
});
} else {
console.log("No discussions this week");
}
if (issues.data.length > 0) {
console.log("\nDirect Issues Created:");
issues.data.forEach(i => {
console.log(`- ${i.title} (${new Date(i.created_at).toDateString()})`);
});
console.log("\nWARNING: Direct issues were created - check workflow configuration");
} else {
console.log("\nNo direct issues created - workflow working correctly");
}
const responseRate = recentDiscussions.length > 0 ?
(recentDiscussions.filter(d => d.comments.totalCount > 0).length / recentDiscussions.length * 100).toFixed(1) : 0;
console.log(`Community Response Rate: ${responseRate}%`);
// You could also create an issue or discussion with this report
// For now, just log it to Actions output
================================================
FILE: .gitignore
================================================
security-crawl-maze/
cmd/katana/katana
katana
*.exe
katana_*/
katana_*/
dist/
.vscode
.devcontainer
.DS_Store
.idea
================================================
FILE: .goreleaser/linux.yml
================================================
version: 2
env:
- GO111MODULE=on
before:
hooks:
- go mod tidy
project_name: katana
builds:
- id: katana-linux-amd64-generic
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
main: ./cmd/katana/main.go
goos:
- linux
goarch:
- amd64
- id: katana-linux-i386-generic
ldflags:
- -s -w
binary: katana
main: ./cmd/katana/main.go
goos:
- linux
goarch:
- 386
- id: katana-linux-arm
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
- CC=aarch64-linux-gnu-gcc
main: ./cmd/katana/main.go
goos:
- linux
goarch:
- arm64
archives:
- formats: zip
checksum:
name_template: "{{ .ProjectName }}-linux-checksums.txt"
announce:
slack:
enabled: true
channel: '#release'
username: GoReleaser
message_template: 'New Release: {{ .ProjectName }} {{.Tag}} is published! Check it out at {{ .ReleaseURL }}'
discord:
enabled: true
message_template: '**New Release: {{ .ProjectName }} {{.Tag}}** is published! Check it out at {{ .ReleaseURL }}'
================================================
FILE: .goreleaser/mac.yml
================================================
version: 2
env:
- GO111MODULE=on
before:
hooks:
- go mod tidy
project_name: katana
builds:
- id: katana-darwin
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
main: ./cmd/katana/main.go
goos:
- darwin
goarch:
- amd64
- arm64
- 386
- arm
archives:
- formats: zip
name_template: '{{ .ProjectName }}_{{ .Version }}_{{ if eq .Os "darwin" }}macOS{{ else }}{{ .Os }}{{ end }}_{{ .Arch }}'
checksum:
name_template: "{{ .ProjectName }}-mac-checksums.txt"
================================================
FILE: .goreleaser/windows.yml
================================================
version: 2
env:
- GO111MODULE=on
before:
hooks:
- go mod tidy
project_name: katana
builds:
- id: katana-windows
ldflags:
- -s -w
binary: katana
env:
- CGO_ENABLED=1
main: ./cmd/katana/main.go
goos:
- windows
goarch:
- 386
- arm64
- amd64
archives:
- formats: zip
checksum:
name_template: "{{ .ProjectName }}-windows-checksums.txt"
================================================
FILE: Dockerfile
================================================
FROM golang:1.25.7-alpine AS build-env
RUN apk add --no-cache git gcc musl-dev
WORKDIR /app
COPY . /app
RUN go mod download
RUN go build ./cmd/katana
FROM alpine:3.23.2
RUN apk add --no-cache bind-tools ca-certificates chromium
COPY --from=build-env /app/katana /usr/local/bin/
ENTRYPOINT ["katana"]
================================================
FILE: LICENSE.md
================================================
MIT License
Copyright (c) 2022 ProjectDiscovery
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: Makefile
================================================
# Go parameters
GOCMD=go
GOBUILD=$(GOCMD) build
GOMOD=$(GOCMD) mod
GOTEST=$(GOCMD) test
GOFLAGS := -v
# This should be disabled if the binary uses pprof
LDFLAGS := -s -w
ifneq ($(shell go env GOOS),darwin)
LDFLAGS := -extldflags "-static"
endif
all: build
build:
$(GOBUILD) $(GOFLAGS) -ldflags '$(LDFLAGS)' -o "katana" cmd/katana/main.go
test:
$(GOTEST) $(GOFLAGS) ./...
integration:
cd integration_tests; bash run.sh cd ..
tidy:
$(GOMOD) tidy
lint:
golangci-lint run ./...
================================================
FILE: README.md
================================================
<h1 align="center">
<img src="https://user-images.githubusercontent.com/8293321/196779266-421c79d4-643a-4f73-9b54-3da379bbac09.png" alt="katana" width="200px">
<br>
</h1>
<h4 align="center">A next-generation crawling and spidering framework</h4>
<p align="center">
<a href="https://goreportcard.com/report/github.com/projectdiscovery/katana"><img src="https://goreportcard.com/badge/github.com/projectdiscovery/katana"></a>
<a href="https://github.com/projectdiscovery/katana/issues"><img src="https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat"></a>
<a href="https://github.com/projectdiscovery/katana/releases"><img src="https://img.shields.io/github/release/projectdiscovery/katana"></a>
<a href="https://twitter.com/pdiscoveryio"><img src="https://img.shields.io/twitter/follow/pdiscoveryio.svg?logo=twitter"></a>
<a href="https://discord.gg/projectdiscovery"><img src="https://img.shields.io/discord/695645237418131507.svg?logo=discord"></a>
</p>
<p align="center">
<a href="#features">Features</a> •
<a href="#installation">Installation</a> •
<a href="#usage">Usage</a> •
<a href="#scope-control">Scope</a> •
<a href="#crawler-configuration">Config</a> •
<a href="#filters">Filters</a> •
<a href="https://discord.gg/projectdiscovery">Join Discord</a>
</p>
# Features
- Fast And fully configurable web crawling
- **Standard** and **Headless** mode
- **JavaScript** parsing / crawling
- Customizable **automatic form filling**
- **Scope control** - Preconfigured field / Regex
- **Customizable output** - Preconfigured fields
- INPUT - **STDIN**, **URL** and **LIST**
- OUTPUT - **STDOUT**, **FILE** and **JSON**
## Installation
katana requires Go 1.25+ to install successfully. If you encounter any installation issues, we recommend trying with the latest available version of Go, as the minimum required version may have changed. Run the command below or download a pre-compiled binary from the [release page](https://github.com/projectdiscovery/katana/releases).
```console
CGO_ENABLED=1 go install github.com/projectdiscovery/katana/cmd/katana@latest
```
**More options to install / run katana-**
<details>
<summary>Docker</summary>
> To install / update docker to latest tag -
```sh
docker pull projectdiscovery/katana:latest
```
> To run katana in standard mode using docker -
```sh
docker run projectdiscovery/katana:latest -u https://tesla.com
```
> To run katana in headless mode using docker -
```sh
docker run projectdiscovery/katana:latest -u https://tesla.com -system-chrome -headless
```
</details>
<details>
<summary>Ubuntu</summary>
> It's recommended to install the following prerequisites -
```sh
sudo apt update
sudo snap refresh
sudo apt install zip curl wget git
sudo snap install golang --classic
wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add -
sudo sh -c 'echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list'
sudo apt update
sudo apt install google-chrome-stable
```
> install katana -
```sh
go install github.com/projectdiscovery/katana/cmd/katana@latest
```
</details>
## Usage
```console
katana -h
```
This will display help for the tool. Here are all the switches it supports.
```console
Katana is a fast crawler focused on execution in automation
pipelines offering both headless and non-headless crawling.
Usage:
./katana [flags]
Flags:
INPUT:
-u, -list string[] target url / list to crawl
-resume string resume scan using resume.cfg
-e, -exclude string[] exclude host matching specified filter ('cdn', 'private-ips', cidr, ip, regex)
CONFIGURATION:
-r, -resolvers string[] list of custom resolver (file or comma separated)
-d, -depth int maximum depth to crawl (default 3)
-jc, -js-crawl enable endpoint parsing / crawling in javascript file
-jsl, -jsluice enable jsluice parsing in javascript file (memory intensive)
-ct, -crawl-duration value maximum duration to crawl the target for (s, m, h, d) (default s)
-kf, -known-files string enable crawling of known files (all,robotstxt,sitemapxml), a minimum depth of 3 is required to ensure all known files are properly crawled.
-mrs, -max-response-size int maximum response size to read (default 4194304)
-timeout int time to wait for request in seconds (default 10)
-aff, -automatic-form-fill enable automatic form filling (experimental)
-fx, -form-extraction extract form, input, textarea & select elements in jsonl output
-retry int number of times to retry the request (default 1)
-proxy string http/socks5 proxy to use
-td, -tech-detect enable technology detection
-H, -headers string[] custom header/cookie to include in all http request in header:value format (file)
-config string path to the katana configuration file
-fc, -form-config string path to custom form configuration file
-flc, -field-config string path to custom field configuration file
-s, -strategy string Visit strategy (depth-first, breadth-first) (default "depth-first")
-iqp, -ignore-query-params Ignore crawling same path with different query-param values
-fsu, -filter-similar filter crawling of similar looking URLs (e.g., /users/123 and /users/456)
-fst, -filter-similar-threshold int number of distinct values before a path position is treated as parameter (default 10)
-tlsi, -tls-impersonate enable experimental client hello (ja3) tls randomization
-dr, -disable-redirects disable following redirects (default false)
-kb, -knowledge-base enable knowledge base classification
DEBUG:
-health-check, -hc run diagnostic check up
-elog, -error-log string file to write sent requests error log
-pprof-server enable pprof server
HEADLESS:
-hl, -headless enable headless hybrid crawling (experimental)
-sc, -system-chrome use local installed chrome browser instead of katana installed
-sb, -show-browser show the browser on the screen with headless mode
-ho, -headless-options string[] start headless chrome with additional options
-nos, -no-sandbox start headless chrome in --no-sandbox mode
-cdd, -chrome-data-dir string path to store chrome browser data
-scp, -system-chrome-path string use specified chrome browser for headless crawling
-noi, -no-incognito start headless chrome without incognito mode
-cwu, -chrome-ws-url string use chrome browser instance launched elsewhere with the debugger listening at this URL
-xhr, -xhr-extraction extract xhr request url,method in jsonl output
-csp, -captcha-solver-provider string captcha solver provider (e.g. capsolver)
-csk, -captcha-solver-key string captcha solver provider api key
SCOPE:
-cs, -crawl-scope string[] in scope url regex to be followed by crawler
-cos, -crawl-out-scope string[] out of scope url regex to be excluded by crawler
-fs, -field-scope string pre-defined scope field (dn,rdn,fqdn) or custom regex (e.g., '(company-staging.io|company.com)') (default "rdn")
-ns, -no-scope disables host based default scope
-do, -display-out-scope display external endpoint from scoped crawling
FILTER:
-mr, -match-regex string[] regex or list of regex to match on output url (cli, file)
-fr, -filter-regex string[] regex or list of regex to filter on output url (cli, file)
-f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir) (Deprecated: use -output-template instead)
-sf, -store-field string field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
-em, -extension-match string[] match output for given extension (eg, -em php,html,js)
-ef, -extension-filter string[] filter output for given extension (eg, -ef png,css)
-ndef, -no-default-ext-filter bool remove default extensions from the filter list
-mdc, -match-condition string match response with dsl based condition
-fdc, -filter-condition string filter response with dsl based condition
-duf, -disable-unique-filter disable duplicate content filtering
-fpt, -filter-page-type string[] filter response with page type (e.g. error,captcha,parked)
RATE-LIMIT:
-c, -concurrency int number of concurrent fetchers to use (default 10)
-p, -parallelism int number of concurrent inputs to process (default 10)
-rd, -delay int request delay between each request in seconds
-rl, -rate-limit int maximum requests to send per second (default 150)
-rlm, -rate-limit-minute int maximum number of requests to send per minute
UPDATE:
-up, -update update katana to latest version
-duc, -disable-update-check disable automatic katana update check
OUTPUT:
-o, -output string file to write output to
-ot, -output-template string custom output template
-sr, -store-response store http requests/responses
-srd, -store-response-dir string store http requests/responses to custom directory
-ncb, -no-clobber do not overwrite output file
-sfd, -store-field-dir string store per-host field to custom directory
-or, -omit-raw omit raw requests/responses from jsonl output
-ob, -omit-body omit response body from jsonl output
-lof, -list-output-fields list available fields for jsonl output format
-eof, -exclude-output-fields exclude fields from jsonl output
-j, -jsonl write output in jsonl format
-nc, -no-color disable output content coloring (ANSI escape codes)
-silent display output only
-v, -verbose display verbose output
-debug display debug output
-version display project version
```
## Running Katana
### Input for katana
**katana** requires **url** or **endpoint** to crawl and accepts single or multiple inputs.
Input URL can be provided using `-u` option, and multiple values can be provided using comma-separated input, similarly **file** input is supported using `-list` option and additionally piped input (stdin) is also supported.
#### URL Input
```sh
katana -u https://tesla.com
```
#### Multiple URL Input (comma-separated)
```sh
katana -u https://tesla.com,https://google.com
```
#### List Input
```bash
$ cat url_list.txt
https://tesla.com
https://google.com
```
```
katana -list url_list.txt
```
#### STDIN (piped) Input
```sh
echo https://tesla.com | katana
```
```sh
cat domains | httpx | katana
```
Example running katana -
```console
katana -u https://youtube.com
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/ v0.0.1
projectdiscovery.io
[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
https://www.youtube.com/
https://www.youtube.com/about/
https://www.youtube.com/about/press/
https://www.youtube.com/about/copyright/
https://www.youtube.com/t/contact_us/
https://www.youtube.com/creators/
https://www.youtube.com/ads/
https://www.youtube.com/t/terms
https://www.youtube.com/t/privacy
https://www.youtube.com/about/policies/
https://www.youtube.com/howyoutubeworks?utm_campaign=ytgen&utm_source=ythp&utm_medium=LeftNav&utm_content=txt&u=https%3A%2F%2Fwww.youtube.com%2Fhowyoutubeworks%3Futm_source%3Dythp%26utm_medium%3DLeftNav%26utm_campaign%3Dytgen
https://www.youtube.com/new
https://m.youtube.com/
https://www.youtube.com/s/desktop/4965577f/jsbin/desktop_polymer.vflset/desktop_polymer.js
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-home-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/cssbin/www-onepick.css
https://www.youtube.com/s/_/ytmainappweb/_/ss/k=ytmainappweb.kevlar_base.0Zo5FUcPkCg.L.B1.O/am=gAE/d=0/rs=AGKMywG5nh5Qp-BGPbOaI1evhF5BVGRZGA
https://www.youtube.com/opensearch?locale=en_GB
https://www.youtube.com/manifest.webmanifest
https://www.youtube.com/s/desktop/4965577f/cssbin/www-main-desktop-watch-page-skeleton.css
https://www.youtube.com/s/desktop/4965577f/jsbin/web-animations-next-lite.min.vflset/web-animations-next-lite.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/custom-elements-es5-adapter.vflset/custom-elements-es5-adapter.js
https://www.youtube.com/s/desktop/4965577f/jsbin/webcomponents-sd.vflset/webcomponents-sd.js
https://www.youtube.com/s/desktop/4965577f/jsbin/intersection-observer.min.vflset/intersection-observer.min.js
https://www.youtube.com/s/desktop/4965577f/jsbin/scheduler.vflset/scheduler.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-i18n-constants-en_GB.vflset/www-i18n-constants.js
https://www.youtube.com/s/desktop/4965577f/jsbin/www-tampering.vflset/www-tampering.js
https://www.youtube.com/s/desktop/4965577f/jsbin/spf.vflset/spf.js
https://www.youtube.com/s/desktop/4965577f/jsbin/network.vflset/network.js
https://www.youtube.com/howyoutubeworks/
https://www.youtube.com/trends/
https://www.youtube.com/jobs/
https://www.youtube.com/kids/
```
## Crawling Mode
### Standard Mode
Standard crawling modality uses the standard go http library under the hood to handle HTTP requests/responses. This modality is much faster as it doesn't have the browser overhead. Still, it analyzes HTTP responses body as is, without any javascript or DOM rendering, potentially missing post-dom-rendered endpoints or asynchronous endpoint calls that might happen in complex web applications depending, for example, on browser-specific events.
### Headless Mode
Headless mode hooks internal headless calls to handle HTTP requests/responses directly within the browser context. This offers two advantages:
- The HTTP fingerprint (TLS and user agent) fully identify the client as a legitimate browser
- Better coverage since the endpoints are discovered analyzing the standard raw response, as in the previous modality, and also the browser-rendered one with javascript enabled.
Headless crawling is optional and can be enabled using `-headless` option.
Here are other headless CLI options -
```console
katana -h headless
Flags:
HEADLESS:
-hl, -headless enable headless hybrid crawling (experimental)
-sc, -system-chrome use local installed chrome browser instead of katana installed
-sb, -show-browser show the browser on the screen with headless mode
-ho, -headless-options string[] start headless chrome with additional options
-nos, -no-sandbox start headless chrome in --no-sandbox mode
-cdd, -chrome-data-dir string path to store chrome browser data
-scp, -system-chrome-path string use specified chrome browser for headless crawling
-noi, -no-incognito start headless chrome without incognito mode
-cwu, -chrome-ws-url string use chrome browser instance launched elsewhere with the debugger listening at this URL
-xhr, -xhr-extraction extract xhr requests
-csp, -captcha-solver-provider string captcha solver provider (e.g. capsolver)
-csk, -captcha-solver-key string captcha solver provider api key
```
*`-no-sandbox`*
----
Runs headless chrome browser with **no-sandbox** option, useful when running as root user.
```console
katana -u https://tesla.com -headless -no-sandbox
```
*`-no-incognito`*
----
Runs headless chrome browser without incognito mode, useful when using the local browser.
```console
katana -u https://tesla.com -headless -no-incognito
```
*`-headless-options`*
----
When crawling in headless mode, additional chrome options can be specified using `-headless-options`, for example -
```console
katana -u https://tesla.com -headless -system-chrome -headless-options --disable-gpu,proxy-server=http://127.0.0.1:8080
```
### Captcha Solving
Katana supports automatic captcha detection and solving during headless crawling. When a captcha page is encountered, katana identifies the captcha provider, solves it via an external service, and continues crawling.
Supported captcha types: **reCAPTCHA v2**, **reCAPTCHA v3**, **reCAPTCHA Enterprise**, **Cloudflare Turnstile**, **hCaptcha**
*`-captcha-solver-provider`*
----
Option to specify the captcha solver provider. Currently supported: `capsolver`.
*`-captcha-solver-key`*
----
API key for the captcha solver provider.
```console
katana -u https://example.com -headless -csp capsolver -csk YOUR_API_KEY
```
The provider and key can also be set via environment variables:
```console
export CAPTCHA_SOLVER_PROVIDER=capsolver
export CAPTCHA_SOLVER_KEY=YOUR_API_KEY
katana -u https://example.com -headless
```
## Scope Control
Crawling can be endless if not scoped, as such katana comes with multiple support to define the crawl scope.
*`-field-scope`*
----
Most handy option to define scope with predefined field name, `rdn` being default option for field scope.
- `rdn` - crawling scoped to root domain name and all subdomains (e.g. `*example.com`) (default)
- `fqdn` - crawling scoped to given sub(domain) (e.g. `www.example.com` or `api.example.com`)
- `dn` - crawling scoped to domain name keyword (e.g. `example`)
```console
katana -u https://tesla.com -fs dn
```
*`-crawl-scope`*
------
For advanced scope control, `-cs` option can be used that comes with **regex** support.
```console
katana -u https://tesla.com -cs login
```
For multiple in scope rules, file input with multiline string / regex can be passed.
```bash
$ cat in_scope.txt
login/
admin/
app/
wordpress/
```
```console
katana -u https://tesla.com -cs in_scope.txt
```
*`-crawl-out-scope`*
-----
For defining what not to crawl, `-cos` option can be used and also support **regex** input.
```console
katana -u https://tesla.com -cos logout
```
For multiple out of scope rules, file input with multiline string / regex can be passed.
```bash
$ cat out_of_scope.txt
/logout
/log_out
```
```console
katana -u https://tesla.com -cos out_of_scope.txt
```
*`-no-scope`*
----
Katana is default to scope `*.domain`, to disable this `-ns` option can be used and also to crawl the internet.
```console
katana -u https://tesla.com -ns
```
*`-display-out-scope`*
----
As default, when scope option is used, it also applies for the links to display as output, as such **external URLs are default to exclude** and to overwrite this behavior, `-do` option can be used to display all the external URLs that exist in targets scoped URL / Endpoint.
```
katana -u https://tesla.com -do
```
Here is all the CLI options for the scope control -
```console
katana -h scope
Flags:
SCOPE:
-cs, -crawl-scope string[] in scope url regex to be followed by crawler
-cos, -crawl-out-scope string[] out of scope url regex to be excluded by crawler
-fs, -field-scope string pre-defined scope field (dn,rdn,fqdn) (default "rdn")
-ns, -no-scope disables host based default scope
-do, -display-out-scope display external endpoint from scoped crawling
```
## Crawler Configuration
Katana comes with multiple options to configure and control the crawl as the way we want.
*`-depth`*
----
Option to define the `depth` to follow the urls for crawling, the more depth the more number of endpoint being crawled + time for crawl.
```
katana -u https://tesla.com -d 5
```
*`-js-crawl`*
----
Option to enable JavaScript file parsing + crawling the endpoints discovered in JavaScript files, disabled as default.
```
katana -u https://tesla.com -jc
```
*`-crawl-duration`*
----
Option to predefined crawl duration, disabled as default.
```
katana -u https://tesla.com -ct 2
```
*`-known-files`*
----
Option to enable crawling `robots.txt` and `sitemap.xml` file, disabled as default.
```
katana -u https://tesla.com -kf robotstxt,sitemapxml
```
*`-automatic-form-fill`*
----
Option to enable automatic form filling for known / unknown fields, known field values can be customized as needed by updating form config file at `$HOME/.config/katana/form-config.yaml`.
Automatic form filling is experimental feature.
```
katana -u https://tesla.com -aff
```
*`-filter-similar`*
----
Option to filter crawling of similar looking URLs by normalizing variable path segments. This detects IDs, UUIDs, hashes, dates, and other dynamic values, and also learns repeating patterns at runtime. For example, `/users/123` and `/users/456` are treated as the same endpoint.
```
katana -u https://tesla.com -fsu
```
The promotion threshold (how many distinct values at a path position before it's treated as a parameter) can be tuned with `-fst`. Lower values are more aggressive (fewer URLs crawled), higher values are more permissive. Default is `10`.
```
katana -u https://tesla.com -fsu -fst 5
```
## Authenticated Crawling
Authenticated crawling involves including custom headers or cookies in HTTP requests to access protected resources. These headers provide authentication or authorization information, allowing you to crawl authenticated content / endpoint. You can specify headers directly in the command line or provide them as a file with katana to perform authenticated crawling.
> **Note**: User needs to be manually perform the authentication and export the session cookie / header to file to use with katana.
*`-headers`*
----
Option to add a custom header or cookie to the request.
> Syntax of [headers](https://datatracker.ietf.org/doc/html/rfc7230#section-3.2) in the HTTP specification
Here is an example of adding a cookie to the request:
```
katana -u https://tesla.com -H 'Cookie: usrsess=AmljNrESo'
```
It is also possible to supply headers or cookies as a file. For example:
```
$ cat cookie.txt
Cookie: PHPSESSIONID=XXXXXXXXX
X-API-KEY: XXXXX
TOKEN=XX
```
```
katana -u https://tesla.com -H cookie.txt
```
There are more options to configure when needed, here is all the config related CLI options -
```console
katana -h config
Flags:
CONFIGURATION:
-r, -resolvers string[] list of custom resolver (file or comma separated)
-d, -depth int maximum depth to crawl (default 3)
-jc, -js-crawl enable endpoint parsing / crawling in javascript file
-ct, -crawl-duration int maximum duration to crawl the target for
-kf, -known-files string enable crawling of known files (all,robotstxt,sitemapxml)
-mrs, -max-response-size int maximum response size to read (default 9223372036854775807)
-timeout int time to wait for request in seconds (default 10)
-aff, -automatic-form-fill enable automatic form filling (experimental)
-fx, -form-extraction enable extraction of form, input, textarea & select elements
-retry int number of times to retry the request (default 1)
-proxy string http/socks5 proxy to use
-H, -headers string[] custom header/cookie to include in request
-config string path to the katana configuration file
-fc, -form-config string path to custom form configuration file
-flc, -field-config string path to custom field configuration file
-s, -strategy string Visit strategy (depth-first, breadth-first) (default "depth-first")
-iqp, -ignore-query-params Ignore crawling same path with different query-param values
-fsu, -filter-similar filter crawling of similar looking URLs (e.g., /users/123 and /users/456)
-fst, -filter-similar-threshold int number of distinct values before a path position is treated as parameter (default 10)
```
### Connecting to Active Browser Session
Katana can also connect to active browser session where user is already logged in and authenticated. and use it for crawling. The only requirement for this is to start browser with remote debugging enabled.
Here is an example of starting chrome browser with remote debugging enabled and using it with katana -
**step 1) First Locate path of chrome executable**
| Operating System | Chromium Executable Location | Google Chrome Executable Location |
|------------------|------------------------------|-----------------------------------|
| Windows (64-bit) | `C:\Program Files (x86)\Google\Chromium\Application\chrome.exe` | `C:\Program Files (x86)\Google\Chrome\Application\chrome.exe` |
| Windows (32-bit) | `C:\Program Files\Google\Chromium\Application\chrome.exe` | `C:\Program Files\Google\Chrome\Application\chrome.exe` |
| macOS | `/Applications/Chromium.app/Contents/MacOS/Chromium` | `/Applications/Google Chrome.app/Contents/MacOS/Google Chrome` |
| Linux | `/usr/bin/chromium` | `/usr/bin/google-chrome` |
**step 2) Start chrome with remote debugging enabled and it will return websocker url. For example, on MacOS, you can start chrome with remote debugging enabled using following command** -
```console
$ /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222
DevTools listening on ws://127.0.0.1:9222/devtools/browser/c5316c9c-19d6-42dc-847a-41d1aeebf7d6
```
> Now login to the website you want to crawl and keep the browser open.
**step 3) Now use the websocket url with katana to connect to the active browser session and crawl the website**
```console
katana -headless -u https://tesla.com -cwu ws://127.0.0.1:9222/devtools/browser/c5316c9c-19d6-42dc-847a-41d1aeebf7d6 -no-incognito
```
> **Note**: you can use `-cdd` option to specify custom chrome data directory to store browser data and cookies but that does not save session data if cookie is set to `Session` only or expires after certain time.
## Filters
*`-field`*
----
> [!WARNING]
> Deprecated: use [**`-output-template`**](#-output-template) instead. The field flag is still supported for backward compatibility.
Katana comes with built in fields that can be used to filter the output for the desired information, `-f` option can be used to specify any of the available fields.
```
-f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,key,value,kv,dir,udir)
```
Here is a table with examples of each field and expected output when used -
| FIELD | DESCRIPTION | EXAMPLE |
| ------- | --------------------------- | ------------------------------------------------------------ |
| `url` | URL Endpoint | `https://admin.projectdiscovery.io/admin/login?user=admin&password=admin` |
| `qurl` | URL including query param | `https://admin.projectdiscovery.io/admin/login.php?user=admin&password=admin` |
| `qpath` | Path including query param | `/login?user=admin&password=admin` |
| `path` | URL Path | `https://admin.projectdiscovery.io/admin/login` |
| `fqdn` | Fully Qualified Domain name | `admin.projectdiscovery.io` |
| `rdn` | Root Domain name | `projectdiscovery.io` |
| `rurl` | Root URL | `https://admin.projectdiscovery.io` |
| `ufile` | URL with File | `https://admin.projectdiscovery.io/login.js` |
| `file` | Filename in URL | `login.php` |
| `key` | Parameter keys in URL | `user,password` |
| `value` | Parameter values in URL | `admin,admin` |
| `kv` | Keys=Values in URL | `user=admin&password=admin` |
| `dir` | URL Directory name | `/admin/` |
| `udir` | URL with Directory | `https://admin.projectdiscovery.io/admin/` |
Here is an example of using field option to only display all the urls with query parameter in it -
```
katana -u https://tesla.com -f qurl -silent
https://shop.tesla.com/en_au?redirect=no
https://shop.tesla.com/en_nz?redirect=no
https://shop.tesla.com/product/men_s-raven-lightweight-zip-up-bomber-jacket?sku=1740250-00-A
https://shop.tesla.com/product/tesla-shop-gift-card?sku=1767247-00-A
https://shop.tesla.com/product/men_s-chill-crew-neck-sweatshirt?sku=1740176-00-A
https://www.tesla.com/about?redirect=no
https://www.tesla.com/about/legal?redirect=no
https://www.tesla.com/findus/list?redirect=no
```
### Custom Fields
You can create custom fields to extract and store specific information from page responses using regex rules. These custom fields are defined using a YAML config file and are loaded from the default location at `$HOME/.config/katana/field-config.yaml`. Alternatively, you can use the `-flc` option to load a custom field config file from a different location.
Here is example custom field.
```yaml
- name: email
type: regex
regex:
- '([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'
- '([a-zA-Z0-9+._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9_-]+)'
- name: phone
type: regex
regex:
- '\d{3}-\d{8}|\d{4}-\d{7}'
```
When defining custom fields, following attributes are supported:
- **name** (required)
> The value of **name** attribute is used as the `-field` cli option value.
- **type** (required)
> The type of custom attribute, currently supported option - `regex`
- **part** (optional)
> The part of the response to extract the information from. The default value is `response`, which includes both the header and body. Other possible values are `header` and `body`.
- group (optional)
> You can use this attribute to select a specific matched group in regex, for example: `group: 1`
#### Running katana using custom field:
```console
katana -u https://tesla.com -f email,phone
```
*`-store-field`*
---
To compliment `field` option which is useful to filter output at run time, there is `-sf, -store-fields` option which works exactly like field option except instead of filtering, it stores all the information on the disk under `katana_field` directory sorted by target url. Use `-sfd` or `-store-field-dir` to store data in a different location.
```
katana -u https://tesla.com -sf key,fqdn,qurl -silent
```
```bash
$ ls katana_field/
https_www.tesla.com_fqdn.txt
https_www.tesla.com_key.txt
https_www.tesla.com_qurl.txt
```
The `-store-field` option can be useful for collecting information to build a targeted wordlist for various purposes, including but not limited to:
- Identifying the most commonly used parameters
- Discovering frequently used paths
- Finding commonly used files
- Identifying related or unknown subdomains
### Katana Filters
*`-extension-match`*
---
Crawl output can be easily matched for specific extension using `-em` option to ensure to display only output containing given extension.
```
katana -u https://tesla.com -silent -em js,jsp,json
```
*`-extension-filter`*
---
Crawl output can be easily filtered for specific extension using `-ef` option which ensure to remove all the urls containing given extension.
```
katana -u https://tesla.com -silent -ef css,txt,md
```
*`-no-default-ext-filter`*
---
Katana filters several extensions by default. This can be disabled with the `-ndef` option.
```
katana -u https://tesla.com -silent -ndef
```
*`-match-regex`*
---
The `-match-regex` or `-mr` flag allows you to filter output URLs using regular expressions. When using this flag, only URLs that match the specified regular expression will be printed in the output.
```
katana -u https://tesla.com -mr 'https://shop\.tesla\.com/*' -silent
```
*`-filter-regex`*
---
The `-filter-regex` or `-fr` flag allows you to filter output URLs using regular expressions. When using this flag, it will skip the URLs that are match the specified regular expression.
```
katana -u https://tesla.com -fr 'https://www\.tesla\.com/*' -silent
```
### Advance Filtering
Katana supports DSL-based expressions for advanced matching and filtering capabilities:
- To match endpoints with a 200 status code:
```shell
katana -u https://www.hackerone.com -mdc 'status_code == 200'
```
- To match endpoints that contain "default" and have a status code other than 403:
```shell
katana -u https://www.hackerone.com -mdc 'contains(endpoint, "default") && status_code != 403'
```
- To match endpoints with PHP technologies:
```shell
katana -u https://www.hackerone.com -mdc 'contains(to_lower(technologies), "php")'
```
- To filter out endpoints running on Cloudflare:
```shell
katana -u https://www.hackerone.com -fdc 'contains(to_lower(technologies), "cloudflare")'
```
DSL functions can be applied to any keys in the jsonl output. For more information on available DSL functions, please visit the [dsl project](https://github.com/projectdiscovery/dsl).
Here are additional filter options -
```console
katana -h filter
Flags:
FILTER:
-mr, -match-regex string[] regex or list of regex to match on output url (cli, file)
-fr, -filter-regex string[] regex or list of regex to filter on output url (cli, file)
-f, -field string field to display in output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
-sf, -store-field string field to store in per-host output (url,path,fqdn,rdn,rurl,qurl,qpath,file,ufile,key,value,kv,dir,udir)
-em, -extension-match string[] match output for given extension (eg, -em php,html,js)
-ef, -extension-filter string[] filter output for given extension (eg, -ef png,css)
-ndef, -no-default-ext-filter bool remove default extensions from the filter list
-mdc, -match-condition string match response with dsl based condition
-fdc, -filter-condition string filter response with dsl based condition
-duf, -disable-unique-filter disable duplicate content filtering
```
## Rate Limit
It's easy to get blocked / banned while crawling if not following target websites limits, katana comes with multiple option to tune the crawl to go as fast / slow we want.
*`-delay`*
-----
option to introduce a delay in seconds between each new request katana makes while crawling, disabled as default.
```
katana -u https://tesla.com -delay 20
```
*`-concurrency`*
-----
option to control the number of urls per target to fetch at the same time.
```
katana -u https://tesla.com -c 20
```
*`-parallelism`*
-----
option to define number of target to process at same time from list input.
```
katana -u https://tesla.com -p 20
```
*`-rate-limit`*
-----
option to use to define max number of request can go out per second.
```
katana -u https://tesla.com -rl 100
```
*`-rate-limit-minute`*
-----
option to use to define max number of request can go out per minute.
```
katana -u https://tesla.com -rlm 500
```
Here is all long / short CLI options for rate limit control -
```console
katana -h rate-limit
Flags:
RATE-LIMIT:
-c, -concurrency int number of concurrent fetchers to use (default 10)
-p, -parallelism int number of concurrent inputs to process (default 10)
-rd, -delay int request delay between each request in seconds
-rl, -rate-limit int maximum requests to send per second (default 150)
-rlm, -rate-limit-minute int maximum number of requests to send per minute
```
## Output
Katana support both file output in plain text format as well as JSON which includes additional information like, `source`, `tag`, and `attribute` name to co-related the discovered endpoint.
*`-output`*
---
By default, katana outputs the crawled endpoints in plain text format. The results can be written to a file by using the -output option.
```console
katana -u https://example.com -no-scope -output example_endpoints.txt
```
*`-output-template`*
---
The `-output-template` option allows you to customize the output format using template, providing flexibility in defining the output structure. This option replaces the deprecated `-field` flag for filtering output. Instead of relying on predefined fields, you can specify a custom template directly in the command line to control how the extracted data is presented.
Example of using the `-output-template` option:
```sh
katana -u https://example.com -output-template '{{email}} - {{url}}'
```
In this example, `email` represents a [custom field](#custom-fields) that extracts and displays email addresses found within the source `url`.
> [!NOTE]
> If a specified field does not exist or does not contain a value, it will simply be omitted from the output.
This option can effectively structure the output in a way that best suits your use case, making data extraction more intuitive and customizable.
*`-jsonl`*
---
```console
katana -u https://example.com -jsonl | jq .
```
```json
{
"timestamp": "2023-03-20T16:23:58.027559+05:30",
"request": {
"method": "GET",
"endpoint": "https://example.com",
"raw": "GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36\r\nAccept-Encoding: gzip\r\n\r\n"
},
"response": {
"status_code": 200,
"headers": {
"accept_ranges": "bytes",
"expires": "Mon, 27 Mar 2023 10:53:58 GMT",
"last_modified": "Thu, 17 Oct 2019 07:18:26 GMT",
"content_type": "text/html; charset=UTF-8",
"server": "ECS (dcb/7EA3)",
"vary": "Accept-Encoding",
"etag": "\"3147526947\"",
"cache_control": "max-age=604800",
"x_cache": "HIT",
"date": "Mon, 20 Mar 2023 10:53:58 GMT",
"age": "331239"
},
"body": "<!doctype html>\n<html>\n<head>\n <title>Example Domain</title>\n\n <meta charset=\"utf-8\" />\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n <style type=\"text/css\">\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n </style> \n</head>\n\n<body>\n<div>\n <h1>Example Domain</h1>\n <p>This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.</p>\n <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n",
"technologies": [
"Azure",
"Amazon ECS",
"Amazon Web Services",
"Docker",
"Azure CDN"
],
"raw": "HTTP/1.1 200 OK\r\nContent-Length: 1256\r\nAccept-Ranges: bytes\r\nAge: 331239\r\nCache-Control: max-age=604800\r\nContent-Type: text/html; charset=UTF-8\r\nDate: Mon, 20 Mar 2023 10:53:58 GMT\r\nEtag: \"3147526947\"\r\nExpires: Mon, 27 Mar 2023 10:53:58 GMT\r\nLast-Modified: Thu, 17 Oct 2019 07:18:26 GMT\r\nServer: ECS (dcb/7EA3)\r\nVary: Accept-Encoding\r\nX-Cache: HIT\r\n\r\n<!doctype html>\n<html>\n<head>\n <title>Example Domain</title>\n\n <meta charset=\"utf-8\" />\n <meta http-equiv=\"Content-type\" content=\"text/html; charset=utf-8\" />\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n <style type=\"text/css\">\n body {\n background-color: #f0f0f2;\n margin: 0;\n padding: 0;\n font-family: -apple-system, system-ui, BlinkMacSystemFont, \"Segoe UI\", \"Open Sans\", \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n \n }\n div {\n width: 600px;\n margin: 5em auto;\n padding: 2em;\n background-color: #fdfdff;\n border-radius: 0.5em;\n box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\n }\n a:link, a:visited {\n color: #38488f;\n text-decoration: none;\n }\n @media (max-width: 700px) {\n div {\n margin: 0 auto;\n width: auto;\n }\n }\n </style> \n</head>\n\n<body>\n<div>\n <h1>Example Domain</h1>\n <p>This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.</p>\n <p><a href=\"https://www.iana.org/domains/example\">More information...</a></p>\n</div>\n</body>\n</html>\n"
}
}
```
*`-store-response`*
----
The `-store-response` option allows for writing all crawled endpoint requests and responses to a text file. When this option is used, text files including the request and response will be written to the **katana_response** directory. If you would like to specify a custom directory, you can use the `-store-response-dir` option.
```console
katana -u https://example.com -no-scope -store-response
```
```bash
$ cat katana_response/index.txt
katana_response/example.com/327c3fda87ce286848a574982ddd0b7c7487f816.txt https://example.com (200 OK)
katana_response/www.iana.org/bfc096e6dd93b993ca8918bf4c08fdc707a70723.txt http://www.iana.org/domains/reserved (200 OK)
```
**Note:**
*`-store-response` option is not supported in `-headless` mode.*
*`-list-output-fields`*
----
The `-list-output-fields` or `-lof` flag displays all available fields that can be used in JSONL output format. This is useful for understanding what data is available when using custom output templates or when excluding specific fields.
```console
katana -lof
```
*`-exclude-output-fields`*
----
The `-exclude-output-fields` or `-eof` flag allows you to exclude specific fields from the JSONL output. This is useful for reducing output size or focusing on specific data by removing unwanted fields.
```console
katana -u https://example.com -jsonl -eof raw,body
```
Here are additional CLI options related to output -
```console
katana -h output
OUTPUT:
-o, -output string file to write output to
-sr, -store-response store http requests/responses
-srd, -store-response-dir string store http requests/responses to custom directory
-lof, -list-output-fields list available fields for jsonl output format
-eof, -exclude-output-fields exclude fields from jsonl output
-j, -json write output in JSONL(ines) format
-nc, -no-color disable output content coloring (ANSI escape codes)
-silent display output only
-v, -verbose display verbose output
-version display project version
```
## Katana as a library
`katana` can be used as a library by creating an instance of the `Option` struct and populating it with the same options that would be specified via CLI. Using the options you can create `crawlerOptions` and so standard or hybrid `crawler`.
`crawler.Crawl` method should be called to crawl the input.
```go
package main
import (
"math"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/katana/pkg/engine/standard"
"github.com/projectdiscovery/katana/pkg/output"
"github.com/projectdiscovery/katana/pkg/types"
)
func main() {
options := &types.Options{
MaxDepth: 3, // Maximum depth to crawl
FieldScope: "rdn", // Crawling Scope Field
BodyReadSize: math.MaxInt, // Maximum response size to read
Timeout: 10, // Timeout is the time to wait for request in seconds
Concurrency: 10, // Concurrency is the number of concurrent crawling goroutines
Parallelism: 10, // Parallelism is the number of urls processing goroutines
Delay: 0, // Delay is the delay between each crawl requests in seconds
RateLimit: 150, // Maximum requests to send per second
Strategy: "depth-first", // Visit strategy (depth-first, breadth-first)
OnResult: func(result output.Result) { // Callback function to execute for result
gologger.Info().Msg(result.Request.URL)
},
}
crawlerOptions, err := types.NewCrawlerOptions(options)
if err != nil {
gologger.Fatal().Msg(err.Error())
}
defer crawlerOptions.Close()
crawler, err := standard.New(crawlerOptions)
if err != nil {
gologger.Fatal().Msg(err.Error())
}
defer crawler.Close()
var input = "https://www.hackerone.com"
err = crawler.Crawl(input)
if err != nil {
gologger.Warning().Msgf("Could not crawl %s: %s", input, err.Error())
}
}
```
## Reporting Issues & Feature Requests
To maintain issue tracking and improve triage efficiency:
**All reports start as [GitHub Discussions](https://github.com/projectdiscovery/katana/discussions)**
- **Bug Reports** → [Start a Q&A Discussion](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)
- **Feature Requests** → [Start an Ideas Discussion](https://github.com/projectdiscovery/katana/discussions/new?category=ideas)
- **Questions** → [Start a Q&A Discussion](https://github.com/projectdiscovery/katana/discussions/new?category=q-a)
**Why Discussions First?**
- **Community can help** with quick questions and troubleshooting
- **Better triage** - confirmed bugs/features become tracked issues
- **Cleaner issue tracker** - focus on actionable items only
Maintainers will convert discussions to issues when appropriate after proper review.
--------
<div align="center">
katana is made with ❤️ by the [projectdiscovery](https://projectdiscovery.io) team and distributed under [MIT License](LICENSE.md).
<a href="https://discord.gg/projectdiscovery"><img src="https://raw.githubusercontent.com/projectdiscovery/nuclei-burp-plugin/main/static/join-discord.png" width="300" alt="Join Discord"></a>
</div>
================================================
FILE: SECURITY.md
================================================
# Security Policy
## Reporting a Vulnerability
DO NOT CREATE AN ISSUE to report a security problem. Instead, please send an email to security@projectdiscovery.io, and we will acknowledge it within 3 working days.
================================================
FILE: cmd/functional-test/main.go
================================================
package main
import (
"flag"
"fmt"
"log"
"os"
"strings"
"github.com/logrusorgru/aurora"
"github.com/pkg/errors"
"github.com/projectdiscovery/katana/internal/testutils"
)
var (
debug = os.Getenv("DEBUG") == "true"
success = aurora.Green("[✓]").String()
failed = aurora.Red("[✘]").String()
errored = false
devKatanaBinary = flag.String("dev", "", "Dev Branch Katana Binary")
)
func main() {
flag.Parse()
if err := runFunctionalTests(); err != nil {
log.Fatalf("Could not run functional tests: %s\n", err)
}
if errored {
os.Exit(1)
}
}
func runFunctionalTests() error {
for _, testcase := range testutils.TestCases {
if err := runIndividualTestCase(testcase); err != nil {
errored = true
fmt.Fprintf(os.Stderr, "%s Test \"%s\" failed: %s\n", failed, testcase.Name, err)
} else {
fmt.Printf("%s Test \"%s\" passed!\n", success, testcase.Name)
}
}
return nil
}
func runIndividualTestCase(testcase testutils.TestCase) error {
argsParts := strings.Fields(testcase.Args)
devOutput, err := testutils.RunKatanaBinaryAndGetResults(testcase.Target, *devKatanaBinary, debug, argsParts)
if err != nil {
return errors.Wrap(err, "could not run Katana dev test")
}
if testcase.CompareFunc != nil {
return testcase.CompareFunc(testcase.Target, devOutput)
}
if !testutils.CompareOutput(devOutput, testcase.Expected) {
return errors.Errorf("expected output %s, got %s", testcase.Expected, devOutput)
}
return nil
}
================================================
FILE: cmd/functional-test/run.sh
================================================
#!/bin/bash
# reading os type from arguments
CURRENT_OS=$1
if [ "${CURRENT_OS}" == "windows-latest" ];then
extension=.exe
fi
echo "::group::Building functional-test binary"
go build -o functional-test$extension
echo "::endgroup::"
echo "::group::Building katana binary from current branch"
go build -o katana_dev$extension ../katana
echo "::endgroup::"
echo 'Starting katana functional test'
./functional-test$extension -dev ./katana_dev$extension
================================================
FILE: cmd/integration-test/filters.go
================================================
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"os"
"os/exec"
"strings"
"sync/atomic"
"github.com/projectdiscovery/katana/internal/runner"
"github.com/projectdiscovery/katana/pkg/output"
"github.com/projectdiscovery/katana/pkg/types"
)
var filtersTestcases = map[string]TestCase{
"match condition": &matchConditionIntegrationTest{},
"filter condition": &filterConditionIntegrationTest{},
"unique filter": &uniqueFilterIntegrationTest{},
}
type matchConditionIntegrationTest struct{}
// Execute executes a test case and returns an error if occurred
// Execute the docs at ../README.md if the code stops working for integration.
func (h *matchConditionIntegrationTest) Execute() error {
results, _ := RunKatanaAndGetResults(false,
"-u", "scanme.sh",
"-match-condition", "status_code == 200 && contains(body, 'ok')",
)
if len(results) != 1 {
return fmt.Errorf("match condition failed")
}
return nil
}
type filterConditionIntegrationTest struct{}
// Execute executes a test case and returns an error if occurred
func (h *filterConditionIntegrationTest) Execute() error {
results, _ := RunKatanaAndGetResults(false,
"-u", "scanme.sh",
"-filter-condition", "status_code == 200 && contains(body, 'ok')",
)
if len(results) != 0 {
return fmt.Errorf("filter condition failed")
}
return nil
}
type uniqueFilterIntegrationTest struct{}
func (h *uniqueFilterIntegrationTest) Execute() error {
// Create a test server that returns 404 for all paths except root
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/" {
w.WriteHeader(http.StatusOK)
_, _ = fmt.Fprint(w, `<html><body>
<a href="/page1">Page 1</a>
<a href="/page2">Page 2</a>
<a href="/page3">Page 3</a>
<a href="/page4">Page 4</a>
</body></html>`)
} else {
w.WriteHeader(http.StatusNotFound)
// Return identical 404 content for all missing pages
_, _ = fmt.Fprint(w, `<html><body><h1>404 - Page Not Found</h1></body></html>`)
}
}))
defer server.Close()
options := types.DefaultOptions
options.URLs = []string{server.URL}
options.MaxDepth = 2
options.Concurrency = 1
options.DisableUniqueFilter = true
var fourOhFourCount atomic.Int32
options.OnResult = func(result output.Result) {
if result.Response.StatusCode == http.StatusNotFound {
fourOhFourCount.Add(1)
}
}
katanaRunner, err := runner.New(&options)
if err != nil {
return fmt.Errorf("could not create runner: %v", err)
}
defer func() {
_ = katanaRunner.Close()
}()
if err := katanaRunner.ExecuteCrawling(); err != nil {
return fmt.Errorf("could not execute crawling: %v", err)
}
if fourOhFourCount.Load() != 4 {
return fmt.Errorf("expected 4 404 responses, got %d", fourOhFourCount.Load())
}
return nil
}
// ExtraArgs
var ExtraDebugArgs = []string{}
func RunKatanaAndGetResults(debug bool, extra ...string) ([]string, error) {
cmd := exec.Command("./katana")
extra = append(extra, ExtraDebugArgs...)
cmd.Args = append(cmd.Args, extra...)
cmd.Args = append(cmd.Args, "-duc") // disable auto updates
if debug {
cmd.Args = append(cmd.Args, "-debug")
cmd.Stderr = os.Stderr
fmt.Println(cmd.String())
} else {
cmd.Args = append(cmd.Args, "-silent")
}
data, err := cmd.Output()
if debug {
fmt.Println(string(data))
}
if len(data) < 1 && err != nil {
return nil, fmt.Errorf("%v: %v", err.Error(), string(data))
}
var parts []string
items := strings.Split(string(data), "\n")
for _, i := range items {
if i != "" {
parts = append(parts, i)
}
}
return parts, nil
}
================================================
FILE: cmd/integration-test/integration-test.go
================================================
package main
import (
"fmt"
"os"
"strings"
"github.com/logrusorgru/aurora"
)
type TestCase interface {
// Execute executes a test case and returns any errors if occurred
Execute() error
}
var (
debug = os.Getenv("DEBUG") == "true"
customTest = os.Getenv("TEST")
errored = false
success = aurora.Green("[✓]").String()
failed = aurora.Red("[✘]").String()
tests = map[string]map[string]TestCase{
"code": libraryTestcases,
"filters": filtersTestcases,
}
)
func main() {
for name, tests := range tests {
fmt.Printf("Running test cases for \"%s\"\n", aurora.Blue(name))
if customTest != "" && !strings.Contains(name, customTest) {
continue // only run tests user asked
}
for name, test := range tests {
err := test.Execute()
if err != nil {
fmt.Fprintf(os.Stderr, "%s Test \"%s\" failed: %s\n", failed, name, err)
errored = true
} else {
fmt.Printf("%s Test \"%s\" passed!\n", success, name)
}
}
}
if errored {
os.Exit(1)
}
}
================================================
FILE: cmd/integration-test/library.go
================================================
package main
import (
"fmt"
"math"
"github.com/projectdiscovery/katana/pkg/engine/standard"
"github.com/projectdiscovery/katana/pkg/output"
"github.com/projectdiscovery/katana/pkg/types"
"github.com/projectdiscovery/katana/pkg/utils/queue"
)
var libraryTestcases = map[string]TestCase{
"katana as library": &goIntegrationTest{},
}
type goIntegrationTest struct{}
// Execute executes a test case and returns an error if occurred
// Execute the docs at ../README.md if the code stops working for integration.
func (h *goIntegrationTest) Execute() error {
var crawledURLs []string
options := &types.Options{
MaxDepth: 1,
FieldScope: "rdn",
BodyReadSize: math.MaxInt,
RateLimit: 150,
Verbose: debug,
Strategy: queue.DepthFirst.String(),
OnResult: func(r output.Result) {
crawledURLs = append(crawledURLs, r.Request.URL)
},
}
crawlerOptions, err := types.NewCrawlerOptions(options)
if err != nil {
return err
}
defer func() {
if err := crawlerOptions.Close(); err != nil {
fmt.Printf("Error closing crawler options: %v\n", err)
}
}()
crawler, err := standard.New(crawlerOptions)
if err != nil {
return err
}
defer func() {
if err := crawler.Close(); err != nil {
fmt.Printf("Error closing crawler: %v\n", err)
}
}()
var input = "https://public-firing-range.appspot.com"
err = crawler.Crawl(input)
if err != nil {
return err
}
if len(crawledURLs) == 0 {
return fmt.Errorf("no URLs crawled")
}
return nil
}
================================================
FILE: cmd/tools/crawl-maze-score/main.go
================================================
package main
import (
"bufio"
"fmt"
"log"
"math"
"os"
"strings"
"github.com/logrusorgru/aurora"
"github.com/projectdiscovery/gologger"
urlutil "github.com/projectdiscovery/utils/url"
)
// expectedResults is the list of expected endpoints from security-crawl-maze
// blueprint directory.
// https://github.com/google/security-crawl-maze/blob/master/blueprints/utils/resources/expected-results.json
var expectedResults = []string{
"/css/font-face.found",
"/headers/content-location.found",
"/headers/link.found",
"/headers/location.found",
"/headers/refresh.found",
"/html/doctype.found",
"/html/manifest.found",
"/html/body/background.found",
"/html/body/a/href.found",
"/html/body/a/ping.found",
"/html/body/audio/src.found",
"/html/body/audio/source/src.found",
"/html/body/audio/source/srcset1x.found",
"/html/body/audio/source/srcset2x.found",
"/html/body/applet/archive.found",
"/html/body/applet/codebase.found",
"/html/body/blockquote/cite.found",
"/html/body/embed/src.found",
"/html/body/form/action-get.found",
"/html/body/form/action-post.found",
"/html/body/form/button/formaction.found",
"/html/body/frameset/frame/src.found",
"/html/body/iframe/src.found",
"/html/body/iframe/srcdoc.found",
"/html/body/img/dynsrc.found",
"/html/body/img/lowsrc.found",
"/html/body/img/longdesc.found",
"/html/body/img/src-data.found",
"/html/body/img/src.found",
"/html/body/img/srcset1x.found",
"/html/body/img/srcset2x.found",
"/html/body/input/src.found",
"/html/body/isindex/action.found",
"/html/body/map/area/ping.found",
"/html/body/object/data.found",
"/html/body/object/codebase.found",
"/html/body/object/param/value.found",
"/html/body/script/src.found",
"/html/body/svg/image/xlink.found",
"/html/body/svg/script/xlink.found",
"/html/body/table/background.found",
"/html/body/table/td/background.found",
"/html/body/video/src.found",
"/html/body/video/track/src.found",
"/html/body/video/poster.found",
"/html/head/profile.found",
"/html/head/base/href.found",
"/html/head/comment-conditional.found",
"/html/head/import/implementation.found",
"/html/head/link/href.found",
"/html/head/meta/content-csp.found",
"/html/head/meta/content-pinned-websites.found",
"/html/head/meta/content-reading-view.found",
"/html/head/meta/content-redirect.found",
"/html/misc/url/full-url.found",
"/html/misc/url/path-relative-url.found",
"/html/misc/url/protocol-relative-url.found",
"/html/misc/url/root-relative-url.found",
"/html/misc/string/dot-dot-slash-prefix.found",
"/html/misc/string/dot-slash-prefix.found",
"/html/misc/string/url-string.found",
"/html/misc/string/string-known-extension.pdf",
"/javascript/misc/automatic-post.found",
"/javascript/misc/comment.found",
"/javascript/misc/string-variable.found",
"/javascript/misc/string-concat-variable.found",
"/javascript/frameworks/angular/event-handler.found",
"/javascript/frameworks/angular/router-outlet.found",
"/javascript/frameworks/angularjs/ng-href.found",
"/javascript/frameworks/polymer/event-handler.found",
"/javascript/frameworks/polymer/polymer-router.found",
"/javascript/frameworks/react/route-path.found",
"/javascript/frameworks/react/index.html/search.found",
"/javascript/interactive/js-delete.found",
"/javascript/interactive/js-post.found",
"/javascript/interactive/js-post-event-listener.found",
"/javascript/interactive/js-put.found",
"/javascript/interactive/listener-and-event-attribute-first.found",
"/javascript/interactive/listener-and-event-attribute-second.found",
"/javascript/interactive/multi-step-request-event-attribute.found",
"/test/javascript/interactive/multi-step-request-event-listener-div-dom.found",
"/test/javascript/interactive/multi-step-request-event-listener-div.found",
"/javascript/interactive/multi-step-request-event-listener-dom.found",
"/javascript/interactive/multi-step-request-event-listener.found",
"/javascript/interactive/multi-step-request-redefine-event-attribute.found",
"/javascript/interactive/multi-step-request-remove-button.found",
"/javascript/interactive/multi-step-request-remove-event-listener.found",
"/javascript/interactive/two-listeners-first.found",
"/javascript/interactive/two-listeners-second.found",
"/misc/known-files/robots.txt.found",
"/misc/known-files/sitemap.xml.found",
}
func main() {
if err := process(); err != nil {
log.Fatalf("%s\n", err)
}
}
var urlTestPrefix = "/test"
func process() error {
if len(os.Args) < 3 {
fmt.Printf("Usage: crawl-maze-score output.txt output_headless.txt")
return nil
}
input := os.Args[1]
inputHeadless := os.Args[2]
links, err := readFoundLinks(input)
if err != nil {
return err
}
linksHeadless, err := readFoundLinks(inputHeadless)
if err != nil {
return err
}
linksMap := make(map[string]struct{})
linksHeadlessMap := make(map[string]struct{})
for _, link := range links {
linksMap[link] = struct{}{}
}
for _, link := range linksHeadless {
linksHeadlessMap[link] = struct{}{}
}
matches, matchesHeadless := 0, 0
for _, expected := range expectedResults {
expected = urlTestPrefix + expected
_, normalOk := linksMap[expected]
_, headlessOk := linksHeadlessMap[expected]
if normalOk {
matches++
}
if headlessOk {
matchesHeadless++
}
fmt.Printf("[%s] [%s] %s\n", colorizeText("standard", normalOk), colorizeText("headless", headlessOk), expected)
}
fmt.Printf("[info] Total links (%d): Standard=>%d Headless=>%d\n", len(expectedResults), len(links), len(linksHeadless))
fmt.Printf("[info] Total: %d NormalMatches=>%d HeadlessMatches=>%d\n", len(expectedResults), matches, matchesHeadless)
fmt.Printf("[info] Score: Normal=>%.2f%% Headless=>%.2f%%\n", math.Round(float64(matches*100/len(expectedResults))), math.Round(float64(matchesHeadless*100/len(expectedResults))))
return nil
}
func colorizeText(text string, value bool) string {
if value {
return aurora.Green(text + ":yes").String()
}
return aurora.Red(text + ":no").String()
}
func strippedLink(link string) string {
parsed, err := urlutil.Parse(link)
if err != nil {
gologger.Warning().Msgf("failed to parse link while extracting path: %v", err)
}
return parsed.Path
}
func readFoundLinks(input string) ([]string, error) {
file, err := os.Open(input)
if err != nil {
return nil, err
}
defer func() {
if err := file.Close(); err != nil {
gologger.Error().Msgf("Error closing file: %v\n", err)
}
}()
scanner := bufio.NewScanner(file)
var links []string
for scanner.Scan() {
text := scanner.Text()
if text == "" {
break
}
if strings.Contains(text, ".found") {
links = append(links, strippedLink(text))
}
}
return links, nil
}
================================================
FILE: go.mod
================================================
module github.com/projectdiscovery/katana
go 1.25.7
require (
github.com/BishopFox/jsluice v0.0.0-20240110145140-0ddfab153e06
github.com/PuerkitoBio/goquery v1.11.0
github.com/adrianbrad/queue v1.3.0
github.com/dominikbraun/graph v0.23.0
github.com/go-rod/rod v0.116.2
github.com/happyhackingspace/dit v0.0.14
github.com/hashicorp/golang-lru/v2 v2.0.7
github.com/json-iterator/go v1.1.12
github.com/lmittmann/tint v1.0.6
github.com/logrusorgru/aurora v2.0.3+incompatible
github.com/lukasbob/srcset v0.0.0-20190730101422-86b742e617f3
github.com/mfonda/simhash v0.0.0-20151007195837-79f94a1100d6
github.com/mitchellh/mapstructure v1.5.0
github.com/pkg/errors v0.9.1
github.com/projectdiscovery/dsl v0.8.5
github.com/projectdiscovery/fastdialer v0.5.2
github.com/projectdiscovery/goflags v0.1.74
github.com/projectdiscovery/gologger v1.1.67
github.com/projectdiscovery/hmap v0.0.99
github.com/projectdiscovery/mapcidr v1.1.97
github.com/projectdiscovery/ratelimit v0.0.82
github.com/projectdiscovery/retryablehttp-go v1.3.2
github.com/projectdiscovery/utils v0.8.0
github.com/projectdiscovery/wappalyzergo v0.2.62
github.com/remeh/sizedwaitgroup v1.0.0
github.com/rs/xid v1.5.0
github.com/stoewer/go-strcase v1.3.0
github.com/stretchr/testify v1.11.1
github.com/valyala/fasttemplate v1.2.2
go.uber.org/multierr v1.11.0
golang.org/x/net v0.51.0
gopkg.in/yaml.v3 v3.0.1
)
require (
aead.dev/minisign v0.2.0 // indirect
github.com/Knetic/govaluate v3.0.0+incompatible // indirect
github.com/Masterminds/semver/v3 v3.4.0 // indirect
github.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057 // indirect
github.com/STARRY-S/zip v0.2.3 // indirect
github.com/VividCortex/ewma v1.2.0 // indirect
github.com/alecthomas/chroma/v2 v2.14.0 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/bodgit/plumbing v1.3.0 // indirect
github.com/bodgit/sevenzip v1.6.1 // indirect
github.com/bodgit/windows v1.0.1 // indirect
github.com/brianvoe/gofakeit/v7 v7.2.1 // indirect
github.com/charmbracelet/glamour v0.8.0 // indirect
github.com/charmbracelet/lipgloss v0.13.0 // indirect
github.com/charmbracelet/x/ansi v0.3.2 // indirect
github.com/cheggaaa/pb/v3 v3.1.4 // indirect
github.com/cloudflare/circl v1.6.1 // indirect
github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c // indirect
github.com/djherbis/times v1.6.0 // indirect
github.com/dlclark/regexp2 v1.11.5 // indirect
github.com/docker/go-units v0.5.0 // indirect
github.com/fatih/color v1.15.0 // indirect
github.com/felixge/fgprof v0.9.5 // indirect
github.com/gaissmai/bart v0.26.0 // indirect
github.com/google/go-github/v30 v30.1.0 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/pprof v0.0.0-20250423184734-337e5dd93bb4 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gosimple/slug v1.15.0 // indirect
github.com/gosimple/unidecode v1.0.1 // indirect
github.com/hashicorp/go-version v1.8.0 // indirect
github.com/hdm/jarm-go v0.0.7 // indirect
github.com/iangcarroll/cookiemonster v1.6.0 // indirect
github.com/kataras/jwt v0.1.8 // indirect
github.com/klauspost/compress v1.18.2 // indirect
github.com/klauspost/pgzip v1.2.6 // indirect
github.com/kr/pretty v0.3.1 // indirect
github.com/logrusorgru/aurora/v4 v4.0.0 // indirect
github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/mholt/archives v0.1.5 // indirect
github.com/mikelolasagasti/xz v1.0.1 // indirect
github.com/minio/minlz v1.0.1 // indirect
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 // indirect
github.com/muesli/reflow v0.3.0 // indirect
github.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a // indirect
github.com/nwaples/rardecode/v2 v2.2.2 // indirect
github.com/pierrec/lz4/v4 v4.1.23 // indirect
github.com/projectdiscovery/asnmap v1.1.1 // indirect
github.com/projectdiscovery/blackrock v0.0.1 // indirect
github.com/projectdiscovery/gostruct v0.0.2 // indirect
github.com/projectdiscovery/machineid v0.0.0-20250715113114-c77eb3567582 // indirect
github.com/refraction-networking/utls v1.7.1 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.12.0 // indirect
github.com/sashabaranov/go-openai v1.37.0 // indirect
github.com/shirou/gopsutil/v3 v3.23.7 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/smacker/go-tree-sitter v0.0.0-20230720070738-0d0a9f78d8f8 // indirect
github.com/sorairolake/lzip-go v0.3.8 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
github.com/spf13/afero v1.15.0 // indirect
github.com/tidwall/btree v1.6.0 // indirect
github.com/tidwall/buntdb v1.3.0 // indirect
github.com/tidwall/gjson v1.18.0 // indirect
github.com/tidwall/grect v0.1.4 // indirect
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/rtred v0.1.2 // indirect
github.com/tidwall/tinyqueue v0.1.1 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/vulncheck-oss/go-exploit v1.51.0 // indirect
github.com/xdg-go/pbkdf2 v1.0.0 // indirect
github.com/ysmood/fetchup v0.2.3 // indirect
github.com/ysmood/got v0.40.0 // indirect
github.com/yuin/goldmark v1.7.4 // indirect
github.com/yuin/goldmark-emoji v1.0.3 // indirect
github.com/zcalusic/sysinfo v1.0.2 // indirect
go4.org v0.0.0-20230225012048-214862532bf5 // indirect
golang.org/x/oauth2 v0.34.0 // indirect
golang.org/x/sync v0.19.0 // indirect
golang.org/x/term v0.40.0 // indirect
golang.org/x/time v0.14.0 // indirect
)
require (
github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 // indirect
github.com/akrylysov/pogreb v0.10.1 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aymerick/douceur v0.2.0 // indirect
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/dimchansky/utfbom v1.1.1 // indirect
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/gorilla/css v1.0.1 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/microcosm-cc/bluemonday v1.0.27 // indirect
github.com/miekg/dns v1.1.62 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
github.com/projectdiscovery/networkpolicy v0.1.33
github.com/projectdiscovery/retryabledns v1.0.112 // indirect
github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d // indirect
github.com/syndtr/goleveldb v1.0.0 // indirect
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/ulikunitz/xz v0.5.15 // indirect
github.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39 // indirect
github.com/ysmood/goob v0.4.0 // indirect
github.com/ysmood/gson v0.7.3 // indirect
github.com/ysmood/leakless v0.9.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248 // indirect
github.com/zmap/zcrypto v0.0.0-20230422215203-9a665e1e9968 // indirect
go.etcd.io/bbolt v1.3.7 // indirect
golang.org/x/crypto v0.48.0 // indirect
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b // indirect
golang.org/x/mod v0.32.0 // indirect
golang.org/x/sys v0.41.0 // indirect
golang.org/x/text v0.34.0 // indirect
golang.org/x/tools v0.41.0 // indirect
gopkg.in/yaml.v2 v2.4.0
)
================================================
FILE: go.sum
================================================
aead.dev/minisign v0.2.0 h1:kAWrq/hBRu4AARY6AlciO83xhNnW9UaC8YipS2uhLPk=
aead.dev/minisign v0.2.0/go.mod h1:zdq6LdSd9TbuSxchxwhpA9zEb9YXcVGoE8JakuiGaIQ=
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/compute/metadata v0.2.0/go.mod h1:zFmK7XCadkQkj6TtorcaGlCW1hT1fIilQDwofLpJ20k=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/BishopFox/jsluice v0.0.0-20240110145140-0ddfab153e06 h1:xa/dJgg1qpWdIyr7tQcTV2TUPgBK/f0TTMLMmD5GqjQ=
github.com/BishopFox/jsluice v0.0.0-20240110145140-0ddfab153e06/go.mod h1:ENDk4KXEVPZTZPygQAEWJK0BlyEWAyQZhxwCMc+o6A0=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/Knetic/govaluate v3.0.0+incompatible h1:7o6+MAPhYTCF0+fdvoz1xDedhRb4f6s9Tn1Tt7/WTEg=
github.com/Knetic/govaluate v3.0.0+incompatible/go.mod h1:r7JcOSlj0wfOMncg0iLm8Leh48TZaKVeNIfJntJ2wa0=
github.com/Masterminds/semver/v3 v3.4.0 h1:Zog+i5UMtVoCU8oKka5P7i9q9HgrJeGzI9SA1Xbatp0=
github.com/Masterminds/semver/v3 v3.4.0/go.mod h1:4V+yj/TJE1HU9XfppCwVMZq3I84lprf4nC11bSS5beM=
github.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057 h1:KFac3SiGbId8ub47e7kd2PLZeACxc1LkiiNoDOFRClE=
github.com/Mzack9999/gcache v0.0.0-20230410081825-519e28eab057/go.mod h1:iLB2pivrPICvLOuROKmlqURtFIEsoJZaMidQfCG1+D4=
github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809 h1:ZbFL+BDfBqegi+/Ssh7im5+aQfBRx6it+kHnC7jaDU8=
github.com/Mzack9999/go-http-digest-auth-client v0.6.1-0.20220414142836-eb8883508809/go.mod h1:upgc3Zs45jBDnBT4tVRgRcgm26ABpaP7MoTSdgysca4=
github.com/PuerkitoBio/goquery v1.11.0 h1:jZ7pwMQXIITcUXNH83LLk+txlaEy6NVOfTuP43xxfqw=
github.com/PuerkitoBio/goquery v1.11.0/go.mod h1:wQHgxUOU3JGuj3oD/QFfxUdlzW6xPHfqyHre6VMY4DQ=
github.com/RumbleDiscovery/rumble-tools v0.0.0-20201105153123-f2adbb3244d2/go.mod h1:jD2+mU+E2SZUuAOHZvZj4xP4frlOo+N/YrXDvASFhkE=
github.com/STARRY-S/zip v0.2.3 h1:luE4dMvRPDOWQdeDdUxUoZkzUIpTccdKdhHHsQJ1fm4=
github.com/STARRY-S/zip v0.2.3/go.mod h1:lqJ9JdeRipyOQJrYSOtpNAiaesFO6zVDsE8GIGFaoSk=
github.com/VividCortex/ewma v1.2.0 h1:f58SaIzcDXrSy3kWaHNvuJgJ3Nmz59Zji6XoJR/q1ow=
github.com/VividCortex/ewma v1.2.0/go.mod h1:nz4BbCtbLyFDeC9SUHbtcT5644juEuWfUAUnGx7j5l4=
github.com/adrianbrad/queue v1.3.0 h1:8FH1N+93HXbqta5+URa1AL+diV7MP3VDXAEnP+DNp48=
github.com/adrianbrad/queue v1.3.0/go.mod h1:wYiPC/3MPbyT45QHLrPR4zcqJWPePubM1oEP/xTwhUs=
github.com/akrylysov/pogreb v0.10.1 h1:FqlR8VR7uCbJdfUob916tPM+idpKgeESDXOA1K0DK4w=
github.com/akrylysov/pogreb v0.10.1/go.mod h1:pNs6QmpQ1UlTJKDezuRWmaqkgUE2TuU0YTWyqJZ7+lI=
github.com/alecthomas/assert/v2 v2.7.0 h1:QtqSACNS3tF7oasA8CU6A6sXZSBDqnm7RfpLl9bZqbE=
github.com/alecthomas/assert/v2 v2.7.0/go.mod h1:Bze95FyfUr7x34QZrjL+XP+0qgp/zg8yS+TtBj1WA3k=
github.com/alecthomas/chroma/v2 v2.14.0 h1:R3+wzpnUArGcQz7fCETQBzO5n9IMNi13iIs46aU4V9E=
github.com/alecthomas/chroma/v2 v2.14.0/go.mod h1:QolEbTfmUHIMVpBqxeDnNBj2uoeI4EbYP4i6n68SG4I=
github.com/alecthomas/repr v0.4.0 h1:GhI2A8MACjfegCPVq9f1FLvIBS+DrQ2KQBFZP1iFzXc=
github.com/alecthomas/repr v0.4.0/go.mod h1:Fr0507jx4eOXV7AlPV6AVZLYrLIuIeSOWtW57eE/O/4=
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
github.com/aymanbagabas/go-udiff v0.2.0 h1:TK0fH4MteXUDspT88n8CKzvK0X9O2xu9yQjWpi6yML8=
github.com/aymanbagabas/go-udiff v0.2.0/go.mod h1:RE4Ex0qsGkTAJoQdQQCA0uG+nAzJO/pI/QwceO5fgrA=
github.com/aymerick/douceur v0.2.0 h1:Mv+mAeH1Q+n9Fr+oyamOlAkUNPWPlA8PPGR0QAaYuPk=
github.com/aymerick/douceur v0.2.0/go.mod h1:wlT5vV2O3h55X9m7iVYN0TBM0NH/MmbLnd30/FjWUq4=
github.com/bits-and-blooms/bitset v1.13.0 h1:bAQ9OPNFYbGHV6Nez0tmNI0RiEu7/hxlYJRUA0wFAVE=
github.com/bits-and-blooms/bitset v1.13.0/go.mod h1:7hO7Gc7Pp1vODcmWvKMRA9BNmbv6a/7QIWpPxHddWR8=
github.com/bits-and-blooms/bloom/v3 v3.5.0 h1:AKDvi1V3xJCmSR6QhcBfHbCN4Vf8FfxeWkMNQfmAGhY=
github.com/bits-and-blooms/bloom/v3 v3.5.0/go.mod h1:Y8vrn7nk1tPIlmLtW2ZPV+W7StdVMor6bC1xgpjMZFs=
github.com/bodgit/plumbing v1.3.0 h1:pf9Itz1JOQgn7vEOE7v7nlEfBykYqvUYioC61TwWCFU=
github.com/bodgit/plumbing v1.3.0/go.mod h1:JOTb4XiRu5xfnmdnDJo6GmSbSbtSyufrsyZFByMtKEs=
github.com/bodgit/sevenzip v1.6.1 h1:kikg2pUMYC9ljU7W9SaqHXhym5HyKm8/M/jd31fYan4=
github.com/bodgit/sevenzip v1.6.1/go.mod h1:GVoYQbEVbOGT8n2pfqCIMRUaRjQ8F9oSqoBEqZh5fQ8=
github.com/bodgit/windows v1.0.1 h1:tF7K6KOluPYygXa3Z2594zxlkbKPAOvqr97etrGNIz4=
github.com/bodgit/windows v1.0.1/go.mod h1:a6JLwrB4KrTR5hBpp8FI9/9W9jJfeQ2h4XDXU74ZCdM=
github.com/brianvoe/gofakeit/v7 v7.2.1 h1:AGojgaaCdgq4Adzrd2uWdbGNDyX6MWNhHdQBraNfOHI=
github.com/brianvoe/gofakeit/v7 v7.2.1/go.mod h1:QXuPeBw164PJCzCUZVmgpgHJ3Llj49jSLVkKPMtxtxA=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/charmbracelet/glamour v0.8.0 h1:tPrjL3aRcQbn++7t18wOpgLyl8wrOHUEDS7IZ68QtZs=
github.com/charmbracelet/glamour v0.8.0/go.mod h1:ViRgmKkf3u5S7uakt2czJ272WSg2ZenlYEZXT2x7Bjw=
github.com/charmbracelet/lipgloss v0.13.0 h1:4X3PPeoWEDCMvzDvGmTajSyYPcZM4+y8sCA/SsA3cjw=
github.com/charmbracelet/lipgloss v0.13.0/go.mod h1:nw4zy0SBX/F/eAO1cWdcvy6qnkDUxr8Lw7dvFrAIbbY=
github.com/charmbracelet/x/ansi v0.3.2 h1:wsEwgAN+C9U06l9dCVMX0/L3x7ptvY1qmjMwyfE6USY=
github.com/charmbracelet/x/ansi v0.3.2/go.mod h1:dk73KoMTT5AX5BsX0KrqhsTqAnhZZoCBjs7dGWp4Ktw=
github.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a h1:G99klV19u0QnhiizODirwVksQB91TJKV/UaTnACcG30=
github.com/charmbracelet/x/exp/golden v0.0.0-20240806155701-69247e0abc2a/go.mod h1:wDlXFlCrmJ8J+swcL/MnGUuYnqgQdW9rhSD61oNMb6U=
github.com/cheggaaa/pb/v3 v3.1.4 h1:DN8j4TVVdKu3WxVwcRKu0sG00IIU6FewoABZzXbRQeo=
github.com/cheggaaa/pb/v3 v3.1.4/go.mod h1:6wVjILNBaXMs8c21qRiaUM8BR82erfgau1DQ4iUXmSA=
github.com/chromedp/cdproto v0.0.0-20230802225258-3cf4e6d46a89/go.mod h1:GKljq0VrfU4D5yc+2qA6OVr8pmO/MBbPEWqWQ/oqGEs=
github.com/chromedp/chromedp v0.9.2/go.mod h1:LkSXJKONWTCHAfQasKFUZI+mxqS4tZqhmtGzzhLsnLs=
github.com/chromedp/sysutil v1.0.0/go.mod h1:kgWmDdq8fTzXYcKIBqIYvRRTnYb9aNS9moAV0xufSww=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/logex v1.2.1/go.mod h1:JLbx6lG2kDbNRFnfkgvh4eRJRPX1QCoOIWomwysCBrQ=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/readline v1.5.1/go.mod h1:Eh+b79XXUwfKfcPLepksvw2tcLE/Ct21YObkaSkeBlk=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/chzyer/test v1.0.0/go.mod h1:2JlltgoNkt4TW/z9V/IzDdFaMTM2JPIi26O1pF38GC8=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cloudflare/circl v1.6.1 h1:zqIqSPIndyBh1bjLVVDHMPpVKqp8Su/V+6MeDzzQBQ0=
github.com/cloudflare/circl v1.6.1/go.mod h1:uddAzsPgqdMAYatqJ0lsjX1oECcQLIlRpzZh3pJrofs=
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08 h1:ox2F0PSMlrAAiAdknSRMDrAr8mfxPCfSZolH+/qQnyQ=
github.com/cnf/structhash v0.0.0-20201127153200-e1b16c1ebc08/go.mod h1:pCxVEbcm3AMg7ejXyorUXi6HQCzOIBf7zEDVPtw0/U4=
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dimchansky/utfbom v1.1.1 h1:vV6w1AhK4VMnhBno/TPVCoK9U/LP0PkLCS9tbxHdi/U=
github.com/dimchansky/utfbom v1.1.1/go.mod h1:SxdoEBH5qIqFocHMyGOXVAybYJdr71b1Q/j0mACtrfE=
github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c h1:+Zo5Ca9GH0RoeVZQKzFJcTLoAixx5s5Gq3pTIS+n354=
github.com/ditashi/jsbeautifier-go v0.0.0-20141206144643-2520a8026a9c/go.mod h1:HJGU9ULdREjOcVGZVPB5s6zYmHi1RxzT71l2wQyLmnE=
github.com/djherbis/times v1.6.0 h1:w2ctJ92J8fBvWPxugmXIv7Nz7Q3iDMKNx9v5ocVH20c=
github.com/djherbis/times v1.6.0/go.mod h1:gOHeRAz2h+VJNZ5Gmc/o7iD9k4wW7NMVqieYCY99oc0=
github.com/dlclark/regexp2 v1.11.5 h1:Q/sSnsKerHeCkc/jSTNq1oCm7KiVgUMZRDUoRu0JQZQ=
github.com/dlclark/regexp2 v1.11.5/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/dominikbraun/graph v0.23.0 h1:TdZB4pPqCLFxYhdyMFb1TBdFxp8XLcJfTTBQucVPgCo=
github.com/dominikbraun/graph v0.23.0/go.mod h1:yOjYyogZLY1LSG9E33JWZJiq5k83Qy2C6POAuiViluc=
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707 h1:2tV76y6Q9BB+NEBasnqvs7e49aEBFI8ejC89PSnWH+4=
github.com/dsnet/compress v0.0.2-0.20230904184137-39efe44ab707/go.mod h1:qssHWj60/X5sZFNxpG4HBPDHVqxNm4DfnCKgrbZOT+s=
github.com/dsnet/golib v0.0.0-20171103203638-1ea166775780/go.mod h1:Lj+Z9rebOhdfkVLjJ8T6VcRQv3SXugXy999NBtR9aFY=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/fatih/color v1.15.0 h1:kOqh6YHBtK8aywxGerMG2Eq3H6Qgoqeo13Bk2Mv/nBs=
github.com/fatih/color v1.15.0/go.mod h1:0h5ZqXfHYED7Bhv2ZJamyIOUej9KtShiJESRwBDUSsw=
github.com/felixge/fgprof v0.9.5 h1:8+vR6yu2vvSKn08urWyEuxx75NWPEvybbkBirEpsbVY=
github.com/felixge/fgprof v0.9.5/go.mod h1:yKl+ERSa++RYOs32d8K6WEXCB4uXdLls4ZaZPpayhMM=
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/gaissmai/bart v0.26.0 h1:xOZ57E9hJLBiQaSyeZa9wgWhGuzfGACgqp4BE77OkO0=
github.com/gaissmai/bart v0.26.0/go.mod h1:GREWQfTLRWz/c5FTOsIw+KkscuFkIV5t8Rp7Nd1Td5c=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-ole/go-ole v1.2.6 h1:/Fpf6oFPoeFik9ty7siob0G6Ke8QvQEuVcuChpwXzpY=
github.com/go-ole/go-ole v1.2.6/go.mod h1:pprOEPIfldk/42T2oK7lQ4v4JSDwmV0As9GaiUsvbm0=
github.com/go-rod/rod v0.116.2 h1:A5t2Ky2A+5eD/ZJQr1EfsQSe5rms5Xof/qj296e+ZqA=
github.com/go-rod/rod v0.116.2/go.mod h1:H+CMO9SCNc2TJ2WfrG+pKhITz57uGNYU43qYHh438Mg=
github.com/gobwas/httphead v0.1.0/go.mod h1:O/RXo79gxV8G+RqlR/otEwx4Q36zl9rqC5u12GKvMCM=
github.com/gobwas/pool v0.2.1/go.mod h1:q8bcK0KcYlCgd9e7WYLm9LpyS+YeLd8JVDW6WezmKEw=
github.com/gobwas/ws v1.2.1/go.mod h1:hRKAFb8wOxFROYNsT1bqfWnhX+b5MFeJM9r2ZSwg/KY=
github.com/gofrs/uuid v3.3.0+incompatible/go.mod h1:b2aQJv3Z4Fp6yNu3cdSllBxTCLRxnplIgP/c0N/04lM=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.8/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/go-github/v30 v30.1.0 h1:VLDx+UolQICEOKu2m4uAoMti1SxuEBAl7RSEG16L+Oo=
github.com/google/go-github/v30 v30.1.0/go.mod h1:n8jBpHl45a/rlBUtRJMOG4GhNADUQFEufcolZ95JfU8=
github.com/google/go-github/v50 v50.1.0/go.mod h1:Ev4Tre8QoKiolvbpOSG3FIi4Mlon3S2Nt9W5JYqKiwA=
github.com/google/go-querystring v1.0.0/go.mod h1:odCYkC5MyYFN7vkCjXpyrEuKhc/BUO6wN/zVPAxq5ck=
github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD/fhyJ8=
github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20240227163752-401108e1b7e7/go.mod h1:czg5+yv1E0ZGTi6S6vVK1mke0fV+FaUhNGcd6VRS9Ik=
github.com/google/pprof v0.0.0-20250423184734-337e5dd93bb4 h1:gD0vax+4I+mAj+jEChEf25Ia07Jq7kYOFO5PPhAxFl4=
github.com/google/pprof v0.0.0-20250423184734-337e5dd93bb4/go.mod h1:5hDyRhoBCxViHszMt12TnOpEI4VVi+U8Gm9iphldiMA=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/gorilla/css v1.0.1 h1:ntNaBIghp6JmvWnxbZKANoLyuXTPZ4cAMlo6RyhlbO8=
github.com/gorilla/css v1.0.1/go.mod h1:BvnYkspnSzMmwRK+b8/xgNPLiIuNZr6vbZBTPQ2A3b0=
github.com/gosimple/slug v1.15.0 h1:wRZHsRrRcs6b0XnxMUBM6WK1U1Vg5B0R7VkIf1Xzobo=
github.com/gosimple/slug v1.15.0/go.mod h1:UiRaFH+GEilHstLUmcBgWcI42viBN7mAb818JrYOeFQ=
github.com/gosimple/unidecode v1.0.1 h1:hZzFTMMqSswvf0LBJZCZgThIZrpDHFXux9KeGmn6T/o=
github.com/gosimple/unidecode v1.0.1/go.mod h1:CP0Cr1Y1kogOtx0bJblKzsVWrqYaqfNOnHzpgWw4Awc=
github.com/happyhackingspace/dit v0.0.14 h1:rkIu0HuFqvqr8F2PJgG0F+lx6DbX/tQE1hXKwIF2NQQ=
github.com/happyhackingspace/dit v0.0.14/go.mod h1:+WeAxrX7QYeiDmXLVaDgrqpyfD4O/sHlOL4wtbiIpUQ=
github.com/hashicorp/go-version v1.8.0 h1:KAkNb1HAiZd1ukkxDFGmokVZe1Xy9HG6NUp+bPle2i4=
github.com/hashicorp/go-version v1.8.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hdm/jarm-go v0.0.7 h1:Eq0geenHrBSYuKrdVhrBdMMzOmA+CAMLzN2WrF3eL6A=
github.com/hdm/jarm-go v0.0.7/go.mod h1:kinGoS0+Sdn1Rr54OtanET5E5n7AlD6T6CrJAKDjJSQ=
github.com/hexops/gotextdiff v1.0.3 h1:gitA9+qJrrTCsiCl7+kh75nPqQt1cx4ZkudSTLoUqJM=
github.com/hexops/gotextdiff v1.0.3/go.mod h1:pSWU5MAI3yDq+fZBTazCSJysOMbxWL1BSow5/V2vxeg=
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
github.com/iangcarroll/cookiemonster v1.6.0 h1:NPFkn/ZZYZgzXhJ1awRnYhZ3fJK3hKWgbctfTW21kew=
github.com/iangcarroll/cookiemonster v1.6.0/go.mod h1:n3MvoAq56NkNyCEyhcYs3ZJMzTc9rL3w7IaITI0apMg=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/ianlancetaylor/demangle v0.0.0-20230524184225-eabc099b10ab/go.mod h1:gx7rwoVhcfuVKG5uya9Hs3Sxj7EIvldVofAWIUtGouw=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/julienschmidt/httprouter v1.3.0 h1:U0609e9tgbseu3rBINet9P48AI/D3oJs4dN7jwJOQ1U=
github.com/julienschmidt/httprouter v1.3.0/go.mod h1:JR6WtHb+2LUe8TCKY3cZOxFyyO8IZAc4RVcycCCAKdM=
github.com/kataras/jwt v0.1.8 h1:u71baOsYD22HWeSOg32tCHbczPjdCk7V4MMeJqTtmGk=
github.com/kataras/jwt v0.1.8/go.mod h1:Q5j2IkcIHnfwy+oNY3TVWuEBJNw0ADgCcXK9CaZwV4o=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.4.1/go.mod h1:RyIbtBH6LamlWaDj8nUwkbUhJ87Yi3uG0guNDohfE1A=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid v1.2.0/go.mod h1:Pj4uuM528wm8OyEC2QMXAi2YiTZ96dNQPGgoMS4s3ek=
github.com/klauspost/pgzip v1.2.6 h1:8RXeL5crjEUFnR2/Sn6GJNWtSQ3Dk8pq4CL3jvdDyjU=
github.com/klauspost/pgzip v1.2.6/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/ledongthuc/pdf v0.0.0-20220302134840-0c2507a12d80/go.mod h1:imJHygn/1yfhB7XSJJKlFZKl/J+dCPAknuiaGOshXAs=
github.com/lmittmann/tint v1.0.6 h1:vkkuDAZXc0EFGNzYjWcV0h7eEX+uujH48f/ifSkJWgc=
github.com/lmittmann/tint v1.0.6/go.mod h1:HIS3gSy7qNwGCj+5oRjAutErFBl4BzdQP6cJZ0NfMwE=
github.com/logrusorgru/aurora v2.0.3+incompatible h1:tOpm7WcpBTn4fjmVfgpQq0EfczGlG91VSDkswnjF5A8=
github.com/logrusorgru/aurora v2.0.3+incompatible/go.mod h1:7rIyQOR62GCctdiQpZ/zOJlFyk6y+94wXzv6RNZgaR4=
github.com/logrusorgru/aurora/v4 v4.0.0 h1:sRjfPpun/63iADiSvGGjgA1cAYegEWMPCJdUpJYn9JA=
github.com/logrusorgru/aurora/v4 v4.0.0/go.mod h1:lP0iIa2nrnT/qoFXcOZSrZQpJ1o6n2CUf/hyHi2Q4ZQ=
github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY=
github.com/lucasb-eyer/go-colorful v1.2.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
github.com/lukasbob/srcset v0.0.0-20190730101422-86b742e617f3 h1:l1rIRmxNhzeQM+qA3D0CsDLo0Hx45q9JmK0BlCjt6Ks=
github.com/lukasbob/srcset v0.0.0-20190730101422-86b742e617f3/go.mod h1:j16TYl5p17+vBMyaL6Nu4ojlOnfX8lc2k2cfmw6m5TQ=
github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.12/go.mod h1:RAqKPSqVFrSLVXbA8x7dzmKdmGzieGRCM46jaSJTDAk=
github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mfonda/simhash v0.0.0-20151007195837-79f94a1100d6 h1:bjfMeqxWEJ6IRUvGkiTkSwx0a6UdQJsbirRSoXogteY=
github.com/mfonda/simhash v0.0.0-20151007195837-79f94a1100d6/go.mod h1:WVJJvUw/pIOcwu2O8ZzHEhmigq2jzwRNfJVRMJB7bR8=
github.com/mholt/archives v0.1.5 h1:Fh2hl1j7VEhc6DZs2DLMgiBNChUux154a1G+2esNvzQ=
github.com/mholt/archives v0.1.5/go.mod h1:3TPMmBLPsgszL+1As5zECTuKwKvIfj6YcwWPpeTAXF4=
github.com/microcosm-cc/bluemonday v1.0.27 h1:MpEUotklkwCSLeH+Qdx1VJgNqLlpY2KXwXFM08ygZfk=
github.com/microcosm-cc/bluemonday v1.0.27/go.mod h1:jFi9vgW+H7c3V0lb6nR74Ib/DIB5OBs92Dimizgw2cA=
github.com/miekg/dns v1.1.35/go.mod h1:KNUDUusw/aVsxyTYZM1oqvCicbwhgbNgztCETuNZ7xM=
github.com/miekg/dns v1.1.62 h1:cN8OuEF1/x5Rq6Np+h1epln8OiyPWV+lROx9LxcGgIQ=
github.com/miekg/dns v1.1.62/go.mod h1:mvDlcItzm+br7MToIKqkglaGhlFMHJ9DTNNWONWXbNQ=
github.com/mikelolasagasti/xz v1.0.1 h1:Q2F2jX0RYJUG3+WsM+FJknv+6eVjsjXNDV0KJXZzkD0=
github.com/mikelolasagasti/xz v1.0.1/go.mod h1:muAirjiOUxPRXwm9HdDtB3uoRPrGnL85XHtokL9Hcgc=
github.com/minio/minlz v1.0.1 h1:OUZUzXcib8diiX+JYxyRLIdomyZYzHct6EShOKtQY2A=
github.com/minio/minlz v1.0.1/go.mod h1:qT0aEB35q79LLornSzeDH75LBf3aH1MV+jB5w9Wasec=
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7 h1:yRZGarbxsRytL6EGgbqK2mCY+Lk5MWKQYKJT2gEglhc=
github.com/minio/selfupdate v0.6.1-0.20230907112617-f11e74f84ca7/go.mod h1:bO02GTIPCMQFTEvE5h4DjYB58bCoZ35XLeBf0buTDdM=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/mreiferson/go-httpclient v0.0.0-20160630210159-31f0106b4474/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=
github.com/mreiferson/go-httpclient v0.0.0-20201222173833-5e475fde3a4d/go.mod h1:OQA4XLvDbMgS8P0CevmM4m9Q3Jq4phKUzcocxuGJ5m8=
github.com/muesli/reflow v0.3.0 h1:IFsN6K9NfGtjeggFP+68I4chLZV2yIKsXJFNZ+eWh6s=
github.com/muesli/reflow v0.3.0/go.mod h1:pbwTDkVPibjO2kyvBQRBxTWEEGDGq0FlB1BIKtnHY/8=
github.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a h1:2MaM6YC3mGu54x+RKAA6JiFFHlHDY1UbkxqppT7wYOg=
github.com/muesli/termenv v0.15.3-0.20240618155329-98d742f6907a/go.mod h1:hxSnBBYLK21Vtq/PHd0S2FYCxBXzBua8ov5s1RobyRQ=
github.com/nwaples/rardecode/v2 v2.2.2 h1:/5oL8dzYivRM/tqX9VcTSWfbpwcbwKG1QtSJr3b3KcU=
github.com/nwaples/rardecode/v2 v2.2.2/go.mod h1:7uz379lSxPe6j9nvzxUZ+n7mnJNgjsRNb6IbvGVHRmw=
github.com/nxadm/tail v1.4.11 h1:8feyoE3OzPrcshW5/MJ4sGESc5cqmGkGCWlco4l0bqY=
github.com/nxadm/tail v1.4.11/go.mod h1:OTaG3NK980DZzxbRq6lEuzgU+mug70nY11sMd4JXXHc=
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.7.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
github.com/onsi/ginkgo v1.16.4 h1:29JGrr5oVBm5ulCWet69zQkzWipVXIol6ygQUe/EzNc=
github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
github.com/onsi/gomega v1.4.3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
github.com/onsi/gomega v1.27.6 h1:ENqfyGeS5AX/rlXDd/ETokDz93u0YufY1Pgxuy/PvWE=
github.com/onsi/gomega v1.27.6/go.mod h1:PIQNjfQwkP3aQAH7lf7j87O/5FiNr+ZR8+ipb+qQlhg=
github.com/op/go-logging v0.0.0-20160315200505-970db520ece7/go.mod h1:HzydrMdWErDVzsI23lYNej1Htcns9BCg93Dk0bBINWk=
github.com/orisano/pixelmatch v0.0.0-20220722002657-fb0b55479cde/go.mod h1:nZgzbfBr3hhjoZnS66nKrHmduYNpc34ny7RK4z5/HM0=
github.com/pierrec/lz4/v4 v4.1.23 h1:oJE7T90aYBGtFNrI8+KbETnPymobAhzRrR8Mu8n1yfU=
github.com/pierrec/lz4/v4 v4.1.23/go.mod h1:EoQMVJgeeEOMsCqCzqFm2O0cJvljX2nGZjcRIPL34O4=
github.com/pkg/diff v0.0.0-20210226163009-20ebb0f2a09e/go.mod h1:pJLUxLENpZxwdsKMEsNbx1VGcRFpLqf3715MtcvvzbA=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c h1:ncq/mPwQF4JjgDlrVEn3C11VoGHZN7m8qihwgMEtzYw=
github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c/go.mod h1:OmDBASR4679mdNQnz2pUhc2G8CO2JrUAVFDRBDP/hJE=
github.com/projectdiscovery/asnmap v1.1.1 h1:ImJiKIaACOT7HPx4Pabb5dksolzaFYsD1kID2iwsDqI=
github.com/projectdiscovery/asnmap v1.1.1/go.mod h1:QT7jt9nQanj+Ucjr9BqGr1Q2veCCKSAVyUzLXfEcQ60=
github.com/projectdiscovery/blackrock v0.0.1 h1:lHQqhaaEFjgf5WkuItbpeCZv2DUIE45k0VbGJyft6LQ=
github.com/projectdiscovery/blackrock v0.0.1/go.mod h1:ANUtjDfaVrqB453bzToU+YB4cUbvBRpLvEwoWIwlTss=
github.com/projectdiscovery/dsl v0.8.5 h1:f3opg8Jzikwx6VXC+CbgseUmSUqdfCnfGT08Syhp0sw=
github.com/projectdiscovery/dsl v0.8.5/go.mod h1:AuUq18cpLJJ0uAjJZKaLrdyAgDHrnQAjLMZtPEyMoJw=
github.com/projectdiscovery/fastdialer v0.5.2 h1:BrK23yWc0XD57DMLqnF5oM5tBy8xx9brin+zoSo6gCw=
github.com/projectdiscovery/fastdialer v0.5.2/go.mod h1:euoxS1E93LDnl0OnNN0UALedAFF+EehBxyU3z+79l0g=
github.com/projectdiscovery/goflags v0.1.74 h1:n85uTRj5qMosm0PFBfsvOL24I7TdWRcWq/1GynhXS7c=
github.com/projectdiscovery/goflags v0.1.74/go.mod h1:UMc9/7dFz2oln+10tv6cy+7WZKTHf9UGhaNkF95emh4=
github.com/projectdiscovery/gologger v1.1.67 h1:GZU3AjYiJvcwJT5TlfIv+152/TVmaz62Zyn3/wWXlig=
github.com/projectdiscovery/gologger v1.1.67/go.mod h1:35oeQP6wvj58S+o+Km6boED/t786FXQkI0exhFHJbNE=
github.com/projectdiscovery/gostruct v0.0.2 h1:s8gP8ApugGM4go1pA+sVlPDXaWqNP5BBDDSv7VEdG1M=
github.com/projectdiscovery/gostruct v0.0.2/go.mod h1:H86peL4HKwMXcQQtEa6lmC8FuD9XFt6gkNR0B/Mu5PE=
github.com/projectdiscovery/hmap v0.0.99 h1:XPfLnD3CUrMqVCIdpK9ozD7Xmp3simx3T+2j4WWhHnU=
github.com/projectdiscovery/hmap v0.0.99/go.mod h1:koyUJi83K5G3w35ZLFXOYZIyYJsO+6hQrgDDN1RBrVE=
github.com/projectdiscovery/machineid v0.0.0-20250715113114-c77eb3567582 h1:eR+0HE//Ciyfwy3HC7fjRyKShSJHYoX2Pv7pPshjK/Q=
github.com/projectdiscovery/machineid v0.0.0-20250715113114-c77eb3567582/go.mod h1:3G3BRKui7nMuDFAZKR/M2hiOLtaOmyukT20g88qRQjI=
github.com/projectdiscovery/mapcidr v1.1.97 h1:7FkxNNVXp+m1rIu5Nv/2SrF9k4+LwP8QuWs2puwy+2w=
github.com/projectdiscovery/mapcidr v1.1.97/go.mod h1:9dgTJh1SP02gYZdpzMjm6vtYFkEHQHoTyaVNvaeJ7lA=
github.com/projectdiscovery/networkpolicy v0.1.33 h1:bVgp+XpLEsQ7ZEJt3UaUqIwhI01MMdt7F2dfIKFQg/w=
github.com/projectdiscovery/networkpolicy v0.1.33/go.mod h1:YAPddAXUc/lhoU85AFdvgOQKx8Qh8r0vzSjexRWk6Yk=
github.com/projectdiscovery/ratelimit v0.0.82 h1:rtO5SQf5uQFu5zTahTaTcO06OxmG8EIF1qhdFPIyTak=
github.com/projectdiscovery/ratelimit v0.0.82/go.mod h1:z076BrLkBb5yS7uhHNoCTf8X/BvFSGRxwQ8EzEL9afM=
github.com/projectdiscovery/retryabledns v1.0.112 h1:4iCiuo6jMnw/pdOZRzBQrbUOUu5tOeuvGupxVV8RDLw=
github.com/projectdiscovery/retryabledns v1.0.112/go.mod h1:xsJTKbo+KGqd7+88z1naEUFJybLH2yjB/zUyOweA7k0=
github.com/projectdiscovery/retryablehttp-go v1.3.2 h1:Rv2gw/8t3QZz+WIuHUspVBoRrpBWpVOhzh/wLUGYSVM=
github.com/projectdiscovery/retryablehttp-go v1.3.2/go.mod h1:q1EQ+FX9JP5Z0EqLXDf+8b6XdzWmBXIMPowpI6hQ9aU=
github.com/projectdiscovery/utils v0.8.0 h1:8d79OCs5xGDNXdKxMUKMY/lgQSUWJMYB1B2Sx+oiqkQ=
github.com/projectdiscovery/utils v0.8.0/go.mod h1:CU6tjtyTRxBrnNek+GPJplw4IIHcXNZNKO09kWgqTdg=
github.com/projectdiscovery/wappalyzergo v0.2.62 h1:SMZ70bLCj6jHnFgjanuiaQpqUXY6aiEC3YoM0ZSvYes=
github.com/projectdiscovery/wappalyzergo v0.2.62/go.mod h1:8FtSVcmPRZU0g1euBpdSYEBHIvB7Zz9MOb754ZqZmfU=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/refraction-networking/utls v1.7.1 h1:dxg+jla3uocgN8HtX+ccwDr68uCBBO3qLrkZUbqkcw0=
github.com/refraction-networking/utls v1.7.1/go.mod h1:TUhh27RHMGtQvjQq+RyO11P6ZNQNBb3N0v7wsEjKAIQ=
github.com/remeh/sizedwaitgroup v1.0.0 h1:VNGGFwNo/R5+MJBf6yrsr110p0m4/OX4S3DCy7Kyl5E=
github.com/remeh/sizedwaitgroup v1.0.0/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
github.com/rogpeppe/go-internal v1.12.0 h1:exVL4IDcn6na9z1rAb56Vxr+CgyK3nn3O+epU5NdKM8=
github.com/rogpeppe/go-internal v1.12.0/go.mod h1:E+RYuTGaKKdloAfM02xzb0FW3Paa99yedzYV+kq4uf4=
github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=
github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d h1:hrujxIzL1woJ7AwssoOcM/tq5JjjG2yYOc8odClEiXA=
github.com/saintfish/chardet v0.0.0-20230101081208-5e3ef4b5456d/go.mod h1:uugorj2VCxiV1x+LzaIdVa9b4S4qGAcH6cbhh4qVxOU=
github.com/sashabaranov/go-openai v1.37.0 h1:hQQowgYm4OXJ1Z/wTrE+XZaO20BYsL0R3uRPSpfNZkY=
github.com/sashabaranov/go-openai v1.37.0/go.mod h1:lj5b/K+zjTSFxVLijLSTDZuP7adOgerWeFyZLUhAKRg=
github.com/shirou/gopsutil/v3 v3.23.7 h1:C+fHO8hfIppoJ1WdsVm1RoI0RwXoNdfTK7yWXV0wVj4=
github.com/shirou/gopsutil/v3 v3.23.7/go.mod h1:c4gnmoRC0hQuaLqvxnx1//VXQ0Ms/X9UnJF8pddY5z4=
github.com/shoenig/go-m1cpu v0.1.6 h1:nxdKQNcEB6vzgA2E2bvzKIYRuNj7XNJ4S/aRSwKzFtM=
github.com/shoenig/go-m1cpu v0.1.6/go.mod h1:1JJMcUBvfNwpq05QDQVAnx3gUHr9IYF7GNg9SUEw2VQ=
github.com/shoenig/test v0.6.4 h1:kVTaSd7WLz5WZ2IaoM0RSzRsUD+m8wRR+5qvntpn4LU=
github.com/shoenig/test v0.6.4/go.mod h1:byHiCGXqrVaflBLAMq/srcZIHynQPQgeyvkvXnjqq0k=
github.com/sirupsen/logrus v1.3.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
github.com/sirupsen/logrus v1.9.0/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/smacker/go-tree-sitter v0.0.0-20230720070738-0d0a9f78d8f8 h1:DxgjlvWYsb80WEN2Zv3WqJFAg2DKjUQJO6URGdf1x6Y=
github.com/smacker/go-tree-sitter v0.0.0-20230720070738-0d0a9f78d8f8/go.mod h1:q99oHDsbP0xRwmn7Vmob8gbSMNyvJ83OauXPSuHQuKE=
github.com/sorairolake/lzip-go v0.3.8 h1:j5Q2313INdTA80ureWYRhX+1K78mUXfMoPZCw/ivWik=
github.com/sorairolake/lzip-go v0.3.8/go.mod h1:JcBqGMV0frlxwrsE9sMWXDjqn3EeVf0/54YPsw66qkU=
github.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=
github.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spf13/afero v1.15.0 h1:b/YBCLWAJdFWJTN9cLhiXXcD7mzKn9Dm86dNnfyQw1I=
github.com/spf13/afero v1.15.0/go.mod h1:NC2ByUVxtQs4b3sIUphxK0NioZnmxgyCrfzeuq8lxMg=
github.com/stoewer/go-strcase v1.3.0 h1:g0eASXYtp+yvN9fK8sH94oCIk0fau9uV1/ZdJ0AVEzs=
github.com/stoewer/go-strcase v1.3.0/go.mod h1:fAH5hQ5pehh+j3nZfvwdk2RgEgQjAoM8wodgtPmh1xo=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2 h1:xuMeJ0Sdp5ZMRXx/aWO6RZxdr3beISkG5/G/aIRr3pY=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.4/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/syndtr/goleveldb v1.0.0 h1:fBdIW9lB4Iz0n9khmH8w27SJ3QEJ7+IgjPEwGSZiFdE=
github.com/syndtr/goleveldb v1.0.0/go.mod h1:ZVVdQEZoIme9iO1Ch2Jdy24qqXrMMOU6lpPAyBWyWuQ=
github.com/tidwall/assert v0.1.0 h1:aWcKyRBUAdLoVebxo95N7+YZVTFF/ASTr7BN4sLP6XI=
github.com/tidwall/assert v0.1.0/go.mod h1:QLYtGyeqse53vuELQheYl9dngGCJQ+mTtlxcktb+Kj8=
github.com/tidwall/btree v1.6.0 h1:LDZfKfQIBHGHWSwckhXI0RPSXzlo+KYdjK7FWSqOzzg=
github.com/tidwall/btree v1.6.0/go.mod h1:twD9XRA5jj9VUQGELzDO4HPQTNJsoWWfYEL+EUQ2cKY=
github.com/tidwall/buntdb v1.3.0 h1:gdhWO+/YwoB2qZMeAU9JcWWsHSYU3OvcieYgFRS0zwA=
github.com/tidwall/buntdb v1.3.0/go.mod h1:lZZrZUWzlyDJKlLQ6DKAy53LnG7m5kHyrEHvvcDmBpU=
github.com/tidwall/gjson v1.12.1/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/gjson v1.18.0 h1:FIDeeyB800efLX89e5a8Y0BNH+LOngJyGrIWxG2FKQY=
github.com/tidwall/gjson v1.18.0/go.mod h1:/wbyibRr2FHMks5tjHJ5F8dMZh3AcwJEMf5vlfC0lxk=
github.com/tidwall/grect v0.1.4 h1:dA3oIgNgWdSspFzn1kS4S/RDpZFLrIxAZOdJKjYapOg=
github.com/tidwall/grect v0.1.4/go.mod h1:9FBsaYRaR0Tcy4UwefBX/UDcDcDy9V5jUcxHzv2jd5Q=
github.com/tidwall/lotsa v1.0.2 h1:dNVBH5MErdaQ/xd9s769R31/n2dXavsQ0Yf4TMEHHw8=
github.com/tidwall/lotsa v1.0.2/go.mod h1:X6NiU+4yHA3fE3Puvpnn1XMDrFZrE9JO2/w+UMuqgR8=
github.com/tidwall/match v1.1.1 h1:+Ho715JplO36QYgwN9PGYNhgZvoUSc9X2c80KVTi+GA=
github.com/tidwall/match v1.1.1/go.mod h1:eRSPERbgtNPcGhD8UCthc6PmLEQXEWd3PRB5JTxsfmM=
github.com/tidwall/pretty v1.2.0/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/pretty v1.2.1 h1:qjsOFOWWQl+N3RsoF5/ssm1pHmJJwhjlSbZ51I6wMl4=
github.com/tidwall/pretty v1.2.1/go.mod h1:ITEVvHYasfjBbM0u2Pg8T2nJnzm8xPwvNhhsoaGGjNU=
github.com/tidwall/rtred v0.1.2 h1:exmoQtOLvDoO8ud++6LwVsAMTu0KPzLTUrMln8u1yu8=
github.com/tidwall/rtred v0.1.2/go.mod h1:hd69WNXQ5RP9vHd7dqekAz+RIdtfBogmglkZSRxCHFQ=
github.com/tidwall/tinyqueue v0.1.1 h1:SpNEvEggbpyN5DIReaJ2/1ndroY8iyEGxPYxoSaymYE=
github.com/tidwall/tinyqueue v0.1.1/go.mod h1:O/QNHwrnjqr6IHItYrzoHAKYhBkLI67Q096fQP5zMYw=
github.com/tklauser/go-sysconf v0.3.11/go.mod h1:GqXfhXY3kiPa0nAXPDIQIWzJbMCB7AmcWpGR8lSZfqI=
github.com/tklauser/go-sysconf v0.3.12 h1:0QaGUFOdQaIVdPgfITYzaTegZvdCjmYO52cSFAEVmqU=
github.com/tklauser/go-sysconf v0.3.12/go.mod h1:Ho14jnntGE1fpdOqQEEaiKRpvIavV0hSfmBq8nJbHYI=
github.com/tklauser/numcpus v0.6.0/go.mod h1:FEZLMke0lhOUG6w2JadTzp0a+Nl8PF/GFkQ5UVIcaL4=
github.com/tklauser/numcpus v0.6.1 h1:ng9scYS7az0Bk4OZLvrNXNSAO2Pxr1XXRAPyjhIx+Fk=
github.com/tklauser/numcpus v0.6.1/go.mod h1:1XfjsgE2zo8GVw7POkMbHENHzVg3GzmoZ9fESEdAacY=
github.com/ulikunitz/xz v0.5.8/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/ulikunitz/xz v0.5.15 h1:9DNdB5s+SgV3bQ2ApL10xRc35ck0DuIX/isZvIk+ubY=
github.com/ulikunitz/xz v0.5.15/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oWt/14=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasttemplate v1.2.2 h1:lxLXG0uE3Qnshl9QyaK6XJxMXlQZELvChBOCmQD0Loo=
github.com/valyala/fasttemplate v1.2.2/go.mod h1:KHLXt3tVN2HBp8eijSv/kGJopbvo7S+qRAEEKiv+SiQ=
github.com/vulncheck-oss/go-exploit v1.51.0 h1:HTmJ4Q94tbEDPb35mQZn6qMg4rT+Sw9n+L7g3Pjr+3o=
github.com/vulncheck-oss/go-exploit v1.51.0/go.mod h1:J28w0dLnA6DnCrnBm9Sbt6smX8lvztnnN2wCXy7No6c=
github.com/weppos/publicsuffix-go v0.13.0/go.mod h1:z3LCPQ38eedDQSwmsSRW4Y7t2L8Ln16JPQ02lHAdn5k=
github.com/weppos/publicsuffix-go v0.30.1-0.20230422193905-8fecedd899db/go.mod h1:aiQaH1XpzIfgrJq3S1iw7w+3EDbRP7mF5fmwUhWyRUs=
github.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39 h1:Bz/zVM/LoGZ9IztGBHrq2zlFQQbEG8dBYnxb4hamIHM=
github.com/weppos/publicsuffix-go v0.40.3-0.20250408071509-6074bbe7fd39/go.mod h1:2oFzEwGYI7lhiqG0YkkcKa6VcpjVinQbWxaPzytDmLA=
github.com/xdg-go/pbkdf2 v1.0.0 h1:Su7DPu48wXMwC3bs7MCNG+z4FhcyEuz5dlvchbq0B0c=
github.com/xdg-go/pbkdf2 v1.0.0/go.mod h1:jrpuAogTd400dnrH08LKmI/xc1MbPOebTwRqcT5RDeI=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/yl2chen/cidranger v1.0.2 h1:lbOWZVCG1tCRX4u24kuM1Tb4nHqWkDxwLdoS+SevawU=
github.com/yl2chen/cidranger v1.0.2/go.mod h1:9U1yz7WPYDwf0vpNWFaeRh0bjwz5RVgRy/9UEQfHl0g=
github.com/ysmood/fetchup v0.2.3 h1:ulX+SonA0Vma5zUFXtv52Kzip/xe7aj4vqT5AJwQ+ZQ=
github.com/ysmood/fetchup v0.2.3/go.mod h1:xhibcRKziSvol0H1/pj33dnKrYyI2ebIvz5cOOkYGns=
github.com/ysmood/goob v0.4.0 h1:HsxXhyLBeGzWXnqVKtmT9qM7EuVs/XOgkX7T6r1o1AQ=
github.com/ysmood/goob v0.4.0/go.mod h1:u6yx7ZhS4Exf2MwciFr6nIM8knHQIE22lFpWHnfql18=
github.com/ysmood/gop v0.2.0 h1:+tFrG0TWPxT6p9ZaZs+VY+opCvHU8/3Fk6BaNv6kqKg=
github.com/ysmood/gop v0.2.0/go.mod h1:rr5z2z27oGEbyB787hpEcx4ab8cCiPnKxn0SUHt6xzk=
github.com/ysmood/got v0.40.0 h1:ZQk1B55zIvS7zflRrkGfPDrPG3d7+JOza1ZkNxcc74Q=
github.com/ysmood/got v0.40.0/go.mod h1:W7DdpuX6skL3NszLmAsC5hT7JAhuLZhByVzHTq874Qg=
github.com/ysmood/gotrace v0.6.0 h1:SyI1d4jclswLhg7SWTL6os3L1WOKeNn/ZtzVQF8QmdY=
github.com/ysmood/gotrace v0.6.0/go.mod h1:TzhIG7nHDry5//eYZDYcTzuJLYQIkykJzCRIo4/dzQM=
github.com/ysmood/gson v0.7.3 h1:QFkWbTH8MxyUTKPkVWAENJhxqdBa4lYTQWqZCiLG6kE=
github.com/ysmood/gson v0.7.3/go.mod h1:3Kzs5zDl21g5F/BlLTNcuAGAYLKt2lV5G8D1zF3RNmg=
github.com/ysmood/leakless v0.9.0 h1:qxCG5VirSBvmi3uynXFkcnLMzkphdh3xx5FtrORwDCU=
github.com/ysmood/leakless v0.9.0/go.mod h1:R8iAXPRaG97QJwqxs74RdwzcRHT1SWCGTNqY8q0JvMQ=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
github.com/yuin/goldmark v1.7.1/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=
github.com/yuin/goldmark v1.7.4 h1:BDXOHExt+A7gwPCJgPIIq7ENvceR7we7rOS9TNoLZeg=
github.com/yuin/goldmark v1.7.4/go.mod h1:uzxRWxtg69N339t3louHJ7+O03ezfj6PlliRlaOzY1E=
github.com/yuin/goldmark-emoji v1.0.3 h1:aLRkLHOuBR2czCY4R8olwMjID+tENfhyFDMCRhbIQY4=
github.com/yuin/goldmark-emoji v1.0.3/go.mod h1:tTkZEbwu5wkPmgTcitqddVxY9osFZiavD+r4AzQrh1U=
github.com/yusufpapurcu/wmi v1.2.3/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
github.com/yusufpapurcu/wmi v1.2.4 h1:zFUKzehAFReQwLys1b/iSMl+JQGSCSjtVqQn9bBrPo0=
github.com/yusufpapurcu/wmi v1.2.4/go.mod h1:SBZ9tNy3G9/m5Oi98Zks0QjeHVDvuK0qfxQmPyzfmi0=
github.com/zcalusic/sysinfo v1.0.2 h1:nwTTo2a+WQ0NXwo0BGRojOJvJ/5XKvQih+2RrtWqfxc=
github.com/zcalusic/sysinfo v1.0.2/go.mod h1:kluzTYflRWo6/tXVMJPdEjShsbPpsFRyy+p1mBQPC30=
github.com/zmap/rc2 v0.0.0-20131011165748-24b9757f5521/go.mod h1:3YZ9o3WnatTIZhuOtot4IcUfzoKVjUHqu6WALIyI0nE=
github.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248 h1:Nzukz5fNOBIHOsnP+6I79kPx3QhLv8nBy2mfFhBRq30=
github.com/zmap/rc2 v0.0.0-20190804163417-abaa70531248/go.mod h1:3YZ9o3WnatTIZhuOtot4IcUfzoKVjUHqu6WALIyI0nE=
github.com/zmap/zcertificate v0.0.0-20180516150559-0e3d58b1bac4/go.mod h1:5iU54tB79AMBcySS0R2XIyZBAVmeHranShAFELYx7is=
github.com/zmap/zcertificate v0.0.1/go.mod h1:q0dlN54Jm4NVSSuzisusQY0hqDWvu92C+TWveAxiVWk=
github.com/zmap/zcrypto v0.0.0-20201128221613-3719af1573cf/go.mod h1:aPM7r+JOkfL+9qSB4KbYjtoEzJqUK50EXkkJabeNJDQ=
github.com/zmap/zcrypto v0.0.0-20201211161100-e54a5822fb7e/go.mod h1:aPM7r+JOkfL+9qSB4KbYjtoEzJqUK50EXkkJabeNJDQ=
github.com/zmap/zcrypto v0.0.0-20230422215203-9a665e1e9968 h1:YOQ1vXEwE4Rnj+uQ/3oCuJk5wgVsvUyW+glsndwYuyA=
github.com/zmap/zcrypto v0.0.0-20230422215203-9a665e1e9968/go.mod h1:xIuOvYCZX21S5Z9bK1BMrertTGX/F8hgAPw7ERJRNS0=
github.com/zmap/zlint/v3 v3.0.0/go.mod h1:paGwFySdHIBEMJ61YjoqT4h7Ge+fdYG4sUQhnTb1lJ8=
go.etcd.io/bbolt v1.3.7 h1:j+zJOnnEjF/kyHlDDgGnVL/AIqIJPq8UoB2GSNfkUfQ=
go.etcd.io/bbolt v1.3.7/go.mod h1:N9Mkw9X8x5fupy0IKsmuqVtoGDyxsaDlbk4Rd05IAQw=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.uber.org/multierr v1.11.0 h1:blXXJkSxSSfBVBlC76pxqeO+LN3aDfLQo+309xJstO0=
go.uber.org/multierr v1.11.0/go.mod h1:20+QtiLqy0Nd6FdQB9TLXag12DsQkrbs3htMFfDN80Y=
go4.org v0.0.0-20230225012048-214862532bf5 h1:nifaUDeh+rPaBCMPMQHZmvJf+QdpLFnuQPwx+LxVmtc=
go4.org v0.0.0-20230225012048-214862532bf5/go.mod h1:F57wTi5Lrj6WLyswp5EYV1ncrEbFGHD4hhz6S1ZYeaU=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200510223506-06a226fb4e37/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201124201722-c8d3bf9c5392/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20201208171446-5f87f3452ae9/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211209193657-4570a0811e8b/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.6.0/go.mod h1:OFC/31mSvZgRz0V1QTNCzfAI1aIRzbiufJtkMIlEp58=
golang.org/x/crypto v0.7.0/go.mod h1:pYwdfH91IfpZVANVyUOhSIPZaFoJGxTFbZhFTx+dXZU=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b h1:M2rDM6z3Fhozi9O7NWsxAkg/yqS/lQJ6PmkyIV3YP+o=
golang.org/x/exp v0.0.0-20250620022241-b7579e27df2b/go.mod h1:3//PLf8L/X+8b4vuAfHzxeRUl04Adcb341+IGKfnqS8=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.32.0 h1:9F4d3PHLljb6x//jOyokMv3eX+YDeepZSEo3mFJy93c=
golang.org/x/mod v0.32.0/go.mod h1:SgipZ/3h2Ci89DlEtEXWUk/HteuRin+HHhN+WbNhguU=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200528225125-3c3fba18258b/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20201110031124-69a78807bb2b/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.8.0/go.mod h1:QVkue5JL9kW//ek3r6jTKnTFis1tRmNAW2P1shuFdJc=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.5.0/go.mod h1:9/XBHVqLaWO3/BRHs5jbpYCnOZVjj5V0ndyaAM7KB4I=
golang.org/x/oauth2 v0.34.0 h1:hqK/t4AKgbqWkdkcAeI8XLmbK+4m4G5YeQRrmiotGlw=
golang.org/x/oauth2 v0.34.0/go.mod h1:lzm5WQJQwKZ3nwavOZ3IS5Aulzxi68dUSgRHujetwEA=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.19.0 h1:vV+1eWNmZ5geRlYjzm2adRgW2/mcpevXNg50YZtPCE4=
golang.org/x/sync v0.19.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190916202348-b4ddaad3f8a3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201126233918-771906719818/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201204225414-ed752295db88/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210228012217-479acdf4ea46/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220310020820-b874c991c1a5/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220615213510-4f61da869c0c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.2.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.10.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.6.0/go.mod h1:m6U89DPEgQRMq3DNkDClhWw02AUbt2daBVO4cn4Hv9U=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/term v0.40.0 h1:36e4zGLqU4yhjlmxEaagx2KuYbJq3EwY8K943ZsHcvg=
golang.org/x/term v0.40.0/go.mod h1:w2P8uVp06p2iyKKuvXIm7N/y0UCRt3UfJTfZ7oOpglM=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.8.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216052735-49a3e744a425/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.41.0 h1:a9b8iMweWG+S0OBnlU36rzLp20z1Rp10w+IY2czHTQc=
golang.org/x/tools v0.41.0/go.mod h1:XSY6eDqxVNiYgezAVqqCeihT4j1U2CCsqvH3WhQpnlg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
google.golang.org/protobuf v1.28.0/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
google.golang.org/protobuf v1.28.1/go.mod h1:HV8QOd/L58Z+nl8r43ehVNZIU/HEI6OcFqwMG9pJV4I=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=
================================================
FILE: integration_tests/run.sh
================================================
#!/bin/bash
echo "::group::Build katana"
rm integration-test katana 2>/dev/null
cd ../cmd/katana
go build
mv katana ../../integration_tests/katana
echo "::endgroup::"
echo "::group::Build katana integration-test"
cd ../integration-test
go build
mv integration-test ../../integration_tests/integration-test
cd ../../integration_tests
echo "::endgroup::"
./integration-test
if [ $? -eq 0 ]
then
exit 0
else
exit 1
fi
================================================
FILE: internal/runner/banner.go
================================================
package runner
import (
"github.com/projectdiscovery/gologger"
updateutils "github.com/projectdiscovery/utils/update"
)
var banner = (`
__ __
/ /_____ _/ /____ ____ ___ _
/ '_/ _ / __/ _ / _ \/ _ /
/_/\_\\_,_/\__/\_,_/_//_/\_,_/
`)
var version = "v1.5.0"
// showBanner is used to show the banner to the user
func showBanner() {
gologger.Print().Msgf("%s\n", banner)
gologger.Print().Msgf("\t\tprojectdiscovery.io\n\n")
}
// GetUpdateCallback returns a callback function that updates katana
func GetUpdateCallback() func() {
return func() {
showBanner()
updateutils.GetUpdateToolCallback("katana", version)()
}
}
================================================
FILE: internal/runner/executer.go
================================================
package runner
import (
"strings"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/utils/errkit"
urlutil "github.com/projectdiscovery/utils/url"
"github.com/remeh/sizedwaitgroup"
)
// ExecuteCrawling executes the crawling main loop
func (r *Runner) ExecuteCrawling() error {
if r.crawler == nil {
return errkit.New("crawler is not initialized")
}
inputs := r.parseInputs()
if len(inputs) == 0 {
return errkit.New("no input provided for crawling")
}
for _, input := range inputs {
_ = r.state.InFlightUrls.Set(addSchemeIfNotExists(input), struct{}{})
}
defer func() {
if err := r.crawler.Close(); err != nil {
gologger.Error().Msgf("Error closing crawler: %v\n", err)
}
}()
wg := sizedwaitgroup.New(r.options.Parallelism)
for _, input := range inputs {
if !r.networkpolicy.Validate(input) {
gologger.Info().Msgf("Skipping excluded host %s", input)
continue
}
wg.Add()
input = addSchemeIfNotExists(input)
go func(input string) {
defer wg.Done()
if err := r.crawler.Crawl(input); err != nil {
gologger.Warning().Msgf("Could not crawl %s: %s", input, err)
}
r.state.InFlightUrls.Delete(input)
}(input)
}
wg.Wait()
return nil
}
// scheme less urls are skipped and are required for headless mode and other purposes
// this method adds scheme if given input does not have any
func addSchemeIfNotExists(inputURL string) string {
if strings.HasPrefix(inputURL, urlutil.HTTP) || strings.HasPrefix(inputURL, urlutil.HTTPS) {
return inputURL
}
parsed, err := urlutil.Parse(inputURL)
if err != nil {
gologger.Warning().Msgf("input %v is not a valid url got %v", inputURL, err)
return inputURL
}
if parsed.Port() != "" && (parsed.Port() == "80" || parsed.Port() == "8080") {
return urlutil.HTTP + urlutil.SchemeSeparator + inputURL
} else {
return urlutil.HTTPS + urlutil.SchemeSeparator + inputURL
}
}
================================================
FILE: internal/runner/healthcheck.go
================================================
package runner
import (
"fmt"
"net"
"os/exec"
"runtime"
"strings"
"github.com/projectdiscovery/goflags"
"github.com/projectdiscovery/katana/pkg/types"
fileutil "github.com/projectdiscovery/utils/file"
permissionutil "github.com/projectdiscovery/utils/permission"
)
func DoHealthCheck(options *types.Options, flagSet *goflags.FlagSet) string {
// RW permissions on config file
cfgFilePath, _ := flagSet.GetConfigFilePath()
var test strings.Builder
_, _ = fmt.Fprintf(&test, "Version: %s\n", version)
_, _ = fmt.Fprintf(&test, "Operative System: %s\n", runtime.GOOS)
_, _ = fmt.Fprintf(&test, "Architecture: %s\n", runtime.GOARCH)
_, _ = fmt.Fprintf(&test, "Go Version: %s\n", runtime.Version())
_, _ = fmt.Fprintf(&test, "Compiler: %s\n", runtime.Compiler)
var testResult string
if permissionutil.IsRoot {
testResult = "Ok"
} else {
testResult = "Ko"
}
_, _ = fmt.Fprintf(&test, "root: %s\n", testResult)
ok, err := fileutil.IsReadable(cfgFilePath)
if ok {
testResult = "Ok"
} else {
testResult = "Ko"
}
if err != nil {
testResult += fmt.Sprintf(" (%s)", err)
}
_, _ = fmt.Fprintf(&test, "Config file \"%s\" Read => %s\n", cfgFilePath, testResult)
ok, err = fileutil.IsWriteable(cfgFilePath)
if ok {
testResult = "Ok"
} else {
testResult = "Ko"
}
if err != nil {
testResult += fmt.Sprintf(" (%s)", err)
}
fmt.Fprintf(&test, "Config file \"%s\" Write => %s\n", cfgFilePath, testResult)
c4, err := net.Dial("tcp4", "scanme.sh:80")
if err == nil && c4 != nil {
_ = c4.Close()
}
testResult = "Ok"
if err != nil {
testResult = fmt.Sprintf("Ko (%s)", err)
}
fmt.Fprintf(&test, "TCP IPv4 connectivity to scanme.sh:80 => %s\n", testResult)
c6, err := net.Dial("tcp6", "scanme.sh:80")
if err == nil && c6 != nil {
_ = c6.Close()
}
testResult = "Ok"
if err != nil {
testResult = fmt.Sprintf("Ko (%s)", err)
}
_, _ = fmt.Fprintf(&test, "TCP IPv6 connectivity to scanme.sh:80 => %s\n", testResult)
u4, err := net.Dial("udp4", "scanme.sh:53")
if err == nil && u4 != nil {
_ = u4.Close()
}
testResult = "Ok"
if err != nil {
testResult = fmt.Sprintf("Ko (%s)", err)
}
_, _ = fmt.Fprintf(&test, "UDP IPv4 connectivity to scanme.sh:80 => %s\n", testResult)
u6, err := net.Dial("udp6", "scanme.sh:80")
if err == nil && u6 != nil {
_ = u6.Close()
}
testResult = "Ok"
if err != nil {
testResult = fmt.Sprintf("Ko (%s)", err)
}
_, _ = fmt.Fprintf(&test, "UDP IPv6 connectivity to scanme.sh:80 => %s\n", testResult)
// attempt to identify if chome is installed locally
if chromePath, err := exec.LookPath("chrome"); err == nil {
_, _ = fmt.Fprintf(&test, "Potential chrome binary path (linux/osx) => %s\n", chromePath)
}
if chromePath, err := exec.LookPath("chrome.exe"); err == nil {
_, _ = fmt.Fprintf(&test, "Potential chrome.exe binary path (windows) => %s\n", chromePath)
}
return test.String()
}
================================================
FILE: internal/runner/options.go
================================================
package runner
import (
"bufio"
"os"
"path/filepath"
"regexp"
"strings"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/gologger/formatter"
"github.com/projectdiscovery/katana/pkg/types"
"github.com/projectdiscovery/katana/pkg/utils"
"github.com/projectdiscovery/utils/errkit"
fileutil "github.com/projectdiscovery/utils/file"
"gopkg.in/yaml.v3"
)
// validateOptions validates the provided options for crawler
func validateOptions(options *types.Options) error {
if options.MaxDepth <= 0 && options.CrawlDuration.Seconds() <= 0 {
return errkit.New("either max-depth or crawl-duration must be specified")
}
if len(options.URLs) == 0 && !fileutil.HasStdin() {
return errkit.New("no inputs specified for crawler")
}
// Disabling automatic form fill (-aff) for headless navigation due to incorrect implementation.
// Form filling should be handled via headless actions within the page context
if options.HeadlessHybrid && options.AutomaticFormFill {
options.AutomaticFormFill = false
gologger.Info().Msgf("Automatic form fill (-aff) has been disabled for headless navigation.")
}
// Disallow ambiguous engine selection
if options.Headless && options.HeadlessHybrid {
return errkit.New("flags -hl (headless) and -hh (hybrid) are mutually exclusive")
}
if (options.HeadlessOptionalArguments != nil || options.HeadlessNoSandbox || options.SystemChromePath != "") &&
!options.Headless && !options.HeadlessHybrid {
return errkit.New("headless (-hl) or hybrid (-hh) mode is required if -ho, -nos or -scp are set")
}
if (options.HeadlessOptionalArguments != nil || options.HeadlessNoSandbox || options.SystemChromePath != "") && !options.Headless && !options.HeadlessHybrid {
return errkit.New("headless mode (-hl) is required if -ho, -nos or -scp are set")
}
if options.SystemChromePath != "" {
if !fileutil.FileExists(options.SystemChromePath) {
return errkit.New("specified system chrome binary does not exist")
}
}
if options.StoreResponseDir != "" && !options.StoreResponse {
gologger.Debug().Msgf("store response directory specified, enabling \"sr\" flag automatically\n")
options.StoreResponse = true
}
for _, mr := range options.OutputMatchRegex {
cr, err := regexp.Compile(mr)
if err != nil {
return errkit.Wrap(err, "Invalid value for match regex option")
}
options.MatchRegex = append(options.MatchRegex, cr)
}
for _, fr := range options.OutputFilterRegex {
cr, err := regexp.Compile(fr)
if err != nil {
return errkit.Wrap(err, "Invalid value for filter regex option")
}
options.FilterRegex = append(options.FilterRegex, cr)
}
if options.KnownFiles != "" && options.MaxDepth < 3 {
gologger.Info().Msgf("Depth automatically set to 3 to accommodate the `--known-files` option (originally set to %d).", options.MaxDepth)
options.MaxDepth = 3
}
gologger.DefaultLogger.SetFormatter(formatter.NewCLI(options.NoColors))
return nil
}
// readCustomFormConfig reads custom form fill config
func readCustomFormConfig(formConfig string) error {
file, err := os.Open(formConfig)
if err != nil {
return errkit.Wrap(err, "could not read form config")
}
defer func() {
if err := file.Close(); err != nil {
gologger.Error().Msgf("Error closing file: %v\n", err)
}
}()
var data utils.FormFillData
if err := yaml.NewDecoder(file).Decode(&data); err != nil {
return errkit.Wrap(err, "could not decode form config")
}
utils.FormData = data
return nil
}
// parseInputs parses the inputs returning a slice of URLs
func (r *Runner) parseInputs() []string {
values := make(map[string]struct{})
for _, url := range r.options.URLs {
if url == "" {
continue
}
value := normalizeInput(url)
if _, ok := values[value]; !ok {
values[value] = struct{}{}
}
}
if r.stdin {
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
value := normalizeInput(scanner.Text())
if _, ok := values[value]; !ok {
values[value] = struct{}{}
}
}
}
final := make([]string, 0, len(values))
for k := range values {
final = append(final, k)
}
return final
}
func normalizeInput(value string) string {
return strings.TrimSpace(value)
}
func initExampleFormFillConfig() error {
homedir, err := os.UserHomeDir()
if err != nil {
return errkit.Wrap(err, "could not get home directory")
}
defaultConfig := filepath.Join(homedir, ".config", "katana", "form-config.yaml")
if fileutil.FileExists(defaultConfig) {
return readCustomFormConfig(defaultConfig)
}
if err := os.MkdirAll(filepath.Dir(defaultConfig), 0775); err != nil {
return err
}
exampleConfig, err := os.Create(defaultConfig)
if err != nil {
return errkit.Wrap(err, "could not get home directory")
}
defer func() {
if err := exampleConfig.Close(); err != nil {
gologger.Error().Msgf("Error closing example config: %v\n", err)
}
}()
err = yaml.NewEncoder(exampleConfig).Encode(utils.DefaultFormFillData)
return err
}
================================================
FILE: internal/runner/runner.go
================================================
package runner
import (
"encoding/json"
"os"
"strconv"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/katana/pkg/engine"
"github.com/projectdiscovery/katana/pkg/engine/headless"
"github.com/projectdiscovery/katana/pkg/engine/hybrid"
"github.com/projectdiscovery/katana/pkg/engine/standard"
"github.com/projectdiscovery/katana/pkg/types"
"github.com/projectdiscovery/mapcidr"
"github.com/projectdiscovery/mapcidr/asn"
"github.com/projectdiscovery/networkpolicy"
"github.com/projectdiscovery/utils/errkit"
fileutil "github.com/projectdiscovery/utils/file"
iputil "github.com/projectdiscovery/utils/ip"
mapsutil "github.com/projectdiscovery/utils/maps"
updateutils "github.com/projectdiscovery/utils/update"
"go.uber.org/multierr"
)
// Runner creates the required resources for crawling
// and executes the crawl process.
type Runner struct {
crawlerOptions *types.CrawlerOptions
stdin bool
crawler engine.Engine
options *types.Options
state *RunnerState
networkpolicy *networkpolicy.NetworkPolicy
}
type RunnerState struct {
InFlightUrls *mapsutil.SyncLockMap[string, struct{}]
}
// New returns a new crawl runner structure
func New(options *types.Options) (*Runner, error) {
// create the resume configuration structure
if options.ShouldResume() {
gologger.Info().Msg("Resuming from save checkpoint")
file, err := os.ReadFile(options.Resume)
if err != nil {
return nil, err
}
runnerState := &RunnerState{}
err = json.Unmarshal(file, runnerState)
if err != nil {
return nil, err
}
options.URLs = mapsutil.GetKeys(runnerState.InFlightUrls.GetAll())
}
options.ConfigureOutput()
showBanner()
if options.Version {
gologger.Info().Msgf("Current version: %s", version)
return nil, nil
}
if !options.DisableUpdateCheck {
latestVersion, err := updateutils.GetToolVersionCallback("katana", version)()
if err != nil {
if options.Verbose {
gologger.Error().Msgf("katana version check failed: %v", err.Error())
}
} else {
gologger.Info().Msgf("Current katana version %v %v", version, updateutils.GetVersionDescription(version, latestVersion))
}
}
if err := initExampleFormFillConfig(); err != nil {
return nil, errkit.Wrap(err, "could not init default config")
}
if err := validateOptions(options); err != nil {
return nil, errkit.Wrap(err, "could not validate options")
}
if options.FormConfig != "" {
if err := readCustomFormConfig(options.FormConfig); err != nil {
return nil, err
}
}
crawlerOptions, err := types.NewCrawlerOptions(options)
if err != nil {
return nil, errkit.Wrap(err, "could not create crawler options")
}
var crawler engine.Engine
switch {
case options.ChromeWSUrl != "":
// When connecting to existing browser via WebSocket URL,
// use hybrid engine regardless of other flags
// (ChromeWSUrl takes precedence over -headless flag)
crawler, err = hybrid.New(crawlerOptions)
case options.Headless:
crawler, err = headless.New(crawlerOptions)
case options.HeadlessHybrid:
crawler, err = hybrid.New(crawlerOptions)
default:
crawler, err = standard.New(crawlerOptions)
}
if err != nil {
return nil, errkit.Wrap(err, "could not create standard crawler")
}
var npOptions networkpolicy.Options
for _, exclude := range options.Exclude {
switch {
case exclude == "cdn":
//implement cdn check in netoworkpolicy pkg??
continue
case exclude == "private-ips":
npOptions.DenyList = append(npOptions.DenyList, networkpolicy.DefaultIPv4Denylist...)
npOptions.DenyList = append(npOptions.DenyList, networkpolicy.DefaultIPv4DenylistRanges...)
npOp
gitextract_rfj92yh6/
├── .github/
│ ├── ISSUE_TEMPLATE/
│ │ ├── README.md
│ │ ├── config.yml
│ │ ├── feature_request.md.disabled
│ │ └── issue-report.md.disabled
│ ├── MAINTAINER_GUIDE.md
│ ├── PULL_REQUEST_TEMPLATE.md
│ ├── dependabot.yml
│ ├── release.yml
│ └── workflows/
│ ├── build-test.yml
│ ├── codeql-analysis.yml
│ ├── compat-checks.yaml
│ ├── dep-auto-merge.yml
│ ├── discussion-triage.yml
│ ├── dockerhub-push.yml
│ ├── functional-test.yml
│ ├── release-binary.yml
│ ├── release-test.yml
│ ├── security-crawl-maze-score.yaml
│ ├── stale.yml
│ └── workflow-monitor.yml
├── .gitignore
├── .goreleaser/
│ ├── linux.yml
│ ├── mac.yml
│ └── windows.yml
├── Dockerfile
├── LICENSE.md
├── Makefile
├── README.md
├── SECURITY.md
├── cmd/
│ ├── functional-test/
│ │ ├── main.go
│ │ └── run.sh
│ ├── integration-test/
│ │ ├── filters.go
│ │ ├── integration-test.go
│ │ └── library.go
│ └── tools/
│ └── crawl-maze-score/
│ └── main.go
├── go.mod
├── go.sum
├── integration_tests/
│ └── run.sh
├── internal/
│ ├── runner/
│ │ ├── banner.go
│ │ ├── executer.go
│ │ ├── healthcheck.go
│ │ ├── options.go
│ │ └── runner.go
│ └── testutils/
│ ├── helper.go
│ ├── integration.go
│ └── testutils.go
└── pkg/
├── engine/
│ ├── common/
│ │ ├── base.go
│ │ ├── error.go
│ │ └── http.go
│ ├── engine.go
│ ├── headless/
│ │ ├── TODOS.md
│ │ ├── browser/
│ │ │ ├── browser.go
│ │ │ ├── cookie/
│ │ │ │ ├── cookie.go
│ │ │ │ ├── cookie_test.go
│ │ │ │ └── rules.json
│ │ │ ├── element.go
│ │ │ └── stealth/
│ │ │ └── assets.go
│ │ ├── captcha/
│ │ │ ├── capsolver/
│ │ │ │ ├── capsolver.go
│ │ │ │ └── capsolver_test.go
│ │ │ ├── captcha.go
│ │ │ ├── helpers_test.go
│ │ │ ├── identify.go
│ │ │ ├── identify_test.go
│ │ │ ├── inject_test.go
│ │ │ ├── injection_test.go
│ │ │ ├── integration_test.go
│ │ │ ├── js/
│ │ │ │ ├── identify.js
│ │ │ │ ├── inject-hcaptcha.js
│ │ │ │ ├── inject-recaptcha.js
│ │ │ │ ├── inject-turnstile.js
│ │ │ │ └── js.go
│ │ │ ├── solver.go
│ │ │ └── solver_test.go
│ │ ├── crawler/
│ │ │ ├── crawler.go
│ │ │ ├── diagnostics/
│ │ │ │ └── diagnostics.go
│ │ │ ├── formfill.go
│ │ │ ├── normalizer/
│ │ │ │ ├── dom_utils.go
│ │ │ │ ├── dom_utils_test.go
│ │ │ │ ├── helpers.go
│ │ │ │ ├── normalizer.go
│ │ │ │ ├── simhash/
│ │ │ │ │ ├── simhash.go
│ │ │ │ │ └── simhash_test.go
│ │ │ │ ├── text_utils.go
│ │ │ │ └── text_utils_test.go
│ │ │ ├── state.go
│ │ │ └── state_test.go
│ │ ├── debugger.go
│ │ ├── graph/
│ │ │ └── graph.go
│ │ ├── headless.go
│ │ ├── js/
│ │ │ ├── js.go
│ │ │ ├── page-init.js
│ │ │ └── utils.js
│ │ └── types/
│ │ └── types.go
│ ├── hybrid/
│ │ ├── crawl.go
│ │ ├── doc.go
│ │ ├── hijack.go
│ │ └── hybrid.go
│ ├── parser/
│ │ ├── files/
│ │ │ ├── request.go
│ │ │ ├── robotstxt.go
│ │ │ ├── robotstxt_test.go
│ │ │ ├── sitemapxml.go
│ │ │ └── sitemapxml_test.go
│ │ ├── parser.go
│ │ ├── parser_generic.go
│ │ ├── parser_nojs.go
│ │ └── parser_test.go
│ └── standard/
│ ├── crawl.go
│ ├── doc.go
│ └── standard.go
├── navigation/
│ ├── request.go
│ └── response.go
├── output/
│ ├── custom_field.go
│ ├── error.go
│ ├── fields.go
│ ├── fields_test.go
│ ├── file_writer.go
│ ├── format_json.go
│ ├── format_screen.go
│ ├── format_template.go
│ ├── options.go
│ ├── output.go
│ ├── responses.go
│ └── result.go
├── types/
│ ├── crawler_options.go
│ ├── default.go
│ ├── options.go
│ └── options_test.go
└── utils/
├── extensions/
│ ├── extensions.go
│ └── extensions_test.go
├── filters/
│ ├── filters.go
│ ├── filters_test.go
│ └── simple.go
├── formfields.go
├── formfields_test.go
├── formfill.go
├── formfill_test.go
├── jsluice.go
├── jsluice_test.go
├── maps.go
├── maps_test.go
├── pathtrie.go
├── pathtrie_test.go
├── queue/
│ ├── priority_queue.go
│ ├── priority_queue_test.go
│ ├── queue.go
│ ├── stack.go
│ ├── stack_test.go
│ └── strategy.go
├── regex.go
├── regex_test.go
├── scope/
│ ├── scope.go
│ └── scope_test.go
├── urlfingerprint.go
├── urlfingerprint_test.go
├── utils.go
└── utils_test.go
SYMBOL INDEX (610 symbols across 112 files)
FILE: cmd/functional-test/main.go
function main (line 23) | func main() {
function runFunctionalTests (line 33) | func runFunctionalTests() error {
function runIndividualTestCase (line 45) | func runIndividualTestCase(testcase testutils.TestCase) error {
FILE: cmd/integration-test/filters.go
type matchConditionIntegrationTest (line 23) | type matchConditionIntegrationTest struct
method Execute (line 27) | func (h *matchConditionIntegrationTest) Execute() error {
type filterConditionIntegrationTest (line 39) | type filterConditionIntegrationTest struct
method Execute (line 42) | func (h *filterConditionIntegrationTest) Execute() error {
type uniqueFilterIntegrationTest (line 54) | type uniqueFilterIntegrationTest struct
method Execute (line 56) | func (h *uniqueFilterIntegrationTest) Execute() error {
function RunKatanaAndGetResults (line 110) | func RunKatanaAndGetResults(debug bool, extra ...string) ([]string, erro...
FILE: cmd/integration-test/integration-test.go
type TestCase (line 11) | type TestCase interface
function main (line 30) | func main() {
FILE: cmd/integration-test/library.go
type goIntegrationTest (line 17) | type goIntegrationTest struct
method Execute (line 21) | func (h *goIntegrationTest) Execute() error {
FILE: cmd/tools/crawl-maze-score/main.go
function main (line 113) | func main() {
function process (line 121) | func process() error {
function colorizeText (line 167) | func colorizeText(text string, value bool) string {
function strippedLink (line 174) | func strippedLink(link string) string {
function readFoundLinks (line 182) | func readFoundLinks(input string) ([]string, error) {
FILE: internal/runner/banner.go
function showBanner (line 18) | func showBanner() {
function GetUpdateCallback (line 24) | func GetUpdateCallback() func() {
FILE: internal/runner/executer.go
method ExecuteCrawling (line 13) | func (r *Runner) ExecuteCrawling() error {
function addSchemeIfNotExists (line 55) | func addSchemeIfNotExists(inputURL string) string {
FILE: internal/runner/healthcheck.go
function DoHealthCheck (line 16) | func DoHealthCheck(options *types.Options, flagSet *goflags.FlagSet) str...
FILE: internal/runner/options.go
function validateOptions (line 20) | func validateOptions(options *types.Options) error {
function readCustomFormConfig (line 79) | func readCustomFormConfig(formConfig string) error {
method parseInputs (line 99) | func (r *Runner) parseInputs() []string {
function normalizeInput (line 126) | func normalizeInput(value string) string {
function initExampleFormFillConfig (line 130) | func initExampleFormFillConfig() error {
FILE: internal/runner/runner.go
type Runner (line 27) | type Runner struct
method Close (line 152) | func (r *Runner) Close() error {
method SaveState (line 159) | func (r *Runner) SaveState(resumeFilename string) error {
type RunnerState (line 36) | type RunnerState struct
function New (line 41) | func New(options *types.Options) (*Runner, error) {
function expandCIDRInputValue (line 165) | func expandCIDRInputValue(value string) []string {
function expandASNInputValue (line 174) | func expandASNInputValue(value string) []string {
FILE: internal/testutils/helper.go
function CompareOutput (line 3) | func CompareOutput(input, expected []string) bool {
FILE: internal/testutils/integration.go
function RunKatanaBinaryAndGetResults (line 9) | func RunKatanaBinaryAndGetResults(target string, katanaBinary string, de...
FILE: internal/testutils/testutils.go
type TestCase (line 9) | type TestCase struct
FILE: pkg/engine/common/base.go
type Shared (line 31) | type Shared struct
method Enqueue (line 83) | func (s *Shared) Enqueue(queue *queue.Queue, navigationRequests ...*na...
method ValidateScope (line 169) | func (s *Shared) ValidateScope(URL string, root string) bool {
method Output (line 183) | func (s *Shared) Output(navigationRequest *navigation.Request, navigat...
method NewCrawlSessionWithURL (line 225) | func (s *Shared) NewCrawlSessionWithURL(URL string) (*CrawlSession, er...
method Do (line 304) | func (s *Shared) Do(crawlSession *CrawlSession, doRequest DoRequestFun...
function NewShared (line 42) | func NewShared(options *types.CrawlerOptions) (*Shared, error) {
type CrawlSession (line 206) | type CrawlSession struct
type DoRequestFunc (line 294) | type DoRequestFunc
FILE: pkg/engine/common/http.go
type RedirectCallback (line 20) | type RedirectCallback
function BuildHttpClient (line 23) | func BuildHttpClient(dialer *fastdialer.Dialer, options *types.Options, ...
FILE: pkg/engine/engine.go
type Engine (line 3) | type Engine interface
FILE: pkg/engine/headless/browser/browser.go
type Launcher (line 37) | type Launcher struct
method ScopeValidator (line 72) | func (l *Launcher) ScopeValidator() ScopeValidator {
method launchBrowserWithDataDir (line 76) | func (l *Launcher) launchBrowserWithDataDir(userDataDir string) (*rod....
method Close (line 143) | func (l *Launcher) Close() {
method createBrowserPageFunc (line 270) | func (l *Launcher) createBrowserPageFunc() (*BrowserPage, error) {
method GetPageFromPool (line 363) | func (l *Launcher) GetPageFromPool() (*BrowserPage, error) {
method PutBrowserToPool (line 541) | func (l *Launcher) PutBrowserToPool(browser *BrowserPage) {
type LauncherOptions (line 44) | type LauncherOptions struct
type ScopeValidator (line 60) | type ScopeValidator
function NewLauncher (line 63) | func NewLauncher(opts LauncherOptions) (*Launcher, error) {
type BrowserPage (line 152) | type BrowserPage struct
method WaitPageLoadHeurisitics (line 191) | func (b *BrowserPage) WaitPageLoadHeurisitics() error {
method WaitPageLoadHeuristicsFallback (line 234) | func (b *BrowserPage) WaitPageLoadHeuristicsFallback() error {
method WaitNewStable (line 245) | func (p *BrowserPage) WaitNewStable(d time.Duration) error {
method handlePageDialogBoxes (line 382) | func (b *BrowserPage) handlePageDialogBoxes() error {
method CloseBrowserPage (line 583) | func (b *BrowserPage) CloseBrowserPage() {
type WaitOptions (line 164) | type WaitOptions struct
function backoffCountSleeper (line 375) | func backoffCountSleeper(initInterval, maxInterval time.Duration, maxAtt...
function fetchContinueRequest (line 482) | func fetchContinueRequest(page *rod.Page, e *proto.FetchRequestPaused) e...
function fetchGetResponseBody (line 489) | func fetchGetResponseBody(page *rod.Page, e *proto.FetchRequestPaused) (...
function netHTTPRequestFromProto (line 509) | func netHTTPRequestFromProto(e *proto.NetworkRequest) (*http.Request, er...
function netHTTPResponseFromProto (line 524) | func netHTTPResponseFromProto(e *proto.FetchRequestPaused, body []byte) ...
function isBrowserConnected (line 572) | func isBrowserConnected(browser *rod.Browser) bool {
FILE: pkg/engine/headless/browser/cookie/cookie.go
type CookieConsentBlockRequest (line 14) | type CookieConsentBlockRequest struct
type Action (line 20) | type Action struct
type Condition (line 24) | type Condition struct
function init (line 36) | func init() {
function ShouldBlockRequest (line 48) | func ShouldBlockRequest(url string, resourceType proto.NetworkResourceTy...
function matchesRule (line 59) | func matchesRule(rule CookieConsentBlockRequest, url string, resourceTyp...
function getResourceType (line 101) | func getResourceType(resourceType proto.NetworkResourceType) string {
FILE: pkg/engine/headless/browser/cookie/cookie_test.go
function TestShouldBlockRequest (line 9) | func TestShouldBlockRequest(t *testing.T) {
FILE: pkg/engine/headless/browser/element.go
constant buttonsCSSSelector (line 14) | buttonsCSSSelector = "button, input[type='button'], input[type='submit']"
constant linksCSSSelector (line 16) | linksCSSSelector = "a"
function isElementDisabled (line 20) | func isElementDisabled(element *types.HTMLElement) bool {
method FindNavigations (line 59) | func (b *BrowserPage) FindNavigations() ([]*types.Action, error) {
method GetAllElements (line 178) | func (b *BrowserPage) GetAllElements(selector string) ([]*types.HTMLElem...
method GetElementFromXpath (line 191) | func (b *BrowserPage) GetElementFromXpath(xpath string) (*types.HTMLElem...
method GetAllForms (line 204) | func (b *BrowserPage) GetAllForms() ([]*types.HTMLForm, error) {
method GetEventListeners (line 218) | func (b *BrowserPage) GetEventListeners() ([]*types.EventListener, error) {
type NavigatedLink (line 257) | type NavigatedLink struct
method GetNavigatedLinks (line 263) | func (b *BrowserPage) GetNavigatedLinks() ([]*NavigatedLink, error) {
function resolveURL (line 309) | func resolveURL(baseURLStr, href string) (string, error) {
FILE: pkg/engine/headless/browser/stealth/assets.go
constant JS (line 5) | JS = `;(() => {
FILE: pkg/engine/headless/captcha/capsolver/capsolver.go
constant defaultPollInterval (line 16) | defaultPollInterval = 3 * time.Second
constant defaultTimeout (line 17) | defaultTimeout = 120 * time.Second
function SetBaseURL (line 22) | func SetBaseURL(url string) {
type Solver (line 35) | type Solver struct
method Solve (line 53) | func (s *Solver) Solve(ctx context.Context, info *captcha.Info) (*capt...
method createTask (line 101) | func (s *Solver) createTask(ctx context.Context, task map[string]any) ...
method pollResult (line 137) | func (s *Solver) pollResult(ctx context.Context, taskID string, provid...
method getTaskResult (line 164) | func (s *Solver) getTaskResult(ctx context.Context, taskID string) (*g...
function init (line 40) | func init() {
function New (line 46) | func New(apiKey string) *Solver {
type createTaskRequest (line 76) | type createTaskRequest struct
type createTaskResponse (line 81) | type createTaskResponse struct
type getTaskResultRequest (line 88) | type getTaskResultRequest struct
type getTaskResultResponse (line 93) | type getTaskResultResponse struct
function extractToken (line 197) | func extractToken(solution map[string]any, provider captcha.Provider) (*...
FILE: pkg/engine/headless/captcha/capsolver/capsolver_test.go
function TestSolve (line 16) | func TestSolve(t *testing.T) {
function TestCreateTaskError (line 76) | func TestCreateTaskError(t *testing.T) {
function TestTurnstileToken (line 100) | func TestTurnstileToken(t *testing.T) {
function TestExtractToken (line 135) | func TestExtractToken(t *testing.T) {
function TestUnsupportedProvider (line 182) | func TestUnsupportedProvider(t *testing.T) {
FILE: pkg/engine/headless/captcha/captcha.go
type Handler (line 14) | type Handler struct
method HandleIfCaptcha (line 29) | func (h *Handler) HandleIfCaptcha(ctx context.Context, page *rod.Page,...
method solveCaptcha (line 46) | func (h *Handler) solveCaptcha(ctx context.Context, page *rod.Page, in...
function NewHandler (line 18) | func NewHandler(solverProvider, apiKey string) (*Handler, error) {
function injectToken (line 63) | func injectToken(page *rod.Page, solution *Solution) error {
function injectionScript (line 72) | func injectionScript(provider Provider) (string, error) {
FILE: pkg/engine/headless/captcha/helpers_test.go
function setupBrowser (line 13) | func setupBrowser(t *testing.T) *rod.Browser {
function servePage (line 25) | func servePage(t *testing.T, html string) string {
FILE: pkg/engine/headless/captcha/identify.go
type Provider (line 8) | type Provider
constant ProviderRecaptchaV2 (line 11) | ProviderRecaptchaV2 Provider = "recaptchav2"
constant ProviderRecaptchaV3 (line 12) | ProviderRecaptchaV3 Provider = "recaptchav3"
constant ProviderRecaptchaV2Enterprise (line 13) | ProviderRecaptchaV2Enterprise Provider = "recaptchav2enterprise"
constant ProviderRecaptchaV3Enterprise (line 14) | ProviderRecaptchaV3Enterprise Provider = "recaptchav3enterprise"
constant ProviderTurnstile (line 15) | ProviderTurnstile Provider = "turnstile"
constant ProviderHCaptcha (line 16) | ProviderHCaptcha Provider = "hcaptcha"
type Info (line 19) | type Info struct
function Identify (line 26) | func Identify(page *rod.Page) (*Info, error) {
FILE: pkg/engine/headless/captcha/identify_test.go
function TestDetectCaptchaWithDit (line 11) | func TestDetectCaptchaWithDit(t *testing.T) {
function TestIdentify (line 50) | func TestIdentify(t *testing.T) {
FILE: pkg/engine/headless/captcha/inject_test.go
function TestInjectionScript (line 10) | func TestInjectionScript(t *testing.T) {
FILE: pkg/engine/headless/captcha/injection_test.go
function TestE2E_RecaptchaV2_Injection (line 12) | func TestE2E_RecaptchaV2_Injection(t *testing.T) {
function TestE2E_Turnstile_Injection (line 54) | func TestE2E_Turnstile_Injection(t *testing.T) {
function TestE2E_HCaptcha_Injection (line 96) | func TestE2E_HCaptcha_Injection(t *testing.T) {
function TestE2E_RecaptchaV2_FormSubmitFallback (line 143) | func TestE2E_RecaptchaV2_FormSubmitFallback(t *testing.T) {
FILE: pkg/engine/headless/captcha/integration_test.go
function TestIntegration_DetectRealCaptchaPages (line 10) | func TestIntegration_DetectRealCaptchaPages(t *testing.T) {
FILE: pkg/engine/headless/captcha/solver.go
type Solution (line 8) | type Solution struct
type Solver (line 13) | type Solver interface
function NewSolver (line 17) | func NewSolver(provider, apiKey string) (Solver, error) {
type SolverConstructor (line 25) | type SolverConstructor
function RegisterSolver (line 29) | func RegisterSolver(name string, constructor SolverConstructor) {
FILE: pkg/engine/headless/captcha/solver_test.go
function TestNewSolver_UnsupportedProvider (line 10) | func TestNewSolver_UnsupportedProvider(t *testing.T) {
function TestRegisterSolver (line 16) | func TestRegisterSolver(t *testing.T) {
FILE: pkg/engine/headless/crawler/crawler.go
type Crawler (line 30) | type Crawler struct
method Close (line 138) | func (c *Crawler) Close() {
method GetCrawlGraph (line 147) | func (c *Crawler) GetCrawlGraph() *graph.CrawlGraph {
method Crawl (line 151) | func (c *Crawler) Crawl(URL string) error {
method crawlFn (line 301) | func (c *Crawler) crawlFn(ctx context.Context, action *types.Action, p...
method executeCrawlStateAction (line 448) | func (c *Crawler) executeCrawlStateAction(action *types.Action, page *...
type Options (line 41) | type Options struct
function init (line 73) | func init() {
function New (line 83) | func New(opts Options) (*Crawler, error) {
function isLogoutPage (line 511) | func isLogoutPage(element *types.HTMLElement) bool {
FILE: pkg/engine/headless/crawler/diagnostics/diagnostics.go
type Writer (line 16) | type Writer interface
type PageStateType (line 24) | type PageStateType
type diskWriter (line 31) | type diskWriter struct
method Close (line 68) | func (w *diskWriter) Close() error {
method LogAction (line 95) | func (w *diskWriter) LogAction(action *types.Action) error {
method LogPageState (line 103) | func (w *diskWriter) LogPageState(state *types.PageState, stateType Pa...
method LogNavigations (line 140) | func (w *diskWriter) LogNavigations(pageStateID string, navigations []...
method LogPageStateScreenshot (line 184) | func (w *diskWriter) LogPageStateScreenshot(pageStateID string, screen...
type stateMetadata (line 38) | type stateMetadata struct
type navigationEntry (line 46) | type navigationEntry struct
function NewWriter (line 55) | func NewWriter(directory string) (Writer, error) {
FILE: pkg/engine/headless/crawler/formfill.go
function deriveName (line 15) | func deriveName(e *types.HTMLElement) string {
function copyAttrs (line 22) | func copyAttrs(src map[string]string, skipKeys ...string) mapsutil.Order...
function convertHTMLElementToFormInput (line 36) | func convertHTMLElementToFormInput(element *types.HTMLElement) utilsform...
function convertHTMLElementToFormTextArea (line 45) | func convertHTMLElementToFormTextArea(element *types.HTMLElement) utilsf...
function convertHTMLElementToFormSelect (line 52) | func convertHTMLElementToFormSelect(element *types.HTMLElement) utilsfor...
method processForm (line 60) | func (c *Crawler) processForm(page *browser.BrowserPage, form *types.HTM...
method getFieldName (line 136) | func (c *Crawler) getFieldName(field *types.HTMLElement) string {
method buildFormSelectWithOptions (line 140) | func (c *Crawler) buildFormSelectWithOptions(page *browser.BrowserPage, ...
method applyFormSuggestions (line 172) | func (c *Crawler) applyFormSuggestions(suggestions mapsutil.OrderedMap[s...
FILE: pkg/engine/headless/crawler/normalizer/dom_utils.go
type DOMNormalizer (line 31) | type DOMNormalizer struct
method Apply (line 66) | func (d *DOMNormalizer) Apply(content string) (string, error) {
function NewDOMNormalizer (line 38) | func NewDOMNormalizer() *DOMNormalizer {
type domTransformationFunc (line 87) | type domTransformationFunc
type selectionTransformationFunc (line 89) | type selectionTransformationFunc
function removeComments (line 96) | func removeComments(n *html.Node) {
function removeCommentsDomTransformationFunc (line 106) | func removeCommentsDomTransformationFunc(s *goquery.Selection) {
function removeClassIDDataAttributesDomTransformationFunc (line 132) | func removeClassIDDataAttributesDomTransformationFunc(s *goquery.Selecti...
function removeAttributes (line 140) | func removeAttributes(s *goquery.Selection) {
FILE: pkg/engine/headless/crawler/normalizer/dom_utils_test.go
function TestDOMNormalizer_Apply (line 7) | func TestDOMNormalizer_Apply(t *testing.T) {
FILE: pkg/engine/headless/crawler/normalizer/normalizer.go
type Normalizer (line 18) | type Normalizer struct
method Apply (line 42) | func (n *Normalizer) Apply(text string) (string, error) {
function New (line 24) | func New() (*Normalizer, error) {
function normalizeDocument (line 67) | func normalizeDocument(text string) string {
function replaceHexEscapeSequence (line 88) | func replaceHexEscapeSequence(match string) string {
function convertHexEscapeSequencesToEntities (line 104) | func convertHexEscapeSequencesToEntities(input string) string {
function stripTextContent (line 110) | func stripTextContent(content string) (string, error) {
function removeTextNodesFromSelection (line 127) | func removeTextNodesFromSelection(s *goquery.Selection) {
FILE: pkg/engine/headless/crawler/normalizer/simhash/simhash.go
constant maxTokens (line 23) | maxTokens = 5000
constant featuresBufSize (line 24) | featuresBufSize = 1000
function fingerprintOptimized (line 34) | func fingerprintOptimized(r io.Reader, shingle int) uint64 {
function extractFeatures (line 92) | func extractFeatures(t *html.Token, features *[]string) {
function Fingerprint (line 130) | func Fingerprint(r io.Reader, shingle int) uint64 {
type Oracle (line 134) | type Oracle struct
method See (line 154) | func (n *Oracle) See(f uint64) *Oracle {
method Seen (line 172) | func (n *Oracle) Seen(f uint64, r uint8) bool {
function NewOracle (line 140) | func NewOracle() *Oracle {
function newNode (line 144) | func newNode(f uint64) *Oracle {
function Distance (line 149) | func Distance(a, b uint64) uint8 {
FILE: pkg/engine/headless/crawler/normalizer/simhash/simhash_test.go
function TestFingerprintDeterministic (line 17) | func TestFingerprintDeterministic(t *testing.T) {
function TestFingerprintSimilarity (line 31) | func TestFingerprintSimilarity(t *testing.T) {
function TestOracle (line 47) | func TestOracle(t *testing.T) {
FILE: pkg/engine/headless/crawler/normalizer/text_utils.go
type TextNormalizer (line 30) | type TextNormalizer struct
method Apply (line 56) | func (n *TextNormalizer) Apply(text string) string {
function NewTextNormalizer (line 39) | func NewTextNormalizer() (*TextNormalizer, error) {
FILE: pkg/engine/headless/crawler/normalizer/text_utils_test.go
function TestTextNormalizer_AllPatterns (line 8) | func TestTextNormalizer_AllPatterns(t *testing.T) {
FILE: pkg/engine/headless/crawler/state.go
constant simhashThreshold (line 20) | simhashThreshold = 2
method isCorrectNavigation (line 22) | func (c *Crawler) isCorrectNavigation(page *browser.BrowserPage, action ...
function getPageHash (line 54) | func getPageHash(page *browser.BrowserPage) (string, *types.PageState, e...
function newPageState (line 67) | func newPageState(page *browser.BrowserPage, action *types.Action) (*typ...
function sha256Hash (line 103) | func sha256Hash(item string) string {
function getStrippedDOM (line 110) | func getStrippedDOM(contents string) (string, error) {
method navigateBackToStateOrigin (line 132) | func (c *Crawler) navigateBackToStateOrigin(action *types.Action, page *...
method tryElementNavigation (line 176) | func (c *Crawler) tryElementNavigation(page *browser.BrowserPage, action...
function isElementMatch (line 213) | func isElementMatch(current, target *types.HTMLElement) bool {
method tryBrowserHistoryNavigation (line 237) | func (c *Crawler) tryBrowserHistoryNavigation(page *browser.BrowserPage,...
method isBackNavigationPossible (line 275) | func (c *Crawler) isBackNavigationPossible(page *browser.BrowserPage, or...
method tryShortestPathNavigation (line 294) | func (c *Crawler) tryShortestPathNavigation(action *types.Action, page *...
FILE: pkg/engine/headless/crawler/state_test.go
function TestPageFingerprint_Stability (line 12) | func TestPageFingerprint_Stability(t *testing.T) {
function TestPageFingerprint (line 16) | func TestPageFingerprint(t *testing.T) {
function TestSimHashSimilarity (line 107) | func TestSimHashSimilarity(t *testing.T) {
FILE: pkg/engine/headless/debugger.go
type ActiveURL (line 12) | type ActiveURL struct
type CrawlDebugger (line 20) | type CrawlDebugger struct
method StartURL (line 55) | func (cd *CrawlDebugger) StartURL(url string, depth int) {
method EndURL (line 70) | func (cd *CrawlDebugger) EndURL(url string) {
method GetActiveURLs (line 81) | func (cd *CrawlDebugger) GetActiveURLs() []ActiveURL {
method handleActiveURLs (line 100) | func (cd *CrawlDebugger) handleActiveURLs(w http.ResponseWriter, r *ht...
method handleHealth (line 112) | func (cd *CrawlDebugger) handleHealth(w http.ResponseWriter, r *http.R...
method Close (line 122) | func (cd *CrawlDebugger) Close() {
function NewCrawlDebugger (line 27) | func NewCrawlDebugger(httpPort int) *CrawlDebugger {
FILE: pkg/engine/headless/graph/graph.go
type CrawlGraph (line 15) | type CrawlGraph struct
method GetVertices (line 34) | func (g *CrawlGraph) GetVertices() []string {
method AddPageState (line 47) | func (g *CrawlGraph) AddPageState(n types.PageState) error {
method AddEdge (line 85) | func (g *CrawlGraph) AddEdge(sourceState, targetState string, action *...
method GetPageState (line 105) | func (g *CrawlGraph) GetPageState(id string) (*types.PageState, error) {
method ShortestPath (line 113) | func (g *CrawlGraph) ShortestPath(sourceState, targetState string) ([]...
method DrawGraph (line 133) | func (g *CrawlGraph) DrawGraph(file string) error {
function navigationHasherFunc (line 19) | func navigationHasherFunc(n types.PageState) string {
function NewCrawlGraph (line 24) | func NewCrawlGraph() *CrawlGraph {
FILE: pkg/engine/headless/headless.go
type Headless (line 22) | type Headless struct
method Crawl (line 99) | func (h *Headless) Crawl(URL string) error {
method Close (line 187) | func (h *Headless) Close() error {
method performAdditionalAnalysis (line 194) | func (h *Headless) performAdditionalAnalysis(rr *output.Result) []*out...
function New (line 33) | func New(options *types.CrawlerOptions) (*Headless, error) {
function newLogger (line 54) | func newLogger(options *types.CrawlerOptions) *slog.Logger {
function validateScopeFunc (line 75) | func validateScopeFunc(h *Headless, URL string) browser.ScopeValidator {
FILE: pkg/engine/headless/js/js.go
function InitJavascriptEnv (line 19) | func InitJavascriptEnv(page *rod.Page) error {
FILE: pkg/engine/headless/js/page-init.js
function hookNavigatedLinkSinks (line 24) | function hookNavigatedLinkSinks() {
function hookMiscellaneousUtilities (line 86) | function hookMiscellaneousUtilities() {
function hookAddEventListener (line 128) | function hookAddEventListener() {
FILE: pkg/engine/headless/js/utils.js
function getEventListeners (line 73) | function getEventListeners(element) {
function areNodesSimilar (line 322) | function areNodesSimilar(left, right) {
FILE: pkg/engine/headless/types/types.go
type PageState (line 19) | type PageState struct
type Action (line 35) | type Action struct
method Hash (line 45) | func (a *Action) Hash() string {
method String (line 55) | func (a *Action) String() string {
type ActionType (line 68) | type ActionType
constant ActionTypeUnknown (line 71) | ActionTypeUnknown ActionType = "unknown"
constant ActionTypeLoadURL (line 72) | ActionTypeLoadURL ActionType = "load_url"
constant ActionTypeExecuteJS (line 73) | ActionTypeExecuteJS ActionType = "execute_js"
constant ActionTypeLeftClick (line 74) | ActionTypeLeftClick ActionType = "left_click"
constant ActionTypeLeftClickDown (line 75) | ActionTypeLeftClickDown ActionType = "left_click_down"
constant ActionTypeLeftClickUp (line 76) | ActionTypeLeftClickUp ActionType = "left_click_up"
constant ActionTypeRightClick (line 77) | ActionTypeRightClick ActionType = "right_click"
constant ActionTypeDoubleClick (line 78) | ActionTypeDoubleClick ActionType = "double_click"
constant ActionTypeScroll (line 79) | ActionTypeScroll ActionType = "scroll"
constant ActionTypeSendKeys (line 80) | ActionTypeSendKeys ActionType = "send_keys"
constant ActionTypeKeyUp (line 81) | ActionTypeKeyUp ActionType = "key_up"
constant ActionTypeKeyDown (line 82) | ActionTypeKeyDown ActionType = "key_down"
constant ActionTypeHover (line 83) | ActionTypeHover ActionType = "hover"
constant ActionTypeFocus (line 84) | ActionTypeFocus ActionType = "focus"
constant ActionTypeBlur (line 85) | ActionTypeBlur ActionType = "blur"
constant ActionTypeMouseOverAndOut (line 86) | ActionTypeMouseOverAndOut ActionType = "mouse_over_and_out"
constant ActionTypeMouseWheel (line 87) | ActionTypeMouseWheel ActionType = "mouse_wheel"
constant ActionTypeFillForm (line 88) | ActionTypeFillForm ActionType = "fill_form"
constant ActionTypeWait (line 89) | ActionTypeWait ActionType = "wait"
constant ActionTypeRedirect (line 90) | ActionTypeRedirect ActionType = "redirect"
constant ActionTypeSubRequest (line 91) | ActionTypeSubRequest ActionType = "sub_request"
function ActionFromEventListener (line 94) | func ActionFromEventListener(listener *EventListener) *Action {
type HTMLElement (line 133) | type HTMLElement struct
method String (line 148) | func (e *HTMLElement) String() string {
method Hash (line 165) | func (e *HTMLElement) Hash() string {
type HTMLForm (line 194) | type HTMLForm struct
method Hash (line 208) | func (f *HTMLForm) Hash() string {
function getStableAttributes (line 242) | func getStableAttributes(attrs map[string]string) []string {
type EventListener (line 267) | type EventListener struct
type NavigationType (line 274) | type NavigationType
constant NavigationTypeForm (line 278) | NavigationTypeForm NavigationType = "form"
constant NavigationTypeButton (line 280) | NavigationTypeButton NavigationType = "button"
constant NavigationTypeLink (line 282) | NavigationTypeLink NavigationType = "link"
constant NavigationTypeEventListener (line 284) | NavigationTypeEventListener NavigationType = "eventlistener"
FILE: pkg/engine/hybrid/crawl.go
method navigateRequest (line 29) | func (c *Crawler) navigateRequest(s *common.CrawlSession, request *navig...
method addHeadersToPage (line 356) | func (c *Crawler) addHeadersToPage(page *rod.Page) {
function traverseDOMNode (line 391) | func traverseDOMNode(node *proto.DOMNode, builder *strings.Builder) {
constant elementNode (line 411) | elementNode = 1
function buildDOMFromNode (line 418) | func buildDOMFromNode(node *proto.DOMNode, builder *strings.Builder) {
FILE: pkg/engine/hybrid/hijack.go
function NewHijack (line 11) | func NewHijack(page *rod.Page) *Hijack {
type Hijack (line 22) | type Hijack struct
method SetPattern (line 30) | func (h *Hijack) SetPattern(pattern *proto.FetchRequestPattern) {
method Start (line 37) | func (h *Hijack) Start(handler HijackHandler) func() error {
method Stop (line 63) | func (h *Hijack) Stop() error {
function FetchGetResponseBody (line 71) | func FetchGetResponseBody(page *rod.Page, e *proto.FetchRequestPaused) (...
function FetchContinueRequest (line 92) | func FetchContinueRequest(page *rod.Page, e *proto.FetchRequestPaused) e...
FILE: pkg/engine/hybrid/hybrid.go
type Crawler (line 22) | type Crawler struct
method Close (line 102) | func (c *Crawler) Close() error {
method Crawl (line 113) | func (c *Crawler) Crawl(rootURL string) error {
method Do (line 134) | func (c *Crawler) Do(crawlSession *common.CrawlSession, doRequest comm...
function New (line 36) | func New(options *types.CrawlerOptions) (*Crawler, error) {
function buildChromeLauncher (line 205) | func buildChromeLauncher(options *types.CrawlerOptions, dataStore string...
FILE: pkg/engine/parser/files/request.go
type visitFunc (line 8) | type visitFunc
type KnownFiles (line 10) | type KnownFiles struct
method Request (line 37) | func (k *KnownFiles) Request(URL string) (navigationRequests []*naviga...
function New (line 16) | func New(httpclient *retryablehttp.Client, files string) *KnownFiles {
FILE: pkg/engine/parser/files/robotstxt.go
type robotsTxtCrawler (line 17) | type robotsTxtCrawler struct
method Visit (line 22) | func (r *robotsTxtCrawler) Visit(URL string) ([]*navigation.Request, e...
method parseReader (line 44) | func (r *robotsTxtCrawler) parseReader(reader io.Reader, resp *http.Re...
FILE: pkg/engine/parser/files/robotstxt_test.go
function TestRobotsTxtParseReader (line 13) | func TestRobotsTxtParseReader(t *testing.T) {
FILE: pkg/engine/parser/files/sitemapxml.go
type sitemapXmlCrawler (line 17) | type sitemapXmlCrawler struct
method Visit (line 22) | func (r *sitemapXmlCrawler) Visit(URL string) (navigationRequests []*n...
method parseReader (line 57) | func (r *sitemapXmlCrawler) parseReader(reader io.Reader, resp *http.R...
type sitemapStruct (line 48) | type sitemapStruct struct
type parsedURL (line 53) | type parsedURL struct
FILE: pkg/engine/parser/files/sitemapxml_test.go
function TestSitemapXmlParseReader (line 12) | func TestSitemapXmlParseReader(t *testing.T) {
FILE: pkg/engine/parser/parser.go
type ResponseParserFunc (line 20) | type ResponseParserFunc
type Parser (line 22) | type Parser
method ParseResponse (line 78) | func (p *Parser) ParseResponse(resp *navigation.Response) (navigationR...
type responseParserType (line 24) | type responseParserType
constant headerParser (line 27) | headerParser responseParserType = iota + 1
constant bodyParser (line 28) | bodyParser
constant contentParser (line 29) | contentParser
type responseParser (line 32) | type responseParser struct
function NewResponseParser (line 37) | func NewResponseParser() *Parser {
function appendFiltered (line 93) | func appendFiltered(existing []*navigation.Request, new []*navigation.Re...
function isValidNavigationRequest (line 102) | func isValidNavigationRequest(req *navigation.Request) bool {
function headerContentLocationParser (line 122) | func headerContentLocationParser(resp *navigation.Response) (navigationR...
function headerLinkParser (line 132) | func headerLinkParser(resp *navigation.Response) (navigationRequests []*...
function headerLocationParser (line 145) | func headerLocationParser(resp *navigation.Response) (navigationRequests...
function headerRefreshParser (line 155) | func headerRefreshParser(resp *navigation.Response) (navigationRequests ...
function bodyATagParser (line 173) | func bodyATagParser(resp *navigation.Response) (navigationRequests []*na...
function bodyLinkHrefTagParser (line 188) | func bodyLinkHrefTagParser(resp *navigation.Response) (navigationRequest...
function bodyEmbedTagParser (line 199) | func bodyEmbedTagParser(resp *navigation.Response) (navigationRequests [...
function bodyFrameTagParser (line 210) | func bodyFrameTagParser(resp *navigation.Response) (navigationRequests [...
function bodyIframeTagParser (line 221) | func bodyIframeTagParser(resp *navigation.Response) (navigationRequests ...
function bodyInputSrcTagParser (line 239) | func bodyInputSrcTagParser(resp *navigation.Response) (navigationRequest...
function bodyIsindexActionTagParser (line 250) | func bodyIsindexActionTagParser(resp *navigation.Response) (navigationRe...
function bodyScriptSrcTagParser (line 261) | func bodyScriptSrcTagParser(resp *navigation.Response) (navigationReques...
function bodyBackgroundTagParser (line 272) | func bodyBackgroundTagParser(resp *navigation.Response) (navigationReque...
function bodyAudioTagParser (line 283) | func bodyAudioTagParser(resp *navigation.Response) (navigationRequests [...
function bodyAppletTagParser (line 306) | func bodyAppletTagParser(resp *navigation.Response) (navigationRequests ...
function bodyImgTagParser (line 321) | func bodyImgTagParser(resp *navigation.Response) (navigationRequests []*...
function bodyObjectTagParser (line 354) | func bodyObjectTagParser(resp *navigation.Response) (navigationRequests ...
function bodySvgTagParser (line 375) | func bodySvgTagParser(resp *navigation.Response) (navigationRequests []*...
function bodyTableTagParser (line 394) | func bodyTableTagParser(resp *navigation.Response) (navigationRequests [...
function bodyVideoTagParser (line 411) | func bodyVideoTagParser(resp *navigation.Response) (navigationRequests [...
function bodyBlockquoteCiteTagParser (line 432) | func bodyBlockquoteCiteTagParser(resp *navigation.Response) (navigationR...
function bodyFrameSrcTagParser (line 443) | func bodyFrameSrcTagParser(resp *navigation.Response) (navigationRequest...
function bodyMapAreaPingTagParser (line 454) | func bodyMapAreaPingTagParser(resp *navigation.Response) (navigationRequ...
function bodyBaseHrefTagParser (line 465) | func bodyBaseHrefTagParser(resp *navigation.Response) (navigationRequest...
function bodyImportImplementationTagParser (line 476) | func bodyImportImplementationTagParser(resp *navigation.Response) (navig...
function bodyButtonFormactionTagParser (line 487) | func bodyButtonFormactionTagParser(resp *navigation.Response) (navigatio...
function bodyHtmlManifestTagParser (line 498) | func bodyHtmlManifestTagParser(resp *navigation.Response) (navigationReq...
function bodyHtmlDoctypeTagParser (line 509) | func bodyHtmlDoctypeTagParser(resp *navigation.Response) (navigationRequ...
function bodyFormTagParser (line 525) | func bodyFormTagParser(resp *navigation.Response) (navigationRequests []...
function bodyMetaContentTagParser (line 620) | func bodyMetaContentTagParser(resp *navigation.Response) (navigationRequ...
function bodyHtmxAttrParser (line 634) | func bodyHtmxAttrParser(resp *navigation.Response) (navigationRequests [...
function scriptContentRegexParser (line 677) | func scriptContentRegexParser(resp *navigation.Response) (navigationRequ...
function scriptJSFileRegexParser (line 693) | func scriptJSFileRegexParser(resp *navigation.Response) (navigationReque...
function bodyScrapeEndpointsParser (line 709) | func bodyScrapeEndpointsParser(resp *navigation.Response) (navigationReq...
function customFieldRegexParser (line 718) | func customFieldRegexParser(resp *navigation.Response) (navigationReques...
FILE: pkg/engine/parser/parser_generic.go
type Options (line 15) | type Options struct
method InitWithOptions (line 22) | func (p *Parser) InitWithOptions(options *Options) {
function scriptContentJsluiceParser (line 41) | func scriptContentJsluiceParser(resp *navigation.Response) (navigationRe...
function scriptJSFileJsluiceParser (line 57) | func scriptJSFileJsluiceParser(resp *navigation.Response) (navigationReq...
FILE: pkg/engine/parser/parser_nojs.go
type Options (line 5) | type Options struct
method InitWithOptions (line 12) | func (p *Parser) InitWithOptions(options *Options) {
FILE: pkg/engine/parser/parser_test.go
function TestHeaderParsers (line 16) | func TestHeaderParsers(t *testing.T) {
function TestBodyParsers (line 41) | func TestBodyParsers(t *testing.T) {
function TestScriptParsers (line 393) | func TestScriptParsers(t *testing.T) {
function TestRegexBodyParsers (line 417) | func TestRegexBodyParsers(t *testing.T) {
function TestHtmxBodyParser (line 493) | func TestHtmxBodyParser(t *testing.T) {
function TestDataURIFiltering (line 527) | func TestDataURIFiltering(t *testing.T) {
FILE: pkg/engine/standard/crawl.go
method makeRequest (line 22) | func (c *Crawler) makeRequest(s *common.CrawlSession, request *navigatio...
FILE: pkg/engine/standard/standard.go
type Crawler (line 11) | type Crawler struct
method Close (line 25) | func (c *Crawler) Close() error {
method Crawl (line 30) | func (c *Crawler) Crawl(rootURL string) error {
function New (line 16) | func New(options *types.CrawlerOptions) (*Crawler, error) {
FILE: pkg/navigation/request.go
type Depth (line 9) | type Depth struct
type Request (line 12) | type Request struct
method RequestURL (line 28) | func (n *Request) RequestURL() string {
function NewNavigationRequestURLFromResponse (line 44) | func NewNavigationRequestURLFromResponse(path, source, tag, attribute st...
FILE: pkg/navigation/response.go
type Headers (line 11) | type Headers
method MarshalJSON (line 20) | func (h *Headers) MarshalJSON() ([]byte, error) {
type Form (line 13) | type Form struct
type Response (line 30) | type Response struct
method AbsoluteURL (line 47) | func (n Response) AbsoluteURL(path string) string {
method IsRedirect (line 64) | func (n Response) IsRedirect() bool {
FILE: pkg/output/custom_field.go
type Part (line 19) | type Part
method ToString (line 152) | func (p Part) ToString() string {
constant Header (line 23) | Header Part = "header"
constant Body (line 24) | Body Part = "body"
constant Response (line 25) | Response Part = "response"
type CustomFieldConfig (line 29) | type CustomFieldConfig struct
method SetCompiledRegexp (line 47) | func (c *CustomFieldConfig) SetCompiledRegexp(r *regexp.Regexp) {
method GetName (line 51) | func (c *CustomFieldConfig) GetName() string {
function parseCustomFieldName (line 55) | func parseCustomFieldName(filePath string) error {
function loadCustomFields (line 88) | func loadCustomFields(filePath string, fields string) error {
function initCustomFieldConfigFile (line 122) | func initCustomFieldConfigFile() (string, error) {
FILE: pkg/output/error.go
type Error (line 5) | type Error struct
FILE: pkg/output/fields.go
type fieldOutput (line 35) | type fieldOutput struct
function validateFieldNames (line 41) | func validateFieldNames(names string) error {
function storeFields (line 63) | func storeFields(output *Result, storeFields []string) {
function appendToFileField (line 86) | func appendToFileField(parsed *url.URL, field, data string) {
function formatField (line 102) | func formatField(output *Result, fields string) []fieldOutput {
function getValueForField (line 203) | func getValueForField(output *Result, parsed *url.URL, hostname, rdn, ru...
function getValueForCustomField (line 265) | func getValueForCustomField(output *Result) []fieldOutput {
FILE: pkg/output/fields_test.go
function TestValidateFieldNames (line 10) | func TestValidateFieldNames(t *testing.T) {
function TestFormatField (line 21) | func TestFormatField(t *testing.T) {
FILE: pkg/output/file_writer.go
type fileWriter (line 9) | type fileWriter struct
method Write (line 24) | func (w *fileWriter) Write(data []byte) error {
method Close (line 34) | func (w *fileWriter) Close() error {
function newFileOutputWriter (line 15) | func newFileOutputWriter(file string) (*fileWriter, error) {
FILE: pkg/output/format_json.go
method formatJSON (line 9) | func (w *StandardWriter) formatJSON(output *Result) ([]byte, error) {
FILE: pkg/output/format_screen.go
method formatScreen (line 10) | func (w *StandardWriter) formatScreen(output *Result) ([]byte, error) {
FILE: pkg/output/format_template.go
method formatTemplate (line 12) | func (w *StandardWriter) formatTemplate(output *Result) ([]byte, error) {
FILE: pkg/output/options.go
type Options (line 10) | type Options struct
FILE: pkg/output/output.go
constant indexFile (line 27) | indexFile = "index.txt"
constant DefaultResponseDir (line 28) | DefaultResponseDir = "katana_response"
type Writer (line 37) | type Writer interface
type StandardWriter (line 46) | type StandardWriter struct
method Write (line 167) | func (w *StandardWriter) Write(result *Result) error {
method WriteErr (line 265) | func (w *StandardWriter) WriteErr(errMessage *Error) error {
method Close (line 285) | func (w *StandardWriter) Close() error {
method matchOutput (line 349) | func (w *StandardWriter) matchOutput(event *Result) bool {
method filterOutput (line 368) | func (w *StandardWriter) filterOutput(event *Result) bool {
function New (line 71) | func New(options Options) (Writer, error) {
function createDirNameNoClobber (line 301) | func createDirNameNoClobber(dir string) string {
function removeDirsWithSuffix (line 331) | func removeDirsWithSuffix(dir string) {
function evalDslExpr (line 386) | func evalDslExpr(result *Result, dslExpr string) bool {
function resultToMap (line 401) | func resultToMap(result Result) (map[string]interface{}, error) {
function flatten (line 442) | func flatten(m map[string]any) map[string]any {
function ignoreErr (line 465) | func ignoreErr(err error) bool {
FILE: pkg/output/responses.go
function getResponseHash (line 16) | func getResponseHash(URL string) string {
method formatResult (line 21) | func (w *StandardWriter) formatResult(result *Result) ([]byte, error) {
function getResponseHost (line 36) | func getResponseHost(URL string) (string, error) {
function createHostDir (line 44) | func createHostDir(storeResponseFolder, domain string) string {
function getResponseFile (line 49) | func getResponseFile(storeResponseFolder, URL string) (string, *fileWrit...
function getResponseFileName (line 63) | func getResponseFileName(storeResponseFolder, domain, URL string) string {
function updateIndex (line 69) | func updateIndex(storeResponseFolder string, result *Result) error {
FILE: pkg/output/result.go
type Result (line 10) | type Result struct
method HasResponse (line 18) | func (r *Result) HasResponse() bool {
FILE: pkg/types/crawler_options.go
type CrawlerOptions (line 24) | type CrawlerOptions struct
method Close (line 174) | func (c *CrawlerOptions) Close() error {
method ValidatePath (line 179) | func (c *CrawlerOptions) ValidatePath(path string) bool {
method ClassifyPage (line 187) | func (c *CrawlerOptions) ClassifyPage(body string) map[string]any {
method ValidateScope (line 205) | func (c *CrawlerOptions) ValidateScope(absURL, rootHostname string) (b...
function NewCrawlerOptions (line 54) | func NewCrawlerOptions(options *Options) (*CrawlerOptions, error) {
FILE: pkg/types/default.go
function init (line 7) | func init() {
FILE: pkg/types/options.go
type OnResultCallback (line 17) | type OnResultCallback
type OnSkipURLCallback (line 20) | type OnSkipURLCallback
type Options (line 22) | type Options struct
method ParseCustomHeaders (line 203) | func (options *Options) ParseCustomHeaders() map[string]string {
method ParseHeadlessOptionalArguments (line 213) | func (options *Options) ParseHeadlessOptionalArguments() map[string]st...
method ShouldResume (line 238) | func (options *Options) ShouldResume() bool {
method ConfigureOutput (line 243) | func (options *Options) ConfigureOutput() {
FILE: pkg/types/options_test.go
function TestParseCustomHeaders (line 11) | func TestParseCustomHeaders(t *testing.T) {
function TestParseHeadlessOptionalArguments (line 52) | func TestParseHeadlessOptionalArguments(t *testing.T) {
FILE: pkg/utils/extensions/extensions.go
type Validator (line 15) | type Validator struct
method ValidatePath (line 42) | func (e *Validator) ValidatePath(item string) bool {
function NewValidator (line 21) | func NewValidator(extensionsMatch, extensionsFilter []string, noDefaultE...
function normalizeExtension (line 70) | func normalizeExtension(extension string) string {
FILE: pkg/utils/extensions/extensions_test.go
function TestValidatorValidate (line 9) | func TestValidatorValidate(t *testing.T) {
FILE: pkg/utils/filters/filters.go
type Filter (line 4) | type Filter interface
FILE: pkg/utils/filters/filters_test.go
function TestSimpleFilter (line 9) | func TestSimpleFilter(t *testing.T) {
FILE: pkg/utils/filters/simple.go
constant MaxChromeURLLength (line 13) | MaxChromeURLLength = 2097152
constant MinSequenceLength (line 15) | MinSequenceLength = 10
constant MaxSequenceCount (line 16) | MaxSequenceCount = 10
type Simple (line 23) | type Simple struct
method UniqueURL (line 37) | func (s *Simple) UniqueURL(url string) bool {
method UniqueContent (line 47) | func (s *Simple) UniqueContent(data []byte) bool {
method Close (line 60) | func (s *Simple) Close() {
method IsCycle (line 65) | func (s *Simple) IsCycle(url string) bool {
function NewSimple (line 28) | func NewSimple() (*Simple, error) {
FILE: pkg/utils/formfields.go
function ParseFormFields (line 14) | func ParseFormFields(document *goquery.Document) []navigation.Form {
FILE: pkg/utils/formfields_test.go
function TestParseFormFields (line 32) | func TestParseFormFields(t *testing.T) {
FILE: pkg/utils/formfill.go
function init (line 15) | func init() {
type FormFillData (line 20) | type FormFillData struct
type FormInput (line 37) | type FormInput struct
type SelectOption (line 45) | type SelectOption struct
type FormSelect (line 52) | type FormSelect struct
type FormTextArea (line 58) | type FormTextArea struct
function FormInputFillSuggestions (line 65) | func FormInputFillSuggestions(inputs []FormInput) mapsutil.OrderedMap[st...
function FormSelectFill (line 136) | func FormSelectFill(inputs []FormSelect) mapsutil.OrderedMap[string, str...
function FormTextAreaFill (line 157) | func FormTextAreaFill(inputs []FormTextArea) mapsutil.OrderedMap[string,...
function FormFillSuggestions (line 176) | func FormFillSuggestions(formFields []interface{}) mapsutil.OrderedMap[s...
function ConvertGoquerySelectionToFormInput (line 196) | func ConvertGoquerySelectionToFormInput(item *goquery.Selection) FormInp...
function ConvertGoquerySelectionToSelectOption (line 217) | func ConvertGoquerySelectionToSelectOption(item *goquery.Selection) Sele...
function ConvertGoquerySelectionToFormSelect (line 237) | func ConvertGoquerySelectionToFormSelect(item *goquery.Selection) FormSe...
function ConvertGoquerySelectionToFormTextArea (line 259) | func ConvertGoquerySelectionToFormTextArea(item *goquery.Selection) Form...
function ConvertGoquerySelectionToFormField (line 279) | func ConvertGoquerySelectionToFormField(item *goquery.Selection) interfa...
FILE: pkg/utils/formfill_test.go
function TestFormInputFillSuggestions (line 61) | func TestFormInputFillSuggestions(t *testing.T) {
FILE: pkg/utils/jsluice.go
function IsPathCommonJSLibraryFile (line 18) | func IsPathCommonJSLibraryFile(path string) bool {
type JSLuiceEndpoint (line 22) | type JSLuiceEndpoint struct
function ExtractJsluiceEndpoints (line 35) | func ExtractJsluiceEndpoints(data string) []JSLuiceEndpoint {
FILE: pkg/utils/jsluice_test.go
function TestIsPathCommonJSLibraryFile (line 7) | func TestIsPathCommonJSLibraryFile(t *testing.T) {
FILE: pkg/utils/maps.go
function MergeDataMaps (line 5) | func MergeDataMaps(dataMap1 *mapsutil.OrderedMap[string, string], dataMa...
FILE: pkg/utils/maps_test.go
function TestMergeDataMap (line 10) | func TestMergeDataMap(t *testing.T) {
FILE: pkg/utils/pathtrie.go
constant DefaultPromotionThreshold (line 11) | DefaultPromotionThreshold = 10
constant DefaultMaxHosts (line 15) | DefaultMaxHosts = 10000
type PathTrie (line 24) | type PathTrie struct
method Fingerprint (line 52) | func (t *PathTrie) Fingerprint(host string, segments []string) []string {
type trieNode (line 30) | type trieNode struct
function NewPathTrie (line 38) | func NewPathTrie(threshold int) *PathTrie {
FILE: pkg/utils/pathtrie_test.go
function TestPathTrie_BasicInsertion (line 8) | func TestPathTrie_BasicInsertion(t *testing.T) {
function TestPathTrie_RepeatedInsertionNoop (line 20) | func TestPathTrie_RepeatedInsertionNoop(t *testing.T) {
function TestPathTrie_PromotionExactThreshold (line 36) | func TestPathTrie_PromotionExactThreshold(t *testing.T) {
function TestPathTrie_Promotion (line 56) | func TestPathTrie_Promotion(t *testing.T) {
function TestPathTrie_PromotionDoesNotAffectOtherHosts (line 73) | func TestPathTrie_PromotionDoesNotAffectOtherHosts(t *testing.T) {
function TestPathTrie_DeepPath (line 86) | func TestPathTrie_DeepPath(t *testing.T) {
function TestPathTrie_EmptySegments (line 103) | func TestPathTrie_EmptySegments(t *testing.T) {
function TestPathTrie_SingleSegment (line 111) | func TestPathTrie_SingleSegment(t *testing.T) {
function TestPathTrie_PromotedNodeChildrenNil (line 124) | func TestPathTrie_PromotedNodeChildrenNil(t *testing.T) {
function TestPathTrie_PreviousValueCollapseAfterPromotion (line 144) | func TestPathTrie_PreviousValueCollapseAfterPromotion(t *testing.T) {
function TestPathTrie_SegmentsAfterPromotedNodeTrackedIndependently (line 164) | func TestPathTrie_SegmentsAfterPromotedNodeTrackedIndependently(t *testi...
function TestPathTrie_MultipleBranchesIndependent (line 187) | func TestPathTrie_MultipleBranchesIndependent(t *testing.T) {
function TestPathTrie_MultiplePromotionsAtDifferentDepths (line 212) | func TestPathTrie_MultiplePromotionsAtDifferentDepths(t *testing.T) {
function TestPathTrie_NewHostLazyInit (line 233) | func TestPathTrie_NewHostLazyInit(t *testing.T) {
function TestPathTrie_CustomThreshold (line 248) | func TestPathTrie_CustomThreshold(t *testing.T) {
function TestPathTrie_LRUEviction (line 270) | func TestPathTrie_LRUEviction(t *testing.T) {
FILE: pkg/utils/queue/priority_queue.go
type priorityQueue (line 7) | type priorityQueue struct
method Len (line 16) | func (p *priorityQueue) Len() int {
method Push (line 20) | func (p *priorityQueue) Push(v interface{}, priority int) {
method Pop (line 28) | func (p *priorityQueue) Pop() interface{} {
function newPriorityQueue (line 11) | func newPriorityQueue() *priorityQueue {
type itemHeap (line 36) | type itemHeap
method Len (line 44) | func (ih *itemHeap) Len() int {
method Less (line 48) | func (ih *itemHeap) Less(i, j int) bool {
method Swap (line 54) | func (ih *itemHeap) Swap(i, j int) {
method Push (line 60) | func (ih *itemHeap) Push(x interface{}) {
method Pop (line 66) | func (ih *itemHeap) Pop() interface{} {
type item (line 38) | type item struct
FILE: pkg/utils/queue/priority_queue_test.go
function TestPriorityQueue (line 9) | func TestPriorityQueue(t *testing.T) {
FILE: pkg/utils/queue/queue.go
type Queue (line 19) | type Queue struct
method Len (line 45) | func (q *Queue) Len() int {
method Push (line 60) | func (q *Queue) Push(x interface{}, priority int) {
method Pop (line 74) | func (q *Queue) Pop() chan interface{} {
function New (line 28) | func New(strategyName string, timeout int) (*Queue, error) {
FILE: pkg/utils/queue/stack.go
type stack (line 8) | type stack struct
method Push (line 16) | func (s *stack) Push(x interface{}) {
method Len (line 20) | func (s *stack) Len() int {
method Pop (line 24) | func (s *stack) Pop() interface{} {
function newStack (line 12) | func newStack() *stack {
FILE: pkg/utils/queue/stack_test.go
function TestStack (line 9) | func TestStack(t *testing.T) {
FILE: pkg/utils/queue/strategy.go
type Strategy (line 4) | type Strategy
method String (line 12) | func (s Strategy) String() string {
constant BreadthFirst (line 8) | BreadthFirst Strategy = iota
constant DepthFirst (line 9) | DepthFirst
FILE: pkg/utils/regex.go
function ExtractBodyEndpoints (line 32) | func ExtractBodyEndpoints(data string) []string {
function ExtractRelativeEndpoints (line 51) | func ExtractRelativeEndpoints(data string) []string {
FILE: pkg/utils/regex_test.go
function TestPageBodyRegex (line 9) | func TestPageBodyRegex(t *testing.T) {
function TestRelativeEndpointsRegex (line 68) | func TestRelativeEndpointsRegex(t *testing.T) {
FILE: pkg/utils/scope/scope.go
type Manager (line 14) | type Manager struct
method Validate (line 72) | func (m *Manager) Validate(URL *url.URL, rootHostname string) (bool, e...
method validateURL (line 96) | func (m *Manager) validateURL(URL string) (bool, error) {
method validateDNS (line 120) | func (m *Manager) validateDNS(hostname, rootHostname string) (bool, er...
type dnsScopeField (line 22) | type dnsScopeField
constant dnDnsScopeField (line 25) | dnDnsScopeField dnsScopeField = iota + 1
constant rdnDnsScopeField (line 26) | rdnDnsScopeField
constant fqdnDNSScopeField (line 27) | fqdnDNSScopeField
constant customDNSScopeField (line 28) | customDNSScopeField
function NewManager (line 38) | func NewManager(inScope, outOfScope []string, fieldScope string, noScope...
function getDomainRDNandRDN (line 149) | func getDomainRDNandRDN(domain string) (string, string, error) {
FILE: pkg/utils/scope/scope_test.go
function TestManagerValidate (line 12) | func TestManagerValidate(t *testing.T) {
function TestGetDomainRDNandDN (line 79) | func TestGetDomainRDNandDN(t *testing.T) {
function TestNoScopeWithOutOfScope (line 90) | func TestNoScopeWithOutOfScope(t *testing.T) {
FILE: pkg/utils/urlfingerprint.go
type segmentPattern (line 12) | type segmentPattern struct
function containsHexLetter (line 23) | func containsHexLetter(s string) bool {
function normalizeSegment (line 48) | func normalizeSegment(segment string) (string, bool) {
function FingerprintURL (line 66) | func FingerprintURL(rawURL string, trie *PathTrie) string {
function buildFingerprint (line 106) | func buildFingerprint(u *url.URL, path string) string {
function sortedQueryKeys (line 127) | func sortedQueryKeys(params url.Values) []string {
FILE: pkg/utils/urlfingerprint_test.go
function TestContainsHexLetter (line 9) | func TestContainsHexLetter(t *testing.T) {
function TestNormalizeSegment (line 32) | func TestNormalizeSegment(t *testing.T) {
function TestFingerprintURL_RegexOnly (line 121) | func TestFingerprintURL_RegexOnly(t *testing.T) {
function TestFingerprintURL_Idempotency (line 269) | func TestFingerprintURL_Idempotency(t *testing.T) {
function TestFingerprintURL_SimilarURLsCollapse (line 285) | func TestFingerprintURL_SimilarURLsCollapse(t *testing.T) {
function TestFingerprintURL_DifferentURLsStayDistinct (line 314) | func TestFingerprintURL_DifferentURLsStayDistinct(t *testing.T) {
function TestFingerprintURL_WithTrie_PromotionLifecycle (line 339) | func TestFingerprintURL_WithTrie_PromotionLifecycle(t *testing.T) {
function TestFingerprintURL_WithTrie_RegexSegmentsDontInflateTrie (line 369) | func TestFingerprintURL_WithTrie_RegexSegmentsDontInflateTrie(t *testing...
function TestFingerprintURL_WithTrie_MultipleHosts (line 385) | func TestFingerprintURL_WithTrie_MultipleHosts(t *testing.T) {
function TestFingerprintURL_WithTrie_DeepPromotion (line 408) | func TestFingerprintURL_WithTrie_DeepPromotion(t *testing.T) {
function TestFingerprintURL_WithTrie_CombinedRegexAndTrie (line 429) | func TestFingerprintURL_WithTrie_CombinedRegexAndTrie(t *testing.T) {
function TestFingerprintURL_InvalidURL (line 452) | func TestFingerprintURL_InvalidURL(t *testing.T) {
function TestFingerprintURL_NilTrieSameAsRegexOnly (line 470) | func TestFingerprintURL_NilTrieSameAsRegexOnly(t *testing.T) {
FILE: pkg/utils/utils.go
function IsURL (line 13) | func IsURL(url string) bool {
function ParseSRCSetTag (line 23) | func ParseSRCSetTag(value string) []string {
function ParseLinkTag (line 35) | func ParseLinkTag(value string) []string {
function ParseRefreshTag (line 54) | func ParseRefreshTag(value string) string {
function WebUserAgent (line 68) | func WebUserAgent() string {
function FlattenHeaders (line 72) | func FlattenHeaders(headers map[string][]string) map[string]string {
function ReplaceAllQueryParam (line 81) | func ReplaceAllQueryParam(reqUrl, val string) string {
function ExtractParentPaths (line 96) | func ExtractParentPaths(rawurl string) []string {
FILE: pkg/utils/utils_test.go
function TestParseLinkTag (line 9) | func TestParseLinkTag(t *testing.T) {
function TestParseRefreshTag (line 17) | func TestParseRefreshTag(t *testing.T) {
function TestExtractParentPaths (line 24) | func TestExtractParentPaths(t *testing.T) {
Condensed preview — 156 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,389K chars).
[
{
"path": ".github/ISSUE_TEMPLATE/README.md",
"chars": 2212,
"preview": "# Issue Template Management\n\n## New Issue Workflow (Discussion-First)\n\nTo improve issue triage and reduce noise from que"
},
{
"path": ".github/ISSUE_TEMPLATE/config.yml",
"chars": 1088,
"preview": "blank_issues_enabled: false\n\ncontact_links:\n - name: Report a Bug (Discussion First)\n url: https://github.com/projec"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.md.disabled",
"chars": 890,
"preview": "---\nname: Feature request\nabout: Request feature to implement in this project\nlabels: 'Type: Enhancement'\n---\n\n<!--\n1. P"
},
{
"path": ".github/ISSUE_TEMPLATE/issue-report.md.disabled",
"chars": 1265,
"preview": "---\nname: Issue report\nabout: Create a report to help us to improve the project\nlabels: 'Type: Bug'\n\n---\n\n<!-- \n1. Pleas"
},
{
"path": ".github/MAINTAINER_GUIDE.md",
"chars": 4780,
"preview": "# Maintainer Guide: Discussion-First Issue Management\n\n## Overview\n\nKatana now uses a **discussion-first approach** for "
},
{
"path": ".github/PULL_REQUEST_TEMPLATE.md",
"chars": 936,
"preview": "## Proposed changes\n\n<!-- Describe the overall picture of your modifications to help maintainers understand the pull req"
},
{
"path": ".github/dependabot.yml",
"chars": 1203,
"preview": "# To get started with Dependabot version updates, you'll need to specify which\n# package ecosystems to update and where "
},
{
"path": ".github/release.yml",
"chars": 351,
"preview": "changelog:\n exclude:\n authors:\n - app/dependabot\n - dependabot\n categories:\n - title: 🎉 New Features\n "
},
{
"path": ".github/workflows/build-test.yml",
"chars": 1212,
"preview": "name: 🔨 Build Test\n\non:\n workflow_dispatch:\n pull_request:\n branches:\n - dev\n paths:\n - '**.go'\n "
},
{
"path": ".github/workflows/codeql-analysis.yml",
"chars": 849,
"preview": "name: 🚨 CodeQL Analysis\n\non:\n workflow_dispatch:\n pull_request:\n branches:\n - dev\n paths:\n - '**.go'\n "
},
{
"path": ".github/workflows/compat-checks.yaml",
"chars": 344,
"preview": "name: ♾️ Compatibility Checks\n\non:\n pull_request:\n types: [opened, synchronize]\n branches:\n - dev\n\njobs:\n c"
},
{
"path": ".github/workflows/dep-auto-merge.yml",
"chars": 583,
"preview": "name: 🤖 Auto Merge\n\non:\n pull_request_review:\n types: [submitted]\n workflow_run:\n workflows: [\"♾️ Compatibility "
},
{
"path": ".github/workflows/discussion-triage.yml",
"chars": 4916,
"preview": "name: Discussion Triage Helper\n\non:\n discussion:\n types: [created]\n\npermissions:\n discussions: write\n\njobs:\n triag"
},
{
"path": ".github/workflows/dockerhub-push.yml",
"chars": 1074,
"preview": "name: 🌥 Docker Push\n\non:\n workflow_run:\n workflows: [\"🎉 Release Binary\"]\n types:\n - completed\n workflow_dis"
},
{
"path": ".github/workflows/functional-test.yml",
"chars": 584,
"preview": "name: 🧪 Functional Test\n\non:\n pull_request:\n paths:\n - '**.go'\n - '**.mod'\n workflow_dispatch:\n\njobs: \n "
},
{
"path": ".github/workflows/release-binary.yml",
"chars": 1954,
"preview": "name: 🎉 Release Binary\n\non:\n push:\n tags:\n - v*\n workflow_dispatch:\n\njobs:\n build-mac:\n runs-on: macos-lat"
},
{
"path": ".github/workflows/release-test.yml",
"chars": 1696,
"preview": "name: 🔨 Release Test\n\non:\n pull_request:\n paths:\n - \"**.yml\"\n - \"**.go\"\n - \"**.mod\"\n workflow_dispat"
},
{
"path": ".github/workflows/security-crawl-maze-score.yaml",
"chars": 973,
"preview": "name: 🙏🏻 Security Crawl Maze Score\n\non:\n workflow_dispatch:\n\njobs:\n build:\n name: Run Scoring\n runs-on: ubuntu-l"
},
{
"path": ".github/workflows/stale.yml",
"chars": 1633,
"preview": "name: 💤 Stale\n\non:\n schedule:\n - cron: '0 0 * * 0' # Weekly\n\njobs:\n stale:\n runs-on: ubuntu-latest\n permissio"
},
{
"path": ".github/workflows/workflow-monitor.yml",
"chars": 3650,
"preview": "name: Workflow Monitor\n\non:\n schedule:\n # Run weekly on Mondays at 9 AM UTC\n - cron: '0 9 * * 1'\n workflow_dispa"
},
{
"path": ".gitignore",
"chars": 117,
"preview": "security-crawl-maze/\ncmd/katana/katana\nkatana\n*.exe\nkatana_*/\nkatana_*/\ndist/\n\n.vscode\n.devcontainer\n.DS_Store\n.idea\n"
},
{
"path": ".goreleaser/linux.yml",
"chars": 1117,
"preview": "version: 2\n\nenv:\n - GO111MODULE=on\nbefore:\n hooks:\n - go mod tidy\nproject_name: katana\nbuilds:\n - id: katana-linux"
},
{
"path": ".goreleaser/mac.yml",
"chars": 539,
"preview": "version: 2\n\nenv:\n - GO111MODULE=on\nbefore:\n hooks:\n - go mod tidy\nproject_name: katana\nbuilds:\n - id: katana-darwi"
},
{
"path": ".goreleaser/windows.yml",
"chars": 404,
"preview": "version: 2\n\nenv:\n - GO111MODULE=on\nbefore:\n hooks:\n - go mod tidy\nproject_name: katana\nbuilds:\n - id: katana-windo"
},
{
"path": "Dockerfile",
"chars": 302,
"preview": "FROM golang:1.25.7-alpine AS build-env\nRUN apk add --no-cache git gcc musl-dev\nWORKDIR /app\nCOPY . /app\nRUN go mod downl"
},
{
"path": "LICENSE.md",
"chars": 1073,
"preview": "MIT License\n\nCopyright (c) 2022 ProjectDiscovery\n\nPermission is hereby granted, free of charge, to any person obtaining "
},
{
"path": "Makefile",
"chars": 481,
"preview": "# Go parameters\nGOCMD=go\nGOBUILD=$(GOCMD) build\nGOMOD=$(GOCMD) mod\nGOTEST=$(GOCMD) test\nGOFLAGS := -v\n# This should be d"
},
{
"path": "README.md",
"chars": 46859,
"preview": "<h1 align=\"center\">\n <img src=\"https://user-images.githubusercontent.com/8293321/196779266-421c79d4-643a-4f73-9b54-3da3"
},
{
"path": "SECURITY.md",
"chars": 214,
"preview": "# Security Policy\n\n## Reporting a Vulnerability\nDO NOT CREATE AN ISSUE to report a security problem. Instead, please sen"
},
{
"path": "cmd/functional-test/main.go",
"chars": 1491,
"preview": "package main\n\nimport (\n\t\"flag\"\n\t\"fmt\"\n\t\"log\"\n\t\"os\"\n\t\"strings\"\n\n\t\"github.com/logrusorgru/aurora\"\n\t\"github.com/pkg/errors\""
},
{
"path": "cmd/functional-test/run.sh",
"chars": 458,
"preview": "#!/bin/bash\n\n# reading os type from arguments\nCURRENT_OS=$1\n\nif [ \"${CURRENT_OS}\" == \"windows-latest\" ];then\n extensi"
},
{
"path": "cmd/integration-test/filters.go",
"chars": 3604,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"os\"\n\t\"os/exec\"\n\t\"strings\"\n\t\"sync/atomic\"\n\n\t\"github.com/"
},
{
"path": "cmd/integration-test/integration-test.go",
"chars": 996,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"strings\"\n\n\t\"github.com/logrusorgru/aurora\"\n)\n\ntype TestCase interface {\n\t// Execut"
},
{
"path": "cmd/integration-test/library.go",
"chars": 1492,
"preview": "package main\n\nimport (\n\t\"fmt\"\n\t\"math\"\n\n\t\"github.com/projectdiscovery/katana/pkg/engine/standard\"\n\t\"github.com/projectdis"
},
{
"path": "cmd/tools/crawl-maze-score/main.go",
"chars": 6680,
"preview": "package main\n\nimport (\n\t\"bufio\"\n\t\"fmt\"\n\t\"log\"\n\t\"math\"\n\t\"os\"\n\t\"strings\"\n\n\t\"github.com/logrusorgru/aurora\"\n\t\"github.com/pr"
},
{
"path": "go.mod",
"chars": 8207,
"preview": "module github.com/projectdiscovery/katana\n\ngo 1.25.7\n\nrequire (\n\tgithub.com/BishopFox/jsluice v0.0.0-20240110145140-0ddf"
},
{
"path": "go.sum",
"chars": 73512,
"preview": "aead.dev/minisign v0.2.0 h1:kAWrq/hBRu4AARY6AlciO83xhNnW9UaC8YipS2uhLPk=\naead.dev/minisign v0.2.0/go.mod h1:zdq6LdSd9Tbu"
},
{
"path": "integration_tests/run.sh",
"chars": 422,
"preview": "#!/bin/bash\n\necho \"::group::Build katana\"\nrm integration-test katana 2>/dev/null\ncd ../cmd/katana\ngo build\nmv katana ../"
},
{
"path": "internal/runner/banner.go",
"chars": 669,
"preview": "package runner\n\nimport (\n\t\"github.com/projectdiscovery/gologger\"\n\tupdateutils \"github.com/projectdiscovery/utils/update\""
},
{
"path": "internal/runner/executer.go",
"chars": 1900,
"preview": "package runner\n\nimport (\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/utils/errkit\""
},
{
"path": "internal/runner/healthcheck.go",
"chars": 2892,
"preview": "package runner\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"os/exec\"\n\t\"runtime\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/goflags\"\n\t\"github"
},
{
"path": "internal/runner/options.go",
"chars": 4945,
"preview": "package runner\n\nimport (\n\t\"bufio\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t"
},
{
"path": "internal/runner/runner.go",
"chars": 5444,
"preview": "package runner\n\nimport (\n\t\"encoding/json\"\n\t\"os\"\n\t\"strconv\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/project"
},
{
"path": "internal/testutils/helper.go",
"chars": 212,
"preview": "package testutils\n\nfunc CompareOutput(input, expected []string) bool {\n\tif len(input) != len(expected) {\n\t\treturn false\n"
},
{
"path": "internal/testutils/integration.go",
"chars": 578,
"preview": "package testutils\n\nimport (\n\t\"fmt\"\n\t\"os/exec\"\n\t\"strings\"\n)\n\nfunc RunKatanaBinaryAndGetResults(target string, katanaBinar"
},
{
"path": "internal/testutils/testutils.go",
"chars": 719,
"preview": "package testutils\n\nimport (\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/utils/errkit\"\n)\n\ntype TestCase struct {\n\tName "
},
{
"path": "pkg/engine/common/base.go",
"chars": 12900,
"preview": "package common\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"io\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"time\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"g"
},
{
"path": "pkg/engine/common/error.go",
"chars": 80,
"preview": "package common\n\nimport \"errors\"\n\nvar ErrOutOfScope = errors.New(\"out of scope\")\n"
},
{
"path": "pkg/engine/common/http.go",
"chars": 2494,
"preview": "package common\n\nimport (\n\t\"context\"\n\t\"crypto/tls\"\n\t\"net\"\n\t\"net/http\"\n\t\"net/url\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/f"
},
{
"path": "pkg/engine/engine.go",
"chars": 78,
"preview": "package engine\n\ntype Engine interface {\n\tCrawl(string) error\n\tClose() error\n}\n"
},
{
"path": "pkg/engine/headless/TODOS.md",
"chars": 14124,
"preview": "# 🗺️ Headless-Crawler Road-Map \n---\n\n## 🥇 Core Improvements (High-impact / first passes)\n\n- [ ] After clicking on elem"
},
{
"path": "pkg/engine/headless/browser/browser.go",
"chars": 18470,
"preview": "package browser\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/base64\"\n\t\"fmt\"\n\t\"io\"\n\t\"log/slog\"\n\t\"net/http\"\n\t\"net/http/httputi"
},
{
"path": "pkg/engine/headless/browser/cookie/cookie.go",
"chars": 3176,
"preview": "// Package cookie implements cookie consent handling for the\n// crawler. Its a partial port of I-Still-Dont-Care-About-C"
},
{
"path": "pkg/engine/headless/browser/cookie/cookie_test.go",
"chars": 1286,
"preview": "package cookie\n\nimport (\n\t\"testing\"\n\n\t\"github.com/go-rod/rod/lib/proto\"\n)\n\nfunc TestShouldBlockRequest(t *testing.T) {\n\t"
},
{
"path": "pkg/engine/headless/browser/cookie/rules.json",
"chars": 429510,
"preview": "[\n {\n \"id\": 1,\n \"priority\": 1,\n \"action\": {\n \"type\": \"block\"\n },\n \"condition\": {\n \"urlFilter\":"
},
{
"path": "pkg/engine/headless/browser/element.go",
"chars": 7741,
"preview": "package browser\n\nimport (\n\t\"net/url\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/pkg/errors\"\n\t\"github.com/projectdiscovery/katan"
},
{
"path": "pkg/engine/headless/browser/stealth/assets.go",
"chars": 185765,
"preview": "// from stealth go-rod\npackage stealth\n\n// JS for stealth\nconst JS = `;(() => {\n\n(({_utilsFns:_utilsFns,_mainFunction:_m"
},
{
"path": "pkg/engine/headless/captcha/capsolver/capsolver.go",
"chars": 5420,
"preview": "package capsolver\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"time\"\n\n\t\"github.com/projectd"
},
{
"path": "pkg/engine/headless/captcha/capsolver/capsolver_test.go",
"chars": 4718,
"preview": "package capsolver\n\nimport (\n\t\"context\"\n\t\"encoding/json\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"sync/atomic\"\n\t\"testing\"\n\n\t\"gi"
},
{
"path": "pkg/engine/headless/captcha/captcha.go",
"chars": 2202,
"preview": "package captcha\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/go-rod/rod\"\n\tditcaptcha \"github.com/happyhackingspa"
},
{
"path": "pkg/engine/headless/captcha/helpers_test.go",
"chars": 779,
"preview": "package captcha\n\nimport (\n\t\"fmt\"\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"testing\"\n\n\t\"github.com/go-rod/rod\"\n\t\"github.com/go-r"
},
{
"path": "pkg/engine/headless/captcha/identify.go",
"chars": 1095,
"preview": "package captcha\n\nimport (\n\t\"github.com/go-rod/rod\"\n\tcaptchajs \"github.com/projectdiscovery/katana/pkg/engine/headless/ca"
},
{
"path": "pkg/engine/headless/captcha/identify_test.go",
"chars": 4082,
"preview": "package captcha\n\nimport (\n\t\"testing\"\n\n\tditcaptcha \"github.com/happyhackingspace/dit/captcha\"\n\t\"github.com/stretchr/testi"
},
{
"path": "pkg/engine/headless/captcha/inject_test.go",
"chars": 1274,
"preview": "package captcha\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/require\"\n)\n\nfu"
},
{
"path": "pkg/engine/headless/captcha/injection_test.go",
"chars": 5578,
"preview": "package captcha\n\nimport (\n\t\"net/http\"\n\t\"net/http/httptest\"\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.co"
},
{
"path": "pkg/engine/headless/captcha/integration_test.go",
"chars": 1316,
"preview": "package captcha\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/require\"\n)\n\nfu"
},
{
"path": "pkg/engine/headless/captcha/js/identify.js",
"chars": 1571,
"preview": "() => {\n const hcap = document.querySelector('.h-captcha[data-sitekey]');\n if (hcap) {\n return { provider: \"hcaptch"
},
{
"path": "pkg/engine/headless/captcha/js/inject-hcaptcha.js",
"chars": 749,
"preview": "(token) => {\n document.querySelectorAll('textarea[name=\"h-captcha-response\"]').forEach(el => { el.value = token; });\n "
},
{
"path": "pkg/engine/headless/captcha/js/inject-recaptcha.js",
"chars": 1900,
"preview": "(token) => {\n document.querySelectorAll('[id=\"g-recaptcha-response\"], [name=\"g-recaptcha-response\"]').forEach(el => {\n "
},
{
"path": "pkg/engine/headless/captcha/js/inject-turnstile.js",
"chars": 651,
"preview": "(token) => {\n document.querySelectorAll('input[name=\"cf-turnstile-response\"]').forEach(el => { el.value = token; });\n\n "
},
{
"path": "pkg/engine/headless/captcha/js/js.go",
"chars": 256,
"preview": "package js\n\nimport _ \"embed\"\n\nvar (\n\t//go:embed identify.js\n\tIdentifyJS string\n\n\t//go:embed inject-recaptcha.js\n\tInjectR"
},
{
"path": "pkg/engine/headless/captcha/solver.go",
"chars": 637,
"preview": "package captcha\n\nimport (\n\t\"context\"\n\t\"fmt\"\n)\n\ntype Solution struct {\n\tToken string\n\tProvider Provider\n}\n\ntype Solver"
},
{
"path": "pkg/engine/headless/captcha/solver_test.go",
"chars": 540,
"preview": "package captcha\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n\t\"github.com/stretchr/testify/require\"\n)\n\nfu"
},
{
"path": "pkg/engine/headless/crawler/crawler.go",
"chars": 14152,
"preview": "package crawler\n\nimport (\n\t\"context\"\n\t\"fmt\"\n\t\"log/slog\"\n\t\"os\"\n\t\"os/user\"\n\t\"path/filepath\"\n\t\"regexp\"\n\t\"sync\"\n\t\"time\"\n\n\t\"g"
},
{
"path": "pkg/engine/headless/crawler/diagnostics/diagnostics.go",
"chars": 5053,
"preview": "package diagnostics\n\nimport (\n\t\"encoding/json\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/kat"
},
{
"path": "pkg/engine/headless/crawler/formfill.go",
"chars": 6440,
"preview": "package crawler\n\nimport (\n\t\"fmt\"\n\t\"log/slog\"\n\n\t\"github.com/go-rod/rod\"\n\t\"github.com/go-rod/rod/lib/proto\"\n\t\"github.com/p"
},
{
"path": "pkg/engine/headless/crawler/normalizer/dom_utils.go",
"chars": 4256,
"preview": "package normalizer\n\nimport (\n\t\"strings\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"golang.org/x/net/html\"\n)\n\n// DefaultDOMTran"
},
{
"path": "pkg/engine/headless/crawler/normalizer/dom_utils_test.go",
"chars": 1972,
"preview": "package normalizer\n\nimport (\n\t\"testing\"\n)\n\nfunc TestDOMNormalizer_Apply(t *testing.T) {\n\ttype args struct {\n\t\tcontent st"
},
{
"path": "pkg/engine/headless/crawler/normalizer/helpers.go",
"chars": 1378,
"preview": "package normalizer\n\n// dateTimePatterns contains regex patterns for various date and time formats\n// The ordering is imp"
},
{
"path": "pkg/engine/headless/crawler/normalizer/normalizer.go",
"chars": 3476,
"preview": "package normalizer\n\nimport (\n\t\"fmt\"\n\t\"html\"\n\t\"net/url\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"strings\"\n\n\t\"github.com/PuerkitoBio/goquery"
},
{
"path": "pkg/engine/headless/crawler/normalizer/simhash/simhash.go",
"chars": 4361,
"preview": "// Package simhash implements SimHash algorithm for near-duplicate detection.\n//\n// The original algorithm is taken from"
},
{
"path": "pkg/engine/headless/crawler/normalizer/simhash/simhash_test.go",
"chars": 2011,
"preview": "package simhash\n\nimport (\n\t\"strings\"\n\t\"testing\"\n)\n\nvar (\n\thtmlA = `<html><body><p>Hello World</p></body></html>`\n\t// htm"
},
{
"path": "pkg/engine/headless/crawler/normalizer/text_utils.go",
"chars": 1957,
"preview": "package normalizer\n\nimport (\n\t\"fmt\"\n\t\"regexp\"\n\t\"slices\"\n)\n\n// DefaultTextPatterns is a list of regex patterns for the te"
},
{
"path": "pkg/engine/headless/crawler/normalizer/text_utils_test.go",
"chars": 2419,
"preview": "package normalizer\n\nimport (\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestTextNormalizer_AllPatterns(t *testing.T) {\n\tnormalizer, e"
},
{
"path": "pkg/engine/headless/crawler/state.go",
"chars": 10551,
"preview": "package crawler\n\nimport (\n\t\"crypto/sha256\"\n\t\"encoding/hex\"\n\t\"fmt\"\n\t\"log/slog\"\n\t\"strings\"\n\n\tgraphlib \"github.com/dominikb"
},
{
"path": "pkg/engine/headless/crawler/state_test.go",
"chars": 4922,
"preview": "package crawler\n\nimport (\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/pkg/errors\"\n\t\"github.com/projectdiscovery/katana/pkg/engin"
},
{
"path": "pkg/engine/headless/debugger.go",
"chars": 2997,
"preview": "package headless\n\nimport (\n\t\"encoding/json\"\n\t\"fmt\"\n\t\"net/http\"\n\t\"sync\"\n\t\"time\"\n)\n\n// ActiveURL represents a URL currentl"
},
{
"path": "pkg/engine/headless/graph/graph.go",
"chars": 3599,
"preview": "// Package graph implements a Directed Graph for storing\n// state information during crawling of a Web Application.\npack"
},
{
"path": "pkg/engine/headless/headless.go",
"chars": 5849,
"preview": "package headless\n\nimport (\n\t\"log/slog\"\n\t\"net/url\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/lmittmann/tint\"\n\t\"github.com/projectdiscov"
},
{
"path": "pkg/engine/headless/js/js.go",
"chars": 613,
"preview": "package js\n\nimport (\n\t_ \"embed\"\n\n\t\"github.com/go-rod/rod\"\n\t\"github.com/pkg/errors\"\n)\n\nvar (\n\t//go:embed utils.js\n\tutilsJ"
},
{
"path": "pkg/engine/headless/js/page-init.js",
"chars": 8168,
"preview": "// This script initializes the page and hooks up event listeners\n// and other interesting stuff needed to make the crawl"
},
{
"path": "pkg/engine/headless/js/utils.js",
"chars": 12194,
"preview": "// This file contains utility JS functions that are utilised by\n// the main crawling JS code to perform actions.\n(functi"
},
{
"path": "pkg/engine/headless/types/types.go",
"chars": 8456,
"preview": "package types\n\nimport (\n\t\"crypto/md5\"\n\t\"encoding/hex\"\n\t\"fmt\"\n\t\"os\"\n\t\"sort\"\n\t\"strings\"\n)\n\nvar (\n\tIsDiagnosticEnabled = fa"
},
{
"path": "pkg/engine/hybrid/crawl.go",
"chars": 13559,
"preview": "package hybrid\n\nimport (\n\t\"bytes\"\n\t\"errors\"\n\t\"io\"\n\t\"net/http\"\n\t\"net/http/httputil\"\n\t\"net/url\"\n\t\"slices\"\n\t\"strings\"\n\t\"tim"
},
{
"path": "pkg/engine/hybrid/doc.go",
"chars": 222,
"preview": "// Package hybrid implements the functionality for a hybrid-headless crawler.\n// It uses both headless browser and net/h"
},
{
"path": "pkg/engine/hybrid/hijack.go",
"chars": 1855,
"preview": "package hybrid\n\nimport (\n\t\"encoding/base64\"\n\n\t\"github.com/go-rod/rod\"\n\t\"github.com/go-rod/rod/lib/proto\"\n)\n\n// NewHijack"
},
{
"path": "pkg/engine/hybrid/hybrid.go",
"chars": 7270,
"preview": "package hybrid\n\nimport (\n\t\"fmt\"\n\t\"os\"\n\t\"time\"\n\n\t\"github.com/go-rod/rod\"\n\t\"github.com/go-rod/rod/lib/launcher\"\n\t\"github.c"
},
{
"path": "pkg/engine/parser/files/request.go",
"chars": 1341,
"preview": "package files\n\nimport (\n\t\"github.com/projectdiscovery/katana/pkg/navigation\"\n\t\"github.com/projectdiscovery/retryablehttp"
},
{
"path": "pkg/engine/parser/files/robotstxt.go",
"chars": 1946,
"preview": "package files\n\nimport (\n\t\"bufio\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.c"
},
{
"path": "pkg/engine/parser/files/robotstxt_test.go",
"chars": 969,
"preview": "package files\n\nimport (\n\t\"net/http\"\n\n\t\"strings\"\n\t\"testing\"\n\n\turlutil \"github.com/projectdiscovery/utils/url\"\n\t\"github.co"
},
{
"path": "pkg/engine/parser/files/sitemapxml.go",
"chars": 2508,
"preview": "package files\n\nimport (\n\t\"encoding/xml\"\n\t\"fmt\"\n\t\"io\"\n\t\"net/http\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"g"
},
{
"path": "pkg/engine/parser/files/sitemapxml_test.go",
"chars": 1012,
"preview": "package files\n\nimport (\n\t\"net/http\"\n\t\"strings\"\n\t\"testing\"\n\n\turlutil \"github.com/projectdiscovery/utils/url\"\n\t\"github.com"
},
{
"path": "pkg/engine/parser/parser.go",
"chars": 29123,
"preview": "package parser\n\nimport (\n\t\"mime/multipart\"\n\t\"net/http\"\n\t\"strings\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"github.com/projec"
},
{
"path": "pkg/engine/parser/parser_generic.go",
"chars": 2669,
"preview": "//go:build !(386 || windows)\n\npackage parser\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"github.com"
},
{
"path": "pkg/engine/parser/parser_nojs.go",
"chars": 701,
"preview": "//go:build windows || 386\n\npackage parser\n\ntype Options struct {\n\tAutomaticFormFill bool\n\tScrapeJSLuiceResponses bo"
},
{
"path": "pkg/engine/parser/parser_test.go",
"chars": 36125,
"preview": "package parser\n\nimport (\n\t\"net/http\"\n\t\"regexp\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"github.com/pro"
},
{
"path": "pkg/engine/standard/crawl.go",
"chars": 4218,
"preview": "package standard\n\nimport (\n\t\"bytes\"\n\t\"context\"\n\t\"io\"\n\t\"net/http\"\n\t\"net/http/httputil\"\n\t\"net/url\"\n\t\"strings\"\n\n\t\"github.co"
},
{
"path": "pkg/engine/standard/doc.go",
"chars": 174,
"preview": "// Package standard implements the functionality for a non-headless crawler.\n// It uses net/http for making requests and"
},
{
"path": "pkg/engine/standard/standard.go",
"chars": 1067,
"preview": "package standard\n\nimport (\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectdiscovery/katana/pkg/engine/commo"
},
{
"path": "pkg/navigation/request.go",
"chars": 1695,
"preview": "package navigation\n\nimport (\n\t\"net/http\"\n\t\"strings\"\n)\n\n// Depth is the depth of a navigation\ntype Depth struct{}\n\n// Req"
},
{
"path": "pkg/navigation/response.go",
"chars": 1930,
"preview": "package navigation\n\nimport (\n\t\"net/http\"\n\t\"strings\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\tjsoniter \"github.com/json-iterat"
},
{
"path": "pkg/output/custom_field.go",
"chars": 4377,
"preview": "package output\n\nimport (\n\t\"os\"\n\t\"path/filepath\"\n\t\"regexp\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.com/projectd"
},
{
"path": "pkg/output/error.go",
"chars": 245,
"preview": "package output\n\nimport \"time\"\n\ntype Error struct {\n\tTimestamp time.Time `json:\"timestamp,omitempty\"`\n\tEndpoint string "
},
{
"path": "pkg/output/fields.go",
"chars": 7966,
"preview": "package output\n\nimport (\n\t\"fmt\"\n\t\"net/url\"\n\t\"os\"\n\t\"path\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\t\"github.co"
},
{
"path": "pkg/output/fields_test.go",
"chars": 1514,
"preview": "package output\n\nimport (\n\t\"testing\"\n\n\t\"github.com/projectdiscovery/katana/pkg/navigation\"\n\t\"github.com/stretchr/testify/"
},
{
"path": "pkg/output/file_writer.go",
"chars": 889,
"preview": "package output\n\nimport (\n\t\"bufio\"\n\t\"os\"\n)\n\n// fileWriter is a concurrent file based output writer.\ntype fileWriter struc"
},
{
"path": "pkg/output/format_json.go",
"chars": 1059,
"preview": "package output\n\nimport (\n\tjsoniter \"github.com/json-iterator/go\"\n\t\"github.com/projectdiscovery/utils/structs\"\n)\n\n// form"
},
{
"path": "pkg/output/format_screen.go",
"chars": 1366,
"preview": "package output\n\nimport (\n\t\"bytes\"\n\t\"fmt\"\n\t\"strconv\"\n)\n\n// formatScreen formats the output for showing on screen.\nfunc (w"
},
{
"path": "pkg/output/format_template.go",
"chars": 977,
"preview": "package output\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"io\"\n\t\"strings\"\n\n\t\"github.com/valyala/fasttemplate\"\n)\n\nfunc (w *StandardWrite"
},
{
"path": "pkg/output/options.go",
"chars": 871,
"preview": "package output\n\nimport (\n\t\"regexp\"\n\n\t\"github.com/projectdiscovery/katana/pkg/utils/extensions\"\n)\n\n// Options contains th"
},
{
"path": "pkg/output/output.go",
"chars": 12694,
"preview": "package output\n\nimport (\n\t\"errors\"\n\t\"fmt\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"regexp\"\n\t\"strconv\"\n\t\"strings\"\n\t\"sync\"\n\n\tjsoniter \"git"
},
{
"path": "pkg/output/responses.go",
"chars": 2522,
"preview": "package output\n\nimport (\n\t\"bytes\"\n\t\"crypto/sha1\"\n\t\"encoding/hex\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"strings\"\n\n\t\"github.com/project"
},
{
"path": "pkg/output/result.go",
"chars": 526,
"preview": "package output\n\nimport (\n\t\"time\"\n\n\t\"github.com/projectdiscovery/katana/pkg/navigation\"\n)\n\n// Result of the crawling\ntype"
},
{
"path": "pkg/types/crawler_options.go",
"chars": 6796,
"preview": "package types\n\nimport (\n\t\"context\"\n\t\"log/slog\"\n\t\"os/user\"\n\t\"regexp\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/fastdialer/fa"
},
{
"path": "pkg/types/default.go",
"chars": 392,
"preview": "package types\n\nimport \"github.com/projectdiscovery/katana/pkg/utils/queue\"\n\nvar DefaultOptions Options\n\nfunc init() {\n\tD"
},
{
"path": "pkg/types/options.go",
"chars": 9634,
"preview": "package types\n\nimport (\n\t\"regexp\"\n\t\"strings\"\n\t\"time\"\n\n\t\"github.com/projectdiscovery/goflags\"\n\t\"github.com/projectdiscove"
},
{
"path": "pkg/types/options_test.go",
"chars": 2516,
"preview": "package types\n\nimport (\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/projectdiscovery/goflags\"\n\t\"github.com/stretchr/testify/requ"
},
{
"path": "pkg/utils/extensions/extensions.go",
"chars": 2512,
"preview": "package extensions\n\nimport (\n\t\"path\"\n\t\"strings\"\n\n\t\"github.com/projectdiscovery/gologger\"\n\turlutil \"github.com/projectdis"
},
{
"path": "pkg/utils/extensions/extensions_test.go",
"chars": 1213,
"preview": "package extensions\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/require\"\n)\n\nfunc TestValidatorValidate(t *testing"
},
{
"path": "pkg/utils/filters/filters.go",
"chars": 1070,
"preview": "package filters\n\n// Filter is an interface implemented by deduplication mechanism\ntype Filter interface {\n\t// Close clos"
},
{
"path": "pkg/utils/filters/filters_test.go",
"chars": 432,
"preview": "package filters\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/require\"\n)\n\nfunc TestSimpleFilter(t *testing.T) {\n\ts"
},
{
"path": "pkg/utils/filters/simple.go",
"chars": 1688,
"preview": "package filters\n\nimport (\n\t\"crypto/md5\"\n\t\"encoding/hex\"\n\n\t\"github.com/projectdiscovery/hmap/store/hybrid\"\n\tstringsutil \""
},
{
"path": "pkg/utils/formfields.go",
"chars": 2207,
"preview": "package utils\r\n\r\nimport (\r\n\t\"strings\"\r\n\r\n\t\"github.com/projectdiscovery/katana/pkg/navigation\"\r\n\r\n\t\"github.com/PuerkitoBi"
},
{
"path": "pkg/utils/formfields_test.go",
"chars": 2055,
"preview": "package utils\r\n\r\nimport (\r\n\t\"net/url\"\r\n\t\"strings\"\r\n\t\"testing\"\r\n\r\n\t\"github.com/PuerkitoBio/goquery\"\r\n\t\"github.com/stretch"
},
{
"path": "pkg/utils/formfill.go",
"chars": 9404,
"preview": "package utils\n\nimport (\n\t\"fmt\"\n\t\"strconv\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\tmapsutil \"github.com/projectdiscovery/util"
},
{
"path": "pkg/utils/formfill_test.go",
"chars": 3053,
"preview": "package utils\n\nimport (\n\t\"net/url\"\n\t\"strings\"\n\t\"testing\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"github.com/stretchr/testif"
},
{
"path": "pkg/utils/jsluice.go",
"chars": 3455,
"preview": "//go:build !(386 || windows)\n\npackage utils\n\nimport (\n\t\"regexp\"\n\n\t\"github.com/BishopFox/jsluice\"\n)\n\nvar (\n\t// CommonJSLi"
},
{
"path": "pkg/utils/jsluice_test.go",
"chars": 616,
"preview": "//go:build !(386 || windows)\n\npackage utils\n\nimport \"testing\"\n\nfunc TestIsPathCommonJSLibraryFile(t *testing.T) {\n\ttype "
},
{
"path": "pkg/utils/maps.go",
"chars": 283,
"preview": "package utils\n\nimport mapsutil \"github.com/projectdiscovery/utils/maps\"\n\nfunc MergeDataMaps(dataMap1 *mapsutil.OrderedMa"
},
{
"path": "pkg/utils/maps_test.go",
"chars": 836,
"preview": "package utils\n\nimport (\n\t\"testing\"\n\n\tmapsutil \"github.com/projectdiscovery/utils/maps\"\n\t\"github.com/stretchr/testify/req"
},
{
"path": "pkg/utils/pathtrie.go",
"chars": 2594,
"preview": "package utils\n\nimport (\n\t\"sync\"\n\n\tlru \"github.com/hashicorp/golang-lru/v2\"\n)\n\n// DefaultPromotionThreshold is the number"
},
{
"path": "pkg/utils/pathtrie_test.go",
"chars": 8601,
"preview": "package utils\n\nimport (\n\t\"fmt\"\n\t\"testing\"\n)\n\nfunc TestPathTrie_BasicInsertion(t *testing.T) {\n\ttrie := NewPathTrie(0)\n\ts"
},
{
"path": "pkg/utils/queue/priority_queue.go",
"chars": 1293,
"preview": "package queue\n\nimport (\n\t\"container/heap\"\n)\n\ntype priorityQueue struct {\n\titemHeap *itemHeap\n}\n\nfunc newPriorityQueue() "
},
{
"path": "pkg/utils/queue/priority_queue_test.go",
"chars": 461,
"preview": "package queue\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/require\"\n)\n\nfunc TestPriorityQueue(t *testing.T) {\n\tqu"
},
{
"path": "pkg/utils/queue/queue.go",
"chars": 2186,
"preview": "package queue\n\nimport (\n\t\"errors\"\n\t\"sync\"\n\t\"time\"\n)\n\n// Queue is a queue that implements bucket based depth-first\n// or "
},
{
"path": "pkg/utils/queue/stack.go",
"chars": 457,
"preview": "package queue\n\nimport (\n\t\"container/list\"\n)\n\n// Taken from https://stackoverflow.com/a/64641330/9546749\ntype stack struc"
},
{
"path": "pkg/utils/queue/stack_test.go",
"chars": 413,
"preview": "package queue\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/require\"\n)\n\nfunc TestStack(t *testing.T) {\n\tqueue := n"
},
{
"path": "pkg/utils/queue/strategy.go",
"chars": 384,
"preview": "package queue\n\n// Strategy of the queue\ntype Strategy int\n\n// strategies of queues available for selection.\nconst (\n\tBre"
},
{
"path": "pkg/utils/regex.go",
"chars": 2170,
"preview": "package utils\n\nimport (\n\t\"regexp\"\n)\n\nvar (\n\tBodyA0 = `(?:`\n\tBodyB0 = `(`\n\tBodyC0 = `(?:[\\.]{1,2}/[A-Za-z0-9\\-_/\\\\?&@\\.?="
},
{
"path": "pkg/utils/regex_test.go",
"chars": 3652,
"preview": "package utils\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/assert\"\n)\n\nfunc TestPageBodyRegex(t *testing.T) {\n\ttes"
},
{
"path": "pkg/utils/scope/scope.go",
"chars": 4782,
"preview": "package scope\n\nimport (\n\t\"fmt\"\n\t\"net\"\n\t\"net/url\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"golang.org/x/net/publicsuffix\"\n)\n\n// Manager ma"
},
{
"path": "pkg/utils/scope/scope_test.go",
"chars": 7281,
"preview": "package scope\n\nimport (\n\t\"testing\"\n\n\turlutil \"github.com/projectdiscovery/utils/url\"\n\t\"github.com/stretchr/testify/requi"
},
{
"path": "pkg/utils/urlfingerprint.go",
"chars": 4010,
"preview": "package utils\n\nimport (\n\t\"net/url\"\n\t\"regexp\"\n\t\"sort\"\n\t\"strings\"\n)\n\n// segmentPattern defines a regex pattern and its rep"
},
{
"path": "pkg/utils/urlfingerprint_test.go",
"chars": 14457,
"preview": "package utils\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\t\"testing\"\n)\n\nfunc TestContainsHexLetter(t *testing.T) {\n\ttests := []struct {\n"
},
{
"path": "pkg/utils/utils.go",
"chars": 2831,
"preview": "package utils\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n\n\t\"github.com/lukasbob/srcset\"\n\t\"github.com/projectdiscovery/gologger\"\n\turluti"
},
{
"path": "pkg/utils/utils_test.go",
"chars": 998,
"preview": "package utils\n\nimport (\n\t\"testing\"\n\n\t\"github.com/stretchr/testify/require\"\n)\n\nfunc TestParseLinkTag(t *testing.T) {\n\thea"
}
]
About this extraction
This page contains the full source code of the projectdiscovery/katana GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 156 files (1.2 MB), approximately 375.0k tokens, and a symbol index with 610 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.