[ { "path": ".cargo/config.toml", "content": "[target.x86_64-unknown-linux-gnu]\nrustflags = [\"-Clink-arg=-fuse-ld=mold\", \"-Ctarget-cpu=native\"]" }, { "path": ".gitattributes", "content": "# Auto detect text files and perform LF normalization\n* text eol=lf" }, { "path": ".gitignore", "content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# C extensions\n*.so\n\n# Distribution / packaging\n.Python\nbuild/\ndevelop-eggs/\ndist/\ndownloads/\neggs/\n.eggs/\nlib/\nlib64/\nparts/\nsdist/\nvar/\nwheels/\n*.egg-info/\n.installed.cfg\n*.egg\nMANIFEST\n\n# PyInstaller\n# Usually these files are written by a python script from a template\n# before PyInstaller builds the exe, so as to inject date/other infos into it.\n*.manifest\n*.spec\n\n# Installer logs\npip-log.txt\npip-delete-this-directory.txt\n\n# Unit test / coverage reports\nhtmlcov/\n.tox/\n.nox/\n.coverage\n.coverage.*\n.cache\nnosetests.xml\ncoverage.xml\n*.cover\n.hypothesis/\n.pytest_cache/\n\n# Translations\n*.mo\n*.pot\n\n# Django stuff:\n*.log\nlocal_settings.py\ndb.sqlite3\n\n# Flask stuff:\ninstance/\n.webassets-cache\n\n# Scrapy stuff:\n.scrapy\n\n# Sphinx documentation\ndocs/_build/\n\n# PyBuilder\ntarget/\n\n# Jupyter Notebook\n.ipynb_checkpoints\n\n# IPython\nprofile_default/\nipython_config.py\n\n# pyenv\n.python-version\n\n# celery beat schedule file\ncelerybeat-schedule\n\n# SageMath parsed files\n*.sage.py\n\n# Environments\n.env\n.venv\nenv/\nvenv/\nENV/\nenv.bak/\nvenv.bak/\n\n# Spyder project settings\n.spyderproject\n.spyproject\n\n# Rope project settings\n.ropeproject\n\n# mkdocs documentation\n\n# mypy\n.mypy_cache/\n.dmypy.json\ndmypy.json\n\n# Pyre type checker\n.pyre/\n\ndns_cache.txt\ndata/\ndb/\nextracted/\npassive_dns_db/\ndns_db/\n\n\n#Added by cargo\n\n/target\nCargo.lock\n\n# Logs\nlogs\n*.log\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\nlerna-debug.log*\n.pnpm-debug.log*\n\n# Diagnostic reports (https://nodejs.org/api/report.html)\nreport.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json\n\n# Runtime data\npids\n*.pid\n*.seed\n*.pid.lock\n\n# Directory for instrumented libs generated by jscoverage/JSCover\nlib-cov\n\n# Coverage directory used by tools like istanbul\ncoverage\n*.lcov\n\n# nyc test coverage\n.nyc_output\n\n# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)\n.grunt\n\n# Bower dependency directory (https://bower.io/)\nbower_components\n\n# node-waf configuration\n.lock-wscript\n\n# Compiled binary addons (https://nodejs.org/api/addons.html)\nbuild/Release\n\n# Dependency directories\nnode_modules/\njspm_packages/\n\n# Snowpack dependency directory (https://snowpack.dev/)\nweb_modules/\n\n# TypeScript cache\n*.tsbuildinfo\n\n# Optional npm cache directory\n.npm\n\n# Optional eslint cache\n.eslintcache\n\n# Optional stylelint cache\n.stylelintcache\n\n# Microbundle cache\n.rpt2_cache/\n.rts2_cache_cjs/\n.rts2_cache_es/\n.rts2_cache_umd/\n\n# Optional REPL history\n.node_repl_history\n\n# Output of 'npm pack'\n*.tgz\n\n# Yarn Integrity file\n.yarn-integrity\n\n# dotenv environment variable files\n.env\n.env.development.local\n.env.test.local\n.env.production.local\n.env.local\n\n# parcel-bundler cache (https://parceljs.org/)\n.cache\n.parcel-cache\n\n# Next.js build output\n.next\nout\n\n# Nuxt.js build / generate output\n.nuxt\ndist\n\n# Gatsby files\n.cache/\n# Comment in the public line in if your project uses Gatsby and not Next.js\n# https://nextjs.org/blog/next-9-1#public-directory-support\n# public\n\n# vuepress build output\n.vuepress/dist\n\n# vuepress v2.x temp and cache directory\n.temp\n.cache\n\n# Docusaurus cache and generated files\n.docusaurus\n\n# Serverless directories\n.serverless/\n\n# FuseBox cache\n.fusebox/\n\n# DynamoDB Local files\n.dynamodb/\n\n# TernJS port file\n.tern-port\n\n# Stores VSCode versions used for testing VSCode extensions\n.vscode-test\n\n# yarn v2\n.yarn/cache\n.yarn/unplugged\n.yarn/build-state.yml\n.yarn/install-state.gz\n.pnp.*\ntask_cmd" }, { "path": ".vscode/launch.json", "content": "{\n // Use IntelliSense to learn about possible attributes.\n // Hover to view descriptions of existing attributes.\n // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387\n \"version\": \"0.2.0\",\n \"configurations\": [\n {\n \"type\": \"lldb\",\n \"request\": \"launch\",\n \"name\": \"Debug\",\n \"program\": \"${workspaceFolder}/\",\n \"args\": [],\n \"cwd\": \"${workspaceFolder}\"\n }\n ]\n}" }, { "path": ".vscode/settings.json", "content": "{\n \"rust-analyzer.linkedProjects\": [\n \"./Cargo.toml\",\n ],\n \"rust-analyzer.cargo.buildScripts.overrideCommand\": null,\n \"emmet.includeLanguages\": {\n \"rust\": \"html\",\n \"*.rs\": \"html\"\n },\n \"tailwindCSS.includeLanguages\": {\n \"rust\": \"html\",\n \"*.rs\": \"html\"\n },\n \"files.associations\": {\n \"*.rs\": \"rust\"\n },\n \"editor.quickSuggestions\": {\n \"other\": \"on\",\n \"comments\": \"on\",\n \"strings\": true\n },\n \"css.validate\": false,\n \"editor.fontWeight\": \"normal\",\n}" }, { "path": "CONTRIBUTING.md", "content": "# Contributing\n\nIf you are contibuting, then thank you, it is appreciated. Below is what requirements a change must have for it to be added, and what information would be useful to have. \n\n## Adding to whitelist\nRequirement(s):\n- Adding domain to whitelist must fix broken functionality/blocked first-party websites, advert domains will not be whitelisted.\n\nInformation needed:\n- What website is blocked/broken, for third-party domains, just the blocked domains that need to be whitelisted isn't enough.\n\n## Adding a new filter list\nRequirement(s):\n- URL must be to original host(unless original no longer exists), not a mirror/processed version\n- License must be compatible with GPLv3\n\nInformation needed:\n- URL to filter list\n- Whether filter list is blacklist or whitelist\n- Whether it is a list of malicious paths(will result in base domains being added)\n\n## Adding a domain to blacklist\nRequirement(s):\n- Must not break pages\n\nInformation needed:\n- Reason for adding domain, eg for adverts, example website using it, for malicious domains, virustotal report or other proof that it is malicious\n" }, { "path": "Cargo.toml", "content": "[package]\nname = \"blockconvert\"\nversion = \"0.1.0\"\nedition = \"2021\"\n\n[lib]\ncrate-type = [\"cdylib\", \"lib\"]\n\n[dependencies]\nserde = { version = \"1.0\", features = [\"derive\", \"rc\"] }\nurl = { version = \"2.5.0\", features = [\"serde\"] }\naxum = { version = \"0.7\", optional = true }\nconsole_error_panic_hook = { version = \"0.1\", optional = true }\nleptos = { git = \"https://github.com/leptos-rs/leptos\", version = \"0.6\", features = [\n \"nonce\",\n] }\nleptos_axum = { git = \"https://github.com/leptos-rs/leptos\", version = \"0.6\", features = [\n \"nonce\",\n], optional = true }\nleptos_meta = { git = \"https://github.com/leptos-rs/leptos\", version = \"0.6\", features = [\n] }\nleptos_router = { git = \"https://github.com/leptos-rs/leptos\", version = \"0.6\", features = [\n] }\ntokio = { version = \"1\", features = [\n \"rt-multi-thread\",\n \"parking_lot\",\n \"process\",\n \"signal\",\n], optional = true }\ntokio-util = { version = \"0.7\", optional = true }\ntower = { version = \"0.4\", optional = true }\ntower-http = { version = \"0.5\", features = [\n \"fs\",\n \"compression-br\",\n \"decompression-br\",\n], optional = true }\nwasm-bindgen = { version = \"=0.2.92\", default-features = false, optional = true }\nthiserror = \"1\"\nhttp = \"1\"\ncsv = { version = \"1.3.0\", optional = true }\nenv_logger = { version = \"0.11.3\", optional = true }\nlog = \"0.4.21\"\nconsole_log = { version = \"1.0\", features = [\"color\"], optional = true }\nsqlx = { version = \"0.7\", features = [\n \"runtime-tokio\",\n \"chrono\",\n \"postgres\",\n \"ipnetwork\",\n], optional = true }\nreqwest = { version = \"0.11.26\", features = [\n \"native-tls\",\n \"brotli\",\n], default-features = false, optional = true }\nchrono = { version = \"0.4\", features = [\"serde\"] }\ndotenvy = { version = \"0.15.7\", optional = true }\nmimalloc = { version = \"0.1.39\", default-features = false, optional = true }\nhickory-resolver = { version = \"0.24.0\", features = [\n \"tokio-runtime\",\n], optional = true }\naddr = \"0.15.6\"\nipnetwork = \"0.20.0\"\nfutures = { version = \"0.3.30\", optional = true }\nhickory-proto = { version = \"0.24.0\", default-features = false }\nhumantime = \"2.1.0\"\nnotify = { version = \"6.1.1\", optional = true }\nasync-channel = { version = \"2.2.0\", optional = true }\ntokio-tungstenite = { version = \"0.21.0\", features = [\n \"native-tls\",\n], optional = true }\nserde_json = { version = \"1.0.114\", optional = true }\nmetrics = { version = \"0.22\", optional = true }\nmetrics-exporter-prometheus = { version = \"0.13.1\", default-features = false, features = [\n \"push-gateway\",\n], optional = true }\nclap = { version = \"4.5.2\", features = [\"derive\"], optional = true }\nrand = {version = \"0.8\", optional = true}\ntoml = {version = \"0.8\", optional = true}\n\n[features]\nhydrate = [\n \"leptos/hydrate\",\n \"leptos_meta/hydrate\",\n \"leptos_router/hydrate\",\n \"dep:console_log\",\n \"dep:console_error_panic_hook\",\n \"dep:wasm-bindgen\",\n]\nssr = [\n \"dep:axum\",\n \"dep:tokio\",\n \"dep:tokio-util\",\n \"dep:tower\",\n \"dep:tower-http\",\n \"dep:leptos_axum\",\n \"leptos/ssr\",\n \"leptos_meta/ssr\",\n \"leptos_router/ssr\",\n \"dep:csv\",\n \"dep:env_logger\",\n \"dep:sqlx\",\n \"dep:reqwest\",\n \"dep:dotenvy\",\n \"dep:mimalloc\",\n \"dns_resolver\",\n \"dep:notify\",\n \"dep:async-channel\",\n \"dep:tokio-tungstenite\",\n \"dep:serde_json\",\n \"dep:futures\",\n \"dep:metrics\",\n \"dep:metrics-exporter-prometheus\",\n \"dep:clap\",\n \"dep:rand\",\n \"dep:toml\",\n]\ndns_resolver = [\"dep:hickory-resolver\"]\ndefault = [\"ssr\"]\n\n# Defines a size-optimized profile for the WASM bundle in release mode\n[profile.wasm-release]\ninherits = \"release\"\nopt-level = 'z'\nlto = true\ncodegen-units = 1\npanic = \"abort\"\n\n[profile.wasm-dev]\ninherits = \"dev\"\ndebug = 0\n\n[profile.wasm-dev.package.\"*\"]\nopt-level = 's'\n\n[profile.server-dev]\ninherits = \"dev\"\nopt-level = 1\ndebug = 0\nlto = \"off\"\n\n[profile.dev.package.\"*\"]\nopt-level = 3\n\n[package.metadata.leptos]\n# The name used by wasm-bindgen/cargo-leptos for the JS/WASM bundle. Defaults to the crate name\noutput-name = \"blockconvert\"\n\n# The site root folder is where cargo-leptos generate all output. WARNING: all content of this folder will be erased on a rebuild. Use it in your server setup.\nsite-root = \"target/blockconvert\"\n\n# The site-root relative folder where all compiled output (JS, WASM and CSS) is written\n# Defaults to pkg\nsite-pkg-dir = \"pkg\"\n\n# [Optional] The source CSS file. If it ends with .sass or .scss then it will be compiled by dart-sass into CSS. The CSS is optimized by Lightning CSS before being written to //app.css\nstyle-file = \"style/main.scss\"\n# Assets source dir. All files found here will be copied and synchronized to site-root.\n# The assets-dir cannot have a sub directory with the same name/path as site-pkg-dir.\n#\n# Optional. Env: LEPTOS_ASSETS_DIR.\nassets-dir = \"public\"\n\n# The IP and port (ex: 127.0.0.1:3000) where the server serves the content. Use it in your server setup.\nsite-addr = \"127.0.0.1:3000\"\n\n# The port to use for automatic reload monitoring\nreload-port = 3001\n\n# [Optional] Command to use when running end2end tests. It will run in the end2end dir.\n# [Windows] for non-WSL use \"npx.cmd playwright test\"\n# This binary name can be checked in Powershell with Get-Command npx\nend2end-cmd = \"npx playwright test\"\nend2end-dir = \"end2end\"\n\n# The browserlist query used for optimizing the CSS.\nbrowserquery = \"defaults\"\n\n# Set by cargo-leptos watch when building with that tool. Controls whether autoreload JS will be included in the head\nwatch = false\n\n# The environment Leptos will run in, usually either \"DEV\" or \"PROD\"\nenv = \"DEV\"\n\n# The features to use when compiling the bin target\n#\n# Optional. Can be over-ridden with the command line parameter --bin-features\nbin-features = [\"ssr\"]\n\n# If the --no-default-features flag should be used when compiling the bin target\n#\n# Optional. Defaults to false.\nbin-default-features = false\n\n\nbin-profile-dev = \"server-dev\"\n\n# The features to use when compiling the lib target\n#\n# Optional. Can be over-ridden with the command line parameter --lib-features\nlib-features = [\"hydrate\"]\n\n# If the --no-default-features flag should be used when compiling the lib target\n#\n# Optional. Defaults to false.\nlib-default-features = false\n\n# The profile to use for the lib target when compiling for release\n#\n# Optional. Defaults to \"release\".\nlib-profile-release = \"wasm-release\"\nlib-profile-dev = \"wasm-dev\"\n\n# The tailwind input file.\n#\n# Optional, Activates the tailwind build\ntailwind-input-file = \"style/tailwind.css\"\n# The tailwind config file.\n#\n# Optional, defaults to \"tailwind.config.js\" which if is not present\n# is generated for you\ntailwind-config-file = \"tailwind.config.js\"\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2022 henrik\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n" }, { "path": "README.md", "content": "# BlockConvert\n\nMalware, advert and tracking blocklist which consolidates and improves upon many other [blocklists](https://github.com/mkb2091/blockconvert/blob/master/filterlists.csv).\n\n\n## What this blocks:\n- Malware/Phishing: Many malware lists are used in building this list, including multiple malware IP lists, which are used to find many more malware domains.\n\n- Adverts: Adblock syntax is partially supported, so this list is able to extract some advert domains. This list is pretty good at blocking adverts, but an in-browser adblocker such as uBlock Origin is recommended as well as relying on hosts/DNS blocking.\n\n- Trackers: Many tracking domains are extracted from the lists used, including Privacy Badger data files which automatically identify trackers.\n\n- Coin mining: A few coin mining blocklists are used to block browser-based coin mining from using cpu.\n\n## Advantages of using this list:\n- Conversion of list types. As well as supporting many common filter list formats, it also supports Privacy Badger data file, which uses algorithms to detect trackers allowing newly created trackers to be quickly detected and added to this blocklist without a human needing to spot the tracker.\n\n- Reverse DNS and passive DNS on malware IP addresses. This allows finding all the domains which a malware IP blacklist suggests could be dangerous to be found and blocked. This allows blocking of malware domains that haven't yet been added to other malware domain lists.\n\n- Use of a whitelist. Using a hosts file doesn't allow whitelisting, and many DNS-based blockers don't have great whitelist support. This list has it's own whitelist, as well as using a few others to try to reduce false positives. This list supports \"*\" in subdomain and TLD to aid in easily fixing many false positives at once. (If you do find a false positive(a domain that shouldn't be blocked), then please make an issue and I will remove it)\n\n- Use of DNS to check if domains still exist. Many lists contain domains that have expired and no longer exist. This makes those lists larger than needed which wastes bandwidth, space and can slow blocking.\n\n## How to use:\n- Pi-hole: Go to the web interface. Log in. Settings -> Blocklists. Copy domain list URL(Pi-hole currently only supports domain lists) from below in the links section, and paste it in the textbox. Click Save.\n\n- Blokada: Open Blokada. Click shield with black middle which says \"{number} in blacklist\". Click the plus in the circle at the bottom of the screen. Copy and paste hosts file from link sections. Click save. WARNING: This list is large and might slow down your phone\n\n- uBlock Origin: Click the uBlock Origin logo/uBlock Origin extension. Click open dashboard(3 horizontal lines under the disable uBlock Origin button, on the right). Click Filter lists. Scroll to the bottom, and click Import(in custom section). Copy and paste the Adblock style blocklist from the link section below.\n\n## Links\n\n[Adblock Plus format](https://mkb2091.github.io/blockconvert/output/adblock.txt)\n\n[Hosts file format](https://mkb2091.github.io/blockconvert/output/hosts.txt)\n\nWARNING: Too large for Windows: https://github.com/mkb2091/blockconvert/issues/87\n\n[Domain list](https://mkb2091.github.io/blockconvert/output/domains.txt)\n\n[Blocked IP address list](https://mkb2091.github.io/blockconvert/output/ip_blocklist.txt)\n\n[DNS Response Policy Zone(RPZ) format](https://mkb2091.github.io/blockconvert/output/domains.rpz)\n\nAs well as generating blocklists, this project also generates whitelists which are used in the process. If you maintain your own blocklist, you may find one of the following whitelists useful:\n\n[Whitelisted domains](https://mkb2091.github.io/blockconvert/output/whitelist_domains.txt)\n\n[Whitelisted ABP format](https://mkb2091.github.io/blockconvert/output/whitelist_adblock.txt)\n\n## The Process\n\n1. Download all expired filterlists\n\n2. Combine and split all the filterlists based on their type. This splits the lines into seperate groups: Adblock rules, blocked domains, regexes of blocked domains, allowed domains, regex of allowed domains, ips which are blocked, ips which are allowed, subnets which are blocked, subnets which are allowed.\n\n3. Apply a regex to all the filterlists to extract domains and combine with other domains found via other means.\n\n4. For each of those domains, use DNS to check if the domain is still active. If the domain isn't in the allowed domains list, doesn't match any of the allowed regexes, isn't in allowed by an adblock exception rule and it is blocked, or one of its cnames/ips is blocked then add it to the output.\n\nSources: [Sources](https://github.com/mkb2091/blockconvert/blob/master/filterlists.csv)\n" }, { "path": "_config.yml", "content": "theme: jekyll-theme-cayman" }, { "path": "build.rs", "content": "// generated by `sqlx migrate build-script`\nfn main() {\n // trigger recompilation when a new migration is added\n println!(\"cargo:rerun-if-changed=migrations\");\n}\n" }, { "path": "end2end/package.json", "content": "{\n \"name\": \"end2end\",\n \"version\": \"1.0.0\",\n \"description\": \"\",\n \"main\": \"index.js\",\n \"scripts\": {},\n \"keywords\": [],\n \"author\": \"\",\n \"license\": \"ISC\",\n \"devDependencies\": {\n \"@playwright/test\": \"^1.28.0\"\n }\n}\n" }, { "path": "end2end/playwright.config.ts", "content": "import type { PlaywrightTestConfig } from \"@playwright/test\";\nimport { devices } from \"@playwright/test\";\n\n/**\n * Read environment variables from file.\n * https://github.com/motdotla/dotenv\n */\n// require('dotenv').config();\n\n/**\n * See https://playwright.dev/docs/test-configuration.\n */\nconst config: PlaywrightTestConfig = {\n testDir: \"./tests\",\n /* Maximum time one test can run for. */\n timeout: 30 * 1000,\n expect: {\n /**\n * Maximum time expect() should wait for the condition to be met.\n * For example in `await expect(locator).toHaveText();`\n */\n timeout: 5000,\n },\n /* Run tests in files in parallel */\n fullyParallel: true,\n /* Fail the build on CI if you accidentally left test.only in the source code. */\n forbidOnly: !!process.env.CI,\n /* Retry on CI only */\n retries: process.env.CI ? 2 : 0,\n /* Opt out of parallel tests on CI. */\n workers: process.env.CI ? 1 : undefined,\n /* Reporter to use. See https://playwright.dev/docs/test-reporters */\n reporter: \"html\",\n /* Shared settings for all the projects below. See https://playwright.dev/docs/api/class-testoptions. */\n use: {\n /* Maximum time each action such as `click()` can take. Defaults to 0 (no limit). */\n actionTimeout: 0,\n /* Base URL to use in actions like `await page.goto('/')`. */\n // baseURL: 'http://localhost:3000',\n\n /* Collect trace when retrying the failed test. See https://playwright.dev/docs/trace-viewer */\n trace: \"on-first-retry\",\n },\n\n /* Configure projects for major browsers */\n projects: [\n {\n name: \"chromium\",\n use: {\n ...devices[\"Desktop Chrome\"],\n },\n },\n\n {\n name: \"firefox\",\n use: {\n ...devices[\"Desktop Firefox\"],\n },\n },\n\n {\n name: \"webkit\",\n use: {\n ...devices[\"Desktop Safari\"],\n },\n },\n\n /* Test against mobile viewports. */\n // {\n // name: 'Mobile Chrome',\n // use: {\n // ...devices['Pixel 5'],\n // },\n // },\n // {\n // name: 'Mobile Safari',\n // use: {\n // ...devices['iPhone 12'],\n // },\n // },\n\n /* Test against branded browsers. */\n // {\n // name: 'Microsoft Edge',\n // use: {\n // channel: 'msedge',\n // },\n // },\n // {\n // name: 'Google Chrome',\n // use: {\n // channel: 'chrome',\n // },\n // },\n ],\n\n /* Folder for test artifacts such as screenshots, videos, traces, etc. */\n // outputDir: 'test-results/',\n\n /* Run your local dev server before starting the tests */\n // webServer: {\n // command: 'npm run start',\n // port: 3000,\n // },\n};\n\nexport default config;\n" }, { "path": "end2end/tests/example.spec.ts", "content": "import { test, expect } from \"@playwright/test\";\n\ntest(\"homepage has title and links to intro page\", async ({ page }) => {\n await page.goto(\"http://localhost:3000/\");\n\n await expect(page).toHaveTitle(\"Welcome to Leptos\");\n\n await expect(page.locator(\"h1\")).toHaveText(\"Welcome to Leptos!\");\n});\n" }, { "path": "filterlists.csv", "content": "name,url,author,license,expires,list_type\n🎮 Game Console Adblock List,https://raw.githubusercontent.com/DandelionSprout/adfilt/master/GameConsoleAdblockList.txt,,Dandelicence,345600,Adblock\nuBlock filters – Resource abuse,https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/resource-abuse.txt,,GPLv3,345600,Adblock\nuBlock filters – Privacy,https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt,,GPLv3,345600,Adblock\nuBlock filters – Badware risks,https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt,,GPLv3,345600,Adblock\nuBlock filters,https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt,,GPLv3,345600,Adblock\nmobiletrackers,https://raw.githubusercontent.com/craiu/mobiletrackers/master/list.txt,,GPLv3,604800,DomainBlocklist\niOS Tracker Blocklist,https://raw.githubusercontent.com/jakejarvis/ios-trackers/master/blocklist.txt,jakejarvis,MIT,864000,DomainBlocklistWithoutSubdomains\nhostsVN,https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts,,MIT,86400,Hostfile\nabuse.ch URLhaus Response Policy Zones,https://urlhaus.abuse.ch/downloads/hostfile,,CC0,86400,Hostfile\nabuse.ch SSLBL Botnet C2 IP Blacklist,https://sslbl.abuse.ch/blacklist/sslipblacklist.txt,,CC0,86400,IPBlocklist\nabuse.ch Feodo Tracker Botnet C2 IP Blocklist,https://feodotracker.abuse.ch/downloads/ipblocklist.txt,,CC0,86400,IPBlocklist\nXiaomi DNS Blocklist,https://raw.githubusercontent.com/unknownFalleN/xiaomi-dns-blocklist/master/xiaomi_dns_block.lst,,GPLv3,604800,DomainBlocklist\nWindowsSpyBlocker - Hosts spy rules,https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt,,MIT,86400,Hostfile\nWhitelist,https://raw.githubusercontent.com/ookangzheng/blahdns/master/hosts/whitelist.txt,ookangzheng,CC BY-NC-SA 4.0(I received permission to add to this project),86400,DomainAllowlist\nWhitelist,https://raw.githubusercontent.com/NoExitTV/whitelist/master/domains/whitelist.txt,,MIT,604800,DomainAllowlist\nTrackers,https://git.herrbischoff.com/trackers/plain/trackers.txt,,The Unlicense,864000,DomainBlocklistWithoutSubdomains\nThe Ultimate Hosts Blacklist whitelist,https://raw.githubusercontent.com/Ultimate-Hosts-Blacklist/whitelist/master/domains.list,,MIT,86400,DomainAllowlist\nThe Hosts File Project,https://hblock.molinero.dev/hosts,Héctor Molinero Fernández ,MIT,86400,Hostfile\nThe Block List Project Tracking,https://raw.githubusercontent.com/blocklistproject/Lists/master/tracking.txt,,The Unlicense,86400,Hostfile\nThe Block List Project Phishing,https://raw.githubusercontent.com/blocklistproject/Lists/master/phishing.txt,,The Unlicense,86400,Hostfile\nThe Block List Project Malware,https://raw.githubusercontent.com/blocklistproject/Lists/master/malware.txt,,The Unlicense,86400,Hostfile\nThe Block List Project Ads,https://raw.githubusercontent.com/blocklistproject/Lists/master/ads.txt,,The Unlicense,86400,Hostfile\nStevenBlack/hosts,https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts,,MIT,86400,Hostfile\nSlovenian List,https://raw.githubusercontent.com/betterwebleon/slovenian-list/master/filters.txt,,The Unlicense,259200,Adblock\nScams and Phishing,https://raw.githubusercontent.com/infinitytec/blocklists/master/scams-and-phishing.txt,infinitytec,MIT,864000,Hostfile\nScam Blocklist by DurableNapkin,https://raw.githubusercontent.com/durablenapkin/scamblocklist/master/hosts.txt,DurableNapkin,MIT,86400,Hostfile\nRegex Filters for Pi-hole Whitelist,https://raw.githubusercontent.com/mmotti/pihole-regex/master/whitelist.list,,With Permission from https://github.com/mmotti/pihole-regex/issues/30,864000,DomainAllowlist\nRegex Filters for Pi-hole,https://raw.githubusercontent.com/mmotti/pihole-regex/master/regex.list,,With Permission from https://github.com/mmotti/pihole-regex/issues/30,864000,RegexBlocklist\nPrivacy filters,https://raw.githubusercontent.com/metaphoricgiraffe/tracking-filters/master/trackingfilters.txt,,The Unlicense,86400,Adblock\nPossibilities,https://raw.githubusercontent.com/infinitytec/blocklists/master/possibilities.txt,infinitytec,MIT,864000,Hostfile\nPiHoleBlocklist - SmartTV,https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt,,MIT,86400,DomainBlocklist\nPiHoleBlocklist - SessionReplay,https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SessionReplay.txt,,MIT,86400,DomainBlocklist\nPiHoleBlocklist - AndroidTracking,https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt,,MIT,86400,DomainBlocklist\nPiHoleBlocklist - AmazonFireTV,https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt,,MIT,86400,DomainBlocklist\nPi-Hole blocklist,https://codeberg.org/spootle/blocklist/raw/branch/master/blocklist.txt,,GPL-3.0,86400,DomainBlocklist\nPerflyst's SmartTV Blocklist for Pi-hole - RegEx extension,https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/regex.list,,MIT,86400,RegexBlocklist\nNopelist,https://raw.githubusercontent.com/genediazjr/nopelist/master/nopelist.txt,Gene Diaz,MIT,604800,DomainBlocklist\nNoTrack Tracker Blocklist,https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt,QuidsUp,GPLv3,86400,DomainBlocklist\nNoTrack Malware Blocklist,https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-malware.txt,QuidsUp,GPLv3,86400,DomainBlocklist\nNoCoin Filter List,https://raw.githubusercontent.com/hoshsadiq/adblock-nocoin-list/master/hosts.txt,,MIT,86400,Hostfile\nMinerBlock Filters,https://raw.githubusercontent.com/xd4rker/MinerBlock/master/assets/filters.txt,,MIT,86400,Adblock\nMalicious URL Blocklist,https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-agh.txt,,CC0,86400,Adblock\nMagneto Malware Skanner - Burner Domains,https://raw.githubusercontent.com/gwillem/magento-malware-scanner/master/rules/burner-domains.txt,,GPLv3,864000,DomainBlocklist\nLightswitch05's Ads & Tracking,https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt,Daniel White,Apache2,172800,Hostfile\nLatvian List,https://notabug.org/latvian-list/adblock-latvian/raw/master/lists/latvian-list.txt,,CC-BY-SA-4.0,86400,Adblock\nKADhosts,https://raw.githubusercontent.com/PolishFiltersTeam/KADhosts/master/KADhosts.txt,,CC-BY-SA-4,86400,Hostfile\nInternational List,https://raw.githubusercontent.com/betterwebleon/international-list/master/filters.txt,,The Unlicense,259200,Adblock\nI Hate Tracker,https://raw.githubusercontent.com/pirat28/IHateTracker/master/iHateTracker.txt,Pirat28,MIT,864000,DomainBlocklist\nHerm's Ad Black List,https://raw.githubusercontent.com/hermanjustinm/Herms-Blacklist/master/HermsAdBlacklist.txt,,MPL-2.0,864000,Adblock\nHackerList,https://pfblockerlists.smallbusinesstech.net/hackerlist.txt,Soren Stoutner,GPLv3,86400,IPBlocklist\nFrellwit's Swedish Hosts File,https://raw.githubusercontent.com/lassekongo83/Frellwits-filter-lists/master/Frellwits-Swedish-Hosts-File.txt,,GPL-3.0,86400,Hostfile\nFirst-party trackers host list,https://hostfiles.frogeye.fr/multiparty-trackers.txt,,MIT,604800,DomainBlocklist\nFanboy's Annoyance List,https://easylist.to/easylist/fanboy-annoyance.txt,,GPLv3,345600,Adblock\nEasyPrivacy,https://easylist.to/easylist/easyprivacy.txt,,GPLv3,345600,Adblock\nEasyList Italy,https://easylist-downloads.adblockplus.org/easylistitaly.txt,,GPLv3,86400,Adblock\nEasyList Hebrew,https://raw.githubusercontent.com/easylist/EasyListHebrew/master/EasyListHebrew.txt,,GPLv3,86400,Adblock\nEasyList Germany,https://easylist.to/easylistgermany/easylistgermany.txt,,GPLv3,86400,Adblock\nEasyList Dutch,https://easylist-downloads.adblockplus.org/easylistdutch.txt,,GPLv3,345600,Adblock\nEasyList China,https://easylist-downloads.adblockplus.org/easylistchina.txt,,GPLv3,345600,Adblock\nEasyList,https://easylist.to/easylist/easylist.txt,,GPLv3,345600,Adblock\nCommonly white listed domains for Pi-Hole,https://raw.githubusercontent.com/anudeepND/whitelist/master/domains/whitelist.txt,,MIT,86400,DomainAllowlist\nCJX's Annoyance List,https://raw.githubusercontent.com/cjx82630/cjxlist/master/cjx-annoyance.txt,,LGPLv3,345600,Adblock\nCB-Malicious-Domains,https://raw.githubusercontent.com/cb-software/CB-Malicious-Domains/master/block_lists/domains_only.txt,,MIT,86400,DomainBlocklist\nBlockConvert Internal IP Blocklist,internal/block_ips.txt,mkb2091,MIT License,30,IPBlocklist\nBlockConvert Internal Blocklist,internal/blocklist.txt,mkb2091,MIT License,30,DomainBlocklist\nBlockConvert Internal Allowlist,internal/allowlist.txt,mkb2091,MIT License,30,DomainAllowlist\nBasic tracking list by Disconnect,https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt,,GPLv3,86400,DomainBlocklist\nBarbBlock,https://paulgb.github.io/BarbBlock/blacklists/hosts-file.txt,,MIT,86400,DomainBlocklist\nBAR-list,https://raw.githubusercontent.com/zznidar/BAR/master/BAR-list,,GPL-3.0,864000,DomainBlocklist\nAnudeep's Blacklist,https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt,Anudeep ,MIT,86400,Hostfile\nAnti-WebMiner,https://raw.githubusercontent.com/greatis/Anti-WebMiner/master/blacklist.txt,,Apache2,86400,DomainBlocklist\nAds and Trackers,https://raw.githubusercontent.com/infinitytec/blocklists/master/ads-and-trackers.txt,infinitytec,MIT,864000,Hostfile\nAdguardMobileAds,https://raw.githubusercontent.com/r-a-y/mobile-hosts/master/AdguardMobileAds.txt,,GPLv3,86400,Hostfile\nAdguardDNS,https://raw.githubusercontent.com/r-a-y/mobile-hosts/master/AdguardDNS.txt,,GPLv3,86400,Hostfile\nAdguardApps,https://raw.githubusercontent.com/r-a-y/mobile-hosts/master/AdguardApps.txt,,GPLv3,86400,Hostfile\nAdblock List for Finland,https://raw.githubusercontent.com/finnish-easylist-addition/finnish-easylist-addition/master/Finland_adb.txt,,The Unlicense,432000,Adblock\nAdGuard Simplified Domain Names filter,https://adguardteam.github.io/AdGuardSDNSFilter/Filters/filter.txt,,GPLv3,86400,Adblock\nAdBlock Farsi,https://raw.githubusercontent.com/SlashArash/adblockfa/master/adblockfa.txt,,The Beer-Ware License,432000,Adblock\nAdAway default blocklist,https://adaway.org/hosts.txt,,CC-BY-3,86400,Hostfile\nAd filter list by Disconnect,https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt,,GPLv3,86400,DomainBlocklist\nAZORult Tracker,https://azorult-tracker.net/api/list/domain?format=plain,,Open Database License,86400,DomainBlocklist\nABPindo,https://raw.githubusercontent.com/ABPindo/indonesianadblockrules/master/subscriptions/abpindo.txt,,GPLv3,345600,Adblock\n,https://zerodot1.gitlab.io/CoinBlockerLists/list_browser.txt,,GPLv3,86400,DomainBlocklist\n,https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt,,GPLv3,86400,DomainBlocklist\n,https://raw.githubusercontent.com/xxcriticxx/.pl-host-file/master/hosts.txt,,GPLv3,86400,Hostfile\n,https://raw.githubusercontent.com/mitchellkrogza/The-Big-List-of-Hacked-Malware-Web-Sites/master/hacked-domains.list,Mitchell Krog,MIT,86400,DomainBlocklist\n,https://raw.githubusercontent.com/matomo-org/referrer-spam-blacklist/master/spammers.txt,,Public Domain,86400,DomainBlocklist\n,https://raw.githubusercontent.com/ligyxy/Blocklist/master/BLOCKLIST,,MIT,86400,DomainBlocklist\n,https://raw.githubusercontent.com/EFForg/privacybadger/master/src/data/yellowlist.txt,,GPLv3+,86400,DomainAllowlist\n,https://cdn.jsdelivr.net/gh/realodix/AdBlockID@master/dist/adblockid.adfl.txt,,,86400,Adblock\n,https://blocklists.kitsapcreator.com/scam-spam.txt,,The Unlicense,86400,DomainBlocklist\n,https://blocklists.kitsapcreator.com/malware-malicious.txt,,The Unlicense,86400,DomainBlocklist\n,https://blocklists.kitsapcreator.com/general.txt,,The Unlicense,86400,DomainBlocklist\n,https://blocklists.kitsapcreator.com/ads.txt,,The Unlicense,86400,DomainBlocklist\n" }, { "path": "internal/adblock.txt", "content": "||cedexis.net^$third-party,badfilter\n||episerver.net^$third-party,badfilter" }, { "path": "internal/allow_regex.txt", "content": "^gsp[0-9][0-9]-ssl\\.ls\\.apple\\.com$\n" }, { "path": "internal/allowlist.txt", "content": "*.a-msedge.net\n*.adafruit.com\n*.b.akamaiedge.net\n*.bitcoin.it\n*.burst-alliance.org\n*.c-msedge.net\n*.cloudflare-dns.com # Cloudflare dns server\n*.coingeek.com\n*.delivery.mp.microsoft.com\n*.dl.delivery.mp.microsoft.com\n*.dsx.mp.microsoft.com\n*.e-msedge.net\n*.g.akamai.net\n*.g.akamaiedge.net\n*.gab.com\n*.md.mp.microsoft.com.*\n*.mp.microsoft.com\n*.prod.do.dsp.mp.microsoft.com\n*.s-msedge.net\n*.safelinks.protection.outlook.com\n*.search.msn.com\n*.simply.com\n*.skype.com\n*.smartscreen.microsoft.com\n*.splunk.com\n*.storage.live.com\n*.telecommand.telemetry.microsoft.com.akadns.net\n*.tlu.dl.delivery.mp.microsoft.com\n*.ugc.bazaarvoice.com\n*.update.microsoft.com\n*.vk.com\n*.windowsupdate.com\n*.wns.windows.com\n1.www.s81c.com\n1drv.ms\n2-01-2c3e-003d.cdx.cedexis.net # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-783295583 Breaks LinkedIn.com\n24timezones.com\n33dim-trikal.tri.sch.gr https://github.com/mkb2091/blockconvert/issues/117\n3p.ampproject.net\n79423.analytics.edgekey.net\n91-cdn.com\n91mobiles.com\na-msedge.net\na.espncdn.com # https://github.com/mkb2091/blockconvert/issues/58 Breaks ESPN\na248.e.akamai.net\naaa.com\naax-us-east.amazon-adsystem.com\naccounts.eu1.gigya.com\naccounts.us1.gigya.com # Needed for iRobot app https://github.com/mkb2091/blockconvert/issues/3#issuecomment-683157232\nactivation-v2.sls.microsoft.com\nactivation.sls.microsoft.com\nada.support # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks Cricket cell phone customer support\nadapools.org # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-809546196 Crypto staking\nadaway.org\nadblock.ee\nadd0n.com\naddons.mozilla.org\nadguard.com\nadguard.com # Adblocking tool\nadguardteam.github.io\nadl.windows.com # https://github.com/mkb2091/blockconvert/issues/92 Breaks Windows Update\naeriagames.com\naetna.com # blocks www.aetna.com which is a health website\naftership.com\naka.ms\nakismet.com\nalexa.com\naliexpress.com\nalive.github.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-867284255\nallure.com\namazon.com\nandroidauthority.com\nanswers.com\nanswers.microsoft.com\napi.bing.com\napi.cdp.microsoft.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-739536391 Required for Microsoft Edge Updates\napi.facebook.com\napi.mobile.immobilienscout24.de # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-715523256 Breaks the immoscout24 app\napi.mymonero.com\napi.particle.io\napi.qrserver.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-780859152 Breaks setting up Google 2FA\napi.vanilla.futurecdn.net\napi.videos.oovvuu.com\nappleid.apple.com.akadns.net\nappspot-preview.l.google.com\napt.newrelic.com\narc.msn.com\narc.msn.com.nsatc.net\narizona.edu\nars.smartscreen.microsoft.com\nassets.bwbx.io # Breaks bloomberg.com layout\nassets.micpn.com\nassets.penny-arcade.com\nassets.zendesk.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-867284255\natt.ada.support # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks Cricket cell phone customer support\nattfeedback.uservoice.com\nau.download.windowsupdate.com\nauth.gfx.ms\nawin1.com\nay.gy\naz416426.vo.msecnd.net # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-744499191 Breaks Airplus Mastercard software\nazorult-tracker.net\nb-graph.facebook.com\nb.akamaiedge.net\nbadges.twitch.tv # https://github.com/mkb2091/blockconvert/issues/94 Breaks Twitch TV badges\nbaidu.com\nbattle.net\nbbci.co.uk\nbehance.net # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-703781874 Not malware\nberush.com\nbestbuy.ca\nbetfair.com\nbetter.fyi\nbgp.he.net # https://github.com/mkb2091/blockconvert/issues/86 Part of Hurricane Electric's Internet Toolkit\nbilder.bild.de # https://github.com/mkb2091/blockconvert/issues/57 Breaks images on https://www.bild.de/\nbinarydefense.com\nbing.com\nbitbucket.org\nbitcoin.com\nbitcoin.it\nbitlord.com\nbitly.com\nblackhatworld.com\nblob.weather.microsoft.com\nblog.hubspot.com\nblog.tube8.com\nblogs.sch.gr # https://github.com/mkb2091/blockconvert/issues/117\nbountysource.com\nbrightcove.map.fastly.net # https://github.com/mkb2091/blockconvert/issues/78 Breaks Brightcove video\nbrowser.pipe.aria.microsoft.com\nc-msedge.net\nc.footprint.net # https://github.com/mkb2091/blockconvert/issues/88 Blocks https://www.24ur.com/ on certain systems\nc.paypal.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-939979052 Breaks PayPal on some websites\ncandycrushsoda.king.com\ncdn.ampproject.org # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-687829847 CDN for AMP\ncdn.content.prod.cms.msn.com # Needed for Windows 10 News Images https://github.com/mkb2091/blockconvert/issues/3#issuecomment-687662575\ncdn.onenote.net\ncdn.privacy-mgmt.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-701373707 Breaks closing of a popup\ncdn2.editmysite.com\ncdnjs.cloudflare.com\ncdnp0.stackassets.com # https://github.com/mkb2091/blockconvert/issues/64 Static assets for stacksocial.com\ncdnp1.stackassets.com # https://github.com/mkb2091/blockconvert/issues/64 Static assets for stacksocial.com\ncdnp2.stackassets.com # https://github.com/mkb2091/blockconvert/issues/64 Static assets for stacksocial.com\ncdnp3.stackassets.com # https://github.com/mkb2091/blockconvert/issues/64 Static assets for stacksocial.com\nce1.uicdn.net # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-769915219 Breaks login for many www.ionos.de and www.1und1.de services\ncesanta.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-819106039 Not malware\nchange.org\ncheckappexec.microsoft.com\nchng.it\nchristianpost.com\nci4.googleusercontent.com\nci5.googleusercontent.com\nclickondetroit.com # https://github.com/mkb2091/blockconvert/issues/82 News site\nclient-office365-tas.msedge.net\nclient-s.gateway.messenger.live.com\ncloudflare-dns.com\ncloudfront.net\ncmail19.com\ncmail20.com\ncn-geo1.uber.com # https://github.com/mkb2091/blockconvert/issues/71 Breaks Uber geolocation\ncnet.com\ncode.jquery.com\ncode.tidio.co\ncode.visualstudio.com\ncodeberg.org\ncognito-identity.us-east-1.amazonaws.com # https://github.com/mkb2091/blockconvert/issues/101 Breaks AWS app login\ncoinpot.co\ncointraffic.io\ncompletion.amazon.com\nconfig.edge.skype.com\nconfirmit.com\ncontent.invisioncic.com\ncountryflags.io\ncredit.finance.intuit.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-788266121 Breaks checking credit score on Mint Mobile App\ncricket.att.ada.support # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks Cricket cell phone customer support\ncrunchyroll.com\nct.sendgrid.net\nctldl.windowsupdate.com\ncv.rdtcdn.com\ncy2.displaycatalog.md.mp.microsoft.com.akadns.net\ncy2.licensing.md.mp.microsoft.com.akadns.net\ncy2.settings.data.microsoft.com.akadns.net\ncybercrime-tracker.net\nd2405b0jymm2dk.cloudfront.net\nd25xi40x97liuc.cloudfront.net # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-798791030 Breaks Amazon Video\nd2gatte9o95jao.cloudfront.net\ndailymail.co.uk # Blocked by PrivacyBadger\ndatadoghq-browser-agent.com\ndege.freeweb.hu # https://github.com/mkb2091/blockconvert/issues/81 Site that hosts tools for running old games\ndelivery.mp.microsoft.com\ndesktop.google.com\ndev.maxmind.com # Used for MaxMind developer guide\ndeveloper.spotify.com # Breaks Spotify\ndigicert.com\ndigitalriver.com\ndim-aigeir.ach.sch.gr # https://github.com/mkb2091/blockconvert/issues/117\ndiscord.com\ndiscuss.newrelic.com\ndisplaycatalog.mp.microsoft.com\ndisq.us\ndistilnetworks.com\ndjvbdz1obemzo.cloudfront.net\ndl.delivery.mp.microsoft.com\ndlsrc.getmonero.org\ndm3p.wns.notify.windows.com.akadns.net\ndmd.metaservices.microsoft.com\ndmd.metaservices.microsoft.com.akadns.net\ndmqdd6hw24ucf.cloudfront.net # https://github.com/mkb2091/blockconvert/issues/63 Breaks Amazon Prime Subtitles\ndns.google.com\ndo.co\ndonate.getmonero.org\ndownload.db-ip.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-779885164 For Geo IP database download\ndownload.electrum.org\ndownload.maxmind.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-749884660 Used to download MaxMind GeoIP files\ndownload.newrelic.com\ndownload.visualware.com # https://github.com/mkb2091/blockconvert/issues/123\ndownload.windowsupdate.com\ndownload.windowsupdate.com.c.footprint.net\ndownloads.getmonero.org\ndpreview.com\ndrh.img.digitalriver.com\ndrh1.img.digitalriver.com\ndrudgereport.com\ndsx.mp.microsoft.com\nduckdns.org\nduckduckgo.com\nduken.nl # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-869051876 A forum, not malware\ne-msedge.net\ne785s8hz.micpn.com\neasylist-downloads.adblockplus.org\neasylist.to\necdinterface.philips.com\neec.crunchyroll.com\nelectrum.org\nelinkeu.clickdimensions.com\neloqua.com\nembed.spotify.com # Breaks Spotify\nemdl.ws.microsoft.com\nencrypted-tbn1.gstatic.com\nencrypted-tbn2.gstatic.com\nencrypted-tbn3.gstatic.com\nentireweb.com\neu-central-1.protection.sophos.com # https://github.com/mkb2091/blockconvert/issues/93 Its a cloud security platform\nev.rdtcdn.com\neverest.castbox.fm\nevoke-windowsservices-tas.msedge.net\nf.chtah.com\nf1.media.brightcove.com # https://github.com/mkb2091/blockconvert/issues/78 Breaks Brightcove video\nfaast.schibsted.io\nfacebook.com\nfandom.wikia.com\nfast.fonts.net\nfast.wistia.net\nfaucethub.io\nfe2.update.microsoft.com\nfe2.update.microsoft.com.akadns.net\nfe3.delivery.dsp.mp.microsoft.com.nsatc.net\nfe3.delivery.mp.microsoft.com\nfeodotracker.abuse.ch\nfiddle.jshell.net\nfide.com\nfiles.catbox.moe\nfiles.freedownloadmanager.org\nfiles.slack.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-771638966 Breaks user uploaded content on Slack\nfiles2.freedownloadmanager.org\nfilterlists.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-787428332 Website with list of filterlists\nfiltri-dns.ga\nfirebaseremoteconfig.googleapis.com # Breaks android applications internally updating\nfiverr.com\nflipboard.com\nforms.office.com\nforum.getmonero.org\nforum.overclockers.co.uk # PC forum\nforums.storagereview.com\nfossbytes.com\nfreedownloadmanager.org\nfs.microsoft.com\nftpcontent.worldnow.com\nfunk.eu\ng.akamai.net\ng.akamaiedge.net\ng.live.com\ng.msn.com.nsatc.net\ngadgetsnow.com\ngameofbitcoins.com\ngamespot.com\ngannettdigital.com\ngardenstatehelicopters.com\ngateway.reddit.com\ngcs.sc-cdn.net\ngeo-prod.do.dsp.mp.microsoft.com\ngeo-prod.dodsp.mp.microsoft.com.nsatc.net\ngetmonero.org\ngithub.com\ngithub.developerdan.com\ngitlab.com\ngleam.io # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993\nglobal.edge.bamgrid.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-687908780 Blocks access to Disney Plus\nglobal.ssl.fastly.net\ngo.microsoft.com\ngo2cloud.org\ngoogle.*\ngoogle.co.*\ngoogle.com.*\ngoogletagmanager.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-780859152 Breaks setting up Google 2FA\ngpticketshop.com # https://github.com/mkb2091/blockconvert/issues/111\ngs-loc.apple.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-695044303 Needed for Apple geolocation\ngsp-ssl-geomap.ls-apple.com.akadns.net # Breaks Apple Maps\ngsp-ssl.ls-apple.com.akadns.net # Breaks Apple Maps\ngsp-ssl.ls.apple.com # Break Apple Maps\nguce.advertising.com\nguce.huffpost.com\nh-ebay.online-metrix.net # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks EBay external logins\nhblock.molinero.dev\nhelp.mint.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks Mint help\nheureka.cz\nhighcharts.com\nhomedepot.tt.omtrdc.net\nhostsfile.mine.nu\nhowtogeek.com\nhubpages.com\nhwcdn.net\ni-am3p-cor003.api.p001.1drv.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks OneDrive/OutLook file storage\ni.ibb.co\nibm.com\nign.com\nignorelist.com\nimage.prntscr.com\nimages.gounlimited.to\nimagizer.imageshack.com\nimg-prod-cms-rt-microsoft-com.akamaized.net\nimg-s-msn-com.akamaized.net # https://github.com/mkb2091/blockconvert/issues/67 Breaks msn.com images\nin.appcenter.ms # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-687908780 Blocks access to Outlook 365 mail\ninc.com\ninference.location.live.net\ninformer.com\ninfowars.com\ninfowarsstore.com\ninsight.rapid7.com\ninsights.eu.newrelic.com\ninstagram.com\nint.whiteboard.microsoft.com\nio9.com\nip-adress.com\nip-tracker.org\nip107307705.ahcdn.com # https://github.com/mkb2091/blockconvert/issues/75 Breaks videos on streaming XXX site\nip107316446.ahcdn.com # https://github.com/mkb2091/blockconvert/issues/75 Breaks videos on streaming XXX site\nipv4.login.msa.akadns6.net\nit.toolbox.com\nitch.io\njaleco.com # Breaks download link for GIMP\njarv.is\njoblo.com\njpost.com # News website\njs.t.sinajs.cn\nkaspersky.com\nkasperskycontenthub.com\nknowyourmeme.com\nku6.com\nlarati.net\nleagueoflegends.com\nleanplum.com\nlicensing.mp.microsoft.com\nlinkedin.com\nlinks.mint.com\nlinktr.ee\nlivejasmin.com\nlivejournal.com\nlocation-inference-westus.cloudapp.net\nlocation.services.mozilla.com\nlogin.live.com\nlogin.msa.akadns6.net\nlogin.windows.net\nlogincdn.msauth.net\nlowendbox.com\nm.me\nm.stripe.network\nmail.com\nmail.getmonero.org\nmail.google.com\nmailchi.mp\nmalwaredomainlist.com\nmanager-magazin-de.manager-magazin.de # https://github.com/mkb2091/blockconvert/issues/60 Breaks accepting cookie banner\nmanifest.prod.boltdns.net # https://github.com/mkb2091/blockconvert/issues/66 Breaks BrightCove player\nmaps.windows.com\nmbl.is\nmd.mp.microsoft.com.*\nmedia.pearsoncmg.com\nmediafire.com\nmediaredirect.microsoft.com\nmine.nu\nmiro.medium.com # https://github.com/mkb2091/blockconvert/issues/77 and https://github.com/mkb2091/blockconvert/issues/3#issuecomment-748449378, Breaks images on medium.com\nmirror.co.uk\nmirrorservice.org # Breaks VLC update download\nmobile.tube8.com\nmobile.twitter.com\nmodern.watson.data.microsoft.com.akadns.net\nmonero.org\nmoonbit.co.in\nmotherboard.vice.com\nmovable-ink-397.com\nmp.microsoft.com\nmscrl.microsoft.com\nmsedge.api.cdp.microsoft.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-729147310 Needed for Microsoft Edge to check for updates\nmsedge.net\nmsftconnecttest.com\nmvaali1.ws.byteoversea.net\nmw1.wsj.net # https://github.com/mkb2091/blockconvert/issues/50\nmy.matterport.com # https://github.com/mkb2091/blockconvert/issues/72 Breaks a 3D space capture platform\nmymonero.com\nnamecheap.com\nnarkive.com\nnaver.com\nnetfort.com\nnewrelic.com\nnews.bitcoin.com\nnewyorker.com # News website\nnext-services.apps.microsoft.com\nnexus.ensighten.com\nnexusrules.officeapps.live.com\nning.com\nnotabug.org\nnotifications.google.com\nnow.sh\nocation-inference-westus.cloudapp.net\nocos-office365-s2s.msedge.ne\nocos-office365-s2s.msedge.net\nocsp.digicert.com\nocsp.verisign.com # https://github.com/mkb2091/blockconvert/issues/94 Breaks verifying validity of certificates\nod.lk\noem.twimg.com\nofficeclient.microsoft.com\nogs.google.com\noneclient.sfx.ms\nonesignal.com\nonion.to\nonline.jimmyjohns.com\nopen.spotify.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-788015015 Breaks Spotify\nopenwall.com\noracle.com # Breaks java download\norbitz.com\nosce11-0-en.url.trendmicro.com\noutlook.office365.com\noverclockers.co.uk # PC forum\npastebin.com\npastebin.org\npasteboard.co\npbs.twimg.com\npcmag.com # Blocked by PrivacyBadger\npcmatic.com\nphishtank.com\nplay.spotify.com # Breaks Spotify\nplay.spotify.edgekey.net # Breaks Spotify\npolyfill.io\npool.ntp.org\npornhub.com\npost.spmailtechnol.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Used for links in email\npowr.io\nprivateinternetaccess.com\nprod.do.dsp.mp.microsoft.com\nprod.telemetry.ros.rockstargames.com\nproofpoint.com\npti.store.microsoft.com\npurchase.mp.microsoft.com\npush.prod.netflix.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-809546196 Breaks Netflix DRM\npushbullet.com # https://github.com/mkb2091/blockconvert/issues/115\nqq.com\nquery.prod.cms.rt.microsoft.com\nquickenloans.com\nquora.com\nrakuten.com\nranker.com\nransomwaretracker.abuse.ch\nraw.githubusercontent.com\nreadcomiconline.to\nrecruitics.com\nreddit.app.link\nreddit.com\nredditstatic.s3.amazonaws.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-687908780 Breaks something\nredtube.com\nrefinery29.com # Blog website\nreport-uri.com\nrest-prde.immedia-semi.com\nreturnpath.com\nris-prod-atm.trafficmanager.net\nris.api.iris.microsoft.com\nris.api.iris.microsoft.com.akadns.net\nrouter.utorrent.com\nrumble.com # https://github.com/mkb2091/blockconvert/issues/76 Video Streaming Platform\ns-media-cache-ak0.pinimg.com\ns-msedge.net\ns.imgur.com\ns.ndemiccreations.com\ns.tradingview.com # https://github.com/mkb2091/blockconvert/issues/49\ns0.ipstatp.com # https://github.com/mkb2091/blockconvert/issues/83 Breaks Captcha\ns3-ap-southeast-1.amazonaws.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-793400847 Breaks AWS S3 in SEA\ns3-eu-west-1.amazonaws.com\ns3-us-west-1.amazonaws.com\ns3.amazonaws.com\ns3.tradingview.com # https://github.com/mkb2091/blockconvert/issues/49\ns7.orientaltrading.com\nsafebrowsing-cache.google.com\nsafebrowsing.google.com\nsat1.us1-1.nanorep.co # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks Mint help\nscrapinghub.com\nsdks.shopifycdn.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-707148021 Needed for webshops\nsearch.msn.com\nsearchenginejournal.com\nsecure.indeed.com # https://github.com/mkb2091/blockconvert/issues/84 Breaks login page for indeed.com\nsecure.mbl.is\nsemrush.com\nsendgrid.com # https://github.com/mkb2091/blockconvert/issues/70 Base domain is not used for tracking\nserial.alcohol-soft.com # https://github.com/mkb2091/blockconvert/issues/80 CD and DVD burning software\nservice.weather.microsoft.com\nsettings-win.data.microsoft.com\nsettings.data.microsoft.com\nsfdataservice.microsoft.com # Blocks Microsoft store payments\nsh.st\nshop.app # https://github.com/mkb2091/blockconvert/issues/98#issue-940336845 URL shortener for shopify\nshop.gadgetsnow.com\nsi6ling.incestflix.com # https://github.com/mkb2091/blockconvert/issues/65 Static Assets\nsimilarweb.com\nsina.com.cn\nsinaimg.cn # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-758476709 Breaks images on Chinese social media Weibo\nsj14.mktossl.com # CNAME for safe.riskiq.com\nskype.com\nskypeecs-prod-usw-0-b.cloudapp.net\nsls.update.microsoft.com\nsls.update.microsoft.com.akadns.net\nsmallseotools.com\nsmartscreen-sn3p.smartscreen.microsoft.com\nsmartscreen.microsoft.com\nsocialize.us1.gigya.com # Needed for iRobot app https://github.com/mkb2091/blockconvert/issues/3#issuecomment-683157232\nsoftware.informer.com\nsohu.com\nsom.aeroplan.com\nsoundcloud.com\nsourceforge.net\nspcweb.ch # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-808365475 Breaks swiss parcel tracking\nspeedcurve.com\nspiegel-de.spiegel.de # https://github.com/mkb2091/blockconvert/issues/61\nspotify.com # Breaks Spotify\nsquidblacklist.org\nsrc.ebay-us.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks EBay external logins\nssl.bblck.me\nssl.marfeelcdn.com\nsslbl.abuse.ch\nstacksocial.com\nstaging-discuss.newrelic.com\nstar-mini.c10r.facebook.com\nstatic-dot-virustotalcloud.appspot.com\nstatic-exp1.licdn.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-783295583 Breaks LinkedIn.com\nstatic.bbci.co.uk\nstatic.getmonero.org\nstatic.matterport.com # https://github.com/mkb2091/blockconvert/issues/72 Breaks a 3D space capture platform\nstatic.tacdn.com # https://github.com/mkb2091/blockconvert/issues/52\nstatic.tidiochat.com\nstatic.wikia.nocookie.net # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-744090132 Breaks images on fandom.com wikis\nstatic.zdassets.com\nstatics.streamable.com\nstats.foldingathome.org # https://github.com/mkb2091/blockconvert/issues/74 Regex False Positive\nstats.govt.nz # # https://github.com/mkb2091/blockconvert/issues/99 Government of New Zealand Statistics site\nstats.pingdom.com\nstats.stackexchange.com\nstatsfe2.update.microsoft.com.akadns.net\nstatus.newrelic.com\nstatus.roll20.net\nstatuscake-email.com\nstatuscake.com\nstorage.live.com # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-772293993 Breaks OneDrive/OutLook file storage\nstore-images.microsoft.com\nstore-images.s-microsoft.com\nstorecatalogrevocation.storequality.microsoft.com\nstoreedgefd.dsx.mp.microsoft.com\nstripecdn.map.fastly.net\nsubstrate.ms-acdc.office.com # CNAME for outlook.office.com\nsubstrate.office.com # CNAME for outlook.office.com\nsupervisor.ext-twitch.tv # https://github.com/mkb2091/blockconvert/issues/100 Breaks twitch extensions\nsupport.microsoft.com\nsurvey.euro.confirmit.com\nsurvey.pearsoncmg.com\ntechradar.com\ntecmint.com\nteespring.com\ntelecommand.telemetry.microsoft.com.akadns.net\ntexashomebase.com\nthebot.net\ntheepochtimes.com # https://github.com/mkb2091/blockconvert/issues/73 News website\ntheoldnet.com\ntiktok.com # Causes adblock to block www.tiktok.com\ntile-service.weather.microsoft.com\ntime.samsungcloudsolution.com # Issues #3, blocks services like Plex, YouTube and Amazon Video on Samsung TV\ntime.windows.com\ntlu.dl.delivery.mp.microsoft.com\ntmx.td.com # https://github.com/mkb2091/blockconvert/issues/102 Breaks Canadian finance website\nto-do.microsoft.com\ntoptenreviews.com\ntracker.tvnihon.com\ntracker2.itzmx.com # https://github.com/mkb2091/blockconvert/issues/124\ntracking.epicgames.com\ntransfer.sh\ntravelocity.com\ntrial.alcohol-soft.com # https://github.com/mkb2091/blockconvert/issues/80 CD and DVD burning software\ntrk.klclick1.com\ntsfe.trafficshaping.dsp.mp.microsoft.com\ntube8.com\nuk.com\numich.qualtrics.com\nunitedstates.smartscreen-prod.microsoft.com\nupdate.microsoft.com\nurldefense.proofpoint.com\nurlhaus.abuse.ch\nus-east-1-a.route.herokuapp.com\nus.configsvc1.live.com.akadns.net\nuse.typekit.net\nusers.telenet.be\nuservoice.com\nv.firebog.net\nv10.events.data.microsoft.com\nv10.vortex-win.data.microsoft.com\nv20.events.data.microsoft.com\nvalidation-v2.sls.microsoft.com\nvanilla.futurecdn.net\nvanityfair.com\nversion.hybrid.api.here.com # Needed for \"Here We Go\" map download https://github.com/mkb2091/blockconvert/issues/3#issuecomment-687661883\nvgwort.de\nvideos.oovvuu.com\nvignette.wikia.nocookie.net\nvip5.afdorigin-prod-am02.afdogw.com\nvip5.afdorigin-prod-ch02.afdogw.com\nvirustotalcloud.appspot.com\nvivo.com.br\nvmn.net\nvortex.accuweather.com\nvxvault.net\nw.soundcloud.com # https://github.com/mkb2091/blockconvert/issues/51\nwac.phicdn.net\nwallet.microsoft.com\nwatchdog-basic.energized.pro # Used to check if user is using Energized Protection\nwatchdog-blu.energized.pro # Used to check if user is using Energized Protection\nwatchdog-blugo.energized.pro # Used to check if user is using Energized Protection\nwatchdog-dns.energized.pro # Used to check if user is using Energized Protection\nwatchdog-ext.energized.pro # Used to check if user is using Energized Protection\nwatchdog-porn.energized.pro # Used to check if user is using Energized Protection\nwatchdog-spark.energized.pro # Used to check if user is using Energized Protection\nwatchdog-ultimate.energized.pro # Used to check if user is using Energized Protection\nwatchdog-unified.energized.pro # Used to check if user is using Energized Protection\nwatchdog.energized.pro # Used to check if user is using Energized Protection\nwatson.telemetry.microsoft.com\nwbd.ms\nwd-prod-cp-us-west-1-fe.westus.cloudapp.azure.com\nwd-prod-ss.trafficmanager.net\nwdcp.microsoft.*\nwdcp.microsoft.com\nweb.getmonero.org\nweb.tresorit.com\nwebmasters.stackexchange.com\nwebmd.com\nwhiteboard.microsoft.com\nwhiteboard.ms\nwidget-mediator.zopim.com\nwidgetdata.tradingview.com # https://github.com/mkb2091/blockconvert/issues/49\nwikidot.com\nwildcard.twimg.com\nwindowsupdate.com\nwl.spotify.com # Breaks spotify account confirmation\nwns.windows.com\nworldstream.nl\nwrapper-api.sp-prod.net\nwscont.apps.microsoft.com\nwscont1.apps.microsoft.com\nwscont2.apps.microsoft.com\nwsj.com\nww.youporn.com\nww2.sinaimg.cn # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-758476709 Breaks images on Chinese social media Weibo\nwww.pushbullet.com # https://github.com/mkb2091/blockconvert/issues/115\nwwww.youporn.com\nwx2.sinaimg.cn # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-758476709 Breaks images on Chinese social media Weibo\nwx3.sinaimg.cn # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-758476709 Breaks images on Chinese social media Weibo\nwx4.sinaimg.cn # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-758476709 Breaks images on Chinese social media Weibo\nx1.com\nxnxx.com # https://github.com/mkb2091/blockconvert/issues/56\nyahoo.co.jp\nyahoo.uservoice.com\nyandex.ru # https://github.com/mkb2091/blockconvert/issues/3#issuecomment-786252939 Breaks yandex\nyoast.com\nyouporn.com\nzdnet.com\nzerodot1.gitlab.io\nzerohedge.com\nzeustracker.abuse.ch\nzoho.com" }, { "path": "internal/block_ipnets.txt", "content": "0.0.0.0/8\n10.0.0.0/8\n100.64.0.0/10\n127.0.0.0/8\n169.254.0.0/16\n172.16.0.0/12\n192.0.0.0/24\n192.0.2.0/24\n192.88.99.0/24\n192.168.0.0/16\n198.18.0.0/15\n198.51.100.0/24\n203.0.113.0/24\n224.0.0.0/4\n240.0.0.0/4\n255.255.255.255/32" }, { "path": "internal/block_ips.txt", "content": "\n" }, { "path": "internal/block_regex.txt", "content": "\\.(?:com|co\\.uk|net)\\.(?:login|account|payment|verif)\n[_.-]analytics?[_.-]\n" }, { "path": "internal/blocklist.txt", "content": "*.ct.sendgrid.net # Tracking https://github.com/mkb2091/blockconvert/issues/70\n*.getemails.com # De-anonymization\naf.congressosiv-isv2018.it\nag.bud-info.com.pl\nall-prize-giveaway.life\nan-geburtstag.fun # Spam\napp0973.vnbue32.live\nawfulcruelty9.live\nbestprizesday4.life\nbs.newlifecampania.it # Spam\ncd.nestwood.it\ncf.spettinatidautore.it # Spam\ncl.folignobenecomune.it\ncompetition4138.redirect-servers52.live\nconverseboom24.live\ncp.milano2000.it\ndxnp7918p5axx.cloudfront.net\nee.caleidoscopio-singoli-pisa.it\ngetemails.com # De-anonymization\ninstallflash-s3.com\nmobile-global-app-market1.life\nnoireblur2.live\nnonamedvlp49.live\nreward7978.nonamedvlp49.live\nrododesk.pl\nysmk.pclatorre.it\nzmusic-online.com" }, { "path": "local.toml", "content": "listen_port = 3000\n\n[[peers]]\nurl = \"http://localhost/\"\nget_dns_results = false\n" }, { "path": "local2.toml", "content": "listen_port = 3000\n\n[[peers]]\nurl = \"http://localhost:3000/\"\nget_dns_results = false\n\n[[peers]]\nurl = \"http://localhost:3000/\"\nget_dns_results = false\n" }, { "path": "migrations/20240222234126_filterlists.sql", "content": "CREATE TABLE IF NOT EXISTS filterLists (\n url TEXT PRIMARY KEY NOT NULL UNIQUE,\n contents TEXT NOT NULL,\n lastUpdated INTEGER NOT NULL\n);" }, { "path": "migrations/20240223010915_lastmodified.sql", "content": "-- Add migration script here\nALTER TABLE filterLists ADD COLUMN lastModified INTEGER NOT NULL DEFAULT 0;" }, { "path": "migrations/20240223011106_lastmodified.sql", "content": "-- Add migration script here\nalter table filterLists drop column lastmodified;" }, { "path": "migrations/20240223011400_etag.sql", "content": "-- Add migration script here\n-- Add migration script here\nALTER TABLE filterLists ADD COLUMN etag TEXT;" }, { "path": "migrations/20240224194439_rules.sql", "content": "-- Add migration script here\nCREATE TABLE IF NOT EXISTS Rules (\n id SERIAL PRIMARY KEY,\n rule TEXT NOT NULL UNIQUE\n);" }, { "path": "migrations/20240224195223_filter_list_contents.sql", "content": "-- Add migration script here\n\nALTER TABLE filterLists RENAME TO OldFilterListContents;\n\nCREATE TABLE IF NOT EXISTS filterLists (\n id SERIAL PRIMARY KEY,\n url TEXT NOT NULL UNIQUE,\n format TEXT NOT NULL,\n contents TEXT NOT NULL,\n lastUpdated INTEGER NOT NULL,\n etag TEXT\n);\n\nINSERT INTO filterLists (url, format, contents, lastUpdated, etag) SELECT url, '', contents, lastUpdated, etag FROM filterLists;\n\nDROP TABLE OldFilterListContents;" }, { "path": "migrations/20240224203224_list_rules.sql", "content": "-- Add migration script here\nCREATE table list_rules (\n id SERIAL PRIMARY KEY,\n list_id INTEGER NOT NULL,\n rule_id INTEGER NOT NULL,\n FOREIGN KEY (list_id) REFERENCES filterLists(id),\n FOREIGN KEY (rule_id) REFERENCES rules(id)\n);" }, { "path": "migrations/20240225214254_list_source.sql", "content": "-- Add migration script here\nALTER TABLE list_rules ADD COLUMN source TEXT;" }, { "path": "migrations/20240225230839_index.sql", "content": "-- Add migration script here\ncreate unique index rule_index on Rules(rule);" }, { "path": "migrations/20240225231841_index.sql", "content": "-- Add migration script here\ncreate index list_rules_index on list_rules(list_id, rule_id);" }, { "path": "migrations/20240225232249_index.sql", "content": "-- Add migration script here\ndrop index list_rules_index;\n\ncreate index list_rules_index_list_id on list_rules(list_id);\ncreate index list_rules_index_rule_id on list_rules(rule_id);" }, { "path": "migrations/20240226004619_change_date.sql", "content": "-- Add migration script here\nalter table filterLists alter column lastUpdated type timestamp with time zone using to_timestamp(lastUpdated);" }, { "path": "migrations/20240226013547_source.sql", "content": "-- Add migration script here\nCREATE TABLE rule_source (\n id SERIAL PRIMARY KEY,\n source TEXT NOT NULL UNIQUE\n);\n\nCREATE INDEX idx_rule_source_source ON rule_source (source);\n\nALTER TABLE list_rules DROP COLUMN source;\n\nALTER TABLE list_rules ADD COLUMN source_id INTEGER REFERENCES rule_source(id);\n" }, { "path": "migrations/20240226152939_temp.sql", "content": "-- Add migration script here\nCREATE TABLE temp_rule_source (\n idx SERIAL PRIMARY KEY,\n rule TEXT NOT NULL,\n source TEXT NOT NULL\n);" }, { "path": "migrations/20240226215547_remove_column.sql", "content": "-- Add migration script here\nALTER TABLE temp_rule_source DROP COLUMN idx;" }, { "path": "migrations/20240226215929_remove_id.sql", "content": "-- Add migration script here\nALTER TABLE list_rules DROP COLUMN id;\n" }, { "path": "migrations/20240226220711_remove_fkey.sql", "content": "-- Add migration script here\nALTER TABLE list_rules DROP CONSTRAINT list_rules_list_id_fkey;\nALTER TABLE list_rules DROP CONSTRAINT list_rules_rule_id_fkey;\nALTER TABLE list_rules DROP CONSTRAINT list_rules_source_id_fkey;" }, { "path": "migrations/20240226223817_domain_rules.sql", "content": "-- Add migration script here\nCREATE TABLE domain_rules (\n id SERIAL PRIMARY KEY,\n domain TEXT NOT NULL\n);\nCREATE INDEX domain_rules_idx_domain ON domain_rules (domain);" }, { "path": "migrations/20240226230317_domain_block.sql", "content": "-- Add migration script here\nALTER TABLE domain_rules ADD COLUMN block BOOLEAN NOT NULL;" }, { "path": "migrations/20240226230425_rename.sql", "content": "-- Add migration script here\nALTER TABLE domain_rules RENAME COLUMN id TO rule_id;" }, { "path": "migrations/20240227191530_drop_column.sql", "content": "-- Add migration script here\nCREATE TEMPORARY TABLE temp_rule_source(source TEXT UNIQUE, rule_id INTEGER) ON COMMIT DROP;\n\nINSERT INTO\n temp_rule_source (source, rule_id)\nSELECT\n rule_source.source,\n list_rules.rule_id\nFROM\n list_rules\n INNER JOIN rule_source ON list_rules.source_id = rule_source.id ON CONFLICT DO NOTHING;\n\nDELETE FROM\n rule_source;\n\nALTER TABLE\n rule_source\nADD\n COLUMN rule_id INTEGER NOT NULL;\n\nINSERT INTO\n rule_source (source, rule_id)\nSELECT\n source,\n rule_id\nFROM\n temp_rule_source;" }, { "path": "migrations/20240227194830_drop_column.sql", "content": "-- Add migration script here\nALTER TABLE list_rules DROP COLUMN rule_id;" }, { "path": "migrations/20240227200629_drop_table.sql", "content": "-- Add migration script here\nDROP TABLE temp_rule_source;" }, { "path": "migrations/20240227201410_primary_key.sql", "content": "-- Add migration script here\nDELETE FROM list_rules;\n\nALTER TABLE\n list_rules\nADD\n PRIMARY KEY (list_id, source_id);" }, { "path": "migrations/20240228001134_domain_rules.sql", "content": "-- Add migration script here\nALTER TABLE\n domain_rules\nADD\n COLUMN subdomain BOOLEAN NOT NULL DEFAULT FALSE;\n\nALTER TABLE\n domain_rules\nALTER COLUMN\n subdomain DROP DEFAULT;\n\nCREATE TEMPORARY TABLE temp_domain_rules (\n domain TEXT NOT NULL,\n allow BOOLEAN NOT NULL,\n subdomain BOOLEAN NOT NULL\n);\n\nINSERT INTO\n temp_domain_rules (domain, allow, subdomain)\nSELECT\n domain,\n block,\n subdomain\nFROM\n domain_rules;\n\nDELETE FROM\n domain_rules;\n\nALTER TABLE\n domain_rules\nADD\n COLUMN id SERIAL NOT NULL UNIQUE;\n\nALTER TABLE\n domain_rules RENAME COLUMN block TO allow;\n\nALTER TABLE\n domain_rules DROP column rule_id;\n\nALTER TABLE\n domain_rules\nADD\n CONSTRAINT domain_rule_unique UNIQUE (domain, allow, subdomain);\n\nINSERT INTO\n domain_rules (domain, allow, subdomain)\nSELECT\n domain,\n NOT allow,\n subdomain\nFROM\n temp_domain_rules;" }, { "path": "migrations/20240228004601_extend_rules.sql", "content": "-- Add migration script here\nALTER TABLE\n Rules\nADD\n COLUMN domain_rule_id INTEGER;\n\nCREATE TABLE unknown_rules (\n id SERIAL PRIMARY KEY,\n rule TEXT NOT NULL UNIQUE\n);\n\nDELETE FROM\n Rules;\n\nALTER TABLE\n Rules\nADD\n COLUMN unknown_rule_id INTEGER;\n\nALTER TABLE\n Rules\nADD\n CONSTRAINT unique_rules UNIQUE NULLS NOT DISTINCT (domain_rule_id, unknown_rule_id);" }, { "path": "migrations/20240228005409_drop_rule.sql", "content": "-- Add migration script here\nALTER TABLE Rules DROP COLUMN rule;" }, { "path": "migrations/20240228015906_change_unique.sql", "content": "-- Add migration script here\nALTER TABLE rule_source DROP CONSTRAINT rule_source_source_key;\n\nALTER TABLE rule_source ADD CONSTRAINT rule_source_unique UNIQUE (source, rule_id);\n" }, { "path": "migrations/20240228164553_ip.sql", "content": "-- Add migration script here\nCREATE TABLE ip_rules (\n id SERIAL PRIMARY KEY,\n ip_address inet NOT NULL,\n allow boolean NOT NULL,\n CONSTRAINT ip_rules_unique UNIQUE (ip_address, allow)\n);\n\nALTER TABLE Rules ADD COLUMN ip_rule_id INTEGER;\nALTER TABLE Rules DROP CONSTRAINT unique_rules;\nALTER TABLE Rules ADD CONSTRAINT unique_rules UNIQUE NULLS NOT DISTINCT (domain_rule_id, ip_rule_id, unknown_rule_id);" }, { "path": "migrations/20240228170646_ip.sql", "content": "-- Add migration script here\nALTER TABLE ip_rules RENAME COLUMN ip_address TO ip_network;" }, { "path": "migrations/20240228175807_remove_unknown.sql", "content": "-- Add migration script here\nDROP TABLE unknown_rules;\nALTER TABLE Rules DROP COLUMN unknown_rule_id;" }, { "path": "migrations/20240228180753_index.sql", "content": "-- Add migration script here\nDELETE FROM Rules;\nALTER TABLE Rules ADD CONSTRAINT rules_unique UNIQUE NULLS NOT DISTINCT (domain_rule_id, ip_rule_id);\n" }, { "path": "migrations/20240229210101_domains.sql", "content": "-- Add migration script here\nCREATE TABLE domains (\n id SERIAL PRIMARY KEY,\n domain TEXT NOT NULL UNIQUE\n);\n\nINSERT INTO\n domains (domain)\nSELECT\n domain\nFROM\n domain_rules ON CONFLICT DO NOTHING;\n\nALTER TABLE\n domain_rules RENAME TO domain_rules_old;\n\nCREATE TABLE domain_rules (\n id SERIAL PRIMARY KEY,\n domain_id INTEGER NOT NULL,\n allow BOOLEAN NOT NULL,\n subdomain BOOLEAN NOT NULL,\n CONSTRAINT domain_rules_unique UNIQUE (domain_id, allow, subdomain)\n);\n\nINSERT INTO\n domain_rules (domain_id, allow, subdomain)\nSELECT\n domains.id,\n domain_rules_old.allow,\n domain_rules_old.subdomain\nFROM\n domain_rules_old\n INNER JOIN domains ON domain_rules_old.domain = domains.domain;\n\nDROP TABLE domain_rules_old;" }, { "path": "migrations/20240229212527_subdomains.sql", "content": "-- Add migration script here\nALTER TABLE\n domains\nALTER COLUMN\n id TYPE bigint;\n\nALTER TABLE\n domain_rules\nALTER COLUMN\n domain_id TYPE bigint;\n\nCREATE TABLE subdomains (\n domain_id bigint NOT NULL UNIQUE,\n parent_domain_id bigint\n);" }, { "path": "migrations/20240301000455_more_indexes.sql", "content": "-- Add migration script here\nCREATE INDEX rule_source_rule_idx ON rule_source (rule_id);\nCREATE INDEX domain_rules_domain_idx ON domain_rules (domain_id);\n" }, { "path": "migrations/20240301000900_subdomain_idx.sql", "content": "-- Add migration script here\nCREATE INDEX subdomain_domain_idx ON subdomains (domain_id);\n\nCREATE INDEX subdomain_parent_idx ON subdomains (parent_domain_id);" }, { "path": "migrations/20240301030213_subdomain_inde.sql", "content": "-- Add migration script here\nCREATE INDEX domain_rules_subdomain_idx ON domain_rules (subdomain);\n" }, { "path": "migrations/20240302171950_expanded_subdomains.sql", "content": "-- Add migration script here\nDELETE FROM\n subdomains\nWHERE\n parent_domain_id IS NOT NULL;\n\nALTER TABLE\n subdomains DROP CONSTRAINT subdomains_domain_id_key;\n\nALTER TABLE\n subdomains\nADD\n CONSTRAINT subdomains_domain_id_key UNIQUE NULLS NOT DISTINCT (domain_id, parent_domain_id);" }, { "path": "migrations/20240302184040_processed_subdomains.sql", "content": "DELETE FROM\n domains;\nDELETE FROM subdomains;\nDELETE FROM domain_rules;\nDELETE FROM Rules;\nDELETE FROM rule_source;\n\nALTER TABLE\n domains\nADD\n COLUMN processed_subdomains boolean DEFAULT false;" }, { "path": "migrations/20240302192658_index.sql", "content": "-- Add migration script here\nCREATE INDEX domains_processed_subdomains_idx ON domains(processed_subdomains);" }, { "path": "migrations/20240302194037_domain_rule_id_idx.sql", "content": "-- Add migration script here\nCREATE INDEX rules_domain_rule_id_idx ON rules(domain_rule_id);" }, { "path": "migrations/20240302194733_not_null_parent.sql", "content": "-- Add migration script here\nDELETE FROM subdomains WHERE parent_domain_id IS NULL;\nALTER TABLE subdomains ALTER COLUMN parent_domain_id SET NOT NULL;" }, { "path": "migrations/20240302205633_not_null#.sql", "content": "-- Add migration script here\nALTER TABLE domains ALTER COLUMN processed_subdomains SET NOT NULL;" }, { "path": "migrations/20240302222401_dns.sql", "content": "-- Add migration script here\nALTER TABLE\n domains\nADD\n COLUMN last_checked_dns TIMESTAMP WITH TIME ZONE;\n\nCREATE INDEX domains_last_checked_dns_idx ON domains(last_checked_dns);\n\nCREATE TABLE dns_ips (\n domain_id BIGINT NOT NULL,\n ip_address INET NOT NULL\n);\n\nCREATE INDEX dns_ips_domain_id_idx ON dns_ips(domain_id);\n\nCREATE INDEX dns_ips_ip_address_idx ON dns_ips(ip_address);\n\nCREATE TABLE dns_cnames (\n domain_id BIGINT NOT NULL,\n cname_domain_id BIGINT NOT NULL\n);\n\nCREATE INDEX dns_cnames_domain_id_idx ON dns_cnames(domain_id);\n\nCREATE INDEX dns_cnames_cname_domain_id_idx ON dns_cnames(cname_domain_id);" }, { "path": "migrations/20240304235746_filterlist.sql", "content": "-- Add migration script here\nALTER TABLE filterlists ADD COLUMN name TEXT;\nALTER TABLE filterlists ADD COLUMN author TEXT;\nALTER TABLE filterlists ADD COLUMN expires INTEGER;\nALTER TABLE filterlists ADD COLUMN license TEXT;" }, { "path": "migrations/20240305000257_filterlist.sql", "content": "-- Add migration script here\nALTER TABLE filterlists ALTER COLUMN contents DROP NOT NULL;" }, { "path": "migrations/20240305000612_filterlist.sql", "content": "-- Add migration script here\nALTER TABLE filterlists ALTER COLUMN lastUpdated DROP NOT NULL;" }, { "path": "migrations/20240305003411_filterlist.sql", "content": "-- Add migration script here\nALTER TABLE filterlists ALTER COLUMN expires SET NOT NULL;" }, { "path": "migrations/20240305200551_rule_matches.sql", "content": "-- Add migration script here\nCREATE TABLE rule_matches (\n rule_id INTEGER NOT NULL,\n domain_id BIGINT NOT NULL\n);\n\nALTER TABLE rule_matches ADD CONSTRAINT rule_matches_pk UNIQUE (rule_id, domain_id);\nCREATE INDEX rule_matches_rule_id_idx ON rule_matches (rule_id);\nCREATE INDEX rule_matches_domain_id_idx ON rule_matches (domain_id);\n\nALTER TABLE Rules ADD COLUMN last_checked_matches TIMESTAMP;\n" }, { "path": "migrations/20240306123603_rule_count.sql", "content": "-- Add migration script here\nALTER TABLE filterLists ADD COLUMN rule_count INT NOT NULL DEFAULT 0;\n" }, { "path": "migrations/20240306214217_index.sql", "content": "-- Add migration script here\nCREATE INDEX rules_last_checked_matches_idx ON rules (last_checked_matches);" }, { "path": "migrations/20240307005702_lists.sql", "content": "-- Add migration script here\nCREATE TABLE allow_domains (\n domain_id BIGINT UNIQUE NOT NULL\n);\n\nCREATE TABLE block_domains (\n domain_id BIGINT UNIQUE NOT NULL\n);" }, { "path": "migrations/20240307012450_index.sql", "content": "-- Add migration script here\nCREATE INDEX domain_rules_allow_idx ON domain_rules (allow);\nCREATE INDEX ip_rules_allow_idx ON ip_rules (allow);" }, { "path": "migrations/20240307031445_idx.sql", "content": "-- Add migration script here\nCREATE INDEX ip_rules_network_idx ON ip_rules (ip_network);" }, { "path": "output/allowed_ips.txt", "content": "" }, { "path": "output/domains.rpz", "content": "" }, { "path": "output/hosts.txt", "content": "" }, { "path": "output/ip_blocklist.txt", "content": "" }, { "path": "output/whitelist_adblock.txt", "content": "" }, { "path": "output/whitelist_domains.txt", "content": "" }, { "path": "package.json", "content": "{\n \"devDependencies\": {\n \"@tailwindcss/typography\": \"^0.5.10\",\n \"daisyui\": \"^4.7.2\",\n \"tailwindcss\": \"^3.4.1\"\n }\n}\n" }, { "path": "rust-toolchain.toml", "content": "\n[toolchain]\nchannel = \"stable\"\n" }, { "path": "rustfmt.toml", "content": "edition = \"2021\"\n" }, { "path": "src/app.rs", "content": "use crate::{\n domain::DomainViewPage,\n error_template::{AppError, ErrorTemplate},\n filterlist::FilterListPage,\n home_page::HomePage,\n ip_view::IpView,\n rule::RuleViewPage,\n stats_view::StatsView,\n};\nuse leptos::*;\nuse leptos_meta::*;\nuse leptos_router::*;\n\n#[component]\npub fn App() -> impl IntoView {\n // Provides context that manages stylesheets, titles, meta tags, etc.\n provide_meta_context();\n\n view! {\n \n \n\n \n // content for this welcome page\n <Router fallback=|| {\n let mut outside_errors = Errors::default();\n outside_errors.insert_with_default_key(AppError::NotFound);\n view! { <ErrorTemplate outside_errors/> }.into_view()\n }>\n\n <header class=\"p-4 text-white bg-indigo-600\">\n <nav class=\"container flex items-center justify-between mx-auto\">\n <A href=\"/\" class=\"text-lg font-bold\">\n Home\n </A>\n <div class=\"space-x-4\">\n <A href=\"/tasks\" class=\"hover:text-indigo-300\">\n Tasks\n </A>\n <A href=\"/stats\" class=\"hover:text-indigo-300\">\n Stats\n </A>\n <A\n href=\"/login\"\n class=\"px-4 py-2 text-indigo-600 bg-white rounded hover:bg-indigo-200\"\n >\n Login\n </A>\n </div>\n </nav>\n </header>\n <main>\n <Routes>\n <Route path=\"\" view=HomePage ssr=SsrMode::Async/>\n <Route path=\"tasks\" view=crate::tasks::TaskView ssr=SsrMode::Async/>\n <Route path=\"stats\" view=StatsView ssr=SsrMode::InOrder/>\n <Route path=\"list\" view=FilterListPage ssr=SsrMode::InOrder/>\n <Route path=\"rule/:id\" view=RuleViewPage ssr=SsrMode::InOrder/>\n <Route path=\"domain/:domain\" view=DomainViewPage ssr=SsrMode::InOrder/>\n <Route path=\"ip/:ip\" view=IpView ssr=SsrMode::InOrder/>\n </Routes>\n </main>\n </Router>\n }\n}\n\n#[component]\npub fn Loading() -> impl IntoView {\n view! { <span class=\"loading loading-spinner loading-sm\"></span> }\n}\n" }, { "path": "src/domain.rs", "content": "#[cfg(feature = \"ssr\")]\nuse crate::rule::RuleData;\nuse crate::{\n app::Loading,\n filterlist::{FilterListLink, FilterListUrl},\n rule::DisplayRule,\n rule::RuleId,\n source::SourceId,\n};\nuse leptos::*;\nuse leptos_router::*;\n#[cfg(feature = \"ssr\")]\npub use lookup_dns_task::DomainResolver;\nuse serde::{Deserialize, Serialize};\nuse std::{collections::BTreeSet, net::IpAddr, str::FromStr, sync::Arc};\n\n#[cfg_attr(feature = \"ssr\", derive(sqlx::Encode, sqlx::Decode))]\n#[derive(Serialize, Deserialize, Debug, Copy, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)]\npub struct DomainId(i64);\n\n#[derive(Debug, Clone, thiserror::Error)]\npub enum DomainParseError {\n Addr,\n HickoryProto,\n Custom,\n}\n\nimpl std::fmt::Display for DomainParseError {\n fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n write!(f, \"Invalid domain\")\n }\n}\n\nimpl<'a> From<addr::error::Error<'a>> for DomainParseError {\n fn from(_: addr::error::Error) -> Self {\n DomainParseError::Addr\n }\n}\n\nimpl From<hickory_proto::error::ProtoError> for DomainParseError {\n fn from(_: hickory_proto::error::ProtoError) -> Self {\n DomainParseError::HickoryProto\n }\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord, Hash)]\n#[serde(transparent)]\npub struct Domain(Arc<str>);\n\n#[cfg(feature = \"ssr\")]\nimpl sqlx::Type<sqlx::Postgres> for Domain {\n fn type_info() -> sqlx::postgres::PgTypeInfo {\n <&str as sqlx::Type<sqlx::Postgres>>::type_info()\n }\n\n fn compatible(ty: &sqlx::postgres::PgTypeInfo) -> bool {\n <&str as sqlx::Type<sqlx::Postgres>>::compatible(ty)\n }\n}\n#[cfg(feature = \"ssr\")]\nimpl sqlx::postgres::PgHasArrayType for Domain {\n fn array_type_info() -> sqlx::postgres::PgTypeInfo {\n <&str as sqlx::postgres::PgHasArrayType>::array_type_info()\n }\n\n fn array_compatible(ty: &sqlx::postgres::PgTypeInfo) -> bool {\n <&str as sqlx::postgres::PgHasArrayType>::array_compatible(ty)\n }\n}\n#[cfg(feature = \"ssr\")]\nimpl sqlx::Encode<'_, sqlx::Postgres> for Domain {\n fn encode_by_ref(&self, buf: &mut sqlx::postgres::PgArgumentBuffer) -> sqlx::encode::IsNull {\n <&str as sqlx::Encode<sqlx::Postgres>>::encode(self.as_ref(), buf)\n }\n}\n\nimpl AsRef<str> for Domain {\n fn as_ref(&self) -> &str {\n &self.0\n }\n}\n\nimpl FromStr for Domain {\n type Err = DomainParseError;\n fn from_str(domain: &str) -> Result<Domain, Self::Err> {\n if domain.len() > 253 {\n return Err(DomainParseError::Custom);\n }\n let mut domain: Arc<str> = domain.into();\n Arc::get_mut(&mut domain).unwrap().make_ascii_lowercase();\n\n if domain.starts_with('*') || domain.ends_with('.') {\n return Err(DomainParseError::Custom);\n }\n if !addr::parse_dns_name(&domain)?.has_known_suffix() {\n return Err(DomainParseError::Addr);\n }\n let name = hickory_proto::rr::Name::from_str_relaxed(&domain)?;\n if name.num_labels() < 2 {\n return Err(DomainParseError::Custom);\n }\n\n if domain.contains('/') {\n log::warn!(\"Invalid domain: {:?}\", domain);\n return Err(DomainParseError::Custom);\n }\n Ok(Domain(domain))\n }\n}\n\n#[cfg(test)]\nmod tests {\n use crate::domain::Domain;\n #[test]\n fn valid_domain() {\n for domain in [\n \"amazonaws.com\",\n \"s3-website.us-east-1.amazonaws.com\",\n \"origin-mobile_mob.conduit.com\",\n ] {\n let domain: Result<Domain, _> = domain.parse();\n assert!(domain.is_ok());\n }\n }\n\n #[test]\n fn invalid_domain() {\n for domain_str in [\n \"com\",\n \"@.amazonaws.com\",\n \"1234\",\n \"example.com,google.com\",\n \"example.com.\",\n ] {\n let domain: Result<Domain, _> = domain_str.parse();\n assert!(domain.is_err(), \"{}\", domain_str);\n }\n }\n #[test]\n fn makes_lowercase() {\n let domain: Domain = \"EXAMPLE.COM\".parse().unwrap();\n assert_eq!(domain.as_ref(), \"example.com\");\n }\n}\n\n#[cfg(feature = \"ssr\")]\nmod lookup_dns_task {\n use std::{collections::HashSet, sync::Mutex, time::Duration};\n\n use super::*;\n use hickory_resolver::error::ResolveError;\n use tokio_util::sync::CancellationToken;\n\n fn parse_lookup_result(\n result: Result<hickory_resolver::lookup_ip::LookupIp, ResolveError>,\n ) -> Result<(Vec<ipnetwork::IpNetwork>, Vec<Domain>), ResolveError> {\n match result {\n Ok(result) => {\n let mut ips: Vec<ipnetwork::IpNetwork> = Vec::new();\n let mut cnames = Vec::new();\n let lookup = result.as_lookup();\n for record in lookup.iter() {\n if let Some(a) = record.as_a() {\n let ip: IpAddr = a.0.into();\n ips.push(ip.into());\n } else if let Some(aaaa) = record.as_aaaa() {\n let ip: IpAddr = aaaa.0.into();\n ips.push(ip.into());\n } else if let Some(cname) = record.as_cname() {\n let mut cname = cname.0.to_ascii();\n if cname.ends_with('.') {\n cname.pop();\n }\n if let Ok(cname) = cname.parse() {\n cnames.push(cname);\n } else {\n log::warn!(\"Invalid CNAME {}\", cname);\n }\n } else {\n log::info!(\"Unknown record type {:?}\", record.record_type());\n }\n }\n Ok((ips, cnames))\n }\n Err(err) => match err.kind() {\n hickory_resolver::error::ResolveErrorKind::NoRecordsFound {\n query: _,\n soa: _,\n negative_ttl: _,\n response_code: _,\n trusted: _,\n } => Ok((vec![], vec![])),\n hickory_resolver::error::ResolveErrorKind::Proto(_) => Ok((vec![], vec![])),\n hickory_resolver::error::ResolveErrorKind::Timeout => Err(err),\n _ => Err(err),\n },\n }\n }\n\n #[cfg(feature = \"ssr\")]\n type Resolver = Arc<\n hickory_resolver::AsyncResolver<\n hickory_resolver::name_server::GenericConnector<\n hickory_resolver::name_server::TokioRuntimeProvider,\n >,\n >,\n >;\n\n #[cfg(feature = \"ssr\")]\n type Task = (DomainId, Domain);\n\n #[cfg(feature = \"ssr\")]\n #[derive(Clone)]\n pub struct DomainResolver {\n resolvers: Vec<(Arc<str>, Resolver)>,\n tx: async_channel::Sender<Task>,\n rx: async_channel::Receiver<Task>,\n read_limit: i64,\n failed_cache_size: usize,\n failed_domains: Arc<Mutex<Vec<i64>>>,\n written_domains: Arc<Mutex<Vec<i64>>>,\n bad_domains: Arc<Mutex<Vec<i64>>>,\n looked_up_domains: Arc<Mutex<Vec<i64>>>,\n dns_ips: Arc<Mutex<(Vec<i64>, Vec<ipnetwork::IpNetwork>)>>,\n dns_cnames: Arc<Mutex<(Vec<i64>, Vec<Domain>)>>,\n token: CancellationToken,\n }\n\n #[cfg(feature = \"ssr\")]\n impl DomainResolver {\n pub fn new(token: CancellationToken) -> Result<Self, ServerFnError> {\n let _ = dotenvy::dotenv()?;\n let servers_str = std::env::var(\"DNS_SERVERS\")?;\n\n let read_limit = std::env::var(\"READ_LIMIT\")?.parse::<u32>()? as i64;\n let mut resolvers = Vec::new();\n for server in servers_str.split(',') {\n let server: Arc<str> = server.into();\n let (addr, port) = server\n .split_once(':')\n .ok_or_else(|| ServerFnError::new(\"Bad DNS_SERVER env\"))?;\n let server_conf = hickory_resolver::config::NameServerConfigGroup::from_ips_clear(\n &[addr.parse()?],\n port.parse()?,\n true,\n );\n let config =\n hickory_resolver::config::ResolverConfig::from_parts(None, vec![], server_conf);\n let mut opts = hickory_resolver::config::ResolverOpts::default();\n opts.ip_strategy = hickory_resolver::config::LookupIpStrategy::Ipv4AndIpv6;\n opts.cache_size = 32;\n opts.attempts = 3;\n opts.timeout = std::time::Duration::from_secs_f32(5.0);\n let resolver = Arc::new(hickory_resolver::AsyncResolver::tokio(config, opts));\n resolvers.push((server, resolver));\n }\n\n if resolvers.is_empty() {\n return Err(ServerFnError::new(\"Empty DNS server list\"));\n }\n let (tx, rx) = async_channel::bounded(read_limit as usize);\n\n let bad_domains = Arc::new(Mutex::new(Vec::new()));\n let failed_cache_size = std::env::var(\"FAILED_CACHE_SIZE\")?.parse()?;\n\n Ok(Self {\n resolvers,\n bad_domains,\n tx,\n rx,\n read_limit,\n failed_cache_size,\n failed_domains: Arc::new(Mutex::new(Vec::new())),\n written_domains: Arc::new(Mutex::new(Vec::new())),\n looked_up_domains: Arc::new(Mutex::new(Vec::new())),\n dns_ips: Default::default(),\n dns_cnames: Default::default(),\n token,\n })\n }\n\n pub async fn run(&self) -> Result<(), ServerFnError> {\n dotenvy::dotenv()?;\n let concurrent_lookups: usize = std::env::var(\"CONCURRENT_LOOKUPS\")?.parse()?;\n\n let mut tasks = tokio::task::JoinSet::new();\n\n for resolver in &self.resolvers {\n for _ in 0..concurrent_lookups {\n let resolver_str = resolver.0.clone();\n let resolver = resolver.1.clone();\n let resolver_self = self.clone();\n let task = async move {\n let token = resolver_self.token.clone();\n tokio::select! {\n _ = token.cancelled() => {\n log::info!(\"Shutting down DNS resolver\");\n Ok(())},\n res =\n resolver_self.run_task(resolver_str, resolver) => res}\n };\n tasks.spawn(task);\n }\n }\n let selector = self.clone();\n tasks.spawn(async move {\n let token = selector.token.clone();\n tokio::select! {\n _ = token.cancelled() => {\n log::info!(\"Shutting down DNS selector\");\n Ok(())}\n res = selector.domain_selector() => res\n }\n });\n let writer = self.clone();\n tasks.spawn(async move { writer.write_to_db().await });\n while let Some(result) = tasks.join_next().await {\n let _ = result?;\n }\n Ok(())\n }\n\n async fn domain_selector(&self) -> Result<(), ServerFnError> {\n let pool = crate::server::get_db().await?;\n let mut started_domains = HashSet::new();\n let mut failed_domains = std::collections::VecDeque::<i64>::new();\n loop {\n {\n failed_domains.extend(std::mem::take(&mut *self.failed_domains.lock()?));\n }\n while failed_domains.len() > self.failed_cache_size {\n if let Some(failed) = failed_domains.pop_front() {\n started_domains.remove(&failed);\n }\n }\n let written_domains = std::mem::take(&mut *self.written_domains.lock()?);\n for domain_id in written_domains {\n started_domains.remove(&domain_id);\n }\n let limit = self.read_limit + started_domains.len() as i64;\n let records = sqlx::query!(\n \"SELECT id, domain\n FROM Domains\n WHERE last_checked_dns IS NULL\n ORDER BY id DESC NULLS FIRST\n LIMIT $1\",\n limit\n )\n .fetch_all(&pool)\n .await?;\n let recheck_domains = if records.len() < limit as usize {\n sqlx::query!(\n \"SELECT id, domain\n FROM Domains\n ORDER BY last_checked_dns ASC NULLS FIRST\n LIMIT $1\",\n limit\n )\n .fetch_all(&pool)\n .await?\n } else {\n vec![]\n };\n\n let records = records.into_iter().map(|record| (record.id, record.domain));\n let recheck_domains = recheck_domains\n .into_iter()\n .map(|record| (record.id, record.domain));\n let mut has_domains = false;\n for (domain_id, domain_str) in records.chain(recheck_domains) {\n has_domains = true;\n if !started_domains.insert(domain_id) {\n continue;\n }\n if let Ok(domain) = domain_str.parse::<Domain>() {\n if domain_str == domain.as_ref() {\n self.tx.send((DomainId(domain_id), domain)).await?;\n continue;\n }\n }\n log::warn!(\"Invalid domain: {}\", domain_str);\n self.bad_domains.lock()?.push(domain_id);\n }\n if !has_domains {\n log::info!(\"No domains to check, sleeping\");\n tokio::time::sleep(Duration::from_secs(30)).await;\n }\n }\n }\n\n async fn run_task(\n &self,\n resolver_str: Arc<str>,\n resolver: Resolver,\n ) -> Result<(), ServerFnError> {\n while let Ok(task) = self.rx.recv().await {\n let (domain_id, domain) = task;\n let mut domain_str = domain.as_ref().to_string();\n domain_str.push('.');\n let result = resolver.lookup_ip(&domain_str).await;\n let result = parse_lookup_result(result);\n match result {\n Ok((ips, cnames)) => {\n self.looked_up_domains.lock()?.push(domain_id.0);\n {\n let mut dns_ips = self.dns_ips.lock()?;\n for ip in ips {\n dns_ips.0.push(domain_id.0);\n dns_ips.1.push(ip);\n }\n }\n {\n let mut dns_cnames = self.dns_cnames.lock()?;\n for cname in cnames {\n dns_cnames.0.push(domain_id.0);\n dns_cnames.1.push(cname);\n }\n }\n }\n Err(err) => {\n log::warn!(\n \"Server: {} Error looking up domain {}: {}\",\n resolver_str,\n domain.as_ref(),\n err\n );\n self.failed_domains.lock()?.push(domain_id.0);\n }\n }\n }\n Ok(())\n }\n\n async fn write_to_db(&self) -> Result<(), ServerFnError> {\n let pool = crate::server::get_db().await?;\n let write_frequency: u64 = std::env::var(\"WRITE_FREQUENCY\")?.parse()?;\n let mut interval = tokio::time::interval(Duration::from_secs(write_frequency));\n interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);\n interval.tick().await;\n loop {\n tokio::select! {\n _ = interval.tick() => {},\n _ = self.token.cancelled() => {\n log::info!(\"Shutting down DNS writer\")\n }\n }\n let looked_up_domains = std::mem::take(&mut *self.looked_up_domains.lock()?);\n let looked_up_domains_deduped =\n looked_up_domains.iter().cloned().collect::<HashSet<_>>();\n assert_eq!(looked_up_domains.len(), looked_up_domains_deduped.len());\n let dns_ips = std::mem::take(&mut *self.dns_ips.lock()?);\n let dns_ips_domain_ids = &dns_ips.0;\n let dns_ips_ips = &dns_ips.1;\n let dns_cnames = std::mem::take(&mut *self.dns_cnames.lock()?);\n let dns_cnames_domain_ids = &dns_cnames.0;\n let dns_cnames_cname = &dns_cnames.1;\n let bad_domains = std::mem::take(&mut *self.bad_domains.lock()?);\n let total_cnames = dns_cnames_cname\n .iter()\n .collect::<HashSet<_>>()\n .into_iter()\n .cloned()\n .collect::<Vec<Domain>>();\n let new_domains_from_cnames = sqlx::query!(\n \"INSERT INTO domains(domain)\n SELECT domain FROM UNNEST($1::text[]) as t(domain)\n ON CONFLICT DO NOTHING\",\n &total_cnames[..] as _\n )\n .execute(&pool)\n .await?\n .rows_affected();\n\n let mut tx = pool.begin().await?;\n sqlx::query!(\n \"DELETE FROM dns_ips WHERE domain_id = ANY($1::bigint[])\",\n &looked_up_domains[..]\n )\n .execute(&mut *tx)\n .await?;\n sqlx::query!(\n \"DELETE FROM dns_cnames WHERE domain_id = ANY($1::bigint[])\",\n &looked_up_domains[..]\n )\n .execute(&mut *tx)\n .await?;\n sqlx::query!(\n \"INSERT INTO dns_ips(domain_id, ip_address)\n SELECT domain_id, ip FROM UNNEST($1::bigint[], $2::inet[]) as t(domain_id, ip)\",\n &dns_ips_domain_ids[..],\n &dns_ips_ips[..]\n )\n .execute(&mut *tx)\n .await?;\n sqlx::query!(\n \"INSERT INTO dns_cnames(domain_id, cname_domain_id)\n SELECT domain_id, cname_domains.id FROM UNNEST($1::bigint[], $2::text[]) as t(domain_id, cname)\n INNER JOIN domains AS cname_domains ON cname_domains.domain = t.cname\n \",\n &dns_cnames_domain_ids[..],\n &dns_cnames_cname[..] as _\n )\n .execute(&mut *tx)\n .await?;\n\n let updated_domains = sqlx::query!(\n \"UPDATE domains\n SET last_checked_dns = now()\n WHERE id = ANY($1::bigint[])\",\n &looked_up_domains[..]\n )\n .execute(&mut *tx)\n .await?\n .rows_affected();\n assert_eq!(updated_domains, looked_up_domains.len() as u64);\n\n tx.commit().await?;\n self.written_domains.lock()?.extend(&looked_up_domains);\n\n if !bad_domains.is_empty() {\n log::info!(\"Removing {} bad domains\", bad_domains.len());\n sqlx::query!(\n \"DELETE FROM domains\n WHERE id = ANY($1::bigint[])\",\n &bad_domains[..]\n )\n .execute(&pool)\n .await?;\n }\n\n self.written_domains.lock()?.extend(&bad_domains);\n\n log::info!(\n \"Looked up {} domains, got {} ips, {} cnames ({} new)\",\n looked_up_domains.len(),\n dns_ips_domain_ids.len(),\n dns_cnames_domain_ids.len(),\n new_domains_from_cnames\n );\n if self.token.is_cancelled() {\n return Ok(());\n }\n }\n }\n }\n}\n\n#[server]\nasync fn get_dns_result(\n domain: Domain,\n) -> Result<(BTreeSet<IpAddr>, BTreeSet<(DomainId, String)>), ServerFnError> {\n let records = sqlx::query!(\n r#\"SELECT dns_ips.ip_address as \"ip_address: Option<ipnetwork::IpNetwork>\",\n cname_domains.id as \"cname_domain_id: Option<i64>\",\n cname_domains.domain as \"cname_domain: Option<String>\"\n FROM domains\n LEFT JOIN dns_ips ON dns_ips.domain_id=domains.id\n LEFT JOIN dns_cnames on dns_cnames.domain_id=domains.id\n LEFT JOIN domains AS cname_domains ON cname_domains.id=dns_cnames.cname_domain_id\n WHERE domains.domain = $1\n \"#r,\n domain.as_ref().to_string()\n )\n .fetch_all(&crate::server::get_db().await?)\n .await?;\n let mut ip_addresses = BTreeSet::new();\n let mut cnames = BTreeSet::new();\n for record in records {\n if let Some(ip) = record.ip_address {\n ip_addresses.insert(ip.ip());\n }\n if let (Some(id), Some(cname)) = (record.cname_domain_id, record.cname_domain) {\n cnames.insert((DomainId(id), cname));\n }\n }\n Ok((ip_addresses, cnames))\n}\n\n#[component]\nfn DnsResultView(domain: Domain) -> impl IntoView {\n view! {\n <Await\n future=move || {\n let domain = domain.clone();\n async move { get_dns_result(domain.clone()).await }\n }\n\n let:dns_results\n >\n\n {\n let dns_results = dns_results.clone();\n move || match dns_results.clone() {\n Ok((ips, cnames)) => {\n view! {\n <div class=\"grid grid-cols-2 gap-4\">\n <div>\n <h2 class=\"mb-2 text-lg font-bold\">IP Addresses</h2>\n <ul class=\"grid grid-cols-2\">\n <For\n each=move || { ips.clone() }\n key=|ip| *ip\n children=|ip| {\n let href = format!(\"/ip/{ip}\");\n view! {\n <li>\n <A href=href class=\"link link-neutral\">\n {ip.to_string()}\n </A>\n </li>\n }\n }\n />\n\n </ul>\n </div>\n <div>\n <h2 class=\"mb-2 text-lg font-bold\">CNAMEs</h2>\n <ul>\n <For\n each=move || { cnames.clone() }\n key=|(id, _cname)| *id\n children=|(_id, cname)| {\n let href = format!(\"/domain/{cname}\");\n view! {\n <li>\n <A href=href class=\"link link-neutral\">\n {cname}\n </A>\n </li>\n }\n }\n />\n\n </ul>\n\n </div>\n\n </div>\n }\n .into_view()\n }\n _ => view! { <p>\"Error\"</p> }.into_view(),\n }\n }\n\n </Await>\n }\n}\n\n#[server]\nasync fn get_blocked_by(\n domain: String,\n) -> Result<Vec<(FilterListUrl, RuleId, SourceId, crate::filterlist::RulePair)>, ServerFnError> {\n let records = sqlx::query!(\n r#\"\n SELECT Rules.id as \"rule_id: RuleId\",\n domain_rules_domain.domain as \"domain: Option<String>\", domain_rules.allow as \"domain_allow: Option<bool>\", subdomain as \"subdomain: Option<bool>\",\n ip_rules.ip_network as \"ip_network: Option<ipnetwork::IpNetwork>\", ip_rules.allow as \"ip_allow: Option<bool>\",\n source_id AS \"source_id: SourceId\", source, url\n FROM domains\n INNER JOIN rule_matches ON domains.id = rule_matches.domain_id\n INNER JOIN Rules on Rules.id = rule_matches.rule_id\n INNER JOIN rule_source ON rules.id = rule_source.rule_id\n INNER JOIN list_rules ON rule_source.id = list_rules.source_id\n INNER JOIN filterLists ON list_rules.list_id = filterLists.id\n LEFT JOIN domain_rules ON rules.domain_rule_id = domain_rules.id\n LEFT JOIN domains AS domain_rules_domain ON domain_rules_domain.id = domain_rules.domain_id\n LEFT JOIN ip_rules ON rules.ip_rule_id = ip_rules.id\n WHERE domains.domain = $1\n ORDER BY url\n LIMIT 100\n \"#r,\n domain\n )\n .fetch_all(&crate::server::get_db().await?)\n .await?;\n let rules = records\n .into_iter()\n .map(|record| {\n let rule_data = RuleData {\n rule_id: record.rule_id,\n domain: record.domain.clone(),\n domain_allow: record.domain_allow,\n domain_subdomain: record.subdomain,\n ip_network: record.ip_network,\n ip_allow: record.ip_allow,\n };\n let rule = rule_data.try_into()?;\n let source = record.source.clone();\n let pair = crate::filterlist::RulePair::new(source.into(), rule);\n let url = record.url.clone();\n Ok((url.parse()?, record.rule_id, record.source_id, pair))\n })\n .collect::<Result<Vec<_>, ServerFnError>>()?;\n\n Ok(rules)\n}\n\n#[component]\nfn BlockedBy(get_domain: Box<dyn Fn() -> Result<String, ParamsError>>) -> impl IntoView {\n let blocked_by = create_resource(get_domain, |domain| async move {\n let rules = get_blocked_by(domain?).await?;\n Ok::<_, ServerFnError>(rules)\n });\n view! {\n <Transition fallback=move || {\n view! { <p>\"Loading\" <Loading/></p> }\n }>\n {move || match blocked_by.get() {\n Some(Ok(rules)) => {\n view! {\n <table class=\"table table-zebra\">\n <For\n each=move || { rules.clone() }\n key=|(_url, rule_id, source_id, _pair)| (*rule_id, *source_id)\n children=|(url, rule_id, _source_id, pair)| {\n let source = pair.get_source().to_string();\n let rule = pair.get_rule().clone();\n view! {\n <tr>\n <td>\n <FilterListLink url=url/>\n </td>\n <td>{source}</td>\n <td>\n <A href=rule_id.get_href() class=\"link link-neutral\">\n <DisplayRule rule=rule/>\n </A>\n </td>\n </tr>\n }\n }\n />\n\n </table>\n }\n .into_view()\n }\n _ => view! { <p>\"Error\"</p> }.into_view(),\n }}\n\n </Transition>\n }\n}\n\n#[server]\nasync fn get_subdomains(domain: String) -> Result<Vec<String>, ServerFnError> {\n let records = sqlx::query!(\n \"SELECT subdomain_text.domain\n FROM domains\n INNER JOIN subdomains ON domains.id = subdomains.parent_domain_id\n INNER JOIN domains AS subdomain_text ON subdomains.domain_id = subdomain_text.id\n WHERE domains.domain = $1\n \",\n domain\n )\n .fetch_all(&crate::server::get_db().await?)\n .await?;\n let subdomains = records.into_iter().map(|record| record.domain).collect();\n Ok(subdomains)\n}\n\n#[component]\nfn DisplaySubdomains(get_domain: Box<dyn Fn() -> Result<String, ParamsError>>) -> impl IntoView {\n let subdomains = create_resource(get_domain, |domain| async move {\n let subdomains = get_subdomains(domain?).await?;\n Ok::<_, ServerFnError>(subdomains)\n });\n view! {\n <Transition fallback=move || {\n view! { <p>\"Loading\" <Loading/></p> }\n }>\n {move || match subdomains.get() {\n Some(Ok(subdomains)) => {\n view! {\n <table class=\"table table-zebra\">\n <For\n each=move || { subdomains.clone() }\n key=std::clone::Clone::clone\n children=|subdomain| {\n let domain_href = format!(\"/domain/{subdomain}\");\n view! {\n <tr>\n <td>\n <A href=domain_href class=\"link link-neutral\">\n {subdomain}\n </A>\n </td>\n </tr>\n }\n }\n />\n\n </table>\n }\n .into_view()\n }\n _ => view! { <p>\"Error\"</p> }.into_view(),\n }}\n\n </Transition>\n }\n}\n\n#[derive(Params, PartialEq)]\nstruct DomainParam {\n domain: Option<String>,\n}\n\n#[component]\npub fn DomainViewPage() -> impl IntoView {\n let params = use_params::<DomainParam>();\n let get_domain = move || {\n params.with(|param| {\n param.as_ref().map_err(Clone::clone).and_then(|param| {\n param\n .domain\n .clone()\n .ok_or_else(|| ParamsError::MissingParam(\"No domain\".into()))\n })\n })\n };\n let get_domain_parsed = move || {\n params.with(|param| {\n Ok::<_, ServerFnError>(\n param\n .as_ref()?\n .domain\n .as_ref()\n .ok_or_else(|| ParamsError::MissingParam(\"No domain\".into()))?\n .parse::<Domain>()?,\n )\n })\n };\n view! {\n <div>\n {move || {\n let domain = get_domain_parsed();\n match domain {\n Ok(domain) => {\n view! {\n <h1 class=\"text-3xl\">\"Domain: \" {domain.as_ref().to_string()}</h1>\n <DnsResultView domain=domain.clone()/>\n <p>\"Filtered by\"</p>\n <BlockedBy get_domain=Box::new(get_domain)/>\n <p>\"Subdomains\"</p>\n <DisplaySubdomains get_domain=Box::new(get_domain)/>\n }\n .into_view()\n }\n Err(err) => view! { <p>\"Error: \" {format!(\"{err:?}\")}</p> }.into_view(),\n }\n }}\n\n </div>\n }\n}\n" }, { "path": "src/error_template.rs", "content": "use http::status::StatusCode;\nuse leptos::*;\nuse thiserror::Error;\n\n#[derive(Clone, Debug, Error)]\npub enum AppError {\n #[error(\"Not Found\")]\n NotFound,\n}\n\nimpl AppError {\n pub fn status_code(&self) -> StatusCode {\n match self {\n AppError::NotFound => StatusCode::NOT_FOUND,\n }\n }\n}\n\n// A basic function to display errors served by the error boundaries.\n// Feel free to do more complicated things here than just displaying the error.\n#[component]\npub fn ErrorTemplate(\n #[prop(optional)] outside_errors: Option<Errors>,\n #[prop(optional)] errors: Option<RwSignal<Errors>>,\n) -> impl IntoView {\n let errors = match outside_errors {\n Some(e) => create_rw_signal(e),\n None => match errors {\n Some(e) => e,\n None => panic!(\"No Errors found and we expected errors!\"),\n },\n };\n // Get Errors from Signal\n let errors = errors.get_untracked();\n\n // Downcast lets us take a type that implements `std::error::Error`\n let errors: Vec<AppError> = errors\n .into_iter()\n .filter_map(|(_k, v)| v.downcast_ref::<AppError>().cloned())\n .collect();\n println!(\"Errors: {errors:#?}\");\n\n // Only the response code for the first error is actually sent from the server\n // this may be customized by the specific application\n #[cfg(feature = \"ssr\")]\n {\n use leptos_axum::ResponseOptions;\n let response = use_context::<ResponseOptions>();\n if let Some(response) = response {\n response.set_status(errors[0].status_code());\n }\n }\n\n view! {\n <h1>{if errors.len() > 1 { \"Errors\" } else { \"Error\" }}</h1>\n <For\n // a function that returns the items we're iterating over; a signal is fine\n each=move || { errors.clone().into_iter().enumerate() }\n // a unique key for each item as a reference\n key=|(index, _error)| *index\n // renders each item to a view\n children=move |error| {\n let error_string = error.1.to_string();\n let error_code = error.1.status_code();\n view! {\n <h2>{error_code.to_string()}</h2>\n <p>\"Error: \" {error_string}</p>\n }\n }\n />\n }\n}\n" }, { "path": "src/fileserv.rs", "content": "use crate::app::App;\nuse axum::response::Response as AxumResponse;\nuse axum::{\n body::Body,\n extract::State,\n http::{Request, Response, StatusCode, Uri},\n response::IntoResponse,\n};\nuse leptos::*;\nuse tower::ServiceExt;\nuse tower_http::services::ServeDir;\n\npub async fn file_and_error_handler(\n uri: Uri,\n State(options): State<LeptosOptions>,\n req: Request<Body>,\n) -> AxumResponse {\n let root = options.site_root.clone();\n let res = get_static_file(uri.clone(), &root).await.unwrap();\n\n if res.status() == StatusCode::OK {\n res.into_response()\n } else {\n let handler = leptos_axum::render_app_to_stream(options.clone(), App);\n handler(req).await.into_response()\n }\n}\n\nasync fn get_static_file(uri: Uri, root: &str) -> Result<Response<Body>, (StatusCode, String)> {\n let req = Request::builder()\n .uri(uri.clone())\n .body(Body::empty())\n .unwrap();\n // `ServeDir` implements `tower::Service` so we can call it with `tower::ServiceExt::oneshot`\n // This path is relative to the cargo root\n match ServeDir::new(root).oneshot(req).await {\n Ok(res) => Ok(res.into_response()),\n Err(err) => Err((\n StatusCode::INTERNAL_SERVER_ERROR,\n format!(\"Something went wrong: {err}\"),\n )),\n }\n}\n" }, { "path": "src/filterlist.rs", "content": "pub use crate::domain::Domain;\nuse crate::PAGE_SIZE;\nuse crate::{rule::RuleId, source::SourceId};\nuse leptos::*;\nuse leptos::{server, ServerFnError};\n\nuse crate::rule::DisplayRule;\n#[cfg(feature = \"ssr\")]\nuse crate::rule::RuleData;\n\nuse leptos_router::*;\nuse serde::{Deserialize, Serialize};\nuse std::str::FromStr;\nuse std::sync::Arc;\n\n#[cfg_attr(feature = \"ssr\", derive(sqlx::Encode, sqlx::Decode))]\n#[derive(Serialize, Deserialize, Debug, Copy, Clone, PartialEq, Eq, Hash)]\npub struct ListId(i32);\n\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct FilterListRecord {\n pub name: Arc<str>,\n pub list_format: FilterListType,\n pub author: Arc<str>,\n pub license: Arc<str>,\n pub expires: std::time::Duration,\n pub last_updated: Option<chrono::DateTime<chrono::Utc>>,\n pub list_size: usize,\n}\n\n#[derive(Clone, Debug, Serialize, Deserialize, PartialEq, PartialOrd, Eq, Ord, Hash)]\n#[serde(transparent)]\npub struct FilterListUrl {\n url: Arc<str>,\n}\n\nimpl FilterListUrl {\n pub fn as_str(&self) -> &str {\n self.as_ref()\n }\n pub fn to_internal_path(&self) -> Option<std::path::PathBuf> {\n if self.as_str().starts_with(\"internal/\") {\n Some(std::path::PathBuf::from(self.as_str()))\n } else {\n None\n }\n }\n}\n\nimpl std::ops::Deref for FilterListUrl {\n type Target = str;\n fn deref(&self) -> &Self::Target {\n self.url.as_ref()\n }\n}\n\nimpl FromStr for FilterListUrl {\n type Err = url::ParseError;\n fn from_str(s: &str) -> Result<Self, Self::Err> {\n match s {\n \"internal/blocklist.txt\" | \"internal/block_ips.txt\" | \"internal/allowlist.txt\" => {\n Ok(Self { url: s.into() })\n }\n s => Ok(Self {\n url: url::Url::parse(s)?.as_str().into(),\n }),\n }\n }\n}\n\n#[derive(Copy, Clone, Debug, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord, Hash)]\npub enum FilterListType {\n Adblock,\n DomainBlocklist,\n DomainBlocklistWithoutSubdomains,\n DomainAllowlist,\n IPBlocklist,\n IPAllowlist,\n IPNetBlocklist,\n DenyHosts,\n RegexAllowlist,\n RegexBlocklist,\n Hostfile,\n}\n\nimpl FilterListType {\n pub fn as_str(&self) -> &'static str {\n match self {\n Self::Adblock => \"Adblock\",\n Self::DomainBlocklist => \"DomainBlocklist\",\n Self::DomainBlocklistWithoutSubdomains => \"DomainBlocklistWithoutSubdomains\",\n Self::DomainAllowlist => \"DomainAllowlist\",\n Self::IPBlocklist => \"IPBlocklist\",\n Self::IPAllowlist => \"IPAllowlist\",\n Self::IPNetBlocklist => \"IPNetBlocklist\",\n Self::DenyHosts => \"DenyHosts\",\n Self::RegexAllowlist => \"RegexAllowlist\",\n Self::RegexBlocklist => \"RegexBlocklist\",\n Self::Hostfile => \"Hostfile\",\n }\n }\n}\n#[derive(Debug, thiserror::Error)]\npub struct InvalidFilterListTypeError;\n\nimpl std::fmt::Display for InvalidFilterListTypeError {\n fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {\n write!(f, \"Invalid FilterListType\")\n }\n}\n\nimpl std::str::FromStr for FilterListType {\n type Err = InvalidFilterListTypeError;\n fn from_str(s: &str) -> Result<Self, Self::Err> {\n match s {\n \"Adblock\" => Ok(Self::Adblock),\n \"DomainBlocklist\" => Ok(Self::DomainBlocklist),\n \"DomainBlocklistWithoutSubdomains\" => Ok(Self::DomainBlocklistWithoutSubdomains),\n \"DomainAllowlist\" => Ok(Self::DomainAllowlist),\n \"IPBlocklist\" => Ok(Self::IPBlocklist),\n \"IPAllowlist\" => Ok(Self::IPAllowlist),\n \"IPNetBlocklist\" => Ok(Self::IPNetBlocklist),\n \"DenyHosts\" => Ok(Self::DenyHosts),\n \"RegexAllowlist\" => Ok(Self::RegexAllowlist),\n \"RegexBlocklist\" => Ok(Self::RegexBlocklist),\n \"Hostfile\" => Ok(Self::Hostfile),\n _ => Err(InvalidFilterListTypeError),\n }\n }\n}\n\n#[derive(Clone, Debug, Serialize, Deserialize)]\npub struct FilterListMap(\n pub Vec<(FilterListUrl, FilterListRecord)>,\n // Just so it is consistently ordered\n);\n\n#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord, Hash)]\npub struct DomainRule {\n pub domain: Domain,\n pub allow: bool,\n pub subdomain: bool,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord, Hash)]\npub struct IpRule {\n pub ip: ipnetwork::IpNetwork,\n pub allow: bool,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord, Hash)]\npub enum Rule {\n #[serde(rename = \"d\")]\n Domain(DomainRule),\n IpRule(IpRule),\n #[serde(rename = \"u\")]\n Unknown,\n #[serde(rename = \"i\")]\n Invalid,\n}\n\n#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, PartialOrd, Ord, Hash)]\n#[serde(into = \"(Arc<str>, Rule)\")]\n#[serde(from = \"(Arc<str>, Rule)\")]\npub struct RulePair {\n source: Arc<str>,\n rule: Rule,\n}\n\nimpl RulePair {\n pub fn new(source: Arc<str>, rule: Rule) -> RulePair {\n RulePair { source, rule }\n }\n pub fn get_rule(&self) -> &Rule {\n &self.rule\n }\n pub fn get_source(&self) -> &Arc<str> {\n &self.source\n }\n}\nimpl From<RulePair> for (Arc<str>, Rule) {\n fn from(val: RulePair) -> Self {\n (val.source, val.rule)\n }\n}\nimpl From<(Arc<str>, Rule)> for RulePair {\n fn from((source, rule): (Arc<str>, Rule)) -> Self {\n Self { source, rule }\n }\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_lines(contents: &str, parser: &dyn Fn(&str) -> Option<Rule>) -> Vec<RulePair> {\n let mut rules = vec![];\n for line in contents.lines() {\n let source = line;\n if line.is_empty() {\n continue;\n }\n if let Some(rule) = parser(line) {\n rules.push(RulePair {\n source: source.into(),\n rule,\n });\n }\n }\n rules\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_domain_list_line(line: &str, allow: bool, subdomain: bool) -> Option<Rule> {\n let line = line.split('#').next()?;\n if line.is_empty() {\n return None;\n }\n let line = line.trim();\n let mut segments = line.split_whitespace();\n match (segments.next(), segments.next(), segments.next()) {\n (Some(domain), None, None) | (Some(\"127.0.0.1\" | \"0.0.0.0\"), Some(domain), None) => {\n let (subdomain, domain) = domain\n .strip_prefix(\"*.\")\n .map_or((subdomain, domain), |domain| (true, domain));\n if let Ok(domain) = domain.parse() {\n let domain_rule = DomainRule {\n domain,\n allow,\n subdomain,\n };\n Some(Rule::Domain(domain_rule))\n } else {\n Some(Rule::Invalid)\n }\n }\n _ => Some(Rule::Invalid),\n }\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_domain_list(contents: &str, allow: bool, subdomain: bool) -> Vec<RulePair> {\n parse_lines(contents, &|line| {\n parse_domain_list_line(line, allow, subdomain)\n })\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_adblock_line(line: &str) -> Option<Rule> {\n let rule = line;\n if rule.starts_with('!') // Comment\n || rule.contains('#') // CSS selector\n || !rule.trim_matches('.').contains('.') // Not a domain\n || matches! {rule, \"[Adblock Plus 2.0]\" | \"[Adblock Plus 1.1]\"}\n {\n return None;\n }\n\n let mut match_end_domain = false;\n\n let rule = if let Some((start, tags)) = rule.split_once('$') {\n let mut block_site = false;\n let mut has_specific_filters = false;\n let mut has_unknown_tag = false;\n for tag in tags.split(',') {\n if tag.starts_with('~') // Can't partially block a site\n || tag.starts_with(\"rewrite=\")\n // Can't rewrite a site\n {\n return None;\n } else if let Some(domain_tag) = tag.strip_prefix(\"domain=\") {\n for domain in domain_tag.split('|') {\n if !start.contains(domain) {\n return None;\n }\n }\n } else {\n match tag {\n \"3p\" | \"doc\" | \"document\" | \"all\" => {\n match_end_domain = true;\n block_site = true;\n }\n \"popup\" | \"ghide\" | \"generichide\" | \"genericblock\" | \"image\" | \"script\"\n | \"third-party\" | \"xmlhttprequest\" | \"stylesheet\" | \"subdocument\" | \"media\"\n | \"csp\" => {\n has_specific_filters = true;\n }\n \"important\" => {}\n _ => {\n has_unknown_tag = true;\n }\n }\n }\n }\n if has_specific_filters && !block_site {\n return None;\n }\n if has_unknown_tag {\n return Some(Rule::Unknown);\n }\n start\n } else {\n rule\n };\n\n if let Some(rule) = rule.strip_prefix('/') {\n if let Some(_rule) = rule.strip_suffix('/') {\n // REGEX\n } else {\n return None; // Path selector\n }\n }\n let (rule, exception) = if let Some(rule) = rule.strip_prefix(\"@@\") {\n (rule, true)\n } else {\n (rule, false)\n };\n let (rule, mut match_start_domain, match_exact_start) =\n if let Some(rule) = rule.strip_prefix(\"||\") {\n (rule, true, false)\n } else {\n let (rule, match_exact_start) = rule\n .strip_prefix('|')\n .map_or((rule, false), |rule| (rule, true));\n (rule, false, match_exact_start)\n };\n\n let (rule, match_end_domain_exact) = rule\n .strip_suffix('|')\n .map_or((rule, false), |rule| (rule, true));\n let (mut rule, match_end_domain) = rule\n .strip_suffix('^')\n .map_or((rule, match_end_domain), |rule| (rule, true));\n if rule.contains('/') {\n return None; // Path selector\n }\n if rule.contains('*') {\n return Some(Rule::Unknown);\n }\n if !match_start_domain {\n (match_start_domain, rule) = rule\n .strip_prefix('.')\n .or_else(|| rule.strip_prefix(\"*.\"))\n .map_or((false, rule), |rule| (true, rule));\n }\n if match_start_domain && (match_end_domain | match_end_domain_exact) {\n if let Ok(domain) = rule.parse() {\n let domain_rule = DomainRule {\n domain,\n allow: exception,\n subdomain: !match_exact_start,\n };\n return Some(Rule::Domain(domain_rule));\n } else if let Ok(ip) = rule.parse::<ipnetwork::IpNetwork>() {\n return Some(Rule::IpRule(IpRule {\n ip,\n allow: exception,\n }));\n }\n }\n Some(Rule::Unknown)\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_adblock(contents: &str) -> Vec<RulePair> {\n parse_lines(contents, &parse_adblock_line)\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_regex_line(line: &str) -> Option<Rule> {\n if line.starts_with('#') {\n return None;\n }\n if let Some(rule) = line.strip_prefix(r\"(^|\\.)\") {\n if let Some(rule) = rule.strip_suffix('$') {\n let mut rule = rule.to_string();\n rule.retain(|c| c != '/');\n if let Ok(domain) = rule.parse() {\n let domain_rule = DomainRule {\n domain,\n allow: false,\n subdomain: true,\n };\n return Some(Rule::Domain(domain_rule));\n }\n }\n }\n Some(Rule::Unknown)\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_ip_network_line(line: &str, allow: bool) -> Option<Rule> {\n let line = line.trim();\n if line.is_empty() || line.starts_with('#') {\n return None;\n }\n if let Ok(ip) = line.parse::<ipnetwork::IpNetwork>() {\n Some(Rule::IpRule(IpRule { ip, allow }))\n } else {\n Some(Rule::Unknown)\n }\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_ip_network_list(contents: &str, allow: bool) -> Vec<RulePair> {\n parse_lines(contents, &|line| parse_ip_network_line(line, allow))\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_regex(contents: &str) -> Vec<RulePair> {\n parse_lines(contents, &parse_regex_line)\n}\n\n#[cfg(feature = \"ssr\")]\nfn parse_unknown_lines(contents: &str) -> Vec<RulePair> {\n parse_lines(contents, &|_| Some(Rule::Unknown))\n}\n\n#[cfg(feature = \"ssr\")]\npub fn parse_list_contents(contents: &str, list_format: FilterListType) -> Vec<RulePair> {\n match list_format {\n FilterListType::Adblock => parse_adblock(contents),\n FilterListType::DomainBlocklist => parse_domain_list(contents, false, true),\n FilterListType::DomainBlocklistWithoutSubdomains => {\n parse_domain_list(contents, false, false)\n }\n FilterListType::DomainAllowlist => parse_domain_list(contents, true, false),\n FilterListType::IPBlocklist => parse_ip_network_list(contents, false),\n FilterListType::IPAllowlist => parse_ip_network_list(contents, true),\n FilterListType::IPNetBlocklist => parse_ip_network_list(contents, false),\n FilterListType::DenyHosts => parse_unknown_lines(contents),\n FilterListType::RegexAllowlist => parse_unknown_lines(contents),\n FilterListType::RegexBlocklist => parse_regex(contents),\n FilterListType::Hostfile => parse_domain_list(contents, false, true),\n }\n}\n\n#[server(ParseList)]\npub async fn parse_list(url: FilterListUrl) -> Result<(), ServerFnError> {\n let start = std::time::Instant::now();\n let pool = crate::server::get_db().await?;\n let url_str = url.as_str();\n let record = sqlx::query!(\n \"SELECT id, format, contents FROM filterLists WHERE url = $1\",\n url_str\n )\n .fetch_one(&pool)\n .await?;\n let list_format: FilterListType = record.format.parse()?;\n\n log::info!(\n \"Parsing {} as format {}\",\n url.as_str(),\n list_format.as_str()\n );\n let list_id = record.id;\n let rules = {\n let contents = record\n .contents\n .ok_or_else(|| ServerFnError::new(\"No contents for list\"))?;\n parse_list_contents(&contents, list_format)\n };\n let (mut domain_src, mut domains, mut allow, mut subdomain) = (vec![], vec![], vec![], vec![]);\n let (mut ip_source, mut ips, mut allow_ips) = (vec![], vec![], vec![]);\n let mut other_rules_src = vec![];\n for rule in &rules {\n let source = rule.get_source().as_ref();\n let source = source[..source.len().min(2000)].to_string();\n match rule.get_rule() {\n Rule::Domain(domain_rule) => {\n domain_src.push(source);\n domains.push(domain_rule.domain.clone());\n allow.push(domain_rule.allow);\n subdomain.push(domain_rule.subdomain);\n }\n Rule::IpRule(ip_rule) => {\n ip_source.push(source);\n ips.push(ip_rule.ip);\n allow_ips.push(ip_rule.allow);\n }\n _ => {\n other_rules_src.push(source);\n }\n }\n }\n log::info!(\n \"Inserting {} rules ({} domain rules, {} IP rules, {} unknown)\",\n rules.len(),\n domain_src.len(),\n ip_source.len(),\n other_rules_src.len()\n );\n\n sqlx::query!(\n \"INSERT INTO domains (domain)\n SELECT domain FROM UNNEST($1::text[]) AS t(domain) ON CONFLICT DO NOTHING\",\n &domains[..] as _\n )\n .execute(&pool)\n .await?;\n\n let mut tx = pool.begin().await?;\n sqlx::query! {\"DELETE FROM list_rules WHERE list_id = $1\", list_id}\n .execute(&mut *tx)\n .await?;\n\n sqlx::query!(\"INSERT INTO domain_rules (domain_id, allow, subdomain)\n SELECT domains.id, allow, subdomain FROM UNNEST($1::text[], $2::bool[], $3::bool[]) AS t(domain, allow, subdomain)\n INNER JOIN domains ON domains.domain = t.domain\n ON CONFLICT DO NOTHING\",\n &domains[..] as _,\n &allow[..],\n &subdomain[..]\n ).execute(&mut *tx).await?;\n\n sqlx::query!(\"INSERT INTO Rules (domain_rule_id)\n SELECT domain_rules.id FROM UNNEST($1::text[], $2::bool[], $3::bool[]) AS t(domain, allow, subdomain)\n INNER JOIN domains ON domains.domain = t.domain\n INNER JOIN domain_rules ON domain_rules.domain_id = domains.id AND domain_rules.allow = t.allow AND domain_rules.subdomain = t.subdomain\n ON CONFLICT DO NOTHING\",\n &domains[..] as _,\n &allow[..],\n &subdomain[..]\n ).execute(&mut *tx).await?;\n\n sqlx::query!(\"INSERT INTO rule_source (source, rule_id)\n SELECT source, Rules.id FROM UNNEST ($1::text[], $2::text[], $3::bool[], $4::bool[]) AS t(source, domain, allow, subdomain)\n INNER JOIN domains ON domains.domain = t.domain\n INNER JOIN domain_rules ON domain_rules.domain_id = domains.id AND domain_rules.allow = t.allow AND domain_rules.subdomain = t.subdomain\n INNER JOIN Rules ON Rules.domain_rule_id = domain_rules.id\n ON CONFLICT DO NOTHING\",\n &domain_src[..],\n &domains[..] as _,\n &allow[..],\n &subdomain[..]\n ).execute(&mut *tx).await?;\n\n sqlx::query!(\n \"INSERT INTO ip_rules (ip_network, allow)\n SELECT ip, allow FROM UNNEST($1::inet[], $2::bool[]) AS t(ip, allow)\n ON CONFLICT DO NOTHING\",\n &ips[..],\n &allow_ips[..]\n )\n .execute(&mut *tx)\n .await?;\n\n sqlx::query!(\n \"INSERT INTO Rules (ip_rule_id)\n SELECT ip_rules.id FROM UNNEST($1::inet[], $2::bool[]) AS t(ip, allow)\n INNER JOIN ip_rules ON ip_rules.ip_network = t.ip AND ip_rules.allow = t.allow\n ON CONFLICT DO NOTHING\",\n &ips[..],\n &allow_ips[..]\n )\n .execute(&mut *tx)\n .await?;\n sqlx::query!(\n \"INSERT INTO rule_source (source, rule_id)\n SELECT source, Rules.id FROM UNNEST ($1::text[], $2::inet[], $3::bool[]) AS t(source, ip, allow)\n INNER JOIN ip_rules ON ip_rules.ip_network = t.ip AND ip_rules.allow = t.allow\n INNER JOIN Rules ON Rules.ip_rule_id = ip_rules.id\n ON CONFLICT DO NOTHING\",\n &ip_source[..],\n &ips[..],\n &allow_ips[..]\n )\n .execute(&mut *tx)\n .await?;\n\n sqlx::query!(\"INSERT INTO list_rules (list_id, source_id)\n SELECT $1, rule_source.id FROM UNNEST ($2::text[], $3::inet[], $4::bool[]) AS t(source, ip, allow)\n INNER JOIN ip_rules ON ip_rules.ip_network = t.ip AND ip_rules.allow = t.allow\n INNER JOIN Rules ON Rules.ip_rule_id = ip_rules.id\n INNER JOIN rule_source ON rule_source.rule_id = Rules.id\n WHERE rule_source.source = t.source\n ON CONFLICT DO NOTHING\",\n list_id,\n &ip_source[..],\n &ips[..],\n &allow_ips[..]\n ).execute(&mut *tx).await?;\n\n sqlx::query!(\n \"INSERT INTO Rules (domain_rule_id, ip_rule_id) VALUES (NULL, NULL)\n ON CONFLICT DO NOTHING\",\n )\n .execute(&mut *tx)\n .await?;\n\n sqlx::query!(\n \"INSERT INTO rule_source (source, rule_id)\n SELECT source, Rules.id FROM UNNEST ($1::text[]) AS t(source)\n INNER JOIN Rules ON Rules.domain_rule_id IS NULL AND Rules.ip_rule_id IS NULL\n ON CONFLICT DO NOTHING\",\n &other_rules_src[..],\n )\n .execute(&mut *tx)\n .await?;\n\n sqlx::query!(\n \"INSERT INTO list_rules (list_id, source_id)\n SELECT $1, rule_source.id FROM UNNEST($2::text[], $3::text[], $4::bool[], $5::bool[]) AS t(source, domain, allow, subdomain)\n INNER JOIN domains ON domains.domain = t.domain\n INNER JOIN domain_rules ON domain_rules.domain_id = domains.id AND domain_rules.allow = t.allow AND domain_rules.subdomain = t.subdomain\n INNER JOIN Rules ON Rules.domain_rule_id = domain_rules.id\n INNER JOIN rule_source ON rule_source.rule_id = Rules.id\n WHERE rule_source.source = t.source\n ON CONFLICT DO NOTHING\",\n list_id,\n &domain_src[..],\n &domains[..] as _,\n &allow[..],\n &subdomain[..]\n ).execute(&mut *tx).await?;\n\n sqlx::query!(\n \"INSERT INTO list_rules (list_id, source_id)\n SELECT $1, rule_source.id FROM UNNEST ($2::text[]) AS t(source)\n INNER JOIN Rules ON Rules.domain_rule_id IS NULL AND Rules.ip_rule_id IS NULL\n INNER JOIN rule_source ON rule_source.rule_id = Rules.id\n WHERE rule_source.source = t.source\n ON CONFLICT DO NOTHING\",\n list_id,\n &other_rules_src[..],\n )\n .execute(&mut *tx)\n .await?;\n\n sqlx::query!(\n \"UPDATE filterLists SET rule_count=$2\n WHERE id = $1\",\n list_id,\n rules.len() as i32\n )\n .execute(&mut *tx)\n .await?;\n\n log::info!(\"Inserted list rules\");\n\n tx.commit().await?;\n log::info!(\"Total time: {:?}\", start.elapsed());\n Ok(())\n}\n\n#[derive(Clone, Debug, Serialize, Deserialize)]\nstruct CsvRecord {\n pub name: String,\n pub url: FilterListUrl,\n pub author: String,\n pub license: String,\n pub expires: u64,\n pub list_type: FilterListType,\n}\n\n#[server]\npub async fn load_filter_map() -> Result<(), ServerFnError> {\n dotenvy::dotenv()?;\n let filterlists_path: std::path::PathBuf = std::env::var(\"FILTERLISTS_PATH\")?.parse()?;\n let contents = tokio::fs::read_to_string(filterlists_path).await?;\n let records = csv::Reader::from_reader(contents.as_bytes())\n .deserialize::<CsvRecord>()\n .collect::<Result<Vec<CsvRecord>, _>>()?;\n let mut urls = Vec::new();\n let mut names = Vec::new();\n let mut formats = Vec::new();\n let mut expires_list = Vec::new();\n let mut authors = Vec::new();\n let mut licenses = Vec::new();\n\n for csv_record in &records {\n let url = csv_record.url.as_str().to_string();\n let name = csv_record.name.clone();\n let format = csv_record.list_type.as_str().to_string();\n let expires = csv_record.expires as i32;\n let author = csv_record.author.clone();\n let license = csv_record.license.clone();\n urls.push(url);\n names.push(name);\n formats.push(format);\n expires_list.push(expires);\n authors.push(author);\n licenses.push(license);\n }\n\n let pool = crate::server::get_db().await?;\n sqlx::query!(\n \"INSERT INTO filterLists (url, name, format, expires, author, license)\n SELECT * FROM UNNEST($1::text[], $2::text[], $3::text[], $4::int[], $5::text[], $6::text[])\n ON CONFLICT (url) DO UPDATE\n SET name = EXCLUDED.name, format = EXCLUDED.format, expires = EXCLUDED.expires, author = EXCLUDED.author, license = EXCLUDED.license\n \",\n &urls,\n &names,\n &formats,\n &expires_list,\n &authors,\n &licenses\n ).execute(&pool).await?;\n write_filter_map().await?;\n Ok(())\n}\n\n#[server]\npub async fn watch_filter_map() -> Result<(), ServerFnError> {\n dotenvy::dotenv()?;\n let filterlists_path: std::path::PathBuf = std::env::var(\"FILTERLISTS_PATH\")?.parse()?;\n use notify::Watcher;\n let notify = std::sync::Arc::new(tokio::sync::Notify::new());\n let notify2 = notify.clone();\n load_filter_map().await?;\n let mut watcher = notify::recommended_watcher(move |_| {\n notify.notify_one();\n })?;\n watcher.watch(&filterlists_path, notify::RecursiveMode::NonRecursive)?;\n let mut last_updated = std::time::Instant::now();\n loop {\n notify2.notified().await;\n if last_updated.elapsed() > std::time::Duration::from_millis(200) {\n load_filter_map().await?;\n last_updated = std::time::Instant::now();\n }\n }\n}\n\n#[server]\npub async fn write_filter_map() -> Result<(), ServerFnError> {\n use csv::Writer;\n dotenvy::dotenv()?;\n let filterlists_path: std::path::PathBuf = std::env::var(\"FILTERLISTS_PATH\")?.parse()?;\n let pool = crate::server::get_db().await?;\n let rows = sqlx::query!(\"SELECT url, name, format, expires, author, license FROM filterLists\")\n .fetch_all(&pool)\n .await?;\n let mut records = Vec::new();\n for record in rows {\n records.push(CsvRecord {\n name: record.name.unwrap_or(String::new()),\n url: record.url.parse()?,\n author: record.author.unwrap_or(String::new()),\n license: record.license.unwrap_or(String::new()),\n expires: record.expires as u64,\n list_type: FilterListType::from_str(&record.format)?,\n });\n }\n records.sort_by_key(|record| (record.name.clone(), record.url.clone()));\n records.reverse();\n let mut wtr = Writer::from_path(filterlists_path)?;\n for record in records {\n wtr.serialize(record)?;\n }\n Ok(())\n}\n\n#[server]\npub async fn get_filter_map() -> Result<FilterListMap, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let rows = sqlx::query!(\n \"SELECT url, name, format, expires, author, license, lastupdated, rule_count\n FROM filterLists\"\n )\n .fetch_all(&pool)\n .await?;\n\n let mut filter_list_map = Vec::new();\n for record in rows {\n let url = record.url.parse()?;\n let record = FilterListRecord {\n name: record.name.unwrap_or(String::new()).into(),\n list_format: FilterListType::from_str(&record.format)?,\n author: record.author.unwrap_or(String::new()).into(),\n license: record.license.unwrap_or(String::new()).into(),\n expires: std::time::Duration::from_secs(record.expires as u64),\n last_updated: record.lastupdated,\n list_size: record.rule_count as usize,\n };\n filter_list_map.push((url, record));\n }\n\n Ok(FilterListMap(filter_list_map))\n}\n\n#[cfg(feature = \"ssr\")]\nstruct LastVersionData {\n last_updated: chrono::DateTime<chrono::Utc>,\n etag: Option<String>,\n}\n\n#[cfg(feature = \"ssr\")]\nasync fn get_last_version_data(\n url: &FilterListUrl,\n) -> Result<Option<LastVersionData>, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let url_str = url.as_str();\n #[allow(non_camel_case_types)]\n let last_version_data = sqlx::query!(\n r#\"SELECT lastUpdated as \"last_updated: chrono::DateTime<chrono::Utc>\", etag FROM filterLists WHERE url = $1\"#,\n url_str\n )\n .fetch_one(&pool)\n .await\n .ok();\n let last_version_data = last_version_data.and_then(|row| {\n Some(LastVersionData {\n last_updated: row.last_updated?,\n etag: row.etag,\n })\n });\n Ok(last_version_data)\n}\n\n#[server]\npub async fn get_last_updated(\n url: FilterListUrl,\n) -> Result<Option<chrono::DateTime<chrono::Utc>>, ServerFnError> {\n get_last_version_data(&url)\n .await\n .map(|data| data.map(|data| data.last_updated))\n}\n\n#[cfg(feature = \"ssr\")]\n#[derive(thiserror::Error, Debug)]\nenum UpdateListError {\n #[error(\"Failed to fetch list\")]\n FailedToFetch,\n}\n\n#[server(UpdateListFn)]\npub async fn update_list(url: FilterListUrl) -> Result<(), ServerFnError> {\n let pool = crate::server::get_db().await?;\n let old_contents = sqlx::query!(\n \"SELECT contents FROM filterLists WHERE url = $1\",\n url.as_str()\n )\n .fetch_one(&pool)\n .await?\n .contents;\n if let Some(internal_path) = url.to_internal_path() {\n let contents = tokio::fs::read_to_string(&internal_path).await?;\n let mut lines = contents.lines().collect::<Vec<_>>();\n lines.sort_unstable();\n lines.dedup();\n let sorted_contents = lines.join(\"\\n\");\n tokio::fs::write(internal_path, &sorted_contents).await?;\n let new_last_updated = chrono::Utc::now();\n sqlx::query!(\n \"UPDATE filterLists\n SET lastUpdated = $2, contents = $3\n WHERE url = $1\n \",\n url.as_str(),\n new_last_updated,\n sorted_contents\n )\n .execute(&pool)\n .await?;\n if old_contents != Some(sorted_contents) {\n parse_list(url).await?;\n }\n return Ok(());\n }\n log::info!(\"Updating {}\", url.as_str());\n let url_str = url.as_str();\n let last_updated = get_last_version_data(&url).await?;\n let mut req = reqwest::Client::new().get(url_str);\n if let Some(last_updated) = last_updated {\n req = req.header(\n \"if-modified-since\",\n last_updated\n .last_updated\n .format(\"%a, %d %b %Y %H:%M:%S GMT\")\n .to_string(),\n );\n if let Some(etag) = last_updated.etag {\n req = req.header(\"if-none-match\", etag);\n }\n }\n let response = req.send().await?;\n match response.status() {\n reqwest::StatusCode::NOT_MODIFIED => {\n log::info!(\"Not modified {:?}\", url_str);\n sqlx::query!(\n \"UPDATE filterLists\n SET lastUpdated = NOW()\n WHERE url = $1\n \",\n url_str\n )\n .execute(&pool)\n .await?;\n Ok(())\n }\n reqwest::StatusCode::OK => {\n let headers = response.headers().clone();\n let etag = headers.get(\"etag\").and_then(|item| item.to_str().ok());\n let body = response.text().await?;\n log::info!(\"Updated {} size ({})\", url_str, body.len());\n sqlx::query!(\n \"UPDATE filterLists\n SET contents = $2, etag = $3\n WHERE url = $1\n \",\n url_str,\n body,\n etag\n )\n .execute(&pool)\n .await?;\n if Some(body) == old_contents {\n log::info!(\"No change in contents for {}\", url_str);\n } else {\n parse_list(url.clone()).await?;\n }\n sqlx::query!(\n \"UPDATE filterLists\n SET lastUpdated = NOW()\n WHERE url = $1\n \",\n url_str\n )\n .execute(&pool)\n .await?;\n Ok(())\n }\n status => {\n log::error!(\"Error fetching {}: {:?}\", url_str, status);\n Err(UpdateListError::FailedToFetch.into())\n }\n }\n}\n\n#[server(DeleteListFn)]\npub async fn delete_list(url: FilterListUrl) -> Result<(), ServerFnError> {\n let pool = crate::server::get_db().await?;\n let url_str = url.as_str();\n sqlx::query!(\n \"DELETE FROM list_rules\n WHERE list_rules.list_id IN (\n SELECT id FROM filterLists WHERE url = $1\n )\",\n url_str\n )\n .execute(&pool)\n .await?;\n sqlx::query!(\"DELETE FROM filterLists WHERE url = $1\", url_str)\n .execute(&pool)\n .await?;\n write_filter_map().await?;\n Ok(())\n}\n\n#[component]\npub fn FilterListLink(url: FilterListUrl) -> impl IntoView {\n let href = format!(\n \"/list{}\",\n params_map! {\n \"url\" => url.as_str(),\n }\n .to_query_string(),\n );\n view! {\n <A href=href class=\"link link-neutral\">\n {url.as_str().to_string()}\n </A>\n }\n}\n\n#[server]\nasync fn get_list_size(url: FilterListUrl) -> Result<Option<usize>, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let url_str = url.as_str();\n let record = sqlx::query!(\n \"SELECT id, rule_count FROM filterLists WHERE url = $1\",\n url_str\n )\n .fetch_one(&pool)\n .await?;\n let list_id = record.id;\n let count = record.rule_count;\n if count == 0 {\n let count = sqlx::query!(\n \"SELECT COUNT(*) FROM list_rules WHERE list_id = $1\",\n list_id\n )\n .fetch_one(&pool)\n .await?\n .count;\n if let Some(count) = count {\n sqlx::query!(\n \"UPDATE filterLists SET rule_count = $1 WHERE id = $2\",\n count as i32,\n list_id\n )\n .execute(&pool)\n .await?;\n Ok(Some(count as usize))\n } else {\n Ok(None)\n }\n } else {\n Ok(Some(count as usize))\n }\n}\n\n#[component]\npub fn ListSize(url: FilterListUrl, list_size: Option<usize>) -> impl IntoView {\n if let Some(size) = list_size {\n if size > 0 {\n return size.into_view();\n }\n }\n view! {\n <Await\n future=move || {\n let url = url.clone();\n async { get_list_size(url).await }\n }\n\n let:size\n >\n {match size {\n Err(err) => format!(\"{err:?}\").into_view(),\n Ok(None) => \"Never\".into_view(),\n Ok(Some(size)) => size.into_view(),\n }}\n\n </Await>\n }\n .into_view()\n}\n\n#[server]\nasync fn get_list_page(\n url: FilterListUrl,\n page: Option<usize>,\n page_size: usize,\n) -> Result<Vec<(RuleId, SourceId, crate::filterlist::RulePair)>, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let url_str = url.as_str();\n let id = sqlx::query!(\"SELECT id FROM filterLists WHERE url = $1\", url_str)\n .fetch_one(&pool)\n .await?\n .id;\n let start = page.unwrap_or(0) * page_size;\n let records = sqlx::query!(\n r#\"SELECT Rules.id AS \"rule_id: RuleId\", rule_source.id AS \"source_id: SourceId\", rule_source.source,\n domain as \"domain: Option<String>\" , domain_rules.allow as \"domain_allow: Option<bool>\", subdomain as \"subdomain: Option<bool>\",\n ip_network as \"ip_network: Option<ipnetwork::IpNetwork>\", ip_rules.allow as \"ip_allow: Option<bool>\"\n FROM list_rules\n INNER JOIN rule_source ON rule_source.id = list_rules.source_id\n INNER JOIN Rules ON Rules.id = rule_source.rule_id\n LEFT JOIN domain_rules ON domain_rules.id = Rules.domain_rule_id\n LEFT JOIN domains ON domains.id = domain_rules.domain_id\n LEFT JOIN ip_rules ON ip_rules.id = Rules.ip_rule_id\n WHERE list_id = $1\n ORDER BY list_rules.source_id\n LIMIT $2 OFFSET $3\n \"#r,\n id,\n page_size as i64 ,\n start as i64\n )\n .fetch_all(&pool)\n .await?;\n let rules = records\n .iter()\n .map(|record| {\n let rule_data = RuleData {\n rule_id: record.rule_id,\n domain: record.domain.clone(),\n domain_allow: record.domain_allow,\n domain_subdomain: record.subdomain,\n ip_network: record.ip_network,\n ip_allow: record.ip_allow,\n };\n let rule = rule_data.try_into()?;\n let source = record.source.clone();\n let pair = crate::filterlist::RulePair::new(source.into(), rule);\n Ok((record.rule_id, record.source_id, pair))\n })\n .collect::<Result<Vec<(_, _, _)>, ServerFnError>>();\n\n rules\n}\n\n#[component]\nfn LastUpdatedInner(last_updated: Option<chrono::DateTime<chrono::Utc>>) -> impl IntoView {\n view! {\n {match last_updated {\n Some(last_updated) => {\n view! { <div>{format!(\"{last_updated}\")}</div> }\n }\n None => {\n view! { <div>\"Never\"</div> }\n }\n }}\n }\n}\n\n#[component]\npub fn LastUpdated(url: FilterListUrl, record: Option<FilterListRecord>) -> impl IntoView {\n view! {\n {match record.clone() {\n Some(record) => {\n let last_updated = record.last_updated;\n view! { <LastUpdatedInner last_updated=last_updated/> }\n }\n None => {\n view! {\n <Await\n future={\n let url = url.clone();\n move || {\n let url = url.clone();\n async move {\n crate::filterlist::get_last_updated(url.clone()).await\n }\n }\n }\n\n let:last_version_data\n >\n {match last_version_data {\n Ok(last_updated) => {\n view! { <LastUpdatedInner last_updated=*last_updated/> }.into_view()\n }\n Err(err) => view! { {format!(\"{err:?}\")} }.into_view(),\n }}\n\n </Await>\n }\n }\n }}\n\n <FilterListUpdate url=url.clone()/>\n }\n}\n\n#[component]\npub fn ParseList(url: FilterListUrl) -> impl IntoView {\n let parse_list_action = create_server_action::<crate::filterlist::ParseList>();\n view! {\n <ActionForm action=parse_list_action>\n <button class=\"btn btn-primary\" type=\"submit\">\n <input type=\"hidden\" placeholder=\"url\" id=\"url\" name=\"url\" value=url.to_string()/>\n \"Parse\"\n </button>\n </ActionForm>\n }\n}\n\n#[component]\npub fn FilterListUpdate(url: FilterListUrl) -> impl IntoView {\n let update_list_action = create_server_action::<UpdateListFn>();\n view! {\n <ActionForm action=update_list_action>\n <button class=\"btn btn-primary\" type=\"submit\">\n <input type=\"hidden\" placeholder=\"url\" id=\"url\" name=\"url\" value=url.to_string()/>\n \"Update\"\n </button>\n </ActionForm>\n }\n}\n\n#[component]\nfn Contents(url: FilterListUrl, page: Option<usize>) -> impl IntoView {\n view! {\n <table class=\"table table-zebra\">\n <thead>\n <tr>\n <th>Source</th>\n <th>Rule</th>\n </tr>\n </thead>\n <Await\n future=move || {\n let url = url.clone();\n async move { get_list_page(url, page, PAGE_SIZE).await }\n }\n\n let:contents\n >\n\n {\n let contents = contents.clone();\n move || match contents.clone() {\n Ok(contents) => {\n let contents = contents.clone();\n view! {\n <tbody>\n <For\n each=move || { contents.clone() }\n\n key=|(rule_id, source_id, _)| (*rule_id, *source_id)\n children=|(rule_id, _source_id, pair)| {\n let source = pair.get_source().to_string();\n let rule = pair.get_rule().clone();\n view! {\n <tr>\n <td>{source}</td>\n <td>\n <A href=rule_id.get_href() class=\"link link-neutral\">\n <DisplayRule rule=rule/>\n </A>\n </td>\n </tr>\n }\n }\n />\n\n </tbody>\n }\n .into_view()\n }\n Err(err) => format!(\"{err:?}\").into_view(),\n }\n }\n\n </Await>\n\n </table>\n }\n}\n\n#[component]\nfn FilterListInner(url: FilterListUrl, page: Option<usize>) -> impl IntoView {\n view! {\n <h1>\"Filter List\"</h1>\n <p>\"URL: \" {url.to_string()}</p>\n <p>\"Last Updated: \" <LastUpdated url=url.clone() record=None/></p>\n <p>\"Rule count: \" <ListSize url=url.clone() list_size=None/></p>\n <FilterListUpdate url=url.clone()/>\n <p>\n <ParseList url=url.clone()/>\n </p>\n\n <DeleteListButton url=url.clone()/>\n {if let Some(page) = page {\n view! { <p>\"Page: \" {page}</p> }\n } else {\n view! { <p>\"Page: 0\"</p> }\n }}\n\n {match page {\n None | Some(0) => view! {}.into_view(),\n Some(page) => {\n let params = params_map! {\n \"url\" => url.as_str(), \"page\" => (page.saturating_sub(1)).to_string()\n };\n let href = format!(\"/list{}\", params.to_query_string());\n view! {\n <A href=href class=\"btn btn-neutral\">\n \"Back\"\n </A>\n }\n }\n }}\n\n {\n let params = params_map! {\n \"url\" => url.as_str(), \"page\" => (page.unwrap_or(0) + 1).to_string()\n };\n let href = format!(\"/list{}\", params.to_query_string());\n view! {\n <A href=href class=\"btn btn-neutral\">\n \"Next\"\n </A>\n }\n }\n\n <p>\"Contents: \" <Contents url=url.clone() page=page/></p>\n }\n}\n\n#[derive(Params, PartialEq, Debug)]\nstruct ViewListParams {\n url: Option<String>,\n page: Option<usize>,\n}\n\n#[derive(thiserror::Error, Debug)]\nenum ViewListError {\n #[error(\"Invalid URL\")]\n ParseURL(#[from] url::ParseError),\n #[error(\"Invalid URL\")]\n ParseParam(#[from] leptos_router::ParamsError),\n #[error(\"Invalid FilterListType\")]\n InvalidFilterListType(#[from] InvalidFilterListTypeError),\n}\n\nimpl ViewListParams {\n fn parse(&self) -> Result<FilterListUrl, ViewListError> {\n Ok(self\n .url\n .as_ref()\n .ok_or_else(|| ParamsError::MissingParam(\"Missing Param\".into()))?\n .parse()?)\n }\n}\n\n#[component]\nfn DeleteListButton(url: FilterListUrl) -> impl IntoView {\n let delete_list_action = create_server_action::<DeleteListFn>();\n view! {\n <ActionForm action=delete_list_action>\n <button class=\"btn btn-danger\" type=\"submit\">\n <input type=\"hidden\" placeholder=\"url\" id=\"url\" name=\"url\" value=url.to_string()/>\n \"Delete\"\n </button>\n </ActionForm>\n }\n}\n\n#[component]\npub fn FilterListPage() -> impl IntoView {\n let params = use_query::<ViewListParams>();\n let get_url = move || {\n params.with(|param| {\n param\n .as_ref()\n .ok()\n .map(|param| param.parse().map(|url| (url, param.page)))\n })\n };\n view! {\n <div>\n\n {move || match get_url() {\n None => view! { <p>\"No URL\"</p> }.into_view(),\n Some(Err(err)) => view! { <p>\"Error: \" {format!(\"{err}\")}</p> }.into_view(),\n Some(Ok((url, page))) => view! { <FilterListInner url=url page=page/> }.into_view(),\n }}\n\n </div>\n }\n}\n" }, { "path": "src/home_page.rs", "content": "use crate::filterlist::{FilterListRecord, FilterListUrl, LastUpdated, ListSize};\nuse leptos::*;\nuse leptos_router::*;\n\n#[component]\nfn FilterListSummary(url: FilterListUrl, record: FilterListRecord) -> impl IntoView {\n view! {\n <tr>\n <td class=\"break-normal break-words max-w-20\">\n\n {\n let name = if record.name.is_empty() {\n url.to_string()\n } else {\n record.name.to_string()\n };\n let href = format!(\n \"/list{}\",\n params_map! {\n \"url\" => url.as_str(),\n }\n .to_query_string(),\n );\n view! {\n <A href=href class=\"link link-neutral\">\n {name}\n </A>\n }\n }\n\n </td>\n <td class=\"break-normal break-words max-w-20\">{record.author.to_string()}</td>\n <td class=\"max-w-xl break-normal break-words\">{record.license.to_string()}</td>\n <td>{humantime::format_duration(record.expires).to_string()}</td>\n <td>{format!(\"{:?}\", record.list_format)}</td>\n <td>\n <LastUpdated url=url.clone() record=Some(record.clone())/>\n </td>\n <td class=\"text-right\">\n <ListSize url=url.clone() list_size=Some(record.list_size)/>\n </td>\n </tr>\n }\n}\n\n/// Renders the home page of your application.\n#[component]\npub fn HomePage() -> impl IntoView {\n view! {\n <Await\n future=|| async move { crate::filterlist::get_filter_map().await }\n let:filterlist_map\n >\n {match filterlist_map.clone() {\n Ok(data) => {\n view! {\n <table class=\"table table-zebra\">\n <thead>\n <td>\"Name\"</td>\n <td>\"Author\"</td>\n <td>\"License\"</td>\n <td>\"Update frequency\"</td>\n <td>\"Format\"</td>\n <td>\"Last Updated\"</td>\n <td>\"Size\"</td>\n </thead>\n <tbody>\n <For\n each=move || {\n let mut data = data.0.clone();\n data.sort_unstable_by(|a, b| {\n b.1.last_updated.cmp(&a.1.last_updated)\n });\n data\n }\n\n key=|(url, _)| url.as_str().to_string()\n children=|(url, record)| {\n view! {\n <FilterListSummary url=url.clone() record=record.clone()/>\n }\n }\n />\n\n </tbody>\n </table>\n }\n .into_view()\n }\n Err(err) => view! { <p>\"Error Loading \" {format!(\"{err:?}\")}</p> }.into_view(),\n }}\n\n </Await>\n }\n}\n" }, { "path": "src/ip_view.rs", "content": "use leptos::*;\nuse leptos_router::*;\nuse std::{collections::BTreeSet, net::IpAddr};\n\nuse crate::{app::Loading, domain::DomainId};\n\n#[server]\nasync fn get_domans_which_resolve_to_ip(\n ip: IpAddr,\n) -> Result<BTreeSet<(DomainId, String)>, ServerFnError> {\n let ip: ipnetwork::IpNetwork = ip.into();\n let records = sqlx::query!(\n r#\"SELECT domains.id as \"domain_id: DomainId\", domain\n from dns_ips\n INNER JOIN domains ON dns_ips.domain_id = domains.id\n WHERE dns_ips.ip_address = $1\"#r,\n ip\n )\n .fetch_all(&crate::server::get_db().await?)\n .await?;\n let domains = records\n .into_iter()\n .map(|record| (record.domain_id, record.domain))\n .collect::<BTreeSet<_>>();\n Ok(domains)\n}\n\n#[component]\nfn DomainsWhichResolveTo(get_ip: GetIp) -> impl IntoView {\n let domain_results = create_resource(get_ip, |ip| async move {\n let ip = ip?;\n let results = get_domans_which_resolve_to_ip(ip).await?;\n Ok::<_, ServerFnError>(results)\n });\n view! {\n <Transition fallback=move || {\n view! { <p>\"Loading\" <Loading/></p> }\n }>\n {move || match domain_results.get() {\n Some(Ok(domains)) => {\n view! {\n <div>\n <p>\"Domains which resolve to this IP Address\"</p>\n <ul class=\"grid grid-cols-2 gap-2\">\n <For\n each=move || { domains.clone() }\n key=|(domain_id, _domain)| *domain_id\n children=|(_domain_id, domain)| {\n view! {\n <li>\n <A\n href=format!(\"/domain/{domain}\")\n class=\"link link-neutral\"\n >\n {domain}\n </A>\n </li>\n }\n }\n />\n\n </ul>\n </div>\n }\n .into_view()\n }\n _ => view! { <p>\"Error\"</p> }.into_view(),\n }}\n\n </Transition>\n }\n}\n\ntype GetIp = Box<dyn Fn() -> Result<IpAddr, ParamsError>>;\n\n#[derive(Params, PartialEq)]\nstruct IpParam {\n ip: Option<IpAddr>,\n}\n\n#[component]\npub fn IpView() -> impl IntoView {\n let params = use_params::<IpParam>();\n let get_ip = move || {\n params.with(|param| {\n param.as_ref().map_err(Clone::clone).and_then(|param| {\n param\n .ip\n .ok_or_else(|| ParamsError::MissingParam(\"No domain\".into()))\n })\n })\n };\n view! {\n <div>\n <h1 class=\"text-2xl font-bold text-gray-800\">{\"IP Address: \"} {get_ip}</h1>\n <DomainsWhichResolveTo get_ip=Box::new(get_ip)/>\n </div>\n }\n}\n" }, { "path": "src/lib.rs", "content": "pub mod app;\npub mod domain;\npub mod error_template;\npub mod filterlist;\npub mod home_page;\npub mod ip_view;\npub mod rule;\n#[cfg(feature = \"ssr\")]\npub mod server;\npub mod stats_view;\npub mod tasks;\n\n#[cfg(feature = \"ssr\")]\nuse mimalloc::MiMalloc;\nuse serde::*;\n\n#[cfg(feature = \"ssr\")]\n#[global_allocator]\nstatic GLOBAL: MiMalloc = MiMalloc;\n\nconst PAGE_SIZE: usize = 50;\npub mod source {\n use super::*;\n #[cfg_attr(feature = \"ssr\", derive(sqlx::Encode, sqlx::Decode))]\n #[derive(Serialize, Deserialize, Debug, Copy, Clone, PartialEq, Eq, Hash)]\n pub struct SourceId(i32);\n}\n\n#[derive(thiserror::Error, Debug)]\npub enum DbInitError {\n #[error(\"Sqlx error {0}\")]\n SqlxError(String),\n #[error(\"Missing DATABASE_URL\")]\n MissingDatabaseUrl(String),\n}\n#[cfg(feature = \"ssr\")]\nimpl From<sqlx::Error> for DbInitError {\n fn from(e: sqlx::Error) -> Self {\n Self::SqlxError(e.to_string())\n }\n}\n\n#[cfg(feature = \"ssr\")]\nimpl From<std::env::VarError> for DbInitError {\n fn from(e: std::env::VarError) -> Self {\n Self::MissingDatabaseUrl(e.to_string())\n }\n}\n\n#[cfg(feature = \"ssr\")]\npub mod fileserv;\n\n#[cfg(feature = \"hydrate\")]\n#[wasm_bindgen::prelude::wasm_bindgen]\npub fn hydrate() {\n console_error_panic_hook::set_once();\n let _ = console_log::init_with_level(log::Level::Debug);\n log::info!(\"Hydrating\");\n leptos::mount_to_body(crate::app::App);\n log::info!(\"Mounted\");\n // leptos::leptos_dom::HydrationCtx::stop_hydrating();\n}\n" }, { "path": "src/main.rs", "content": "#[cfg(feature = \"ssr\")]\nuse clap::Parser;\n\n#[cfg(feature = \"ssr\")]\n#[derive(Debug, serde::Deserialize, serde::Serialize)]\nstruct Config {\n listen_port: u16,\n peers: Vec<blockconvert::server::Peer>,\n}\n\nimpl Default for Config {\n fn default() -> Self {\n Self {\n listen_port: 3000,\n peers: vec![Default::default(); 2],\n }\n }\n}\n\n#[cfg(feature = \"ssr\")]\n/// Simple program to greet a person\n#[derive(Parser, Debug)]\n#[command(version, about, long_about = None)]\nstruct Args {\n config_path: std::path::PathBuf,\n #[arg(short, long)]\n create_config: bool,\n #[arg(short, long)]\n run_tasks: bool,\n}\n\n#[cfg(feature = \"ssr\")]\n#[tokio::main]\nasync fn main() {\n use axum::Router;\n use blockconvert::app::App;\n use blockconvert::fileserv::file_and_error_handler;\n use blockconvert::{filterlist, server};\n use leptos::*;\n use leptos_axum::{generate_route_list, LeptosRoutes};\n use tower_http::compression::CompressionLayer;\n use tower_http::compression::CompressionLevel;\n env_logger::init();\n\n let args = Args::parse();\n let config_path = args.config_path.clone();\n let node_conf = if let Ok(conf) = tokio::fs::read_to_string(args.config_path).await {\n let Ok(conf): Result<Config, _> = toml::from_str(&conf) else {\n logging::warn!(\"Error parsing config file\");\n return;\n };\n conf\n } else if args.create_config {\n let conf = Config::default();\n let conf_str = toml::to_string(&conf).unwrap();\n tokio::fs::write(config_path, conf_str).await.unwrap();\n logging::log!(\"Created config file\");\n conf\n } else {\n logging::log!(\"Config file not found\");\n return;\n };\n\n println!(\"Config: {:?}\", node_conf);\n\n let peer_state = server::PeerState::new(&node_conf.peers);\n\n // Setting get_configuration(None) means we'll be using cargo-leptos's env values\n // For deployment these variables are:\n // <https://github.com/leptos-rs/start-axum#executing-a-server-on-a-remote-machine-without-the-toolchain>\n // Alternately a file can be specified such as Some(\"Cargo.toml\")\n // The file would need to be included with the executable when moved to deployment\n let conf = get_configuration(Some(\"Cargo.toml\")).await.unwrap();\n let mut leptos_options = conf.leptos_options;\n let mut addr = leptos_options.site_addr;\n addr.set_port(node_conf.listen_port);\n leptos_options.site_addr = addr;\n let routes = generate_route_list(App);\n //let state = State { leptos_options };\n // build our application with a route\n let app = Router::new()\n .leptos_routes(&leptos_options, routes, App)\n .nest(\"/peer/\", server::get_peer_router(peer_state.clone()))\n .fallback(file_and_error_handler)\n .with_state(leptos_options)\n .layer(CompressionLayer::new().quality(CompressionLevel::Fastest));\n let token = tokio_util::sync::CancellationToken::new();\n let mut tasks = tokio::task::JoinSet::new();\n if args.run_tasks {\n tasks.spawn(filterlist::watch_filter_map());\n tasks.spawn(server::parse_missing_subdomains());\n tasks.spawn(server::check_dns(token.clone()));\n tasks.spawn(server::import_pihole_logs());\n tasks.spawn(blockconvert::rule::find_rule_matches());\n tasks.spawn(server::build_list());\n tasks.spawn(server::update_expired_lists());\n tasks.spawn(server::garbage_collect());\n tasks.spawn(server::run_cmd(token.clone()));\n tasks.spawn(server::certstream(token.clone()));\n }\n\n let listener = tokio::net::TcpListener::bind(&addr).await.unwrap();\n let server = tasks.spawn(async move {\n logging::log!(\"listening on http://{}\", &listener.local_addr()?);\n axum::serve(listener, app.into_make_service()).await?;\n Ok(())\n });\n {\n let token = token.clone();\n tasks.spawn(async move {\n let mut interrupt =\n tokio::signal::unix::signal(tokio::signal::unix::SignalKind::interrupt()).unwrap();\n let mut hangup =\n tokio::signal::unix::signal(tokio::signal::unix::SignalKind::hangup()).unwrap();\n let mut terminate =\n tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()).unwrap();\n tokio::select! {\n _ = tokio::signal::ctrl_c() =>\n logging::log!(\"Ctrl-C received, shutting down\"),\n _ = interrupt.recv() =>\n logging::log!(\"Interrupt received, shutting down\"),\n _ = hangup.recv() =>\n logging::log!(\"Hangup received, shutting down\"),\n _ = terminate.recv() =>\n logging::log!(\"Terminate received, shutting down\"),\n }\n token.cancel();\n Ok(())\n });\n }\n\n while let Some(task) = tasks.join_next().await {\n if let Err(e) = task.unwrap() {\n logging::log!(\"Error: {:?}\", e);\n return;\n }\n logging::log!(\"Task completed\");\n if token.is_cancelled() {\n server.abort();\n logging::log!(\"Shutting down\");\n tokio::select! {\n _ = async move {while tasks.join_next().await.is_some() {}} => {},\n _ = tokio::time::sleep(std::time::Duration::from_secs(5)) => {}\n }\n break;\n }\n }\n logging::log!(\"Exiting\");\n}\n\n#[cfg(not(feature = \"ssr\"))]\npub fn main() {\n // no client-side main function\n // unless we want this to work with e.g., Trunk for a purely client-side app\n // see lib.rs for hydration function instead\n}\n" }, { "path": "src/rule.rs", "content": "use crate::app::Loading;\nuse crate::filterlist::DomainRule;\nuse crate::filterlist::FilterListLink;\nuse crate::filterlist::FilterListUrl;\nuse crate::filterlist::Rule;\nuse crate::{domain::DomainId, filterlist::ListId, source::SourceId};\nuse leptos::*;\nuse leptos_router::*;\nuse serde::{Deserialize, Serialize};\n\n#[cfg_attr(feature = \"ssr\", derive(sqlx::Encode, sqlx::Decode))]\n#[derive(Serialize, Deserialize, Debug, Copy, Clone, PartialEq, Eq, Hash)]\npub struct RuleId(i32);\n\nimpl RuleId {\n pub fn get_href(&self) -> String {\n format!(\"/rule/{}\", self.0)\n }\n}\n\n#[cfg(feature = \"ssr\")]\npub async fn find_rule_matches() -> Result<(), ServerFnError> {\n use std::time::Duration;\n\n dotenvy::dotenv()?;\n let pool = crate::server::get_db().await?;\n let read_limit = std::env::var(\"READ_LIMIT\")?.parse::<u32>()? as i64;\n let interval: u64 = std::env::var(\"RULE_MATCH_CHECK_INTERVAL\")?.parse()?;\n let interval: Duration = Duration::from_secs(interval);\n let mut interval = tokio::time::interval(interval);\n interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);\n loop {\n interval.tick().await;\n let records = sqlx::query!(\n \"SELECT id from Rules\n ORDER BY last_checked_matches ASC NULLS FIRST\n LIMIT $1\",\n read_limit\n )\n .fetch_all(&pool)\n .await?;\n\n let rule_ids = records\n .into_iter()\n .map(|record| record.id)\n .collect::<Vec<_>>();\n\n let mut tx = pool.begin().await?;\n sqlx::query!(\n \"DELETE FROM rule_matches WHERE rule_id = ANY($1::int[])\",\n &rule_ids[..]\n )\n .execute(&mut *tx)\n .await?;\n sqlx::query!(\"\n INSERT INTO rule_matches(rule_id, domain_id)\n SELECT Rules.id AS rule_id, domains.id AS domain_id\n FROM Rules\n LEFT JOIN domain_rules ON Rules.domain_rule_id = domain_rules.id\n LEFT JOIN subdomains ON domain_rules.domain_id = subdomains.parent_domain_id AND domain_rules.subdomain = true\n LEFT JOIN ip_rules ON Rules.ip_rule_id = ip_rules.id AND ip_rules.allow=false\n LEFT JOIN dns_ips ON ip_rules.ip_network = dns_ips.ip_address\n LEFT JOIN dns_cnames ON (dns_cnames.cname_domain_id = domain_rules.domain_id\n OR dns_cnames.cname_domain_id = subdomains.domain_id) AND domain_rules.allow=false\n INNER JOIN domains ON domain_rules.domain_id = domains.id\n OR subdomains.domain_id = domains.id\n OR dns_ips.domain_id = domains.id\n OR dns_cnames.domain_id = domains.id\n INNER JOIN dns_ips AS dns_check ON dns_check.domain_id = domains.id AND dns_check.ip_address IS NOT NULL\n WHERE Rules.id = ANY($1::int[])\n ON CONFLICT DO NOTHING\",\n &rule_ids[..]).execute(&mut *tx).await?;\n let count = sqlx::query!(\n \"SELECT COUNT(*) FROM rule_matches WHERE rule_id = ANY($1::int[])\",\n &rule_ids[..]\n )\n .fetch_one(&mut *tx)\n .await?\n .count\n .unwrap_or(0);\n log::info!(\n \"Checked {} rules and found {} matches\",\n rule_ids.len(),\n count\n );\n sqlx::query!(\n \"UPDATE rules\n SET last_checked_matches = now()\n WHERE id = ANY($1::int[])\",\n &rule_ids[..]\n )\n .execute(&mut *tx)\n .await?;\n tx.commit().await?;\n }\n}\n\n#[server]\npub async fn get_rule(id: RuleId) -> Result<Rule, ServerFnError> {\n let record = sqlx::query!(\n \"SELECT domain_rule_id, ip_rule_id FROM Rules\n WHERE Rules.id = $1\",\n id.0\n )\n .fetch_one(&crate::server::get_db().await?)\n .await?;\n if let Some(domain_rule_id) = record.domain_rule_id {\n let record = sqlx::query!(\n \"SELECT domain, allow, subdomain FROM domain_rules\n INNER JOIN domains ON domains.id = domain_rules.domain_id \n WHERE domain_rules.id = $1\",\n domain_rule_id\n )\n .fetch_one(&crate::server::get_db().await?)\n .await?;\n let domain_rule = DomainRule {\n domain: record.domain.parse()?,\n allow: record.allow,\n subdomain: record.subdomain,\n };\n Ok(Rule::Domain(domain_rule))\n } else if let Some(ip_rule_id) = record.ip_rule_id {\n let record = sqlx::query!(\n \"SELECT ip_network, allow FROM ip_rules WHERE id = $1\",\n ip_rule_id\n )\n .fetch_one(&crate::server::get_db().await?)\n .await?;\n Ok(Rule::IpRule(crate::filterlist::IpRule {\n ip: record.ip_network,\n allow: record.allow,\n }))\n } else {\n Ok(Rule::Invalid)\n }\n}\n\ntype GetId = Box<dyn Fn() -> Result<RuleId, ParamsError>>;\n\n#[server]\nasync fn get_sources(\n id: RuleId,\n) -> Result<Vec<(SourceId, String, ListId, FilterListUrl)>, ServerFnError> {\n let sources = sqlx::query!(\n r#\"SELECT rule_source.id AS \"source_id: SourceId\", source, filterLists.id as \"list_id: ListId\", filterLists.url FROM rule_source\n INNER JOIN list_rules ON rule_source.id = list_rules.source_id\n INNER JOIN filterLists ON list_rules.list_id = filterLists.id\n WHERE rule_id = $1\n ORDER BY (source)\n \"#r,\n id.0\n )\n .fetch_all(&crate::server::get_db().await?)\n .await?;\n sources\n .into_iter()\n .map(|record| {\n Ok((\n record.source_id,\n record.source,\n record.list_id,\n record.url.parse()?,\n ))\n })\n .collect()\n}\n\n#[component]\nfn Sources(get_id: GetId) -> impl IntoView {\n let source_resource = create_resource(get_id, |id| async move {\n let sources = get_sources(id?).await?;\n Ok::<_, ServerFnError>(sources)\n });\n view! {\n <Transition fallback=move || {\n view! { <p>\"Loading\" <Loading/></p> }\n }>\n {move || match source_resource.get() {\n Some(Ok(sources)) => {\n view! {\n <p>\n \"Sources:\" <table class=\"table table-zebra\">\n <thead>\n <tr>\n <th>Source</th>\n <th>List</th>\n </tr>\n </thead>\n <tbody>\n <For\n each=move || { sources.clone() }\n key=|(source_id, _, _, _)| *source_id\n children=|(_, source, _list_id, url)| {\n view! {\n <tr>\n <td>{source}</td>\n <td>\n <FilterListLink url=url/>\n </td>\n </tr>\n }\n }\n />\n\n </tbody>\n </table>\n\n </p>\n }\n .into_view()\n }\n Some(Err(err)) => view! { <p>\"Error: \" {format!(\"{err}\")}</p> }.into_view(),\n None => view! { \"Invalid URL\" }.into_view(),\n }}\n\n </Transition>\n }\n}\n\n#[component]\npub fn DisplayRule(rule: Rule) -> impl IntoView {\n view! {\n {match rule {\n Rule::Domain(domain_rule) => {\n view! {\n {if domain_rule.allow { \"ALLOW: \" } else { \"BLOCK: \" }}\n {if domain_rule.subdomain { \"*.\" } else { \"\" }}\n {domain_rule.domain.as_ref().to_owned()}\n }\n .into_view()\n }\n Rule::IpRule(ip_rule) => {\n view! {\n {if ip_rule.allow { \"ALLOW: \" } else { \"BLOCK: \" }}\n {ip_rule.ip.to_string()}\n }\n .into_view()\n }\n Rule::Unknown => \"Unknown\".into_view(),\n Rule::Invalid => \"Invalid Rule\".into_view(),\n }}\n }\n}\n\n#[component]\nfn RuleRawView(\n rule: Resource<Result<RuleId, ParamsError>, Result<Rule, ServerFnError>>,\n) -> impl IntoView {\n view! {\n <Transition fallback=move || {\n view! { <p>\"Loading\" <Loading/></p> }\n }>\n {move || match rule.get() {\n Some(Ok(rule)) => view! { <DisplayRule rule=rule/> }.into_view(),\n Some(Err(err)) => view! { <p>\"Error: \" {format!(\"{err}\")}</p> }.into_view(),\n None => view! { \"Invalid URL\" }.into_view(),\n }}\n\n </Transition>\n }\n}\n\n#[server]\nasync fn get_rule_blocked_domains(id: RuleId) -> Result<Vec<(DomainId, String)>, ServerFnError> {\n let domains = sqlx::query!(\n r#\"SELECT DISTINCT domains.id as \"domain_id: DomainId\", domain as \"domain: String\"\n FROM Rules\n INNER JOIN rule_matches ON Rules.id = rule_matches.rule_id\n INNER JOIN domains ON rule_matches.domain_id = domains.id\n WHERE Rules.id = $1\"#r,\n id.0\n )\n .fetch_all(&crate::server::get_db().await?)\n .await?;\n Ok(domains\n .into_iter()\n .map(|record| (record.domain_id, record.domain))\n .collect())\n}\n\n#[component]\nfn RuleBlockedDomainsView(get_id: Box<dyn Fn() -> Result<RuleId, ParamsError>>) -> impl IntoView {\n let domains_resource = create_resource(get_id, |id| async move {\n let domains = get_rule_blocked_domains(id?).await?;\n Ok::<_, ServerFnError>(domains)\n });\n view! {\n <Transition fallback=move || {\n view! { <p>\"Loading\" <Loading/></p> }\n }>\n {move || match domains_resource.get() {\n Some(Ok(domains)) => {\n view! {\n <p>\n \"Matched Domains:\" <table class=\"table table-zebra\">\n <thead>\n <tr>\n <th>Domain</th>\n </tr>\n </thead>\n <tbody>\n <For\n each=move || { domains.clone() }\n key=|(id, _)| *id\n children=|(_domain_id, domain)| {\n let domain_href = format!(\"/domain/{domain}\");\n view! {\n <tr>\n <td>\n <A href=domain_href class=\"link link-neutral\">\n {domain}\n </A>\n </td>\n </tr>\n }\n }\n />\n\n </tbody>\n </table>\n\n </p>\n }\n .into_view()\n }\n Some(Err(err)) => view! { <p>\"Error: \" {format!(\"{err}\")}</p> }.into_view(),\n None => \"Invalid URL\".into_view(),\n }}\n\n </Transition>\n }\n}\n\n#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)]\npub struct RuleData {\n pub rule_id: RuleId,\n pub domain: Option<String>,\n pub domain_allow: Option<bool>,\n pub domain_subdomain: Option<bool>,\n pub ip_network: Option<ipnetwork::IpNetwork>,\n pub ip_allow: Option<bool>,\n}\n\nimpl TryInto<Rule> for RuleData {\n type Error = ServerFnError;\n fn try_into(self) -> Result<Rule, Self::Error> {\n match (\n self.domain,\n self.domain_allow,\n self.domain_subdomain,\n self.ip_network,\n self.ip_allow,\n ) {\n (Some(domain), Some(allow), Some(subdomain), None, None) => {\n Ok(Rule::Domain(DomainRule {\n domain: domain.parse()?,\n allow,\n subdomain,\n }))\n }\n (None, None, None, Some(ip_network), Some(allow)) => {\n Ok(Rule::IpRule(crate::filterlist::IpRule {\n ip: ip_network,\n allow,\n }))\n }\n _ => Ok(Rule::Invalid),\n }\n }\n}\n\n#[derive(Params, PartialEq)]\nstruct RuleParam {\n id: Option<i32>,\n}\n\n#[component]\npub fn RuleViewPage() -> impl IntoView {\n let params = use_params::<RuleParam>();\n let get_id = move || {\n params.with(|param| {\n param.as_ref().map_err(Clone::clone).and_then(|param| {\n Ok(RuleId(param.id.ok_or_else(|| {\n ParamsError::MissingParam(\"No id\".into())\n })?))\n })\n })\n };\n let rule_resource = create_resource(get_id, |id| async move {\n let rule = get_rule(id?).await?;\n Ok::<_, ServerFnError>(rule)\n });\n view! {\n <p>\"Rule: \" <RuleRawView rule=rule_resource/></p>\n <Sources get_id=Box::new(get_id)/>\n <RuleBlockedDomainsView get_id=Box::new(get_id)/>\n }\n}\n" }, { "path": "src/server.rs", "content": "use crate::DbInitError;\nuse crate::{domain::Domain, filterlist::FilterListUrl};\n\nuse addr::domain;\nuse axum::body::Bytes;\nuse axum::extract::{Path, State};\nuse leptos::*;\nuse notify::Watcher;\nuse std::collections::HashSet;\nuse tokio::task::JoinSet;\nuse tower_http::decompression::DecompressionLayer;\n\nuse std::sync::{Arc, Mutex, RwLock};\nuse std::time::Duration;\nuse tokio::io::{AsyncBufReadExt, AsyncWriteExt};\nuse tokio_util::sync::CancellationToken;\n\nstatic SQLITE_POOL: tokio::sync::OnceCell<sqlx::PgPool> = tokio::sync::OnceCell::const_new();\n\npub async fn get_db() -> Result<sqlx::PgPool, DbInitError> {\n SQLITE_POOL\n .get_or_try_init(|| {\n let _ = dotenvy::dotenv();\n let db_url = std::env::var(\"DATABASE_URL\");\n async { Ok::<_, DbInitError>(sqlx::PgPool::connect(&db_url?).await?) }\n })\n .await\n .cloned()\n}\n\npub async fn parse_missing_subdomains() -> Result<(), ServerFnError> {\n dotenvy::dotenv()?;\n let read_limit: usize = std::env::var(\"READ_LIMIT\")?.parse()?;\n let pool = get_db().await?;\n loop {\n let records = sqlx::query!(\n \"SELECT domain from domains\n WHERE processed_subdomains = false\n LIMIT $1\",\n read_limit as i64\n )\n .fetch_all(&pool)\n .await?;\n if records.is_empty() {\n tokio::time::sleep(Duration::from_secs(30)).await;\n continue;\n }\n let mut checked_domains = Vec::new();\n\n let mut all_domains = Vec::new();\n let mut all_parents = Vec::new();\n\n for record in records {\n checked_domains.push(record.domain.clone());\n let Ok(domain) = record.domain.parse::<Domain>() else {\n continue;\n };\n let parents = domain\n .as_ref()\n .match_indices('.')\n .map(|(i, _)| record.domain.split_at(i + 1).1)\n .filter_map(|parent| parent.parse::<Domain>().ok());\n for parent in parents {\n all_domains.push(domain.clone());\n all_parents.push(parent);\n }\n }\n let mut parent_set = all_parents\n .iter()\n .cloned()\n .collect::<std::collections::HashSet<_>>();\n for domain in &all_domains {\n parent_set.remove(domain);\n }\n let parent_set = parent_set.into_iter().collect::<Vec<_>>();\n sqlx::query!(\n \"INSERT INTO domains (domain)\n SELECT domain FROM UNNEST($1::text[]) as t(domain)\n ON CONFLICT DO NOTHING\",\n &parent_set[..] as _\n )\n .execute(&pool)\n .await?;\n sqlx::query!(\n \"INSERT INTO subdomains (domain_id, parent_domain_id)\n SELECT domains_with_parents.id, parents.id\n FROM UNNEST($1::text[], $2::text[]) AS t(domain, parent)\n INNER JOIN domains AS domains_with_parents ON domains_with_parents.domain = t.domain\n INNER JOIN domains AS parents ON parents.domain = t.parent\n ON CONFLICT DO NOTHING\",\n &all_domains[..] as _,\n &all_parents[..] as _,\n )\n .execute(&pool)\n .await?;\n sqlx::query!(\n \"INSERT INTO domains (domain)\n SELECT domain FROM UNNEST($1::text[]) as t(domain)\n ON CONFLICT(domain)\n DO UPDATE SET processed_subdomains = true\",\n &checked_domains[..]\n )\n .execute(&pool)\n .await?;\n }\n}\n\npub async fn check_dns(token: CancellationToken) -> Result<(), ServerFnError> {\n let resolver = crate::domain::DomainResolver::new(token)?;\n resolver.run().await?;\n Ok(())\n}\n\npub async fn import_pihole_logs() -> Result<(), ServerFnError> {\n let _ = dotenvy::dotenv()?;\n let Ok(log_path) = std::env::var(\"PIHOLE_LOG_PATH\") else {\n log::info!(\"No PIHOLE_LOG_PATH set, skipping\");\n return Ok(());\n };\n let log_path: std::path::PathBuf = log_path.parse()?;\n let write_frequency: u64 = std::env::var(\"WRITE_FREQUENCY\")?.parse()?;\n let notify = std::sync::Arc::new(tokio::sync::Notify::new());\n let notify2 = notify.clone();\n let mut watcher = notify::recommended_watcher(move |_| {\n notify.notify_one();\n })?;\n watcher.watch(&log_path, notify::RecursiveMode::NonRecursive)?;\n let pool: sqlx::Pool<sqlx::Postgres> = get_db().await?;\n let file = tokio::fs::File::open(log_path).await?;\n let buf = tokio::io::BufReader::new(file);\n let mut lines = buf.lines();\n let mut domains = HashSet::new();\n let mut last_wrote = std::time::Instant::now();\n while let Ok(line) = lines.next_line().await {\n if let Some(line) = line {\n for segment in line.split_whitespace() {\n if let Ok(domain) = segment.parse() {\n let domain: Domain = domain;\n domains.insert(domain);\n }\n }\n } else {\n notify2.notified().await;\n }\n if last_wrote.elapsed().as_secs() > write_frequency {\n let domains_vec = domains.drain().collect::<Vec<_>>();\n let record = sqlx::query!(\n \"INSERT INTO domains (domain)\n SELECT domain FROM UNNEST($1::text[]) as t(domain)\n ON CONFLICT DO NOTHING\",\n &domains_vec[..] as _\n )\n .execute(&pool)\n .await?;\n let inserted = record.rows_affected();\n if inserted != 0 {\n log::info!(\"Inserted {} domains from Pihole logs\", inserted);\n }\n last_wrote = std::time::Instant::now();\n }\n }\n\n Ok(())\n}\n\npub async fn update_expired_lists() -> Result<(), ServerFnError> {\n let pool = get_db().await?;\n loop {\n let record = sqlx::query!(\n r#\"SELECT url, lastupdated+expires * INTERVAL '1 seconds' AS nextupdate\n FROM filterLists ORDER BY nextupdate NULLS FIRST\n LIMIT 1\"#r\n )\n .fetch_one(&pool)\n .await?;\n if let Some(next_update) = record.nextupdate {\n let next_update = next_update - chrono::Utc::now();\n if let Ok(next_update) = next_update.to_std() {\n tokio::time::sleep(next_update).await;\n }\n }\n let url: FilterListUrl = record.url.parse()?;\n if let Err(err) = crate::filterlist::update_list(url.clone()).await {\n log::warn!(\"Error updating list {}: {:?}\", url.as_str(), err);\n }\n }\n}\n\npub async fn build_list() -> Result<(), ServerFnError> {\n // return Ok(());\n dotenvy::dotenv()?;\n let pool = get_db().await?;\n sqlx::query!(\"DELETE FROM allow_domains\")\n .execute(&pool)\n .await?;\n sqlx::query!(\"DELETE FROM block_domains\")\n .execute(&pool)\n .await?;\n let allow_record = sqlx::query!(\n \"INSERT INTO allow_domains(domain_id)\n SELECT rule_matches.domain_id from rule_matches\n INNER JOIN rules ON rule_matches.rule_id = rules.id\n LEFT JOIN domain_rules ON rules.domain_rule_id = domain_rules.id\n LEFT JOIN ip_rules ON rules.ip_rule_id = ip_rules.id\n WHERE domain_rules.allow = true OR ip_rules.allow = true\n ON CONFLICT DO NOTHING\",\n )\n .execute(&pool)\n .await?;\n log::info!(\"Inserted {} allow rules\", allow_record.rows_affected());\n let record = sqlx::query!(\n \"INSERT INTO block_domains(domain_id)\n SELECT rule_matches.domain_id from rule_matches\n INNER JOIN rules ON rule_matches.rule_id = rules.id\n LEFT JOIN domain_rules ON rules.domain_rule_id = domain_rules.id\n LEFT JOIN ip_rules ON rules.ip_rule_id = ip_rules.id\n WHERE domain_rules.allow = false OR ip_rules.allow = false\n ON CONFLICT DO NOTHING\",\n )\n .execute(&pool)\n .await?;\n log::info!(\"Inserted {} block rules\", record.rows_affected());\n\n {\n let records = sqlx::query!(\"select domain from block_domains\n INNER JOIN domains ON block_domains.domain_id = domains.id\n where not exists(select 1 from allow_domains where allow_domains.domain_id=block_domains.domain_id)\n ORDER BY domain\").fetch_all(&pool).await?;\n let domain_file = tokio::fs::File::create(\"output/domains.txt\").await?;\n let mut domain_buf = tokio::io::BufWriter::new(domain_file);\n let adblock_file = tokio::fs::File::create(\"output/adblock.txt\").await?;\n let mut adblock_buf = tokio::io::BufWriter::new(adblock_file);\n let mut count = 0;\n for record in records {\n domain_buf.write_all(record.domain.as_bytes()).await?;\n domain_buf.write_all(b\"\\n\").await?;\n\n adblock_buf.write_all(b\"||\").await?;\n adblock_buf.write_all(record.domain.as_bytes()).await?;\n adblock_buf.write_all(b\"^\\n\").await?;\n\n count += 1;\n }\n domain_buf.flush().await?;\n log::info!(\"Wrote {} rules to output/domains.txt\", count);\n adblock_buf.flush().await?;\n log::info!(\"Wrote {} rules to output/adblock.txt\", count);\n }\n sqlx::query!(\"DELETE FROM allow_domains\")\n .execute(&pool)\n .await?;\n sqlx::query!(\"DELETE FROM block_domains\")\n .execute(&pool)\n .await?;\n Ok(())\n}\n\nasync fn garbage_collect_rule_source(pool: &sqlx::PgPool) -> Result<u64, ServerFnError> {\n let record = sqlx::query!(\n \"delete from rule_source where not exists\n (select 1 from list_rules where source_id=rule_source.id)\"\n )\n .execute(pool)\n .await?;\n Ok(record.rows_affected())\n}\n\nasync fn garbage_collect_rules(pool: &sqlx::PgPool) -> Result<u64, ServerFnError> {\n let record = sqlx::query!(\n \"delete from Rules where not exists\n (select 1 from rule_source where Rules.id=rule_source.rule_id)\"\n )\n .execute(pool)\n .await?;\n Ok(record.rows_affected())\n}\n\nasync fn garbage_collect_rule_matches(pool: &sqlx::PgPool) -> Result<u64, ServerFnError> {\n let record = sqlx::query!(\n \"delete from rule_matches where not exists\n (select 1 from rules where Rules.id=rule_matches.rule_id)\"\n )\n .execute(pool)\n .await?;\n Ok(record.rows_affected())\n}\n\npub async fn garbage_collect() -> Result<(), ServerFnError> {\n let pool = get_db().await?;\n let gc_interval = std::env::var(\"GC_INTERVAL\")?.parse::<u64>()?;\n let mut interval = tokio::time::interval(Duration::from_secs(gc_interval));\n interval.tick().await;\n loop {\n interval.tick().await;\n let rows = garbage_collect_rule_source(&pool).await?;\n if rows > 0 {\n log::info!(\"Garbage collected {} rule sources\", rows);\n }\n interval.tick().await;\n let rows = garbage_collect_rules(&pool).await?;\n if rows > 0 {\n log::info!(\"Garbage collected {} rules\", rows);\n }\n interval.tick().await;\n let rows = garbage_collect_rule_matches(&pool).await?;\n if rows > 0 {\n log::info!(\"Garbage collected {} rule matches\", rows);\n }\n }\n}\n\npub async fn run_cmd(token: CancellationToken) -> Result<(), ServerFnError> {\n dotenvy::dotenv()?;\n let cmd = std::env::var(\"TASK_CMD\")?;\n let mut interval = tokio::time::interval(Duration::from_secs(300));\n loop {\n tokio::select! {\n _ = token.cancelled() => {\n log::info!(\"Shutting down run_cmd\");\n return Ok(());},\n _ = interval.tick() => {}}\n let output = tokio::process::Command::new(&cmd).output().await;\n if let Err(err) = output {\n log::warn!(\"Error running command: {:?}\", err);\n }\n }\n}\n\nconst CERTSTREAM_URL: &str = \"wss://certstream.calidog.io/domains-only\";\n\n#[derive(serde::Deserialize)]\nstruct CertStreamMessage {\n data: Vec<String>,\n}\n\nasync fn stream_certstream(\n domains: tokio::sync::mpsc::UnboundedSender<Domain>,\n) -> Result<(), ServerFnError> {\n use futures::StreamExt;\n let (mut client, _) = tokio_tungstenite::connect_async(CERTSTREAM_URL).await?;\n while let Some(Ok(msg)) = client.next().await {\n if let tokio_tungstenite::tungstenite::protocol::Message::Text(msg) = msg {\n let msg: CertStreamMessage = serde_json::from_str(&msg)?;\n for domain in msg.data {\n let Ok(domain) = domain.parse::<Domain>() else {\n continue;\n };\n domains.send(domain)?;\n }\n }\n }\n Ok(())\n}\n\nasync fn write_certstream(\n mut rx: tokio::sync::mpsc::UnboundedReceiver<Domain>,\n token: CancellationToken,\n) -> Result<(), ServerFnError> {\n dotenvy::dotenv()?;\n let pool = get_db().await?;\n let interval = std::env::var(\"WRITE_FREQUENCY\")?.parse::<u64>()?;\n let mut interval = tokio::time::interval(Duration::from_secs(interval));\n interval.tick().await;\n loop {\n tokio::select! {_ = interval.tick() => {},\n _ = token.cancelled() => log::info!(\"Shutting down certstream writer\")\n }\n let mut domains = Vec::new();\n while let Ok(domain) = rx.try_recv() {\n domains.push(domain.as_ref().to_string());\n }\n if domains.is_empty() {\n continue;\n }\n let record = sqlx::query!(\n \"INSERT INTO domains(domain)\n SELECT domain FROM UNNEST($1::text[]) as t(domain)\n ON CONFLICT DO NOTHING\",\n &domains[..]\n )\n .execute(&pool)\n .await?;\n log::info!(\n \"Certstream inserted {} new domains (out of {} found)\",\n record.rows_affected(),\n domains.len()\n );\n if token.is_cancelled() {\n return Ok(());\n }\n }\n}\n\npub async fn certstream(token: CancellationToken) -> Result<(), ServerFnError> {\n let (tx, rx) = tokio::sync::mpsc::unbounded_channel();\n tokio::spawn(async move {\n loop {\n if let Err(err) = stream_certstream(tx.clone()).await {\n log::warn!(\"Error streaming certstream: {:?}\", err);\n tokio::time::sleep(Duration::from_secs(10)).await;\n }\n }\n });\n write_certstream(rx, token).await?;\n Ok(())\n}\n\nasync fn dns_results(State(peer_state): State<PeerState>) {}\n\n#[derive(Clone, Debug, serde::Deserialize, serde::Serialize)]\npub struct Peer {\n url: url::Url,\n get_dns_results: bool,\n}\n\nimpl Default for Peer {\n fn default() -> Self {\n Self {\n url: \"http://localhost:3000/\".parse().unwrap(),\n get_dns_results: false,\n }\n }\n}\n\n#[derive(Clone)]\npub struct PeerState {\n peers: Arc<[Peer]>,\n}\n#[derive(Debug, thiserror::Error)]\nenum PeerError {\n #[error(\"URL error: {0}\")]\n Url(#[from] url::ParseError),\n #[error(\"Request error: {0}\")]\n Request(#[from] reqwest::Error),\n #[error(\"Multiple errors: {0:?}\")]\n Multiple(Vec<PeerError>),\n}\n\nimpl PeerState {\n pub fn new(peers: &[Peer]) -> Self {\n Self { peers: peers.into() }\n }\n}\n\npub fn get_peer_router(peer_state: PeerState) -> axum::Router<LeptosOptions> {\n axum::Router::new()\n .route(\"/dns-results\", axum::routing::get(dns_results))\n .with_state(peer_state)\n}\n" }, { "path": "src/stats_view.rs", "content": "use leptos::*;\n\n#[server]\nasync fn count_total_rules() -> Result<usize, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let count = sqlx::query!(\n \"SELECT reltuples::bigint AS count\n FROM pg_catalog.pg_class\n WHERE relname = 'rules'\"\n )\n .fetch_one(&pool)\n .await?\n .count\n .ok_or_else(|| ServerFnError::new(\"No count\"))? as usize;\n Ok(count)\n}\n\n#[component]\nfn TotalRuleCount() -> impl IntoView {\n view! {\n <Await future=|| async { count_total_rules().await } let:total_rules>\n {match total_rules {\n Ok(count) => view! { {count} }.into_view(),\n Err(err) => view! { {format!(\"{err:?}\")} }.into_view(),\n }}\n\n </Await>\n }\n}\n\n#[server]\nasync fn get_total_rule_matches() -> Result<usize, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let count = sqlx::query!(\n \"SELECT reltuples::bigint AS count\n FROM pg_catalog.pg_class\n WHERE relname = 'rule_matches'\"\n )\n .fetch_one(&pool)\n .await?\n .count\n .ok_or_else(|| ServerFnError::new(\"No count\"))? as usize;\n Ok(count)\n}\n\n#[component]\nfn TotalRuleMatches() -> impl IntoView {\n view! {\n <Await future=|| async { get_total_rule_matches().await } let:total_rule_matches>\n {match total_rule_matches {\n Ok(count) => view! { {count} }.into_view(),\n Err(err) => view! { {format!(\"{err:?}\")} }.into_view(),\n }}\n\n </Await>\n }\n}\n\n#[server]\nasync fn get_domain_count() -> Result<usize, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let count = sqlx::query!(\n \"SELECT reltuples::bigint AS count\n FROM pg_catalog.pg_class\n WHERE relname = 'domains'\"\n )\n .fetch_one(&pool)\n .await?\n .count\n .ok_or_else(|| ServerFnError::new(\"No count\"))? as usize;\n Ok(count)\n}\n\n#[component]\nfn DomainCount() -> impl IntoView {\n view! {\n <Await future=|| async { get_domain_count().await } let:domain_count>\n {match domain_count {\n Ok(count) => view! { {count} }.into_view(),\n Err(err) => view! { {format!(\"{err:?}\")} }.into_view(),\n }}\n\n </Await>\n }\n}\n#[server]\nasync fn get_subdomains_count() -> Result<usize, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let count = sqlx::query!(\n \"SELECT reltuples::bigint AS count\n FROM pg_catalog.pg_class\n WHERE relname = 'subdomains'\"\n )\n .fetch_one(&pool)\n .await?\n .count\n .ok_or_else(|| ServerFnError::new(\"No count\"))? as usize;\n Ok(count)\n}\n\n#[component]\nfn SubdomainCount() -> impl IntoView {\n view! {\n <Await future=|| async { get_subdomains_count().await } let:subdomain_count>\n {match subdomain_count {\n Ok(count) => view! { {count} }.into_view(),\n Err(err) => view! { {format!(\"{err:?}\")} }.into_view(),\n }}\n\n </Await>\n }\n}\n\n#[server]\nasync fn get_dns_ip_count() -> Result<usize, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let count = sqlx::query!(\n \"SELECT reltuples::bigint AS count\n FROM pg_catalog.pg_class\n WHERE relname = 'dns_ips'\"\n )\n .fetch_one(&pool)\n .await?\n .count\n .ok_or_else(|| ServerFnError::new(\"No count\"))? as usize;\n Ok(count)\n}\n\n#[component]\nfn DnsIpCount() -> impl IntoView {\n view! {\n <Await future=|| async { get_dns_ip_count().await } let:dns_ip_count>\n {match dns_ip_count {\n Ok(count) => view! { {count} }.into_view(),\n Err(err) => view! { {format!(\"{err:?}\")} }.into_view(),\n }}\n\n </Await>\n }\n}\n\n#[server]\nasync fn get_dns_cname_count() -> Result<usize, ServerFnError> {\n let pool = crate::server::get_db().await?;\n let count = sqlx::query!(\n \"SELECT reltuples::bigint AS count\n FROM pg_catalog.pg_class\n WHERE relname = 'dns_cnames'\"\n )\n .fetch_one(&pool)\n .await?\n .count\n .ok_or_else(|| ServerFnError::new(\"No count\"))? as usize;\n Ok(count)\n}\n\n#[component]\nfn DnsCnameCount() -> impl IntoView {\n view! {\n <Await future=|| async { get_dns_cname_count().await } let:dns_ip_count>\n {match dns_ip_count {\n Ok(count) => view! { {count} }.into_view(),\n Err(err) => view! { {format!(\"{err:?}\")} }.into_view(),\n }}\n\n </Await>\n }\n}\n\n#[component]\npub fn StatsView() -> impl IntoView {\n view! {\n <div>\n <h1 class=\"mt-5 mb-5 text-4xl font-bold text-center text-indigo-600\">Stats</h1>\n <table class=\"table max-w-fit\">\n <tr>\n <td>\"Total Domains\"</td>\n <td class=\"text-right\">\n <DomainCount/>\n </td>\n </tr>\n <tr>\n <td>\"Total DNS IPs\"</td>\n <td class=\"text-right\">\n <DnsIpCount/>\n </td>\n </tr>\n <tr>\n <td>\"Total DNS CNAMES\"</td>\n <td class=\"text-right\">\n <DnsCnameCount/>\n </td>\n </tr>\n <tr>\n <td>\"Total Subdomains\"</td>\n <td class=\"text-right\">\n <SubdomainCount/>\n </td>\n </tr>\n <tr>\n <td>\"Total Rules\"</td>\n <td class=\"text-right\">\n <TotalRuleCount/>\n </td>\n </tr>\n <tr>\n <td>\"Total Rule Matches\"</td>\n <td class=\"text-right\">\n <TotalRuleMatches/>\n </td>\n </tr>\n </table>\n </div>\n }\n}\n" }, { "path": "src/tasks.rs", "content": "use leptos::*;\n\ntrait Task {\n type Error;\n fn name(&self) -> &str;\n async fn run_once(&self) -> Result<String, Self::Error>;\n}\n#[cfg(feature = \"ssr\")]\nstruct GarbageCollectRuleSource {}\n\n#[cfg(feature = \"ssr\")]\nimpl Task for GarbageCollectRuleSource {\n type Error = ServerFnError;\n fn name(&self) -> &str {\n \"Garbage collect rule_source\"\n }\n async fn run_once(&self) -> Result<String, Self::Error> {\n let pool = crate::server::get_db().await?;\n let rows_removed = sqlx::query!(\n \"delete from rule_source where not exists\n (select 1 from list_rules where source_id=rule_source.id)\"\n )\n .execute(&pool)\n .await?\n .rows_affected();\n Ok(format!(\n \"Garbage collected {} rows from rule_source\",\n rows_removed\n ))\n }\n}\n\n#[cfg(feature = \"ssr\")]\nasync fn register_task<T: Task>(_task: T) {\n let _pool = crate::server::get_db().await.unwrap();\n}\n\n#[component]\npub fn TaskView() -> impl IntoView {\n view! {\n <div>\n <h1>\"Tasks\"</h1>\n <p>\"This is the tasks view\"</p>\n </div>\n }\n}\n\n\n" }, { "path": "style/main.scss", "content": "" }, { "path": "style/tailwind.css", "content": "@tailwind base;\n@tailwind components;\n@tailwind utilities;" }, { "path": "tailwind.config.js", "content": "/** @type {import('tailwindcss').Config} */\n module.exports = {\n content: {\n relative: true,\n files: [\"*.html\", \"./src/*\", \"./src/**/*.rs\"],\n },\n theme: {\n extend: {},\n },\n plugins: [require(\"daisyui\")],\n }\n " }, { "path": "tld_list.txt", "content": "# Version 2020071800, Last Updated Sat Jul 18 07:07:01 2020 UTC\nAAA\nAARP\nABARTH\nABB\nABBOTT\nABBVIE\nABC\nABLE\nABOGADO\nABUDHABI\nAC\nACADEMY\nACCENTURE\nACCOUNTANT\nACCOUNTANTS\nACO\nACTOR\nAD\nADAC\nADS\nADULT\nAE\nAEG\nAERO\nAETNA\nAF\nAFAMILYCOMPANY\nAFL\nAFRICA\nAG\nAGAKHAN\nAGENCY\nAI\nAIG\nAIRBUS\nAIRFORCE\nAIRTEL\nAKDN\nAL\nALFAROMEO\nALIBABA\nALIPAY\nALLFINANZ\nALLSTATE\nALLY\nALSACE\nALSTOM\nAM\nAMAZON\nAMERICANEXPRESS\nAMERICANFAMILY\nAMEX\nAMFAM\nAMICA\nAMSTERDAM\nANALYTICS\nANDROID\nANQUAN\nANZ\nAO\nAOL\nAPARTMENTS\nAPP\nAPPLE\nAQ\nAQUARELLE\nAR\nARAB\nARAMCO\nARCHI\nARMY\nARPA\nART\nARTE\nAS\nASDA\nASIA\nASSOCIATES\nAT\nATHLETA\nATTORNEY\nAU\nAUCTION\nAUDI\nAUDIBLE\nAUDIO\nAUSPOST\nAUTHOR\nAUTO\nAUTOS\nAVIANCA\nAW\nAWS\nAX\nAXA\nAZ\nAZURE\nBA\nBABY\nBAIDU\nBANAMEX\nBANANAREPUBLIC\nBAND\nBANK\nBAR\nBARCELONA\nBARCLAYCARD\nBARCLAYS\nBAREFOOT\nBARGAINS\nBASEBALL\nBASKETBALL\nBAUHAUS\nBAYERN\nBB\nBBC\nBBT\nBBVA\nBCG\nBCN\nBD\nBE\nBEATS\nBEAUTY\nBEER\nBENTLEY\nBERLIN\nBEST\nBESTBUY\nBET\nBF\nBG\nBH\nBHARTI\nBI\nBIBLE\nBID\nBIKE\nBING\nBINGO\nBIO\nBIZ\nBJ\nBLACK\nBLACKFRIDAY\nBLOCKBUSTER\nBLOG\nBLOOMBERG\nBLUE\nBM\nBMS\nBMW\nBN\nBNPPARIBAS\nBO\nBOATS\nBOEHRINGER\nBOFA\nBOM\nBOND\nBOO\nBOOK\nBOOKING\nBOSCH\nBOSTIK\nBOSTON\nBOT\nBOUTIQUE\nBOX\nBR\nBRADESCO\nBRIDGESTONE\nBROADWAY\nBROKER\nBROTHER\nBRUSSELS\nBS\nBT\nBUDAPEST\nBUGATTI\nBUILD\nBUILDERS\nBUSINESS\nBUY\nBUZZ\nBV\nBW\nBY\nBZ\nBZH\nCA\nCAB\nCAFE\nCAL\nCALL\nCALVINKLEIN\nCAM\nCAMERA\nCAMP\nCANCERRESEARCH\nCANON\nCAPETOWN\nCAPITAL\nCAPITALONE\nCAR\nCARAVAN\nCARDS\nCARE\nCAREER\nCAREERS\nCARS\nCASA\nCASE\nCASEIH\nCASH\nCASINO\nCAT\nCATERING\nCATHOLIC\nCBA\nCBN\nCBRE\nCBS\nCC\nCD\nCEB\nCENTER\nCEO\nCERN\nCF\nCFA\nCFD\nCG\nCH\nCHANEL\nCHANNEL\nCHARITY\nCHASE\nCHAT\nCHEAP\nCHINTAI\nCHRISTMAS\nCHROME\nCHURCH\nCI\nCIPRIANI\nCIRCLE\nCISCO\nCITADEL\nCITI\nCITIC\nCITY\nCITYEATS\nCK\nCL\nCLAIMS\nCLEANING\nCLICK\nCLINIC\nCLINIQUE\nCLOTHING\nCLOUD\nCLUB\nCLUBMED\nCM\nCN\nCO\nCOACH\nCODES\nCOFFEE\nCOLLEGE\nCOLOGNE\nCOM\nCOMCAST\nCOMMBANK\nCOMMUNITY\nCOMPANY\nCOMPARE\nCOMPUTER\nCOMSEC\nCONDOS\nCONSTRUCTION\nCONSULTING\nCONTACT\nCONTRACTORS\nCOOKING\nCOOKINGCHANNEL\nCOOL\nCOOP\nCORSICA\nCOUNTRY\nCOUPON\nCOUPONS\nCOURSES\nCPA\nCR\nCREDIT\nCREDITCARD\nCREDITUNION\nCRICKET\nCROWN\nCRS\nCRUISE\nCRUISES\nCSC\nCU\nCUISINELLA\nCV\nCW\nCX\nCY\nCYMRU\nCYOU\nCZ\nDABUR\nDAD\nDANCE\nDATA\nDATE\nDATING\nDATSUN\nDAY\nDCLK\nDDS\nDE\nDEAL\nDEALER\nDEALS\nDEGREE\nDELIVERY\nDELL\nDELOITTE\nDELTA\nDEMOCRAT\nDENTAL\nDENTIST\nDESI\nDESIGN\nDEV\nDHL\nDIAMONDS\nDIET\nDIGITAL\nDIRECT\nDIRECTORY\nDISCOUNT\nDISCOVER\nDISH\nDIY\nDJ\nDK\nDM\nDNP\nDO\nDOCS\nDOCTOR\nDOG\nDOMAINS\nDOT\nDOWNLOAD\nDRIVE\nDTV\nDUBAI\nDUCK\nDUNLOP\nDUPONT\nDURBAN\nDVAG\nDVR\nDZ\nEARTH\nEAT\nEC\nECO\nEDEKA\nEDU\nEDUCATION\nEE\nEG\nEMAIL\nEMERCK\nENERGY\nENGINEER\nENGINEERING\nENTERPRISES\nEPSON\nEQUIPMENT\nER\nERICSSON\nERNI\nES\nESQ\nESTATE\nET\nETISALAT\nEU\nEUROVISION\nEUS\nEVENTS\nEXCHANGE\nEXPERT\nEXPOSED\nEXPRESS\nEXTRASPACE\nFAGE\nFAIL\nFAIRWINDS\nFAITH\nFAMILY\nFAN\nFANS\nFARM\nFARMERS\nFASHION\nFAST\nFEDEX\nFEEDBACK\nFERRARI\nFERRERO\nFI\nFIAT\nFIDELITY\nFIDO\nFILM\nFINAL\nFINANCE\nFINANCIAL\nFIRE\nFIRESTONE\nFIRMDALE\nFISH\nFISHING\nFIT\nFITNESS\nFJ\nFK\nFLICKR\nFLIGHTS\nFLIR\nFLORIST\nFLOWERS\nFLY\nFM\nFO\nFOO\nFOOD\nFOODNETWORK\nFOOTBALL\nFORD\nFOREX\nFORSALE\nFORUM\nFOUNDATION\nFOX\nFR\nFREE\nFRESENIUS\nFRL\nFROGANS\nFRONTDOOR\nFRONTIER\nFTR\nFUJITSU\nFUJIXEROX\nFUN\nFUND\nFURNITURE\nFUTBOL\nFYI\nGA\nGAL\nGALLERY\nGALLO\nGALLUP\nGAME\nGAMES\nGAP\nGARDEN\nGAY\nGB\nGBIZ\nGD\nGDN\nGE\nGEA\nGENT\nGENTING\nGEORGE\nGF\nGG\nGGEE\nGH\nGI\nGIFT\nGIFTS\nGIVES\nGIVING\nGL\nGLADE\nGLASS\nGLE\nGLOBAL\nGLOBO\nGM\nGMAIL\nGMBH\nGMO\nGMX\nGN\nGODADDY\nGOLD\nGOLDPOINT\nGOLF\nGOO\nGOODYEAR\nGOOG\nGOOGLE\nGOP\nGOT\nGOV\nGP\nGQ\nGR\nGRAINGER\nGRAPHICS\nGRATIS\nGREEN\nGRIPE\nGROCERY\nGROUP\nGS\nGT\nGU\nGUARDIAN\nGUCCI\nGUGE\nGUIDE\nGUITARS\nGURU\nGW\nGY\nHAIR\nHAMBURG\nHANGOUT\nHAUS\nHBO\nHDFC\nHDFCBANK\nHEALTH\nHEALTHCARE\nHELP\nHELSINKI\nHERE\nHERMES\nHGTV\nHIPHOP\nHISAMITSU\nHITACHI\nHIV\nHK\nHKT\nHM\nHN\nHOCKEY\nHOLDINGS\nHOLIDAY\nHOMEDEPOT\nHOMEGOODS\nHOMES\nHOMESENSE\nHONDA\nHORSE\nHOSPITAL\nHOST\nHOSTING\nHOT\nHOTELES\nHOTELS\nHOTMAIL\nHOUSE\nHOW\nHR\nHSBC\nHT\nHU\nHUGHES\nHYATT\nHYUNDAI\nIBM\nICBC\nICE\nICU\nID\nIE\nIEEE\nIFM\nIKANO\nIL\nIM\nIMAMAT\nIMDB\nIMMO\nIMMOBILIEN\nIN\nINC\nINDUSTRIES\nINFINITI\nINFO\nING\nINK\nINSTITUTE\nINSURANCE\nINSURE\nINT\nINTEL\nINTERNATIONAL\nINTUIT\nINVESTMENTS\nIO\nIPIRANGA\nIQ\nIR\nIRISH\nIS\nISMAILI\nIST\nISTANBUL\nIT\nITAU\nITV\nIVECO\nJAGUAR\nJAVA\nJCB\nJCP\nJE\nJEEP\nJETZT\nJEWELRY\nJIO\nJLL\nJM\nJMP\nJNJ\nJO\nJOBS\nJOBURG\nJOT\nJOY\nJP\nJPMORGAN\nJPRS\nJUEGOS\nJUNIPER\nKAUFEN\nKDDI\nKE\nKERRYHOTELS\nKERRYLOGISTICS\nKERRYPROPERTIES\nKFH\nKG\nKH\nKI\nKIA\nKIM\nKINDER\nKINDLE\nKITCHEN\nKIWI\nKM\nKN\nKOELN\nKOMATSU\nKOSHER\nKP\nKPMG\nKPN\nKR\nKRD\nKRED\nKUOKGROUP\nKW\nKY\nKYOTO\nKZ\nLA\nLACAIXA\nLAMBORGHINI\nLAMER\nLANCASTER\nLANCIA\nLAND\nLANDROVER\nLANXESS\nLASALLE\nLAT\nLATINO\nLATROBE\nLAW\nLAWYER\nLB\nLC\nLDS\nLEASE\nLECLERC\nLEFRAK\nLEGAL\nLEGO\nLEXUS\nLGBT\nLI\nLIDL\nLIFE\nLIFEINSURANCE\nLIFESTYLE\nLIGHTING\nLIKE\nLILLY\nLIMITED\nLIMO\nLINCOLN\nLINDE\nLINK\nLIPSY\nLIVE\nLIVING\nLIXIL\nLK\nLLC\nLLP\nLOAN\nLOANS\nLOCKER\nLOCUS\nLOFT\nLOL\nLONDON\nLOTTE\nLOTTO\nLOVE\nLPL\nLPLFINANCIAL\nLR\nLS\nLT\nLTD\nLTDA\nLU\nLUNDBECK\nLUPIN\nLUXE\nLUXURY\nLV\nLY\nMA\nMACYS\nMADRID\nMAIF\nMAISON\nMAKEUP\nMAN\nMANAGEMENT\nMANGO\nMAP\nMARKET\nMARKETING\nMARKETS\nMARRIOTT\nMARSHALLS\nMASERATI\nMATTEL\nMBA\nMC\nMCKINSEY\nMD\nME\nMED\nMEDIA\nMEET\nMELBOURNE\nMEME\nMEMORIAL\nMEN\nMENU\nMERCKMSD\nMETLIFE\nMG\nMH\nMIAMI\nMICROSOFT\nMIL\nMINI\nMINT\nMIT\nMITSUBISHI\nMK\nML\nMLB\nMLS\nMM\nMMA\nMN\nMO\nMOBI\nMOBILE\nMODA\nMOE\nMOI\nMOM\nMONASH\nMONEY\nMONSTER\nMORMON\nMORTGAGE\nMOSCOW\nMOTO\nMOTORCYCLES\nMOV\nMOVIE\nMP\nMQ\nMR\nMS\nMSD\nMT\nMTN\nMTR\nMU\nMUSEUM\nMUTUAL\nMV\nMW\nMX\nMY\nMZ\nNA\nNAB\nNAGOYA\nNAME\nNATIONWIDE\nNATURA\nNAVY\nNBA\nNC\nNE\nNEC\nNET\nNETBANK\nNETFLIX\nNETWORK\nNEUSTAR\nNEW\nNEWHOLLAND\nNEWS\nNEXT\nNEXTDIRECT\nNEXUS\nNF\nNFL\nNG\nNGO\nNHK\nNI\nNICO\nNIKE\nNIKON\nNINJA\nNISSAN\nNISSAY\nNL\nNO\nNOKIA\nNORTHWESTERNMUTUAL\nNORTON\nNOW\nNOWRUZ\nNOWTV\nNP\nNR\nNRA\nNRW\nNTT\nNU\nNYC\nNZ\nOBI\nOBSERVER\nOFF\nOFFICE\nOKINAWA\nOLAYAN\nOLAYANGROUP\nOLDNAVY\nOLLO\nOM\nOMEGA\nONE\nONG\nONL\nONLINE\nONYOURSIDE\nOOO\nOPEN\nORACLE\nORANGE\nORG\nORGANIC\nORIGINS\nOSAKA\nOTSUKA\nOTT\nOVH\nPA\nPAGE\nPANASONIC\nPARIS\nPARS\nPARTNERS\nPARTS\nPARTY\nPASSAGENS\nPAY\nPCCW\nPE\nPET\nPF\nPFIZER\nPG\nPH\nPHARMACY\nPHD\nPHILIPS\nPHONE\nPHOTO\nPHOTOGRAPHY\nPHOTOS\nPHYSIO\nPICS\nPICTET\nPICTURES\nPID\nPIN\nPING\nPINK\nPIONEER\nPIZZA\nPK\nPL\nPLACE\nPLAY\nPLAYSTATION\nPLUMBING\nPLUS\nPM\nPN\nPNC\nPOHL\nPOKER\nPOLITIE\nPORN\nPOST\nPR\nPRAMERICA\nPRAXI\nPRESS\nPRIME\nPRO\nPROD\nPRODUCTIONS\nPROF\nPROGRESSIVE\nPROMO\nPROPERTIES\nPROPERTY\nPROTECTION\nPRU\nPRUDENTIAL\nPS\nPT\nPUB\nPW\nPWC\nPY\nQA\nQPON\nQUEBEC\nQUEST\nQVC\nRACING\nRADIO\nRAID\nRE\nREAD\nREALESTATE\nREALTOR\nREALTY\nRECIPES\nRED\nREDSTONE\nREDUMBRELLA\nREHAB\nREISE\nREISEN\nREIT\nRELIANCE\nREN\nRENT\nRENTALS\nREPAIR\nREPORT\nREPUBLICAN\nREST\nRESTAURANT\nREVIEW\nREVIEWS\nREXROTH\nRICH\nRICHARDLI\nRICOH\nRIGHTATHOME\nRIL\nRIO\nRIP\nRMIT\nRO\nROCHER\nROCKS\nRODEO\nROGERS\nROOM\nRS\nRSVP\nRU\nRUGBY\nRUHR\nRUN\nRW\nRWE\nRYUKYU\nSA\nSAARLAND\nSAFE\nSAFETY\nSAKURA\nSALE\nSALON\nSAMSCLUB\nSAMSUNG\nSANDVIK\nSANDVIKCOROMANT\nSANOFI\nSAP\nSARL\nSAS\nSAVE\nSAXO\nSB\nSBI\nSBS\nSC\nSCA\nSCB\nSCHAEFFLER\nSCHMIDT\nSCHOLARSHIPS\nSCHOOL\nSCHULE\nSCHWARZ\nSCIENCE\nSCJOHNSON\nSCOT\nSD\nSE\nSEARCH\nSEAT\nSECURE\nSECURITY\nSEEK\nSELECT\nSENER\nSERVICES\nSES\nSEVEN\nSEW\nSEX\nSEXY\nSFR\nSG\nSH\nSHANGRILA\nSHARP\nSHAW\nSHELL\nSHIA\nSHIKSHA\nSHOES\nSHOP\nSHOPPING\nSHOUJI\nSHOW\nSHOWTIME\nSHRIRAM\nSI\nSILK\nSINA\nSINGLES\nSITE\nSJ\nSK\nSKI\nSKIN\nSKY\nSKYPE\nSL\nSLING\nSM\nSMART\nSMILE\nSN\nSNCF\nSO\nSOCCER\nSOCIAL\nSOFTBANK\nSOFTWARE\nSOHU\nSOLAR\nSOLUTIONS\nSONG\nSONY\nSOY\nSPACE\nSPORT\nSPOT\nSPREADBETTING\nSR\nSRL\nSS\nST\nSTADA\nSTAPLES\nSTAR\nSTATEBANK\nSTATEFARM\nSTC\nSTCGROUP\nSTOCKHOLM\nSTORAGE\nSTORE\nSTREAM\nSTUDIO\nSTUDY\nSTYLE\nSU\nSUCKS\nSUPPLIES\nSUPPLY\nSUPPORT\nSURF\nSURGERY\nSUZUKI\nSV\nSWATCH\nSWIFTCOVER\nSWISS\nSX\nSY\nSYDNEY\nSYSTEMS\nSZ\nTAB\nTAIPEI\nTALK\nTAOBAO\nTARGET\nTATAMOTORS\nTATAR\nTATTOO\nTAX\nTAXI\nTC\nTCI\nTD\nTDK\nTEAM\nTECH\nTECHNOLOGY\nTEL\nTEMASEK\nTENNIS\nTEVA\nTF\nTG\nTH\nTHD\nTHEATER\nTHEATRE\nTIAA\nTICKETS\nTIENDA\nTIFFANY\nTIPS\nTIRES\nTIROL\nTJ\nTJMAXX\nTJX\nTK\nTKMAXX\nTL\nTM\nTMALL\nTN\nTO\nTODAY\nTOKYO\nTOOLS\nTOP\nTORAY\nTOSHIBA\nTOTAL\nTOURS\nTOWN\nTOYOTA\nTOYS\nTR\nTRADE\nTRADING\nTRAINING\nTRAVEL\nTRAVELCHANNEL\nTRAVELERS\nTRAVELERSINSURANCE\nTRUST\nTRV\nTT\nTUBE\nTUI\nTUNES\nTUSHU\nTV\nTVS\nTW\nTZ\nUA\nUBANK\nUBS\nUG\nUK\nUNICOM\nUNIVERSITY\nUNO\nUOL\nUPS\nUS\nUY\nUZ\nVA\nVACATIONS\nVANA\nVANGUARD\nVC\nVE\nVEGAS\nVENTURES\nVERISIGN\nVERSICHERUNG\nVET\nVG\nVI\nVIAJES\nVIDEO\nVIG\nVIKING\nVILLAS\nVIN\nVIP\nVIRGIN\nVISA\nVISION\nVIVA\nVIVO\nVLAANDEREN\nVN\nVODKA\nVOLKSWAGEN\nVOLVO\nVOTE\nVOTING\nVOTO\nVOYAGE\nVU\nVUELOS\nWALES\nWALMART\nWALTER\nWANG\nWANGGOU\nWATCH\nWATCHES\nWEATHER\nWEATHERCHANNEL\nWEBCAM\nWEBER\nWEBSITE\nWED\nWEDDING\nWEIBO\nWEIR\nWF\nWHOSWHO\nWIEN\nWIKI\nWILLIAMHILL\nWIN\nWINDOWS\nWINE\nWINNERS\nWME\nWOLTERSKLUWER\nWOODSIDE\nWORK\nWORKS\nWORLD\nWOW\nWS\nWTC\nWTF\nXBOX\nXEROX\nXFINITY\nXIHUAN\nXIN\nXN--11B4C3D\nXN--1CK2E1B\nXN--1QQW23A\nXN--2SCRJ9C\nXN--30RR7Y\nXN--3BST00M\nXN--3DS443G\nXN--3E0B707E\nXN--3HCRJ9C\nXN--3OQ18VL8PN36A\nXN--3PXU8K\nXN--42C2D9A\nXN--45BR5CYL\nXN--45BRJ9C\nXN--45Q11C\nXN--4GBRIM\nXN--54B7FTA0CC\nXN--55QW42G\nXN--55QX5D\nXN--5SU34J936BGSG\nXN--5TZM5G\nXN--6FRZ82G\nXN--6QQ986B3XL\nXN--80ADXHKS\nXN--80AO21A\nXN--80AQECDR1A\nXN--80ASEHDB\nXN--80ASWG\nXN--8Y0A063A\nXN--90A3AC\nXN--90AE\nXN--90AIS\nXN--9DBQ2A\nXN--9ET52U\nXN--9KRT00A\nXN--B4W605FERD\nXN--BCK1B9A5DRE4C\nXN--C1AVG\nXN--C2BR7G\nXN--CCK2B3B\nXN--CCKWCXETD\nXN--CG4BKI\nXN--CLCHC0EA0B2G2A9GCD\nXN--CZR694B\nXN--CZRS0T\nXN--CZRU2D\nXN--D1ACJ3B\nXN--D1ALF\nXN--E1A4C\nXN--ECKVDTC9D\nXN--EFVY88H\nXN--FCT429K\nXN--FHBEI\nXN--FIQ228C5HS\nXN--FIQ64B\nXN--FIQS8S\nXN--FIQZ9S\nXN--FJQ720A\nXN--FLW351E\nXN--FPCRJ9C3D\nXN--FZC2C9E2C\nXN--FZYS8D69UVGM\nXN--G2XX48C\nXN--GCKR3F0F\nXN--GECRJ9C\nXN--GK3AT1E\nXN--H2BREG3EVE\nXN--H2BRJ9C\nXN--H2BRJ9C8C\nXN--HXT814E\nXN--I1B6B1A6A2E\nXN--IMR513N\nXN--IO0A7I\nXN--J1AEF\nXN--J1AMH\nXN--J6W193G\nXN--JLQ480N2RG\nXN--JLQ61U9W7B\nXN--JVR189M\nXN--KCRX77D1X4A\nXN--KPRW13D\nXN--KPRY57D\nXN--KPUT3I\nXN--L1ACC\nXN--LGBBAT1AD8J\nXN--MGB9AWBF\nXN--MGBA3A3EJT\nXN--MGBA3A4F16A\nXN--MGBA7C0BBN0A\nXN--MGBAAKC7DVF\nXN--MGBAAM7A8H\nXN--MGBAB2BD\nXN--MGBAH1A3HJKRD\nXN--MGBAI9AZGQP6J\nXN--MGBAYH7GPA\nXN--MGBBH1A\nXN--MGBBH1A71E\nXN--MGBC0A9AZCG\nXN--MGBCA7DZDO\nXN--MGBCPQ6GPA1A\nXN--MGBERP4A5D4AR\nXN--MGBGU82A\nXN--MGBI4ECEXP\nXN--MGBPL2FH\nXN--MGBT3DHD\nXN--MGBTX2B\nXN--MGBX4CD0AB\nXN--MIX891F\nXN--MK1BU44C\nXN--MXTQ1M\nXN--NGBC5AZD\nXN--NGBE9E0A\nXN--NGBRX\nXN--NODE\nXN--NQV7F\nXN--NQV7FS00EMA\nXN--NYQY26A\nXN--O3CW4H\nXN--OGBPF8FL\nXN--OTU796D\nXN--P1ACF\nXN--P1AI\nXN--PGBS0DH\nXN--PSSY2U\nXN--Q7CE6A\nXN--Q9JYB4C\nXN--QCKA1PMC\nXN--QXA6A\nXN--QXAM\nXN--RHQV96G\nXN--ROVU88B\nXN--RVC1E0AM3E\nXN--S9BRJ9C\nXN--SES554G\nXN--T60B56A\nXN--TCKWE\nXN--TIQ49XQYJ\nXN--UNUP4Y\nXN--VERMGENSBERATER-CTB\nXN--VERMGENSBERATUNG-PWB\nXN--VHQUV\nXN--VUQ861B\nXN--W4R85EL8FHU5DNRA\nXN--W4RS40L\nXN--WGBH1C\nXN--WGBL6A\nXN--XHQ521B\nXN--XKC2AL3HYE2A\nXN--XKC2DL3A5EE0H\nXN--Y9A3AQ\nXN--YFRO4I67O\nXN--YGBI2AMMX\nXN--ZFR164B\nXXX\nXYZ\nYACHTS\nYAHOO\nYAMAXUN\nYANDEX\nYE\nYODOBASHI\nYOGA\nYOKOHAMA\nYOU\nYOUTUBE\nYT\nYUN\nZA\nZAPPOS\nZARA\nZERO\nZIP\nZM\nZONE\nZUERICH\nZW\n" }, { "path": "update.sh", "content": "set -e\ngit pull\ncargo run --release generate\ngit commit output -m \"Ran Autoupdate\"\ngit push\n\n" } ]