[
  {
    "path": ".github/ISSUE_TEMPLATE/bug_report.md",
    "content": "---\nname: Bug report\nabout: Create a report to help us improve\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\n**Describe the bug**\nA clear and concise description of what the bug is.\n\n**To Reproduce**\nSteps to reproduce the behavior:\n1. Go to '...'\n2. Click on '....'\n3. Scroll down to '....'\n4. See error\n\n**Expected behavior**\nA clear and concise description of what you expected to happen.\n\n**Screenshots**\nIf applicable, add screenshots to help explain your problem.\n\n**Desktop (please complete the following information):**\n - OS: [e.g. iOS]\n - Browser [e.g. chrome, safari]\n - Version [e.g. 22]\n\n**Smartphone (please complete the following information):**\n - Device: [e.g. iPhone6]\n - OS: [e.g. iOS8.1]\n - Browser [e.g. stock browser, safari]\n - Version [e.g. 22]\n\n**Additional context**\nAdd any other context about the problem here.\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/custom.md",
    "content": "---\nname: Custom issue template\nabout: Describe this issue template's purpose here.\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\n\n"
  },
  {
    "path": ".github/ISSUE_TEMPLATE/feature_request.md",
    "content": "---\nname: Feature request\nabout: Suggest an idea for this project\ntitle: ''\nlabels: ''\nassignees: ''\n\n---\n\n**Is your feature request related to a problem? Please describe.**\nA clear and concise description of what the problem is. Ex. I'm always frustrated when [...]\n\n**Describe the solution you'd like**\nA clear and concise description of what you want to happen.\n\n**Describe alternatives you've considered**\nA clear and concise description of any alternative solutions or features you've considered.\n\n**Additional context**\nAdd any other context or screenshots about the feature request here.\n"
  },
  {
    "path": ".gitignore",
    "content": "cmd/tests\nconfig/config.yaml\nlog/\nresult/\nstore/\nbin/\nbbolt/"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2022 Twiny\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "## Spidy\nA tool that crawl websites to find domain names and checks thier availiabity.\n\n### Install\n\n```sh\ngit clone https://github.com/twiny/spidy.git\ncd ./spidy\n\n# build\ngo build -o bin/spidy -v cmd/spidy/main.go\n\n# run\n./bin/spidy -c config/config.yaml -u https://github.com\n```\n\n## Usage\n\n```sh\nNAME:\n   Spidy - Domain name scraper\n\nUSAGE:\n   spidy [global options] command [command options] [arguments...]\n\nVERSION:\n   2.0.0\n\nCOMMANDS:\n   help, h  Shows a list of commands or help for one command\n\nGLOBAL OPTIONS:\n   --config path, -c path  path to config file\n   --help, -h              show help (default: false)\n   --urls urls, -u urls    urls of page to scrape  (accepts multiple inputs)\n   --version, -v           print the version (default: false)\n```\n\n## Configuration\n\n```yaml\n# main crawler config\ncrawler:\n    max_depth: 10 # max depth of pages to visit per website.\n    # filter: [] # regexp filter\n    rate_limit: \"1/5s\" # 1 request per 5 sec\n    max_body_size: \"20MB\" # max page body size\n    user_agents: # array of user-agents\n      - \"Spidy/2.1; +https://github.com/ twiny/spidy\"\n    # proxies: [] # array of proxy. http(s), SOCKS5\n# Logs\nlog:\n    rotate: 7 # log rotation\n    path: \"./log\" # log directory\n# Store\nstore:\n    ttl: \"24h\" # keep cache for 24h \n    path: \"./store\" # store directory\n# Results\nresult:\n    path: ./result # result directory\nparralle: 3 # number of concurrent workers \ntimeout: \"5m\" # request timeout\ntlds: [\"biz\", \"cc\", \"com\", \"edu\", \"info\", \"net\", \"org\", \"tv\"] # array of domain extension to check.\n```\n\n\n## TODO\n\n- [ ] Add support to more `writers`.\n- [ ] Add terminal logging.\n- [ ] Add test cases.\n\n## Issues\n\nNOTE: This package is provided \"as is\" with no guarantee. Use it at your own risk and always test it yourself before using it in a production environment. If you find any issues, please create a new issue."
  },
  {
    "path": "cmd/spidy/api/spider.go",
    "content": "package api\n\nimport (\n\t\"context\"\n\t_ \"embed\"\n\t\"fmt\"\n\t\"log\"\n\t\"net/http\"\n\t\"os\"\n\t\"os/signal\"\n\t\"strconv\"\n\t\"sync\"\n\t\"syscall\"\n\n\t//\n\t\"github.com/twiny/spidy/v2/internal/pkg/spider/v1\"\n\t\"github.com/twiny/spidy/v2/internal/service/cache\"\n\t\"github.com/twiny/spidy/v2/internal/service/writer\"\n\n\t//\n\t\"github.com/twiny/domaincheck\"\n\t\"github.com/twiny/flog\"\n\t\"github.com/twiny/wbot\"\n)\n\n//go:embed version\nvar Version string\n\n// Spider\ntype Spider struct {\n\twg      *sync.WaitGroup\n\tsetting *spider.Setting\n\tbot     *wbot.WBot\n\tpages   chan *spider.Page\n\tcheck   *domaincheck.Checker\n\tstore   spider.Storage\n\twrite   spider.Writer\n\tlog     *flog.Logger\n}\n\n// NewSpider\nfunc NewSpider(fp string) (*Spider, error) {\n\t// get settings\n\tsetting := spider.ParseSetting(fp)\n\n\t// crawler opts\n\topts := []wbot.Option{\n\t\twbot.SetParallel(setting.Parralle),\n\t\twbot.SetMaxDepth(setting.Crawler.MaxDepth),\n\t\twbot.SetRateLimit(setting.Crawler.Limit.Rate, setting.Crawler.Limit.Interval),\n\t\twbot.SetMaxBodySize(setting.Crawler.MaxBodySize),\n\t\twbot.SetUserAgents(setting.Crawler.UserAgents),\n\t\twbot.SetProxies(setting.Crawler.Proxies),\n\t}\n\n\tbot := wbot.NewWBot(opts...)\n\n\tcheck, err := domaincheck.NewChecker()\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// store\n\tstore, err := cache.NewCache(setting.Store.TTL, setting.Store.Path)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\t// logger\n\tlog, err := flog.NewLogger(setting.Log.Path, \"spidy\", setting.Log.Rotate)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\twrite, err := writer.NewCSVWriter(setting.Result.Path)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &Spider{\n\t\twg:      &sync.WaitGroup{},\n\t\tsetting: setting,\n\t\tbot:     bot,\n\t\tpages:   make(chan *spider.Page, setting.Parralle),\n\t\tcheck:   check,\n\t\tstore:   store,\n\t\twrite:   write,\n\t\tlog:     log,\n\t}, nil\n}\n\n// Start\nfunc (s *Spider) Start(links []string) error {\n\t// go crawl\n\ts.wg.Add(len(links))\n\tfor _, link := range links {\n\t\tgo func(l string) {\n\t\t\tdefer s.wg.Done()\n\t\t\t//\n\t\t\tif err := s.bot.Crawl(l); err != nil {\n\t\t\t\ts.log.Error(err.Error(), map[string]string{\"url\": l})\n\t\t\t}\n\t\t}(link)\n\t}\n\n\t// check domains\n\ts.wg.Add(s.setting.Parralle)\n\tfor i := 0; i < s.setting.Parralle; i++ {\n\t\tgo func() {\n\t\t\tdefer s.wg.Done()\n\t\t\t// results\n\t\t\tfor res := range s.bot.Stream() {\n\t\t\t\t// if response is ok\n\t\t\t\tif res.Status != http.StatusOK {\n\t\t\t\t\ts.log.Info(\"bad HTTP status\", map[string]string{\n\t\t\t\t\t\t\"url\":    res.URL.String(),\n\t\t\t\t\t\t\"status\": strconv.Itoa(res.Status),\n\t\t\t\t\t})\n\t\t\t\t\tcontinue\n\t\t\t\t}\n\n\t\t\t\t// extract domains\n\t\t\t\tdomains := spider.FindDomains(res.Body)\n\n\t\t\t\t// check availability\n\t\t\t\tfor _, domain := range domains {\n\t\t\t\t\troot := fmt.Sprintf(\"%s.%s\", domain.Name, domain.TLD)\n\n\t\t\t\t\t// check if allowed extension\n\t\t\t\t\tif len(s.setting.TLDs) > 0 {\n\t\t\t\t\t\tif ok := s.setting.TLDs[domain.TLD]; !ok {\n\t\t\t\t\t\t\ts.log.Info(\"unsupported domain\", map[string]string{\n\t\t\t\t\t\t\t\t\"domain\": root,\n\t\t\t\t\t\t\t\t\"url\":    res.URL.String(),\n\t\t\t\t\t\t\t})\n\t\t\t\t\t\t\tcontinue\n\t\t\t\t\t\t}\n\t\t\t\t\t}\n\n\t\t\t\t\t// skip if already checked\n\t\t\t\t\tif s.store.HasChecked(root) {\n\t\t\t\t\t\ts.log.Info(\"already checked\", map[string]string{\n\t\t\t\t\t\t\t\"domain\": root,\n\t\t\t\t\t\t\t\"url\":    res.URL.String(),\n\t\t\t\t\t\t})\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\n\t\t\t\t\t//\n\t\t\t\t\tctx, cancel := context.WithTimeout(context.Background(), s.setting.Timeout)\n\t\t\t\t\tdefer cancel()\n\n\t\t\t\t\tstatus, err := s.check.Check(ctx, root)\n\t\t\t\t\tif err != nil {\n\t\t\t\t\t\ts.log.Error(err.Error(), map[string]string{\n\t\t\t\t\t\t\t\"domain\": root,\n\t\t\t\t\t\t\t\"url\":    res.URL.String(),\n\t\t\t\t\t\t})\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\n\t\t\t\t\t// save domain\n\t\t\t\t\tif err := s.write.Write(&spider.Domain{\n\t\t\t\t\t\tURL:    res.URL.String(),\n\t\t\t\t\t\tName:   domain.Name,\n\t\t\t\t\t\tTLD:    domain.TLD,\n\t\t\t\t\t\tStatus: status.String(),\n\t\t\t\t\t}); err != nil {\n\t\t\t\t\t\ts.log.Error(err.Error(), map[string]string{\n\t\t\t\t\t\t\t\"domain\": root,\n\t\t\t\t\t\t\t\"url\":    res.URL.String(),\n\t\t\t\t\t\t})\n\t\t\t\t\t\tcontinue\n\t\t\t\t\t}\n\n\t\t\t\t\t// terminal print\n\t\t\t\t\tfmt.Printf(\"[Spidy] == domain: %s - status %s\\n\", root, status.String())\n\t\t\t\t}\n\t\t\t}\n\t\t}()\n\t}\n\n\ts.wg.Wait()\n\treturn nil\n}\n\n// Shutdown\nfunc (s *Spider) Shutdown() error {\n\t// attempt graceful shutdown\n\tsigs := make(chan os.Signal, 1)\n\tsignal.Notify(sigs, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)\n\n\t<-sigs\n\tlog.Println(\"shutting down ...\")\n\n\t// 2nd ctrl+c kills program\n\tgo func() {\n\t\tsigs := make(chan os.Signal, 1)\n\t\tsignal.Notify(sigs, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)\n\t\t<-sigs\n\t\tlog.Println(\"killing program ...\")\n\t\tos.Exit(0)\n\t}()\n\n\ts.bot.Close()\n\ts.log.Close()\n\tif err := s.store.Close(); err != nil {\n\t\treturn err\n\t}\n\tos.Exit(0)\n\treturn nil\n}\n"
  },
  {
    "path": "cmd/spidy/api/version",
    "content": "2.0.0"
  },
  {
    "path": "cmd/spidy/main.go",
    "content": "package main\n\nimport (\n\t\"log\"\n\t\"os\"\n\n\t//\n\n\t\"github.com/twiny/spidy/v2/cmd/spidy/api\"\n\n\t//\n\t\"github.com/urfave/cli/v2\"\n)\n\n// main\nfunc main() {\n\tapp := &cli.App{\n\t\tName:     \"Spidy\",\n\t\tHelpName: \"spidy\",\n\t\tUsage:    \"Domain name scraper\",\n\t\tVersion:  api.Version,\n\t\tFlags: []cli.Flag{\n\t\t\t&cli.StringFlag{\n\t\t\t\tName:     \"config\",\n\t\t\t\tAliases:  []string{\"c\"},\n\t\t\t\tUsage:    \"`path` to config file\",\n\t\t\t\tRequired: true,\n\t\t\t},\n\t\t\t&cli.StringSliceFlag{\n\t\t\t\tName:     \"urls\",\n\t\t\t\tAliases:  []string{\"u\"},\n\t\t\t\tUsage:    \"`urls` of page to scrape\",\n\t\t\t\tRequired: true,\n\t\t\t},\n\t\t},\n\t\tAction: func(c *cli.Context) error {\n\t\t\ts, err := api.NewSpider(c.String(\"config\"))\n\t\t\tif err != nil {\n\t\t\t\treturn err\n\t\t\t}\n\t\t\tgo s.Shutdown()\n\n\t\t\treturn s.Start(c.StringSlice(\"urls\"))\n\t\t},\n\t}\n\n\tif err := app.Run(os.Args); err != nil {\n\t\tlog.Println(err)\n\t\treturn\n\t}\n}\n"
  },
  {
    "path": "config/example.config.yaml",
    "content": "crawler:\n    max_depth: 10\n    # filter: []\n    rate_limit: \"1/5s\"\n    max_body_size: \"20MB\"\n    user_agents:\n      - \"Spidy/2.1; +https://github.com/twiny/spidy\"\n    # proxies: []\nlog:\n    rotate: 7\n    path: \"./log\"\nstore:\n    ttl: \"24h\"\n    path: \"./store\"\nresult:\n    path: ./result\nparralle: 3\ntimeout: \"5m\"\ntlds: [\"biz\", \"cc\", \"com\", \"edu\", \"info\", \"net\", \"org\", \"tv\"]"
  },
  {
    "path": "go.mod",
    "content": "module github.com/twiny/spidy/v2\n\ngo 1.18\n\nrequire (\n\tgithub.com/PuerkitoBio/goquery v1.8.0\n\tgithub.com/twiny/carbon v1.0.1\n\tgithub.com/twiny/domaincheck v0.1.0\n\tgithub.com/twiny/flog v1.0.3\n\tgithub.com/twiny/wbot v0.1.5\n\tgithub.com/urfave/cli/v2 v2.10.3\n\tgolang.org/x/net v0.0.0-20220513224357-95641704303c\n\tgopkg.in/yaml.v3 v3.0.1\n)\n\nrequire (\n\tgithub.com/andybalholm/cascadia v1.3.1 // indirect\n\tgithub.com/benbjohnson/clock v1.3.0 // indirect\n\tgithub.com/cespare/xxhash v1.1.0 // indirect\n\tgithub.com/cespare/xxhash/v2 v2.1.1 // indirect\n\tgithub.com/cpuguy83/go-md2man/v2 v2.0.2 // indirect\n\tgithub.com/dgraph-io/badger/v3 v3.2103.2 // indirect\n\tgithub.com/dgraph-io/ristretto v0.1.0 // indirect\n\tgithub.com/dustin/go-humanize v1.0.0 // indirect\n\tgithub.com/fatih/color v1.10.0 // indirect\n\tgithub.com/goccy/go-yaml v1.9.4 // indirect\n\tgithub.com/gogo/protobuf v1.3.2 // indirect\n\tgithub.com/golang/glog v0.0.0-20160126235308-23def4e6c14b // indirect\n\tgithub.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 // indirect\n\tgithub.com/golang/protobuf v1.3.1 // indirect\n\tgithub.com/golang/snappy v0.0.3 // indirect\n\tgithub.com/google/flatbuffers v1.12.1 // indirect\n\tgithub.com/klauspost/compress v1.12.3 // indirect\n\tgithub.com/mattn/go-colorable v0.1.8 // indirect\n\tgithub.com/mattn/go-isatty v0.0.12 // indirect\n\tgithub.com/pkg/errors v0.9.1 // indirect\n\tgithub.com/russross/blackfriday/v2 v2.1.0 // indirect\n\tgithub.com/twiny/ratelimit v0.0.0-20220509163414-256d3376b0ac // indirect\n\tgithub.com/twiny/whois/v2 v2.0.1 // indirect\n\tgithub.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect\n\tgo.opencensus.io v0.22.5 // indirect\n\tgolang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e // indirect\n\tgolang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect\n)\n"
  },
  {
    "path": "go.sum",
    "content": "cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=\ngithub.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=\ngithub.com/OneOfOne/xxhash v1.2.2 h1:KMrpdQIwFcEqXDklaen+P1axHaj9BSKzvpUUfnHldSE=\ngithub.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=\ngithub.com/PuerkitoBio/goquery v1.8.0 h1:PJTF7AmFCFKk1N6V6jmKfrNH9tV5pNE6lZMkG0gta/U=\ngithub.com/PuerkitoBio/goquery v1.8.0/go.mod h1:ypIiRMtY7COPGk+I/YbZLbxsxn9g5ejnI2HSMtkjZvI=\ngithub.com/andybalholm/cascadia v1.3.1 h1:nhxRkql1kdYCc8Snf7D5/D3spOX+dBgjA6u8x004T2c=\ngithub.com/andybalholm/cascadia v1.3.1/go.mod h1:R4bJ1UQfqADjvDa4P6HZHLh/3OxWWEqc0Sk8XGwHqvA=\ngithub.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=\ngithub.com/benbjohnson/clock v1.3.0 h1:ip6w0uFQkncKQ979AypyG0ER7mqUSBdKLOgAle/AT8A=\ngithub.com/benbjohnson/clock v1.3.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=\ngithub.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=\ngithub.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=\ngithub.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=\ngithub.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=\ngithub.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=\ngithub.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=\ngithub.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=\ngithub.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=\ngithub.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=\ngithub.com/cpuguy83/go-md2man/v2 v2.0.2 h1:p1EgwI/C7NhT0JmVkwCD2ZBK8j4aeHQX2pMHHBfMQ6w=\ngithub.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=\ngithub.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=\ngithub.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=\ngithub.com/dgraph-io/badger/v3 v3.2103.2 h1:dpyM5eCJAtQCBcMCZcT4UBZchuTJgCywerHHgmxfxM8=\ngithub.com/dgraph-io/badger/v3 v3.2103.2/go.mod h1:RHo4/GmYcKKh5Lxu63wLEMHJ70Pac2JqZRYGhlyAo2M=\ngithub.com/dgraph-io/ristretto v0.1.0 h1:Jv3CGQHp9OjuMBSne1485aDpUkTKEcUqF+jm/LuerPI=\ngithub.com/dgraph-io/ristretto v0.1.0/go.mod h1:fux0lOrBhrVCJd3lcTHsIJhq1T2rokOu6v9Vcb3Q9ug=\ngithub.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2 h1:tdlZCpZ/P9DhczCTSixgIKmwPv6+wP5DGjqLYw5SUiA=\ngithub.com/dgryski/go-farm v0.0.0-20190423205320-6a90982ecee2/go.mod h1:SqUrOPUnsFjfmXRMNPybcSiG0BgUW2AuFH8PAnS2iTw=\ngithub.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo=\ngithub.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=\ngithub.com/fatih/color v1.10.0 h1:s36xzo75JdqLaaWoiEHk767eHiwo0598uUxyfiPkDsg=\ngithub.com/fatih/color v1.10.0/go.mod h1:ELkj/draVOlAH/xkhN6mQ50Qd0MPOk5AAr3maGEBuJM=\ngithub.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=\ngithub.com/go-playground/assert/v2 v2.0.1/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=\ngithub.com/go-playground/locales v0.13.0 h1:HyWk6mgj5qFqCT5fjGBuRArbVDfE4hi8+e8ceBS/t7Q=\ngithub.com/go-playground/locales v0.13.0/go.mod h1:taPMhCMXrRLJO55olJkUXHZBHCxTMfnGwq/HNwmWNS8=\ngithub.com/go-playground/universal-translator v0.17.0 h1:icxd5fm+REJzpZx7ZfpaD876Lmtgy7VtROAbHHXk8no=\ngithub.com/go-playground/universal-translator v0.17.0/go.mod h1:UkSxE5sNxxRwHyU+Scu5vgOQjsIJAF8j9muTVoKLVtA=\ngithub.com/go-playground/validator/v10 v10.4.1 h1:pH2c5ADXtd66mxoE0Zm9SUhxE20r7aM3F26W0hOn+GE=\ngithub.com/go-playground/validator/v10 v10.4.1/go.mod h1:nlOn6nFhuKACm19sB/8EGNn9GlaMV7XkbRSipzJ0Ii4=\ngithub.com/goccy/go-yaml v1.9.4 h1:S0GCYjwHKVI6IHqio7QWNKNThUl6NLzFd/g8Z65Axw8=\ngithub.com/goccy/go-yaml v1.9.4/go.mod h1:U/jl18uSupI5rdI2jmuCswEA2htH9eXfferR3KfscvA=\ngithub.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=\ngithub.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=\ngithub.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=\ngithub.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=\ngithub.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I=\ngithub.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=\ngithub.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=\ngithub.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=\ngithub.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=\ngithub.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=\ngithub.com/golang/snappy v0.0.3 h1:fHPg5GQYlCeLIPB9BZqMVR5nR9A+IM5zcgeTdjMYmLA=\ngithub.com/golang/snappy v0.0.3/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=\ngithub.com/google/flatbuffers v1.12.1 h1:MVlul7pQNoDzWRLTw5imwYsl+usrS1TXG2H4jg6ImGw=\ngithub.com/google/flatbuffers v1.12.1/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=\ngithub.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=\ngithub.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=\ngithub.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=\ngithub.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=\ngithub.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=\ngithub.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=\ngithub.com/klauspost/compress v1.12.3 h1:G5AfA94pHPysR56qqrkO2pxEexdDzrpFJ6yt/VqWxVU=\ngithub.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8eO+e+Dq5Gzg=\ngithub.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=\ngithub.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=\ngithub.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=\ngithub.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=\ngithub.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=\ngithub.com/leodido/go-urn v1.2.0 h1:hpXL4XnriNwQ/ABnpepYM/1vCLWNDfUNts8dX3xTG6Y=\ngithub.com/leodido/go-urn v1.2.0/go.mod h1:+8+nEpDfqqsY+g338gtMEUOtuK+4dEMhiQEgxpxOKII=\ngithub.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=\ngithub.com/mattn/go-colorable v0.1.8 h1:c1ghPdyEDarC70ftn0y+A/Ee++9zz8ljHG1b13eJ0s8=\ngithub.com/mattn/go-colorable v0.1.8/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc=\ngithub.com/mattn/go-isatty v0.0.12 h1:wuysRhFDzyxgEmMf5xjvJ2M9dZoWAXNNr5LSBS7uHXY=\ngithub.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU=\ngithub.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=\ngithub.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=\ngithub.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=\ngithub.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=\ngithub.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=\ngithub.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=\ngithub.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=\ngithub.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=\ngithub.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=\ngithub.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=\ngithub.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=\ngithub.com/spaolacci/murmur3 v1.1.0 h1:7c1g84S4BPRrfL5Xrdp6fOJ206sU9y293DDHaoy0bLI=\ngithub.com/spaolacci/murmur3 v1.1.0/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=\ngithub.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=\ngithub.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=\ngithub.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=\ngithub.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=\ngithub.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=\ngithub.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=\ngithub.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=\ngithub.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=\ngithub.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk=\ngithub.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=\ngithub.com/twiny/carbon v1.0.1 h1:srGnk3N4KbAvCVgieWzYgZkLoBYGjnerTdxqzPy3TQs=\ngithub.com/twiny/carbon v1.0.1/go.mod h1:Ymh/hwZd8cZWYWnSL9xqSaQMd955k9EJx4/YS8wVdv0=\ngithub.com/twiny/domaincheck v0.1.0 h1:ByFbTKzdLymEaEkqAoA+vFuBxi33zOOyXCOTvvAm95c=\ngithub.com/twiny/domaincheck v0.1.0/go.mod h1:vlDqt80kuclqhfG3KrTu/rJd7aZe5P6viJ2acVuUvL4=\ngithub.com/twiny/flog v1.0.3 h1:iBTf+yEm/maBTJYFaMgD2lXIE5g7gSZnaTnmVXbs1tI=\ngithub.com/twiny/flog v1.0.3/go.mod h1:Hi9bzahz0Zmw30XiBT9oqWOrc10ive6L42Owwz02Vp8=\ngithub.com/twiny/ratelimit v0.0.0-20220509163414-256d3376b0ac h1:nT+8DFvrU5Nu3Be2bK7LooU8AslFJeypQoAF+wm1CM0=\ngithub.com/twiny/ratelimit v0.0.0-20220509163414-256d3376b0ac/go.mod h1:C589KqlnfcMeRAJ+evrNJwSf9ddkXO926hRDtgjjoYM=\ngithub.com/twiny/wbot v0.1.5 h1:yTfTv6+tmVHik6aY2DLuJZUG5/WPP37oE2TAgXkXRno=\ngithub.com/twiny/wbot v0.1.5/go.mod h1:JNeqtjncCXLALd0qaKw2q/4kC8F34weLiyf9QOljzQk=\ngithub.com/twiny/whois/v2 v2.0.1 h1:jDqkiq0wv2qdm9d/bquhQpg7AhJDYf89g7ozZElSTuA=\ngithub.com/twiny/whois/v2 v2.0.1/go.mod h1:UeyP4HmWFruXXuYQ722s/BnWgwxi7fRb/bk9Fnqm7OA=\ngithub.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=\ngithub.com/urfave/cli/v2 v2.10.3 h1:oi571Fxz5aHugfBAJd5nkwSk3fzATXtMlpxdLylSCMo=\ngithub.com/urfave/cli/v2 v2.10.3/go.mod h1:f8iq5LtQ/bLxafbdBSLPPNsgaW0l/2fYYEHhAyPlwvo=\ngithub.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=\ngithub.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 h1:bAn7/zixMGCfxrRTfdpNzjtPYqr8smhKouy9mxVdGPU=\ngithub.com/xrash/smetrics v0.0.0-20201216005158-039620a65673/go.mod h1:N3UwUGtsrSj3ccvlPHLoLsHnpR27oXr4ZE984MbSER8=\ngithub.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=\ngithub.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=\ngo.opencensus.io v0.22.5 h1:dntmOdLpSpHlVqbW5Eay97DelsZHe+55D+xC6i0dDS0=\ngo.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=\ngolang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=\ngolang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=\ngolang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=\ngolang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=\ngolang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=\ngolang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=\ngolang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=\ngolang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=\ngolang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=\ngolang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=\ngolang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=\ngolang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=\ngolang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=\ngolang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=\ngolang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=\ngolang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=\ngolang.org/x/net v0.0.0-20220513224357-95641704303c h1:nF9mHSvoKBLkQNQhJZNsc66z2UzAMUbLGjC95CF3pU0=\ngolang.org/x/net v0.0.0-20220513224357-95641704303c/go.mod h1:CfG3xpIq0wQ8r1q4Su4UZFWDARRcnwPjda9FqA0JpMk=\ngolang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=\ngolang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=\ngolang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=\ngolang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=\ngolang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e h1:fLOSk5Q00efkSvAm+4xcoXD+RRmLmmulPn5I3Y9F2EM=\ngolang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=\ngolang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=\ngolang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=\ngolang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=\ngolang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=\ngolang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=\ngolang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=\ngolang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=\ngolang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=\ngolang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=\ngolang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=\ngolang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=\ngolang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=\ngolang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngolang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngolang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngolang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=\ngolang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=\ngoogle.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=\ngoogle.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=\ngoogle.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=\ngoogle.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=\ngoogle.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=\ngoogle.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=\ngopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=\ngopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=\ngopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=\ngopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=\ngopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=\ngopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=\nhonnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=\n"
  },
  {
    "path": "internal/pkg/hbyte/hbyte.go",
    "content": "package hbyte\n\nimport (\n\t\"fmt\"\n\t\"strings\"\n)\n\nconst (\n\tb  = \"b\"\n\tkb = \"kb\"\n\tmb = \"mb\"\n\tgb = \"gb\"\n\ttb = \"tb\"\n)\n\n// type BYTE int64\n\nconst (\n\tB int64 = 1 << (10 * iota)\n\tKB\n\tMB\n\tGB\n\tTB\n)\n\n// Parse\nfunc Parse(s string) int64 {\n\t// lower case\n\ts = strings.ToLower(s)\n\n\tvar n int64\n\tvar unit string\n\n\tfmt.Sscanf(s, \"%d%s\", &n, &unit)\n\n\tswitch unit {\n\tcase b:\n\t\treturn n\n\tcase kb:\n\t\treturn n * KB\n\tcase mb:\n\t\treturn n * MB\n\tcase gb:\n\t\treturn n * GB\n\tcase tb:\n\t\treturn n * TB\n\tdefault:\n\t\treturn n\n\t}\n}\n\n// String\nfunc String(n int64) string {\n\tswitch {\n\tcase n >= TB:\n\t\treturn fmt.Sprintf(\"%d %s\", n/TB, tb)\n\tcase n >= GB:\n\t\treturn fmt.Sprintf(\"%d %s\", n/GB, gb)\n\tcase n >= MB:\n\t\treturn fmt.Sprintf(\"%d %s\", n/MB, mb)\n\tcase n >= KB:\n\t\treturn fmt.Sprintf(\"%d %s\", n/KB, kb)\n\tdefault:\n\t\treturn fmt.Sprintf(\"%d %s\", n, b)\n\t}\n}\n"
  },
  {
    "path": "internal/pkg/spider/v1/domain.go",
    "content": "package spider\n\n// Domain\ntype Domain struct {\n\tURL    string\n\tName   string\n\tTLD    string\n\tStatus string\n}\n\n// CSVRow\nfunc (d Domain) CSVRow() []string {\n\tvar row []string\n\treturn append(row, d.URL, d.Name, d.TLD, d.Status)\n}\n"
  },
  {
    "path": "internal/pkg/spider/v1/page.go",
    "content": "package spider\n\nimport \"net/url\"\n\n// Page\ntype Page struct {\n\tURL    *url.URL\n\tStatus int\n\tBody   []byte\n}\n"
  },
  {
    "path": "internal/pkg/spider/v1/setting.go",
    "content": "package spider\n\nimport (\n\t\"io/ioutil\"\n\t\"runtime\"\n\t\"strconv\"\n\t\"strings\"\n\t\"time\"\n\n\t//\n\t\"github.com/twiny/spidy/v2/internal/pkg/hbyte\"\n\n\t\"gopkg.in/yaml.v3\"\n)\n\n// default cores\nvar core = func() int {\n\tc := runtime.NumCPU()\n\tif c == 1 {\n\t\treturn c\n\t}\n\treturn c - 1\n}()\n\n// defaultSetting\nvar defaultSetting = &Setting{\n\tCrawler: struct {\n\t\tMaxDepth int32\n\t\tFilter   []string\n\t\tLimit    struct {\n\t\t\tRate     int\n\t\t\tInterval time.Duration\n\t\t}\n\t\tMaxBodySize int64\n\t\tUserAgents  []string\n\t\tProxies     []string\n\t}{\n\t\tMaxDepth: 10,\n\t\tFilter:   []string{},\n\t\tLimit: struct {\n\t\t\tRate     int\n\t\t\tInterval time.Duration\n\t\t}{\n\t\t\tRate:     1,\n\t\t\tInterval: time.Second,\n\t\t},\n\t\tMaxBodySize: 10 * 1024 * 1024, // 10 MB\n\t\tUserAgents:  []string{`Spidy/2.1; +https://github.com/twiny/spidy`},\n\t\tProxies:     []string{},\n\t},\n\tLog: struct {\n\t\tRotate int\n\t\tPath   string\n\t}{\n\t\tRotate: 7,\n\t\tPath:   \"./log\",\n\t},\n\tStore: struct {\n\t\tTTL  time.Duration\n\t\tPath string\n\t}{\n\t\tTTL:  6 * time.Hour, // format: 1h, 1d, 1w, 1m - minimum 6h\n\t\tPath: \"./store\",\n\t},\n\tResult: struct{ Path string }{\n\t\tPath: \"./result\",\n\t},\n\tParralle: core,\n\tTimeout:  1 * time.Minute,\n\tTLDs:     tlds,\n}\n\n// Setting\ntype Setting struct {\n\tCrawler struct {\n\t\tMaxDepth int32\n\t\tFilter   []string\n\t\tLimit    struct {\n\t\t\tRate     int\n\t\t\tInterval time.Duration\n\t\t}\n\t\tMaxBodySize int64\n\t\tUserAgents  []string\n\t\tProxies     []string\n\t}\n\tLog struct {\n\t\tRotate int // format: 30d\n\t\tPath   string\n\t}\n\tStore struct {\n\t\tTTL  time.Duration\n\t\tPath string\n\t}\n\tResult struct {\n\t\tPath string\n\t}\n\tParralle int\n\tTimeout  time.Duration\n\tTLDs     map[string]bool\n}\n\n// ParseSetting\nfunc ParseSetting(fp string) *Setting {\n\tdata, err := ioutil.ReadFile(fp)\n\tif err != nil {\n\t\treturn defaultSetting\n\t}\n\n\tvar s = struct {\n\t\tCrawler struct {\n\t\t\tMaxDepth    int32    `yaml:\"max_depth\"`\n\t\t\tFilter      []string `yaml:\"filter,flow\"`\n\t\t\tRateLimit   string   `yaml:\"rate_limit\"` // format: req/time.Duration => 5/1s\n\t\t\tMaxBodySize string   `yaml:\"max_body_size\"`\n\t\t\tUserAgents  []string `yaml:\"user_agents,flow\"`\n\t\t\tProxies     []string `yaml:\"proxies,flow\"`\n\t\t} `yaml:\"crawler\"`\n\t\tLog struct {\n\t\t\tRotate int    `yaml:\"rotate\"` // format: 30d\n\t\t\tPath   string `yaml:\"path\"`\n\t\t} `yaml:\"log\"`\n\t\tStore struct {\n\t\t\tTTL  string `yaml:\"ttl\"` // format: 1h, 24h\n\t\t\tPath string `yaml:\"path\"`\n\t\t} `yaml:\"store\"`\n\t\tResult struct {\n\t\t\tPath string `yaml:\"path\"`\n\t\t} `yaml:\"result\"`\n\t\tParralle int      `yaml:\"parralle\"`\n\t\tTimeout  string   `yaml:\"timeout\"`\n\t\tTLDs     []string `yaml:\"tlds,flow\"`\n\t}{}\n\n\tif err := yaml.Unmarshal(data, &s); err != nil {\n\t\treturn defaultSetting\n\t}\n\n\trate, interval := parseRateLimit(s.Crawler.RateLimit)\n\n\treturn &Setting{\n\t\tCrawler: struct {\n\t\t\tMaxDepth int32\n\t\t\tFilter   []string\n\t\t\tLimit    struct {\n\t\t\t\tRate     int\n\t\t\t\tInterval time.Duration\n\t\t\t}\n\t\t\tMaxBodySize int64\n\t\t\tUserAgents  []string\n\t\t\tProxies     []string\n\t\t}{\n\t\t\tMaxDepth: s.Crawler.MaxDepth,\n\t\t\tFilter:   s.Crawler.Filter,\n\t\t\tLimit: struct {\n\t\t\t\tRate     int\n\t\t\t\tInterval time.Duration\n\t\t\t}{\n\t\t\t\tRate:     rate,\n\t\t\t\tInterval: interval,\n\t\t\t},\n\t\t\tMaxBodySize: parseBodySize(s.Crawler.MaxBodySize),\n\t\t\tUserAgents:  s.Crawler.UserAgents,\n\t\t\tProxies:     s.Crawler.Proxies,\n\t\t},\n\t\tLog: struct {\n\t\t\tRotate int\n\t\t\tPath   string\n\t\t}{\n\t\t\tRotate: s.Log.Rotate,\n\t\t\tPath:   s.Log.Path,\n\t\t},\n\t\tStore: struct {\n\t\t\tTTL  time.Duration\n\t\t\tPath string\n\t\t}{\n\t\t\tTTL:  parseTTL(s.Store.TTL),\n\t\t\tPath: s.Store.Path,\n\t\t},\n\t\tResult: struct{ Path string }{\n\t\t\tPath: s.Result.Path,\n\t\t},\n\t\tParralle: s.Parralle,\n\t\tTimeout:  parseTimeout(s.Timeout),\n\t\tTLDs:     parseTLDs(s.TLDs),\n\t}\n}\n\n// parseRateLimit\nfunc parseRateLimit(s string) (int, time.Duration) {\n\t// default rate limit\n\tdr, di := defaultSetting.Crawler.Limit.Rate, defaultSetting.Crawler.Limit.Interval\n\n\tif s == \"\" {\n\t\treturn dr, di\n\t}\n\n\tparts := strings.Split(s, \"/\")\n\tif len(parts) != 2 {\n\t\treturn dr, di\n\t}\n\n\tr, i := parts[0], parts[1]\n\n\trate, err := strconv.Atoi(r)\n\tif err != nil {\n\t\treturn dr, di\n\t}\n\n\tinterval, err := time.ParseDuration(i)\n\tif err != nil {\n\t\treturn dr, di\n\t}\n\n\treturn rate, interval\n}\n\n// parseTLDs\nfunc parseTLDs(list []string) map[string]bool {\n\tm := map[string]bool{}\n\tfor _, s := range list {\n\t\tm[s] = true\n\t}\n\treturn m\n}\n\n// parseTimeout\nfunc parseTimeout(s string) time.Duration {\n\td, err := time.ParseDuration(s)\n\tif err != nil {\n\t\treturn defaultSetting.Timeout\n\t}\n\treturn d\n}\n\n// parseTTL\nfunc parseTTL(s string) time.Duration {\n\td, err := time.ParseDuration(s)\n\tif err != nil {\n\t\treturn defaultSetting.Timeout\n\t}\n\treturn d\n}\n\n// parseBodySize\nfunc parseBodySize(s string) int64 {\n\tsize := hbyte.Parse(s)\n\tif size == 0 {\n\t\treturn defaultSetting.Crawler.MaxBodySize\n\t}\n\treturn size\n}\n"
  },
  {
    "path": "internal/pkg/spider/v1/store.go",
    "content": "package spider\n\n// Storage\ntype Storage interface {\n\tHasChecked(name string) bool\n\tClose() error\n}\n"
  },
  {
    "path": "internal/pkg/spider/v1/string_replacer.go",
    "content": "package spider\n\nimport \"strings\"\n\n// UnescapeHTML: replace Unicode Character with a whitespace\n// to avoid getting wrong results when extracting domain from text.\nvar UnescapeHTML = strings.NewReplacer(\n\t`\\u002f`, ` `,\n\t`\\u002F`, ` `,\n\t//\n\t`\\u0020`, ` `,\n\t`\\u0021`, ` `,\n\t`\\u0022`, ` `,\n\t`\\u0023`, ` `,\n\t`\\u0024`, ` `,\n\t`\\u0025`, ` `,\n\t`\\u0026`, ` `,\n\t`\\u0027`, ` `,\n\t`\\u0028`, ` `,\n\t`\\u0029`, ` `,\n\t//\n\t`\\u002a`, ` `,\n\t`\\u002A`, ` `,\n\t//\n\t`\\u002b`, ` `,\n\t`\\u002B`, ` `,\n\t//\n\t`\\u002c`, ` `,\n\t`\\u002C`, ` `,\n\t//\n\t`\\u002d`, ` `,\n\t`\\u002D`, ` `,\n\t//\n\t`\\u002e`, ` `,\n\t`\\u002E`, ` `,\n\t//\n\t`\\u0030`, ` `,\n\t`\\u0031`, ` `,\n\t`\\u0032`, ` `,\n\t`\\u0033`, ` `,\n\t`\\u0034`, ` `,\n\t`\\u0035`, ` `,\n\t`\\u0036`, ` `,\n\t`\\u0037`, ` `,\n\t`\\u0038`, ` `,\n\t`\\u0039`, ` `,\n\t//\n\t`\\u003a`, ` `,\n\t`\\u003A`, ` `,\n\t//\n\t`\\u003b`, ` `,\n\t`\\u003B`, ` `,\n\t//\n\t`\\u003c`, ` `,\n\t`\\u003C`, ` `,\n\t//\n\t`\\u003d`, ` `,\n\t`\\u003D`, ` `,\n\t//\n\t`\\u003e`, ` `,\n\t`\\u003E`, ` `,\n\t//\n\t`\\u003f`, ` `,\n\t`\\u003F`, ` `,\n\t//\n\t`\\u0040`, ` `,\n\t`\\u0041`, ` `,\n\t`\\u0042`, ` `,\n\t`\\u0043`, ` `,\n\t`\\u0044`, ` `,\n\t`\\u0045`, ` `,\n\t`\\u0046`, ` `,\n\t`\\u0047`, ` `,\n\t`\\u0048`, ` `,\n\t`\\u0049`, ` `,\n\t//\n\t`\\u004a`, ` `,\n\t`\\u004A`, ` `,\n\t//\n\t`\\u004b`, ` `,\n\t`\\u004B`, ` `,\n\t//\n\t`\\u004c`, ` `,\n\t`\\u004C`, ` `,\n\t//\n\t`\\u004d`, ` `,\n\t`\\u004D`, ` `,\n\t//\n\t`\\u004e`, ` `,\n\t`\\u004E`, ` `,\n\t//\n\t`\\u004f`, ` `,\n\t`\\u004F`, ` `,\n\t//\n\t`\\u0050`, ` `,\n\t`\\u0051`, ` `,\n\t`\\u0052`, ` `,\n\t`\\u0053`, ` `,\n\t`\\u0054`, ` `,\n\t`\\u0055`, ` `,\n\t`\\u0056`, ` `,\n\t`\\u0057`, ` `,\n\t`\\u0058`, ` `,\n\t`\\u0059`, ` `,\n\t//\n\t`\\u005a`, ` `,\n\t`\\u005A`, ` `,\n\t//\n\t`\\u005b`, ` `,\n\t`\\u005B`, ` `,\n\t//\n\t`\\u005c`, ` `,\n\t`\\u005C`, ` `,\n\t//\n\t`\\u005d`, ` `,\n\t`\\u005D`, ` `,\n\t//\n\t`\\u005e`, ` `,\n\t`\\u005E`, ` `,\n\t//\n\t`\\u005f`, ` `,\n\t`\\u005F`, ` `,\n\t//\n\t`\\u0060`, ` `,\n\t`\\u0061`, ` `,\n\t`\\u0062`, ` `,\n\t`\\u0063`, ` `,\n\t`\\u0064`, ` `,\n\t`\\u0065`, ` `,\n\t`\\u0066`, ` `,\n\t`\\u0067`, ` `,\n\t`\\u0068`, ` `,\n\t`\\u0069`, ` `,\n\t//\n\t`\\u006a`, ` `,\n\t`\\u006A`, ` `,\n\t//\n\t`\\u006b`, ` `,\n\t`\\u006B`, ` `,\n\t//\n\t`\\u006c`, ` `,\n\t`\\u006C`, ` `,\n\t//\n\t`\\u006d`, ` `,\n\t`\\u006D`, ` `,\n\t//\n\t`\\u006e`, ` `,\n\t`\\u006E`, ` `,\n\t//\n\t`\\u006f`, ` `,\n\t`\\u006F`, ` `,\n\t//\n\t`\\u0070`, ` `,\n\t`\\u0071`, ` `,\n\t`\\u0072`, ` `,\n\t`\\u0073`, ` `,\n\t`\\u0074`, ` `,\n\t`\\u0075`, ` `,\n\t`\\u0076`, ` `,\n\t`\\u0077`, ` `,\n\t`\\u0078`, ` `,\n\t`\\u0079`, ` `,\n\t//\n\t`\\u007a`, ` `,\n\t`\\u007A`, ` `,\n\t//\n\t`\\u007b`, ` `,\n\t`\\u007B`, ` `,\n\t//\n\t`\\u007c`, ` `,\n\t`\\u007C`, ` `,\n\t//\n\t`\\u007d`, ` `,\n\t`\\u007D`, ` `,\n\t//\n\t`\\u007e`, ` `,\n\t`\\u007E`, ` `,\n\t//\n\t`%20`, ` `,\n\t`%21`, ` `,\n\t`%22`, ` `,\n\t`%23`, ` `,\n\t`%24`, ` `,\n\t`%25`, ` `,\n\t`%26`, ` `,\n\t`%27`, ` `,\n\t`%28`, ` `,\n\t`%29`, ` `,\n\t`%2A`, ` `,\n\t`%2B`, ` `,\n\t`%2C`, ` `,\n\t`%2D`, ` `,\n\t`%2E`, ` `,\n\t`%2F`, ` `,\n\t`%30`, ` `,\n\t`%31`, ` `,\n\t`%32`, ` `,\n\t`%33`, ` `,\n\t`%34`, ` `,\n\t`%35`, ` `,\n\t`%36`, ` `,\n\t`%37`, ` `,\n\t`%38`, ` `,\n\t`%39`, ` `,\n\t`%3A`, ` `,\n\t`%3B`, ` `,\n\t`%3C`, ` `,\n\t`%3D`, ` `,\n\t`%3E`, ` `,\n\t`%3F`, ` `,\n\t`%40`, ` `,\n\t`%41`, ` `,\n\t`%42`, ` `,\n\t`%43`, ` `,\n\t`%44`, ` `,\n\t`%45`, ` `,\n\t`%46`, ` `,\n\t`%47`, ` `,\n\t`%48`, ` `,\n\t`%49`, ` `,\n\t`%4A`, ` `,\n\t`%4B`, ` `,\n\t`%4C`, ` `,\n\t`%4D`, ` `,\n\t`%4E`, ` `,\n\t`%4F`, ` `,\n\t`%50`, ` `,\n\t`%51`, ` `,\n\t`%52`, ` `,\n\t`%53`, ` `,\n\t`%54`, ` `,\n\t`%55`, ` `,\n\t`%56`, ` `,\n\t`%57`, ` `,\n\t`%58`, ` `,\n\t`%59`, ` `,\n\t`%5A`, ` `,\n\t`%5B`, ` `,\n\t`%5C`, ` `,\n\t`%5D`, ` `,\n\t`%5E`, ` `,\n\t`%5F`, ` `,\n\t`%60`, ` `,\n\t`%61`, ` `,\n\t`%62`, ` `,\n\t`%63`, ` `,\n\t`%64`, ` `,\n\t`%65`, ` `,\n\t`%66`, ` `,\n\t`%67`, ` `,\n\t`%68`, ` `,\n\t`%69`, ` `,\n\t`%6A`, ` `,\n\t`%6B`, ` `,\n\t`%6C`, ` `,\n\t`%6D`, ` `,\n\t`%6E`, ` `,\n\t`%6F`, ` `,\n\t`%70`, ` `,\n\t`%71`, ` `,\n\t`%72`, ` `,\n\t`%73`, ` `,\n\t`%74`, ` `,\n\t`%75`, ` `,\n\t`%76`, ` `,\n\t`%77`, ` `,\n\t`%78`, ` `,\n\t`%79`, ` `,\n\t`%7A`, ` `,\n\t`%7B`, ` `,\n\t`%7C`, ` `,\n\t`%7D`, ` `,\n\t`%7E`, ` `,\n\t`%7F`, ` `,\n\t`%80`, ` `,\n\t`%81`, ` `,\n\t`%82`, ` `,\n\t`%83`, ` `,\n\t`%84`, ` `,\n\t`%85`, ` `,\n\t`%86`, ` `,\n\t`%87`, ` `,\n\t`%88`, ` `,\n\t`%89`, ` `,\n\t`%8A`, ` `,\n\t`%8B`, ` `,\n\t`%8C`, ` `,\n\t`%8D`, ` `,\n\t`%8E`, ` `,\n\t`%8F`, ` `,\n\t`%90`, ` `,\n\t`%91`, ` `,\n\t`%92`, ` `,\n\t`%93`, ` `,\n\t`%94`, ` `,\n\t`%95`, ` `,\n\t`%96`, ` `,\n\t`%97`, ` `,\n\t`%98`, ` `,\n\t`%99`, ` `,\n\t`%9A`, ` `,\n\t`%9B`, ` `,\n\t`%9C`, ` `,\n\t`%9D`, ` `,\n\t`%9E`, ` `,\n\t`%9F`, ` `,\n\t`%A0`, ` `,\n\t`%A1`, ` `,\n\t`%A2`, ` `,\n\t`%A3`, ` `,\n\t`%A4`, ` `,\n\t`%A5`, ` `,\n\t`%A6`, ` `,\n\t`%A7`, ` `,\n\t`%A8`, ` `,\n\t`%A9`, ` `,\n\t`%AA`, ` `,\n\t`%AB`, ` `,\n\t`%AC`, ` `,\n\t`%AD`, ` `,\n\t`%AE`, ` `,\n\t`%AF`, ` `,\n\t`%B0`, ` `,\n\t`%B1`, ` `,\n\t`%B2`, ` `,\n\t`%B3`, ` `,\n\t`%B4`, ` `,\n\t`%B5`, ` `,\n\t`%B6`, ` `,\n\t`%B7`, ` `,\n\t`%B8`, ` `,\n\t`%B9`, ` `,\n\t`%BA`, ` `,\n\t`%BB`, ` `,\n\t`%BC`, ` `,\n\t`%BD`, ` `,\n\t`%BE`, ` `,\n\t`%BF`, ` `,\n\t`%C0`, ` `,\n\t`%C1`, ` `,\n\t`%C2`, ` `,\n\t`%C3`, ` `,\n\t`%C4`, ` `,\n\t`%C5`, ` `,\n\t`%C6`, ` `,\n\t`%C7`, ` `,\n\t`%C8`, ` `,\n\t`%C9`, ` `,\n\t`%CA`, ` `,\n\t`%CB`, ` `,\n\t`%CC`, ` `,\n\t`%CD`, ` `,\n\t`%CE`, ` `,\n\t`%CF`, ` `,\n\t`%D0`, ` `,\n\t`%D1`, ` `,\n\t`%D2`, ` `,\n\t`%D3`, ` `,\n\t`%D4`, ` `,\n\t`%D5`, ` `,\n\t`%D6`, ` `,\n\t`%D7`, ` `,\n\t`%D8`, ` `,\n\t`%D9`, ` `,\n\t`%DA`, ` `,\n\t`%DB`, ` `,\n\t`%DC`, ` `,\n\t`%DD`, ` `,\n\t`%DE`, ` `,\n\t`%DF`, ` `,\n\t`%E0`, ` `,\n\t`%E1`, ` `,\n\t`%E2`, ` `,\n\t`%E3`, ` `,\n\t`%E4`, ` `,\n\t`%E5`, ` `,\n\t`%E6`, ` `,\n\t`%E7`, ` `,\n\t`%E8`, ` `,\n\t`%E9`, ` `,\n\t`%EA`, ` `,\n\t`%EB`, ` `,\n\t`%EC`, ` `,\n\t`%ED`, ` `,\n\t`%EE`, ` `,\n\t`%EF`, ` `,\n\t`%F0`, ` `,\n\t`%F1`, ` `,\n\t`%F2`, ` `,\n\t`%F3`, ` `,\n\t`%F4`, ` `,\n\t`%F5`, ` `,\n\t`%F6`, ` `,\n\t`%F7`, ` `,\n\t`%F8`, ` `,\n\t`%F9`, ` `,\n\t`%FA`, ` `,\n\t`%FB`, ` `,\n\t`%FC`, ` `,\n\t`%FD`, ` `,\n\t`%FE`, ` `,\n\t`%FF`, ` `,\n)\n"
  },
  {
    "path": "internal/pkg/spider/v1/tld_list.go",
    "content": "package spider\n\n// allowed TLDs: list of allowed domain tlds to avoid getting bad extensions.\nvar tlds = map[string]bool{\n\t\"ac\":     true,\n\t\"ae\":     true,\n\t\"aero\":   true,\n\t\"af\":     true,\n\t\"ag\":     true,\n\t\"am\":     true,\n\t\"as\":     true,\n\t\"asia\":   true,\n\t\"at\":     true,\n\t\"au\":     true,\n\t\"ax\":     true,\n\t\"be\":     true,\n\t\"bg\":     true,\n\t\"bi\":     true,\n\t\"biz\":    true,\n\t\"bj\":     true,\n\t\"br\":     true,\n\t\"by\":     true,\n\t\"ca\":     true,\n\t\"cat\":    true,\n\t\"cc\":     true,\n\t\"cl\":     true,\n\t\"cn\":     true,\n\t\"co\":     true,\n\t\"com\":    true,\n\t\"coop\":   true,\n\t\"cx\":     true,\n\t\"de\":     true,\n\t\"dk\":     true,\n\t\"dm\":     true,\n\t\"dz\":     true,\n\t\"edu\":    true,\n\t\"ee\":     true,\n\t\"eu\":     true,\n\t\"fi\":     true,\n\t\"fo\":     true,\n\t\"fr\":     true,\n\t\"gb.com\": true,\n\t\"qc.com\": true,\n\t\"ge\":     true,\n\t\"gl\":     true,\n\t\"gov\":    true,\n\t\"gs\":     true,\n\t\"hk\":     true,\n\t\"hr\":     true,\n\t\"hu\":     true,\n\t\"hu.com\": true,\n\t\"id\":     true,\n\t\"ie\":     true,\n\t\"in\":     true,\n\t\"info\":   true,\n\t\"int\":    true,\n\t\"io\":     true,\n\t\"ir\":     true,\n\t\"is\":     true,\n\t\"je\":     true,\n\t\"jobs\":   true,\n\t\"kg\":     true,\n\t\"kr\":     true,\n\t\"la\":     true,\n\t\"lu\":     true,\n\t\"lv\":     true,\n\t\"ly\":     true,\n\t\"ma\":     true,\n\t\"md\":     true,\n\t\"me\":     true,\n\t\"mk\":     true,\n\t\"mobi\":   true,\n\t\"ms\":     true,\n\t\"mu\":     true,\n\t\"mx\":     true,\n\t\"name\":   true,\n\t\"net\":    true,\n\t\"nf\":     true,\n\t\"ng\":     true,\n\t\"no\":     true,\n\t\"no.com\": true,\n\t\"nu\":     true,\n\t\"nz\":     true,\n\t\"org\":    true,\n\t\"pl\":     true,\n\t\"pr\":     true,\n\t\"pro\":    true,\n\t\"pw\":     true,\n\t\"ro\":     true,\n\t\"ru\":     true,\n\t\"sa.com\": true,\n\t\"sc\":     true,\n\t\"se\":     true,\n\t\"se.com\": true,\n\t\"sg\":     true,\n\t\"sh\":     true,\n\t\"si\":     true,\n\t\"sk\":     true,\n\t\"sm\":     true,\n\t\"st\":     true,\n\t\"so\":     true,\n\t\"su\":     true,\n\t\"tc\":     true,\n\t\"tel\":    true,\n\t\"tf\":     true,\n\t\"th\":     true,\n\t\"tk\":     true,\n\t\"tl\":     true,\n\t\"tm\":     true,\n\t\"tn\":     true,\n\t\"travel\": true,\n\t\"tw\":     true,\n\t\"tv\":     true,\n\t\"tz\":     true,\n\t\"ua\":     true,\n\t\"uk\":     true,\n\t\"us\":     true,\n\t\"uy.com\": true,\n\t\"uz\":     true,\n\t\"vc\":     true,\n\t\"ve\":     true,\n\t\"vg\":     true,\n\t\"ws\":     true,\n\t\"xxx\":    true,\n\t\"yu\":     true,\n\t\"za.com\": true,\n}\n"
  },
  {
    "path": "internal/pkg/spider/v1/utils.go",
    "content": "package spider\n\nimport (\n\t\"bytes\"\n\t\"regexp\"\n\t\"strings\"\n\n\t\"github.com/PuerkitoBio/goquery\"\n\t\"golang.org/x/net/publicsuffix\"\n)\n\nvar (\n\t//  domain regexp\n\tdomainRegexp = regexp.MustCompile(`(([[:alnum:]]-?)?([[:alnum:]]-?)+\\.)+[[:alpha:]]{2,4}`)\n)\n\n// FindDomains\nfunc FindDomains(body []byte) (domains []Domain) {\n\tdoc, err := goquery.NewDocumentFromReader(bytes.NewReader(body))\n\tif err != nil {\n\t\treturn\n\t}\n\n\tvar s = UnescapeHTML.Replace(doc.Text())\n\n\tfor _, domain := range domainRegexp.FindAllString(s, -1) {\n\t\tname, tld, ok := splitDomain(domain)\n\t\tif ok {\n\t\t\tdomains = append(domains, Domain{\n\t\t\t\tName: name,\n\t\t\t\tTLD:  tld,\n\t\t\t})\n\t\t}\n\t}\n\n\treturn\n}\n\n// SplitDomain\nfunc splitDomain(d string) (name string, tld string, ok bool) {\n\t// get domain tld\n\troot, err := publicsuffix.EffectiveTLDPlusOne(d)\n\tif err != nil {\n\t\treturn\n\t}\n\n\t//convert to domain name, and tld\n\ti := strings.Index(root, \".\")\n\ttld = root[i+1:]\n\n\tif _, ok = tlds[tld]; !ok {\n\t\treturn\n\t}\n\n\troot = strings.ToLower(root)\n\ttld = strings.ToLower(tld)\n\tname = strings.TrimSuffix(root, \".\"+tld)\n\n\treturn\n}\n"
  },
  {
    "path": "internal/pkg/spider/v1/writer.go",
    "content": "package spider\n\n// Writer\ntype Writer interface {\n\tWrite(*Domain) error\n}\n"
  },
  {
    "path": "internal/service/cache/cache.go",
    "content": "package cache\n\nimport (\n\t\"time\"\n\n\t//\n\n\t//\n\t\"github.com/twiny/carbon\"\n)\n\n// Cache\ntype Cache struct {\n\tttl time.Duration\n\tdb  *carbon.Cache\n}\n\n// NewCache\nfunc NewCache(ttl time.Duration, dir string) (*Cache, error) {\n\tdb, err := carbon.NewCache(dir)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\treturn &Cache{\n\t\tttl: ttl,\n\t\tdb:  db,\n\t}, nil\n}\n\n// HasChecked\nfunc (c *Cache) HasChecked(name string) bool {\n\t// first check if domain is in cache\n\tb, err := c.db.Get(name)\n\tif err != nil || b == nil {\n\t\t// if not found save to cache\n\t\tif err := c.db.Set(name, []byte(name), c.ttl); err != nil {\n\t\t\treturn false\n\t\t}\n\t\treturn false\n\t}\n\treturn true\n}\n\n// Close\nfunc (c *Cache) Close() error {\n\tc.db.Close()\n\treturn nil\n}\n"
  },
  {
    "path": "internal/service/writer/csv_writer.go",
    "content": "package writer\n\nimport (\n\t\"encoding/csv\"\n\t\"os\"\n\t\"path/filepath\"\n\t\"sync\"\n\t\"time\"\n\n\t\"github.com/twiny/spidy/v2/internal/pkg/spider/v1\"\n)\n\n// CSVWriter\ntype CSVWriter struct {\n\tl *sync.Mutex\n\tf *os.File\n\tw *csv.Writer\n}\n\n// NewCSVWriter\nfunc NewCSVWriter(dir string) (*CSVWriter, error) {\n\tif _, err := os.Stat(dir); os.IsNotExist(err) {\n\t\tif err := os.MkdirAll(dir, 0755); err != nil {\n\t\t\treturn nil, err\n\t\t}\n\t}\n\n\tname := time.Now().Format(\"2006-01-02\")\n\tfp := filepath.Join(dir, name+\"_domains.csv\")\n\n\t// open or create log\n\tf, err := os.OpenFile(fp, os.O_APPEND|os.O_CREATE|os.O_WRONLY, os.ModePerm)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\treturn &CSVWriter{\n\t\tl: &sync.Mutex{},\n\t\tf: f,\n\t\tw: csv.NewWriter(f),\n\t}, nil\n}\n\n// Write\nfunc (c *CSVWriter) Write(d *spider.Domain) error {\n\tc.l.Lock()\n\tdefer func() {\n\t\tc.l.Unlock()\n\t\tc.w.Flush()\n\t}()\n\n\treturn c.w.Write([]string{d.Name + \".\" + d.TLD, d.Status})\n}\n\n// Close\nfunc (c *CSVWriter) Close() error {\n\treturn c.f.Close()\n}\n"
  }
]