[
  {
    "path": ".changeset/config.json",
    "content": "{\n  \"$schema\": \"https://unpkg.com/@changesets/config@3.0.0/schema.json\",\n  \"commit\": false,\n  \"fixed\": [[\"google-indexing-script\"]],\n  \"changelog\": [\n    \"@changesets/changelog-github\",\n    { \"repo\": \"goenning/google-indexing-script\" }\n  ],\n  \"linked\": [],\n  \"access\": \"public\",\n  \"baseBranch\": \"main\",\n  \"updateInternalDependencies\": \"patch\"\n}\n"
  },
  {
    "path": ".github/pull_request_template.md",
    "content": "**What did I change?**\n\n<!-- A brief description of the change. -->\n\n**Why did I change it?**\n\n<!-- A brief description of why the change was made. -->"
  },
  {
    "path": ".github/workflows/ci.yml",
    "content": "name: CI\non: [push]\n\njobs:\n  build:\n    name: Build, lint, and test on Node ${{ matrix.node }} and ${{ matrix.os }}\n\n    runs-on: ${{ matrix.os }}\n    strategy:\n      matrix:\n        node: [\"18.x\", \"20.x\"]\n        os: [ubuntu-latest]\n\n    steps:\n      - name: Checkout repo\n        uses: actions/checkout@v4\n\n      - name: Use Node ${{ matrix.node }}\n        uses: actions/setup-node@v4\n        with:\n          node-version: ${{ matrix.node }}\n          cache: \"npm\"\n\n      - name: Install Dependencies\n        run: npm install\n\n      - name: Build\n        run: npm run build\n\n"
  },
  {
    "path": ".github/workflows/release.yml",
    "content": "name: Release\non:\n  push:\n    branches:\n      - main\n\njobs:\n  release:\n    if: github.repository == 'goenning/google-indexing-script'\n\n    runs-on: ubuntu-latest\n\n    steps:\n      - name: Checkout repo\n        uses: actions/checkout@v4\n\n      - name: Use Node\n        uses: actions/setup-node@v4\n        with:\n          cache: \"npm\"\n\n      - name: Install Dependencies\n        run: npm install\n\n      - name: Build\n        run: npm run build\n\n      - name: Create Release Pull Request or Publish to npm\n        uses: changesets/action@v1\n        with:\n          publish: npm run release\n          version: npm run version\n          commit: \"release version\"\n          title: \"release version\"\n        env:\n          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}\n          NPM_TOKEN: ${{ secrets.NPM_TOKEN }}\n"
  },
  {
    "path": ".gitignore",
    "content": "service_account.json\n.cache\nnode_modules\n.vscode\ndist"
  },
  {
    "path": ".nvmrc",
    "content": "20\n"
  },
  {
    "path": "CHANGELOG.md",
    "content": "# google-indexing-script\n\n## 0.4.0\n\n### Minor Changes\n\n- [#68](https://github.com/goenning/google-indexing-script/pull/68) [`caa73f7`](https://github.com/goenning/google-indexing-script/commit/caa73f765b5d494d65a894a83bb8faf351e6d8ae) Thanks [@AntoineKM](https://github.com/AntoineKM)! - Improve CLI with commander\n\n## 0.3.0\n\n### Minor Changes\n\n- [#65](https://github.com/goenning/google-indexing-script/pull/65) [`e0c31f8`](https://github.com/goenning/google-indexing-script/commit/e0c31f837acfe2083843436050b40f21f3806838) Thanks [@AntoineKM](https://github.com/AntoineKM)! - Add custom URLs option\n\n### Patch Changes\n\n- [#65](https://github.com/goenning/google-indexing-script/pull/65) [`e0c31f8`](https://github.com/goenning/google-indexing-script/commit/e0c31f837acfe2083843436050b40f21f3806838) Thanks [@AntoineKM](https://github.com/AntoineKM)! - Fix siteUrls convertions\n\n## 0.2.0\n\n### Minor Changes\n\n- [#62](https://github.com/goenning/google-indexing-script/pull/62) [`93dd956`](https://github.com/goenning/google-indexing-script/commit/93dd956dca4065b97d6076db772560fba57aec50) Thanks [@hasanafzal8485](https://github.com/hasanafzal8485)! - Don't want the same URL use my API limit again until his previous cache limit is completed\n\n## 0.1.0\n\n### Minor Changes\n\n- [#55](https://github.com/goenning/google-indexing-script/pull/55) [`908938a`](https://github.com/goenning/google-indexing-script/commit/908938a701d964b75331e322fbea8d77e6db976e) Thanks [@AntoineKM](https://github.com/AntoineKM)! - feat(get-publish-metadata): optional retries if rate limited\n\n## 0.0.5\n\n### Patch Changes\n\n- [#44](https://github.com/goenning/google-indexing-script/pull/44) [`77b94ed`](https://github.com/goenning/google-indexing-script/commit/77b94edeef863721c07bd3e12d6d38052723f422) Thanks [@AntoineKM](https://github.com/AntoineKM)! - Add site url checker\n\n## 0.0.4\n\n### Patch Changes\n\n- [#40](https://github.com/goenning/google-indexing-script/pull/40) [`074f2c7`](https://github.com/goenning/google-indexing-script/commit/074f2c7ebbafff3a03ebf07baf7b21922a98698d) Thanks [@AntoineKM](https://github.com/AntoineKM)! - Add documentation comments\n\n## 0.0.3\n\n### Patch Changes\n\n- [#39](https://github.com/goenning/google-indexing-script/pull/39) [`9467e82`](https://github.com/goenning/google-indexing-script/commit/9467e82496170aeaa42ecd8ab6b8de4ba8f8315f) Thanks [@AntoineKM](https://github.com/AntoineKM)! - Fix index function to handle options passed\n"
  },
  {
    "path": "CONTRIBUTING.md",
    "content": "# Contributing to Google Indexing Script\n\nBefore jumping into a PR be sure to search [existing PRs](/goenning/google-indexing-script/pulls) or [issues](/goenning/google-indexing-script/issues) for an open or closed item that relates to your submission.\n\n# Developing\n\nAll pull requests should be opened against `main`.\n\n1. Clone the repository\n```bash\ngit clone https://github.com/goenning/google-indexing-script.git\n```\n\n2. Install dependencies\n```bash\nnpm install\n```\n\n3. Install the cli globally\n```bash\nnpm install -g .\n```\n\n4. Run the development bundle\n```bash\nnpm run dev\n```\n\n5. See how to [use it](/README.md#installation) and make your changes !\n\n# Building\n\nAfter making your changes, you can build the project with the following command:\n\n```bash\nnpm run build\n```\n\n# Pull Request\n\n1. Make sure your code is formatted with `prettier`\n2. Make sure your code passes the tests\n3. Make sure you added the changes with `npm run changeset`\n"
  },
  {
    "path": "LICENSE",
    "content": "MIT License\n\nCopyright (c) 2024 Guilherme Oenning\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
  },
  {
    "path": "README.md",
    "content": "# Google Indexing Script\n\nUse this script to get your entire site indexed on Google in less than 48 hours. No tricks, no hacks, just a simple script and a Google API.\n\n> [!IMPORTANT]\n>\n> 1. This script uses [Google Indexing API](https://developers.google.com/search/apis/indexing-api/v3/quickstart) and it only works on pages with either `JobPosting` or `BroadcastEvent` structured data.\n> 2. Indexing != Ranking. This will not help your page rank on Google, it'll just let Google know about the existence of your pages.\n\n## Requirements\n\n- Install [Node.js](https://nodejs.org/en/download)\n- An account on [Google Search Console](https://search.google.com/search-console/about) with the verified sites you want to index\n- An account on [Google Cloud](https://console.cloud.google.com/)\n\n## Preparation\n\n1. Follow this [guide](https://developers.google.com/search/apis/indexing-api/v3/prereqs) from Google. By the end of it, you should have a project on Google Cloud with the Indexing API enabled, a service account with the `Owner` permission on your sites.\n2. Make sure you enable both [`Google Search Console API`](https://console.cloud.google.com/apis/api/searchconsole.googleapis.com) and [`Web Search Indexing API`](https://console.cloud.google.com/apis/api/indexing.googleapis.com) on your [Google Project ➤ API Services ➤ Enabled API & Services](https://console.cloud.google.com/apis/dashboard).\n3. [Download the JSON](https://github.com/goenning/google-indexing-script/issues/2) file with the credentials of your service account and save it in the same folder as the script. The file should be named `service_account.json`\n\n## Installation\n\n### Using CLI\n\nInstall the cli globally on your machine.\n\n```bash\nnpm i -g google-indexing-script\n```\n\n### Using the repository\n\nClone the repository to your machine.\n\n```bash\ngit clone https://github.com/goenning/google-indexing-script.git\ncd google-indexing-script\n```\n\nInstall and build the project.\n\n```bash\nnpm install\nnpm run build\nnpm i -g .\n```\n\n> [!NOTE]\n> Ensure you are using an up-to-date Node.js version, with a preference for v20 or later. Check your current version with `node -v`.\n\n## Usage\n\n<details open>\n<summary>With <code>service_account.json</code> <i>(recommended)</i></summary>\n\nCreate a `.gis` directory in your home folder and move the `service_account.json` file there.\n\n```bash\nmkdir ~/.gis\nmv service_account.json ~/.gis\n```\n\nRun the script with the domain or url you want to index.\n\n```bash\ngis <domain or url>\n# example\ngis seogets.com\n```\n\nHere are some other ways to run the script:\n\n```bash\n# custom path to service_account.json\ngis seogets.com --path /path/to/service_account.json\n# long version command\ngoogle-indexing-script seogets.com\n# cloned repository\nnpm run index seogets.com\n```\n\n</details>\n\n<details>\n<summary>With environment variables</summary>\n\nOpen `service_account.json` and copy the `client_email` and `private_key` values.\n\nRun the script with the domain or url you want to index.\n\n```bash\nGIS_CLIENT_EMAIL=your-client-email GIS_PRIVATE_KEY=your-private-key gis seogets.com\n```\n\n</details>\n\n<details>\n<summary>With arguments <i>(not recommended)</i></summary>\n\nOpen `service_account.json` and copy the `client_email` and `private_key` values.\n\nOnce you have the values, run the script with the domain or url you want to index, the client email and the private key.\n\n```bash\ngis seogets.com --client-email your-client-email --private-key your-private-key\n```\n\n</details>\n\n<details>\n<summary>As a npm module</summary>\n\nYou can also use the script as a [npm module](https://www.npmjs.com/package/google-indexing-script) in your own project.\n\n```bash\nnpm i google-indexing-script\n```\n\n```javascript\nimport { index } from \"google-indexing-script\";\nimport serviceAccount from \"./service_account.json\";\n\nindex(\"seogets.com\", {\n  client_email: serviceAccount.client_email,\n  private_key: serviceAccount.private_key,\n})\n  .then(console.log)\n  .catch(console.error);\n```\n\nRead the [API documentation](https://jsdocs.io/package/google-indexing-script) for more details.\n\n</details>\n\nHere's an example of what you should expect:\n\n![](./output.png)\n\n> [!IMPORTANT]\n>\n> - Your site must have 1 or more sitemaps submitted to Google Search Console. Otherwise, the script will not be able to find the pages to index.\n> - You can run the script as many times as you want. It will only index the pages that are not already indexed.\n> - Sites with a large number of pages might take a while to index, be patient.\n\n## Quota\n\nDepending on your account several quotas are configured for the API (see [docs](https://developers.google.com/search/apis/indexing-api/v3/quota-pricing#quota)). By default the script exits as soon as the rate limit is exceeded. You can configure a retry mechanism for the read requests that apply on a per minute time frame.\n\n<details>\n<summary>With environment variables</summary>\n\n```bash\nexport GIS_QUOTA_RPM_RETRY=true\n```\n\n</details>\n\n<details>\n<summary>As a npm module</summary>\n\n```javascript\nimport { index } from 'google-indexing-script'\nimport serviceAccount from './service_account.json'\n\nindex('seogets.com', {\n  client_email: serviceAccount.client_email,\n  private_key: serviceAccount.private_key\n  quota: {\n    rpmRetry: true\n  }\n})\n  .then(console.log)\n  .catch(console.error)\n```\n\n</details>\n\n## 📄 License\n\nMIT License\n\n## 💖 Sponsor\n\nThis project is sponsored by [SEO Gets](https://seogets.com)\n\n![](https://seogets.com/og.png)\n"
  },
  {
    "path": "package.json",
    "content": "{\n  \"name\": \"google-indexing-script\",\n  \"description\": \"Script to get your site indexed on Google in less than 48 hours\",\n  \"version\": \"0.4.0\",\n  \"main\": \"./dist/index.js\",\n  \"types\": \"./dist/index.d.ts\",\n  \"bin\": {\n    \"google-indexing-script\": \"./dist/bin.js\",\n    \"gis\": \"./dist/bin.js\"\n  },\n  \"keywords\": [\n    \"google\",\n    \"indexing\",\n    \"search-console\",\n    \"sitemap\",\n    \"seo\",\n    \"google-search\",\n    \"cli\",\n    \"typescript\"\n  ],\n  \"license\": \"MIT\",\n  \"scripts\": {\n    \"index\": \"ts-node ./src/cli.ts\",\n    \"build\": \"tsup\",\n    \"dev\": \"tsup --watch\",\n    \"changeset\": \"changeset\",\n    \"version\": \"changeset version\",\n    \"release\": \"changeset publish\"\n  },\n  \"dependencies\": {\n    \"commander\": \"^12.1.0\",\n    \"googleapis\": \"131.0.0\",\n    \"picocolors\": \"^1.0.1\",\n    \"sitemapper\": \"3.2.8\"\n  },\n  \"prettier\": {\n    \"printWidth\": 120\n  },\n  \"devDependencies\": {\n    \"@changesets/changelog-github\": \"^0.5.0\",\n    \"@changesets/cli\": \"^2.27.1\",\n    \"ts-node\": \"^10.9.2\",\n    \"tsup\": \"^8.0.2\",\n    \"typescript\": \"^5.3.3\"\n  }\n}\n"
  },
  {
    "path": "src/bin.ts",
    "content": "#!/usr/bin/env node\nrequire(\"./cli\");\n"
  },
  {
    "path": "src/cli.ts",
    "content": "import { index } from \".\";\nimport { Command } from \"commander\";\nimport packageJson from \"../package.json\";\nimport { green } from \"picocolors\";\n\nconst program = new Command(packageJson.name);\n\nprogram\n  .alias(\"gis\")\n  .version(packageJson.version, \"-v, --version\", \"Output the current version.\")\n  .description(packageJson.description)\n  .argument(\"[input]\")\n  .usage(`${green(\"[input]\")} [options]`)\n  .helpOption(\"-h, --help\", \"Output usage information.\")\n  .option(\"-c, --client-email <email>\", \"The client email for the Google service account.\")\n  .option(\"-k, --private-key <key>\", \"The private key for the Google service account.\")\n  .option(\"-p, --path <path>\", \"The path to the Google service account credentials file.\")\n  .option(\"-u, --urls <urls>\", \"A comma-separated list of URLs to index.\")\n  .option(\"--rpm-retry\", \"Retry when the rate limit is exceeded.\")\n  .action((input, options) => {\n    index(input, {\n      client_email: options.clientEmail,\n      private_key: options.privateKey,\n      path: options.path,\n      urls: options.urls ? options.urls.split(\",\") : undefined,\n      quota: {\n        rpmRetry: options.rpmRetry,\n      },\n    });\n  })\n  .parse(process.argv);\n"
  },
  {
    "path": "src/index.ts",
    "content": "import { getAccessToken } from \"./shared/auth\";\nimport {\n  convertToSiteUrl,\n  getPublishMetadata,\n  requestIndexing,\n  getEmojiForStatus,\n  getPageIndexingStatus,\n  convertToFilePath,\n  checkSiteUrl,\n  checkCustomUrls,\n} from \"./shared/gsc\";\nimport { getSitemapPages } from \"./shared/sitemap\";\nimport { Status } from \"./shared/types\";\nimport { batch } from \"./shared/utils\";\nimport { readFileSync, existsSync, mkdirSync, writeFileSync } from \"fs\";\nimport path from \"path\";\n\nconst CACHE_TIMEOUT = 1000 * 60 * 60 * 24 * 14; // 14 days\nexport const QUOTA = {\n  rpm: {\n    retries: 3,\n    waitingTime: 60000, // 1 minute\n  },\n};\n\nexport type IndexOptions = {\n  client_email?: string;\n  private_key?: string;\n  path?: string;\n  urls?: string[];\n  quota?: {\n    rpmRetry?: boolean; // read requests per minute: retry after waiting time\n  };\n};\n\n/**\n * Indexes the specified domain or site URL.\n * @param input - The domain or site URL to index.\n * @param options - (Optional) Additional options for indexing.\n */\nexport const index = async (input: string = process.argv[2], options: IndexOptions = {}) => {\n  if (!input) {\n    console.error(\"❌ Please provide a domain or site URL as the first argument.\");\n    console.error(\"\");\n    process.exit(1);\n  }\n\n  if (!options.client_email) {\n    options.client_email = process.env.GIS_CLIENT_EMAIL;\n  }\n  if (!options.private_key) {\n    options.private_key = process.env.GIS_PRIVATE_KEY;\n  }\n  if (!options.path) {\n    options.path = process.env.GIS_PATH;\n  }\n  if (!options.urls) {\n    options.urls = process.env.GIS_URLS ? process.env.GIS_URLS.split(\",\") : undefined;\n  }\n  if (!options.quota) {\n    options.quota = {\n      rpmRetry: process.env.GIS_QUOTA_RPM_RETRY === \"true\",\n    };\n  }\n\n  const accessToken = await getAccessToken(options.client_email, options.private_key, options.path);\n  let siteUrl = convertToSiteUrl(input);\n  console.log(`🔎 Processing site: ${siteUrl}`);\n  const cachePath = path.join(\".cache\", `${convertToFilePath(siteUrl)}.json`);\n\n  if (!accessToken) {\n    console.error(\"❌ Failed to get access token, check your service account credentials.\");\n    console.error(\"\");\n    process.exit(1);\n  }\n\n  siteUrl = await checkSiteUrl(accessToken, siteUrl);\n\n  let pages = options.urls || [];\n  if (pages.length === 0) {\n    console.log(`🔎 Fetching sitemaps and pages...`);\n    const [sitemaps, pagesFromSitemaps] = await getSitemapPages(accessToken, siteUrl);\n\n    if (sitemaps.length === 0) {\n      console.error(\"❌ No sitemaps found, add them to Google Search Console and try again.\");\n      console.error(\"\");\n      process.exit(1);\n    }\n\n    pages = pagesFromSitemaps;\n\n    console.log(`👉 Found ${pages.length} URLs in ${sitemaps.length} sitemap`);\n  } else {\n    pages = checkCustomUrls(siteUrl, pages);\n    console.log(`👉 Found ${pages.length} URLs in the provided list`);\n  }\n\n  const statusPerUrl: Record<string, { status: Status; lastCheckedAt: string }> = existsSync(cachePath)\n    ? JSON.parse(readFileSync(cachePath, \"utf8\"))\n    : {};\n  const pagesPerStatus: Record<Status, string[]> = {\n    [Status.SubmittedAndIndexed]: [],\n    [Status.DuplicateWithoutUserSelectedCanonical]: [],\n    [Status.CrawledCurrentlyNotIndexed]: [],\n    [Status.DiscoveredCurrentlyNotIndexed]: [],\n    [Status.PageWithRedirect]: [],\n    [Status.URLIsUnknownToGoogle]: [],\n    [Status.RateLimited]: [],\n    [Status.Forbidden]: [],\n    [Status.Error]: [],\n  };\n\n  const indexableStatuses = [\n    Status.DiscoveredCurrentlyNotIndexed,\n    Status.CrawledCurrentlyNotIndexed,\n    Status.URLIsUnknownToGoogle,\n    Status.Forbidden,\n    Status.Error,\n    Status.RateLimited,\n  ];\n\n  const shouldRecheck = (status: Status, lastCheckedAt: string) => {\n    const shouldIndexIt = indexableStatuses.includes(status);\n    const isOld = new Date(lastCheckedAt) < new Date(Date.now() - CACHE_TIMEOUT);\n    return shouldIndexIt && isOld;\n  };\n\n  await batch(\n    async (url) => {\n      let result = statusPerUrl[url];\n      if (!result || shouldRecheck(result.status, result.lastCheckedAt)) {\n        const status = await getPageIndexingStatus(accessToken, siteUrl, url);\n        result = { status, lastCheckedAt: new Date().toISOString() };\n        statusPerUrl[url] = result;\n      }\n\n      pagesPerStatus[result.status] = pagesPerStatus[result.status] ? [...pagesPerStatus[result.status], url] : [url];\n    },\n    pages,\n    50,\n    (batchIndex, batchCount) => {\n      console.log(`📦 Batch ${batchIndex + 1} of ${batchCount} complete`);\n    }\n  );\n\n  console.log(``);\n  console.log(`👍 Done, here's the status of all ${pages.length} pages:`);\n  mkdirSync(\".cache\", { recursive: true });\n  writeFileSync(cachePath, JSON.stringify(statusPerUrl, null, 2));\n\n  for (const status of Object.keys(pagesPerStatus)) {\n    const pages = pagesPerStatus[status as Status];\n    if (pages.length === 0) continue;\n    console.log(`• ${getEmojiForStatus(status as Status)} ${status}: ${pages.length} pages`);\n  }\n  console.log(\"\");\n\n  const indexablePages = Object.entries(pagesPerStatus).flatMap(([status, pages]) =>\n    indexableStatuses.includes(status as Status) ? pages : []\n  );\n\n  if (indexablePages.length === 0) {\n    console.log(`✨ There are no pages that can be indexed. Everything is already indexed!`);\n  } else {\n    console.log(`✨ Found ${indexablePages.length} pages that can be indexed.`);\n    indexablePages.forEach((url) => console.log(`• ${url}`));\n  }\n  console.log(``);\n\n  for (const url of indexablePages) {\n    console.log(`📄 Processing url: ${url}`);\n    const status = await getPublishMetadata(accessToken, url, {\n      retriesOnRateLimit: options.quota.rpmRetry ? QUOTA.rpm.retries : 0,\n    });\n    if (status === 404) {\n      await requestIndexing(accessToken, url);\n      console.log(\"🚀 Indexing requested successfully. It may take a few days for Google to process it.\");\n    } else if (status < 400) {\n      console.log(`🕛 Indexing already requested previously. It may take a few days for Google to process it.`);\n    }\n    console.log(``);\n  }\n\n  console.log(`👍 All done!`);\n  console.log(`💖 Brought to you by https://seogets.com - SEO Analytics.`);\n  console.log(``);\n};\n\nexport * from \"./shared\";\n"
  },
  {
    "path": "src/shared/auth.ts",
    "content": "import { google } from \"googleapis\";\nimport fs from \"fs\";\nimport path from \"path\";\nimport os from \"os\";\n\n/**\n * Retrieves an access token for Google APIs using service account credentials.\n * @param client_email - The client email of the service account.\n * @param private_key - The private key of the service account.\n * @param customPath - (Optional) Custom path to the service account JSON file.\n * @returns The access token.\n */\nexport async function getAccessToken(client_email?: string, private_key?: string, customPath?: string) {\n  if (!client_email && !private_key) {\n    const filePath = \"service_account.json\";\n    const filePathFromHome = path.join(os.homedir(), \".gis\", \"service_account.json\");\n    const isFile = fs.existsSync(filePath);\n    const isFileFromHome = fs.existsSync(filePathFromHome);\n    const isCustomFile = !!customPath && fs.existsSync(customPath);\n\n    if (!isFile && !isFileFromHome && !isCustomFile) {\n      console.error(`❌ ${filePath} not found, please follow the instructions in README.md`);\n      console.error(\"\");\n      process.exit(1);\n    }\n\n    const key = JSON.parse(\n      fs.readFileSync(!!customPath && isCustomFile ? customPath : isFile ? filePath : filePathFromHome, \"utf8\")\n    );\n    client_email = key.client_email;\n    private_key = key.private_key;\n  } else {\n    if (!client_email) {\n      console.error(\"❌ Missing client_email in service account credentials.\");\n      console.error(\"\");\n      process.exit(1);\n    }\n\n    if (!private_key) {\n      console.error(\"❌ Missing private_key in service account credentials.\");\n      console.error(\"\");\n      process.exit(1);\n    }\n  }\n\n  const jwtClient = new google.auth.JWT(\n    client_email,\n    undefined,\n    private_key,\n    [\"https://www.googleapis.com/auth/webmasters.readonly\", \"https://www.googleapis.com/auth/indexing\"],\n    undefined\n  );\n\n  const tokens = await jwtClient.authorize();\n  return tokens.access_token;\n}\n"
  },
  {
    "path": "src/shared/gsc.ts",
    "content": "import { webmasters_v3 } from \"googleapis\";\nimport { QUOTA } from \"..\";\nimport { Status } from \"./types\";\nimport { fetchRetry } from \"./utils\";\n\n/**\n * Converts a given input string to a valid Google Search Console site URL format.\n * @param input - The input string to be converted.\n * @returns The converted site URL (domain.com or https://domain.com/)\n */\nexport function convertToSiteUrl(input: string) {\n  if (input.startsWith(\"http://\") || input.startsWith(\"https://\")) {\n    return input.endsWith(\"/\") ? input : `${input}/`;\n  }\n  return `sc-domain:${input}`;\n}\n\n/**\n * Converts a given file path to a formatted version suitable for use as a file name.\n * @param path - The url to be converted as a file name\n * @returns The converted file path\n */\nexport function convertToFilePath(path: string) {\n  return path.replace(\"http://\", \"http_\").replace(\"https://\", \"https_\").replaceAll(\"/\", \"_\");\n}\n\n/**\n * Converts an HTTP URL to a sc-domain URL format.\n * @param httpUrl The HTTP URL to be converted.\n * @returns The sc-domain formatted URL.\n */\nexport function convertToSCDomain(httpUrl: string) {\n  return `sc-domain:${httpUrl.replace(\"http://\", \"\").replace(\"https://\", \"\").replace(\"/\", \"\")}`;\n}\n\n/**\n * Converts a domain to an HTTP URL.\n * @param domain The domain to be converted.\n * @returns The HTTP URL.\n */\nexport function convertToHTTP(domain: string) {\n  return `http://${domain}/`;\n}\n\n/**\n * Converts a domain to an HTTPS URL.\n * @param domain The domain to be converted.\n * @returns The HTTPS URL.\n */\nexport function convertToHTTPS(domain: string) {\n  return `https://${domain}/`;\n}\n\n/**\n * Retrieves a list of sites associated with the specified service account from the Google Webmasters API.\n * @param accessToken - The access token for authentication.\n * @returns An array containing the site URLs associated with the service account.\n */\nexport async function getSites(accessToken: string) {\n  const sitesResponse = await fetchRetry(\"https://www.googleapis.com/webmasters/v3/sites\", {\n    headers: {\n      \"Content-Type\": \"application/json\",\n      Authorization: `Bearer ${accessToken}`,\n    },\n  });\n\n  if (sitesResponse.status === 403) {\n    console.error(\"🔐 This service account doesn't have access to any sites.\");\n    return [];\n  }\n\n  const sitesBody: webmasters_v3.Schema$SitesListResponse = await sitesResponse.json();\n\n  if (!sitesBody.siteEntry) {\n    console.error(\"❌ No sites found, add them to Google Search Console and try again.\");\n    return [];\n  }\n\n  return sitesBody.siteEntry.map((x) => x.siteUrl);\n}\n\n/**\n * Checks if the site URL is valid and accessible by the service account.\n * @param accessToken - The access token for authentication.\n * @param siteUrl - The URL of the site to check.\n * @returns The corrected URL if found, otherwise the original site URL.\n */\nexport async function checkSiteUrl(accessToken: string, siteUrl: string) {\n  const sites = await getSites(accessToken);\n  let formattedUrls: string[] = [];\n\n  // Convert the site URL into all possible formats\n  if (siteUrl.startsWith(\"https://\")) {\n    formattedUrls.push(siteUrl);\n    formattedUrls.push(convertToHTTP(siteUrl.replace(\"https://\", \"\")));\n    formattedUrls.push(convertToSCDomain(siteUrl));\n  } else if (siteUrl.startsWith(\"http://\")) {\n    formattedUrls.push(siteUrl);\n    formattedUrls.push(convertToHTTPS(siteUrl.replace(\"http://\", \"\")));\n    formattedUrls.push(convertToSCDomain(siteUrl));\n  } else if (siteUrl.startsWith(\"sc-domain:\")) {\n    formattedUrls.push(siteUrl);\n    formattedUrls.push(convertToHTTP(siteUrl.replace(\"sc-domain:\", \"\")));\n    formattedUrls.push(convertToHTTPS(siteUrl.replace(\"sc-domain:\", \"\")));\n  } else {\n    console.error(\"❌ Unknown site URL format.\");\n    console.error(\"\");\n    process.exit(1);\n  }\n\n  // Check if any of the formatted URLs are accessible\n  for (const formattedUrl of formattedUrls) {\n    if (sites.includes(formattedUrl)) {\n      return formattedUrl;\n    }\n  }\n\n  // If none of the formatted URLs are accessible\n  console.error(\"❌ This service account doesn't have access to this site.\");\n  console.error(\"\");\n  process.exit(1);\n}\n\n/**\n * Checks if the given URLs are valid.\n * @param siteUrl - The URL of the site.\n * @param urls - The URLs to check.\n * @returns An array containing the corrected URLs if found, otherwise the original URLs\n */\nexport function checkCustomUrls(siteUrl: string, urls: string[]) {\n  const protocol = siteUrl.startsWith(\"http://\") ? \"http://\" : \"https://\";\n  const domain = siteUrl.replace(\"https://\", \"\").replace(\"http://\", \"\").replace(\"sc-domain:\", \"\");\n  const formattedUrls: string[] = urls.map((url) => {\n    url = url.trim();\n    if (url.startsWith(\"/\")) {\n      // the url is a relative path (e.g. /about)\n      return `${protocol}${domain}${url}`;\n    } else if (url.startsWith(\"http://\") || url.startsWith(\"https://\")) {\n      // the url is already a full url (e.g. https://domain.com/about)\n      return url;\n    } else if (url.startsWith(domain)) {\n      // the url is a full url without the protocol (e.g. domain.com/about)\n      return `${protocol}${url}`;\n    } else {\n      // the url is a relative path without the leading slash (e.g. about)\n      return `${protocol}${domain}/${url}`;\n    }\n  });\n\n  return formattedUrls;\n}\n\n/**\n * Retrieves the indexing status of a page.\n * @param accessToken - The access token for authentication.\n * @param siteUrl - The URL of the site.\n * @param inspectionUrl - The URL of the page to inspect.\n * @returns A promise resolving to the status of indexing.\n */\nexport async function getPageIndexingStatus(\n  accessToken: string,\n  siteUrl: string,\n  inspectionUrl: string\n): Promise<Status> {\n  try {\n    const response = await fetchRetry(`https://searchconsole.googleapis.com/v1/urlInspection/index:inspect`, {\n      method: \"POST\",\n      headers: {\n        \"Content-Type\": \"application/json\",\n        Authorization: `Bearer ${accessToken}`,\n      },\n      body: JSON.stringify({\n        inspectionUrl,\n        siteUrl,\n      }),\n    });\n\n    if (response.status === 403) {\n      console.error(`🔐 This service account doesn't have access to this site.`);\n      console.error(await response.text());\n\n      return Status.Forbidden;\n    }\n\n    if (response.status >= 300) {\n      if (response.status === 429) {\n        return Status.RateLimited;\n      } else {\n        console.error(`❌ Failed to get indexing status.`);\n        console.error(`Response was: ${response.status}`);\n        console.error(await response.text());\n\n        return Status.Error;\n      }\n    }\n\n    const body = await response.json();\n    return body.inspectionResult.indexStatusResult.coverageState;\n  } catch (error) {\n    console.error(`❌ Failed to get indexing status.`);\n    console.error(`Error was: ${error}`);\n    throw error;\n  }\n}\n\n/**\n * Retrieves an emoji representation corresponding to the given status.\n * @param status - The status for which to retrieve the emoji.\n * @returns The emoji representing the status.\n */\nexport function getEmojiForStatus(status: Status) {\n  switch (status) {\n    case Status.SubmittedAndIndexed:\n      return \"✅\";\n    case Status.DuplicateWithoutUserSelectedCanonical:\n      return \"😵\";\n    case Status.CrawledCurrentlyNotIndexed:\n    case Status.DiscoveredCurrentlyNotIndexed:\n      return \"👀\";\n    case Status.PageWithRedirect:\n      return \"🔀\";\n    case Status.URLIsUnknownToGoogle:\n      return \"❓\";\n    case Status.RateLimited:\n      return \"🚦\";\n    default:\n      return \"❌\";\n  }\n}\n\n/**\n * Retrieves metadata for publishing from the given URL.\n * @param accessToken - The access token for authentication.\n * @param url - The URL for which to retrieve metadata.\n * @param options - The options for the request.\n * @returns The status of the request.\n */\nexport async function getPublishMetadata(accessToken: string, url: string, options?: { retriesOnRateLimit: number }) {\n  const response = await fetchRetry(\n    `https://indexing.googleapis.com/v3/urlNotifications/metadata?url=${encodeURIComponent(url)}`,\n    {\n      method: \"GET\",\n      headers: {\n        \"Content-Type\": \"application/json\",\n        Authorization: `Bearer ${accessToken}`,\n      },\n    }\n  );\n\n  if (response.status === 403) {\n    console.error(`🔐 This service account doesn't have access to this site.`);\n    console.error(`Response was: ${response.status}`);\n    console.error(await response.text());\n  }\n\n  if (response.status === 429) {\n    if (options?.retriesOnRateLimit && options?.retriesOnRateLimit > 0) {\n      const RPM_WATING_TIME = (QUOTA.rpm.retries - options.retriesOnRateLimit + 1) * QUOTA.rpm.waitingTime; // increase waiting time for each retry\n      console.log(\n        `🚦 Rate limit exceeded for read requests. Retries left: ${options.retriesOnRateLimit}. Waiting for ${\n          RPM_WATING_TIME / 1000\n        }sec.`\n      );\n      await new Promise((resolve) => setTimeout(resolve, RPM_WATING_TIME));\n      await getPublishMetadata(accessToken, url, { retriesOnRateLimit: options.retriesOnRateLimit - 1 });\n    } else {\n      console.error(\"🚦 Rate limit exceeded, try again later.\");\n      console.error(\"\");\n      console.error(\"   Quota: https://developers.google.com/search/apis/indexing-api/v3/quota-pricing#quota\");\n      console.error(\"   Usage: https://console.cloud.google.com/apis/enabled\");\n      console.error(\"\");\n      process.exit(1);\n    }\n  }\n\n  if (response.status >= 500) {\n    console.error(`❌ Failed to get publish metadata.`);\n    console.error(`Response was: ${response.status}`);\n    console.error(await response.text());\n  }\n\n  return response.status;\n}\n\n/**\n * Requests indexing for the given URL.\n * @param accessToken - The access token for authentication.\n * @param url - The URL to be indexed.\n */\nexport async function requestIndexing(accessToken: string, url: string) {\n  const response = await fetchRetry(\"https://indexing.googleapis.com/v3/urlNotifications:publish\", {\n    method: \"POST\",\n    headers: {\n      \"Content-Type\": \"application/json\",\n      Authorization: `Bearer ${accessToken}`,\n    },\n    body: JSON.stringify({\n      url: url,\n      type: \"URL_UPDATED\",\n    }),\n  });\n\n  if (response.status === 403) {\n    console.error(`🔐 This service account doesn't have access to this site.`);\n    console.error(`Response was: ${response.status}`);\n  }\n\n  if (response.status >= 300) {\n    if (response.status === 429) {\n      console.error(\"🚦 Rate limit exceeded, try again later.\");\n      console.error(\"\");\n      console.error(\"   Quota: https://developers.google.com/search/apis/indexing-api/v3/quota-pricing#quota\");\n      console.error(\"   Usage: https://console.cloud.google.com/apis/enabled\");\n      console.error(\"\");\n      process.exit(1);\n    } else {\n      console.error(`❌ Failed to request indexing.`);\n      console.error(`Response was: ${response.status}`);\n      console.error(await response.text());\n    }\n  }\n}\n"
  },
  {
    "path": "src/shared/index.ts",
    "content": "export * from \"./auth\";\nexport * from \"./gsc\";\nexport * from \"./sitemap\";\nexport * from \"./types\";\nexport * from \"./utils\";\n"
  },
  {
    "path": "src/shared/sitemap.ts",
    "content": "import Sitemapper from \"sitemapper\";\nimport { fetchRetry } from \"./utils\";\nimport { webmasters_v3 } from \"googleapis\";\n\n/**\n * Retrieves a list of sitemaps associated with the specified site URL from the Google Webmasters API.\n * @param accessToken The access token for authentication.\n * @param siteUrl The URL of the site for which to retrieve the list of sitemaps.\n * @returns An array containing the paths of the sitemaps associated with the site URL.\n */\nasync function getSitemapsList(accessToken: string, siteUrl: string) {\n  const url = `https://www.googleapis.com/webmasters/v3/sites/${encodeURIComponent(siteUrl)}/sitemaps`;\n\n  const response = await fetchRetry(url, {\n    headers: {\n      \"Content-Type\": \"application/json\",\n      Authorization: `Bearer ${accessToken}`,\n    },\n  });\n\n  if (response.status === 403) {\n    console.error(`🔐 This service account doesn't have access to this site.`);\n    return [];\n  }\n\n  if (response.status >= 300) {\n    console.error(`❌ Failed to get list of sitemaps.`);\n    console.error(`Response was: ${response.status}`);\n    console.error(await response.text());\n    return [];\n  }\n\n  const body: webmasters_v3.Schema$SitemapsListResponse = await response.json();\n\n  if (!body.sitemap) {\n    console.error(\"❌ No sitemaps found, add them to Google Search Console and try again.\");\n    return [];\n  }\n\n  return body.sitemap.filter((x) => x.path !== undefined && x.path !== null).map((x) => x.path as string);\n}\n\n/**\n * Retrieves a list of pages from all sitemaps associated with the specified site URL.\n * @param accessToken The access token for authentication.\n * @param siteUrl The URL of the site for which to retrieve the sitemap pages.\n * @returns An array containing the list of sitemaps and an array of unique page URLs extracted from those sitemaps.\n */\nexport async function getSitemapPages(accessToken: string, siteUrl: string) {\n  const sitemaps = await getSitemapsList(accessToken, siteUrl);\n\n  let pages: string[] = [];\n  for (const url of sitemaps) {\n    const Google = new Sitemapper({\n      url,\n    });\n\n    const { sites } = await Google.fetch();\n    pages = [...pages, ...sites];\n  }\n\n  return [sitemaps, [...new Set(pages)]];\n}\n"
  },
  {
    "path": "src/shared/types.ts",
    "content": "/**\n * Enum representing indexing status of a URL\n */\nexport enum Status {\n  SubmittedAndIndexed = \"Submitted and indexed\",\n  DuplicateWithoutUserSelectedCanonical = \"Duplicate without user-selected canonical\",\n  CrawledCurrentlyNotIndexed = \"Crawled - currently not indexed\",\n  DiscoveredCurrentlyNotIndexed = \"Discovered - currently not indexed\",\n  PageWithRedirect = \"Page with redirect\",\n  URLIsUnknownToGoogle = \"URL is unknown to Google\",\n  RateLimited = \"RateLimited\",\n  Forbidden = \"Forbidden\",\n  Error = \"Error\",\n}\n"
  },
  {
    "path": "src/shared/utils.ts",
    "content": "/**\n * Creates an array of chunks from the given array with a specified size.\n * @param arr The array to be chunked.\n * @param size The size of each chunk.\n * @returns An array of chunks.\n */\nconst createChunks = (arr: any[], size: number) =>\n  Array.from({ length: Math.ceil(arr.length / size) }, (_, i) => arr.slice(i * size, i * size + size));\n\n/**\n * Executes tasks on items in batches and invokes a callback upon completion of each batch.\n * @param task The task function to be executed on each item.\n * @param items The array of items on which the task is to be executed.\n * @param batchSize The size of each batch.\n * @param onBatchComplete The callback function invoked upon completion of each batch.\n */\nexport async function batch(\n  task: (url: string) => void,\n  items: string[],\n  batchSize: number,\n  onBatchComplete: (batchIndex: number, batchCount: number) => void\n) {\n  const chunks = createChunks(items, batchSize);\n  for (let i = 0; i < chunks.length; i++) {\n    await Promise.all(chunks[i].map(task));\n    onBatchComplete(i, chunks.length);\n  }\n}\n\n/**\n * Fetches a resource from a URL with retry logic.\n * @param url The URL of the resource to fetch.\n * @param options The options for the fetch request.\n * @param retries The number of retry attempts (default is 5).\n * @returns A Promise resolving to the fetched response.\n * @throws Error when retries are exhausted or server error occurs.\n */\nexport async function fetchRetry(url: string, options: RequestInit, retries: number = 5) {\n  try {\n    const response = await fetch(url, options);\n    if (response.status >= 500) {\n      const body = await response.text();\n      throw new Error(`Server error code ${response.status}\\n${body}`);\n    }\n    return response;\n  } catch (err) {\n    if (retries <= 0) {\n      throw err;\n    }\n    return fetchRetry(url, options, retries - 1);\n  }\n}\n"
  },
  {
    "path": "tsconfig.json",
    "content": "{\n  \"compilerOptions\": {\n    \"target\": \"esnext\",\n    \"module\": \"commonjs\",\n    \"lib\": [\"dom\", \"es6\", \"es2021\", \"esnext.asynciterable\"],\n    \"skipLibCheck\": true,\n    \"sourceMap\": true,\n    \"outDir\": \"./dist\",\n    \"moduleResolution\": \"node\",\n    \"removeComments\": false,\n    \"noImplicitAny\": false,\n    \"strictNullChecks\": true,\n    \"strictFunctionTypes\": true,\n    \"noImplicitThis\": true,\n    \"noUnusedLocals\": true,\n    \"noUnusedParameters\": true,\n    \"noImplicitReturns\": true,\n    \"noFallthroughCasesInSwitch\": true,\n    \"allowSyntheticDefaultImports\": true,\n    \"esModuleInterop\": true,\n    \"emitDecoratorMetadata\": true,\n    \"experimentalDecorators\": true,\n    \"resolveJsonModule\": true\n  },\n  \"include\": [\"**/*.ts\"],\n  \"exclude\": [\"node_modules/\", \"dist/\"]\n}\n"
  },
  {
    "path": "tsup.config.ts",
    "content": "import { defineConfig, Options } from \"tsup\";\n\nconst config: Options = {\n  entry: [\"src/**/*.ts\"],\n  splitting: true,\n  sourcemap: true,\n  clean: true,\n  platform: \"node\",\n  dts: true,\n  minify: true,\n};\n\nexport default defineConfig(config);\n"
  }
]