Repository: felixrieseberg/slack-archive
Branch: main
Commit: 4ce7b36fe08f
Files: 30
Total size: 87.7 KB

Directory structure:
gitextract_oionf7w6/

├── .gitignore
├── .node-version
├── .npmignore
├── README.md
├── bin/
│   └── slack-archive.js
├── package.json
├── src/
│   ├── ambient.d.ts
│   ├── archive-data.ts
│   ├── backup.ts
│   ├── channels.ts
│   ├── cli.ts
│   ├── config.ts
│   ├── create-html.tsx
│   ├── data-load.ts
│   ├── data-write.ts
│   ├── download-files.ts
│   ├── emoji.ts
│   ├── interfaces.ts
│   ├── messages.ts
│   ├── reactions.ts
│   ├── retry.ts
│   ├── search.ts
│   ├── threads.ts
│   ├── timestamp.ts
│   ├── users.ts
│   └── web-client.ts
├── static/
│   ├── scroll.js
│   ├── search.html
│   └── style.css
└── tsconfig.json

================================================
FILE CONTENTS
================================================

================================================
FILE: .gitignore
================================================
node_modules
out
slack-archive
.DS_Store
*.log
.token
lib

================================================
FILE: .node-version
================================================
16.4.0


================================================
FILE: .npmignore
================================================
node_modules
out
slack-archive
.DS_Store
*.log
.token
src

================================================
FILE: README.md
================================================
# Export your Slack workspace as static HTML

Alright, so you want to export all your messages on Slack. You want them in a format that you
can still enjoy in 20 years. This tool will help you do that.

 * **Completely static**: The generated files are pure HTML and will still work in 50 years.
 * **Everything you care about**: This tool downloads messages, files, and avatars.
 * **Nothing you do not care about**: Choose exactly which channels and DMs to download.
 * **All types of conversations**: We'll fetch public channels, private channels, DMs, and multi-person DMs.
 * **Incremental backups**: If you already have local data, we'll extend it - no need to download existing stuff again.
 * **JSON included**: All data is also stored as JSON, so you can consume it with other tools later.
 * **No cloud, free**: Do all of this for free, without giving anyone your information.
 * **Basic search**: Offers basic search functionality.

<img width="1151" alt="Screen Shot 2021-09-09 at 6 43 55 PM" src="https://user-images.githubusercontent.com/1426799/132776566-0f75a1b4-4b9a-4b53-8a39-e44e8a747a68.png">

## Using it

1. Do you already have a user token for your workspace? If not, read on below on how to get a token.
2. Make sure you have [`node` and `npm`](https://nodejs.org/en/) installed, ideally something newer than Node v14.
3. Run `slack-archive`, which will interactively guide you through the options.

```sh
npx slack-archive
```

### Parameters

```
--automatic:                Don't prompt and automatically fetch all messages from all channels.
--use-previous-channel-config: Fetch messages from channels selected in previous run instead of prompting.
--channel-types             Comma-separated list of channel types to fetch messages from.
                            (public_channel, private_channel, mpim, im)
--exclude-channels          Comma-separated list of channels to exclude, in automatic mode
--no-backup:                Don't create backups. Not recommended.
--no-search:                Don't create a search file, saving disk space.
--no-file-download:         Don't download files.
--no-slack-connect:         Don't connect to Slack, just generate HTML from local data.
--force-html-generation:    Force regeneration of HTML files. Useful after slack-archive upgrades.
```

## Getting a token

In order to download messages from private channels and direct messages, we will need a "user
token". Slack uses the token to identify what permissions it'll give this app. We used to be able
to just copy a token out of your Slack app, but now, we'll need to create a custom app and jump
through a few hoops.

This will be mostly painless, I promise.

### 1) Make a custom app

Head over to https://api.slack.com/apps and `Create New App`. Select `From scratch`.
Give it a name and choose the workspace you'd like to export.

Then, from the `Features` menu on the left, select `OAuth & Permission`. 

As a redirect URL, enter something random that doesn't actually exist, or a domain you control. For instace:

```
https://notarealurl.com/
```

(Note that redirects will take a _very_ long time if using a domain that doesn't actually exist)

Then, add the following `User Token Scopes`:

 * channels:history
 * channels:read
 * files:read
 * groups:history
 * groups:read
 * im:history
 * im:read
 * mpim:history
 * mpim:read
 * remote_files:read
 * users:read

Finally, head back to `Basic Information` and make a note of your app's `client
id` and `client secret`. We'll need both later.

### 2) Authorize

Make sure you have your Slack workspace `URL` (aka team name) and your app's `client id`.
Then, in a browser, open this URL - replacing `{your-team-name}` and `{your-client-id}`
with your values.

```
https://{your-team-name}.slack.com/oauth/authorize?client_id={your-client-id}&scope=client
```

Confirm everything until Slack sends you to the mentioned non-existent URL. Look at your
browser's address bar - it should contain an URL that looks like this:

```
https://notarealurl.com/?code={code}&state=
```

Copy everything between `?code=` and `&state`. This is your `code`. We'll need it in the
next step.

Next, we'll exchange your code for a token. To do so, we'll also need your `client secret` 
from the first step when we created your app. In a browser, open this URL - replacing 
`{your-team-name}`, `{your-client-id}`, `{your-code}` and `{your-client-secret}` with 
your values.

```
https://{your-team-name}.slack.com/api/oauth.access?client_id={your-client-id}&client_secret={your-client-secret}&code={your-code}
```

Your browser should now be returning some JSON including a token. Make a note of it - that's what we'll use. Paste it in the command line, OR create a file called `.token` in the slack-archive directory (created when the command is first run) and paste it in there.


================================================
FILE: bin/slack-archive.js
================================================
#!/usr/bin/env node

import('../lib/cli.js')


================================================
FILE: package.json
================================================
{
  "name": "slack-archive",
  "version": "1.6.1",
  "description": "Create static HTML archives for your Slack workspaces",
  "scripts": {
    "prettier": "npx prettier --write src/*",
    "cli": "ts-node src/cli.ts",
    "html": "ts-node src/create-html.tsx",
    "test": "echo \"Error: no test specified\" && exit 1",
    "compile": "tsc",
    "watch": "tsc -w",
    "prepublishOnly": "npm run compile"
  },
  "bin": {
    "slack-archive": "./bin/slack-archive.js"
  },
  "type": "module",
  "keywords": [
    "slack",
    "export",
    "download"
  ],
  "repository": {
    "type": "git",
    "url": "https://github.com/felixrieseberg/slack-archive.git"
  },
  "author": "Felix Rieseberg <felix@felixrieseberg.com>",
  "license": "MIT",
  "dependencies": {
    "@slack/web-api": "^6.7.2",
    "date-fns": "^2.28.0",
    "emoji-datasource": "^14.0.0",
    "es-main": "^1.0.2",
    "fs-extra": "^10.1.0",
    "inquirer": "^8.2.0",
    "lodash-es": "^4.17.21",
    "node-fetch": "^2.6.7",
    "ora": "^6.1.0",
    "react": "^17.0.2",
    "react-dom": "^17.0.2",
    "rimraf": "^5.0.5",
    "slack-markdown": "^0.2.0",
    "trash": "^8.1.0"
  },
  "devDependencies": {
    "@types/date-fns": "^2.6.0",
    "@types/fs-extra": "^9.0.13",
    "@types/inquirer": "^8.1.3",
    "@types/lodash-es": "^4.17.5",
    "@types/node": "^17.0.5",
    "@types/node-fetch": "^2.5.12",
    "@types/react": "^17.0.38",
    "@types/react-dom": "^17.0.11",
    "ts-node": "^10.8.1",
    "tslib": "^2.4.0",
    "typescript": "^4.7.4"
  },
  "ts-node": {
    "files": true
  }
}


================================================
FILE: src/ambient.d.ts
================================================
declare module "slack-markdown";
declare module "es-main";
declare module "emoji-datasource";


================================================
FILE: src/archive-data.ts
================================================
import fs from "fs-extra";

import { SLACK_ARCHIVE_DATA_PATH } from "./config.js";
import { readJSON } from "./data-load.js";
import { write } from "./data-write.js";
import { SlackArchiveData, User } from "./interfaces.js";

export async function getSlackArchiveData(): Promise<SlackArchiveData> {
  const returnIfEmpty: SlackArchiveData = { channels: {} };

  if (!fs.existsSync(SLACK_ARCHIVE_DATA_PATH)) {
    return returnIfEmpty;
  }

  const result = await readJSON<SlackArchiveData>(SLACK_ARCHIVE_DATA_PATH);
  const merged = { channels: result.channels || {}, auth: result.auth };

  return merged;
}

export async function setSlackArchiveData(
  newData: SlackArchiveData
): Promise<void> {
  const oldData = await getSlackArchiveData();
  const dataToWrite = {
    channels: { ...oldData.channels, ...newData.channels },
    auth: newData.auth,
  };

  return write(
    SLACK_ARCHIVE_DATA_PATH,
    JSON.stringify(dataToWrite, undefined, 2)
  );
}


================================================
FILE: src/backup.ts
================================================
import fs from "fs-extra";
import inquirer from "inquirer";
import path from "path";
import trash from "trash";
import { rimraf } from "rimraf";

import { AUTOMATIC_MODE, DATA_DIR, NO_BACKUP, OUT_DIR } from "./config.js";

const { prompt } = inquirer;

let backupDir = `${DATA_DIR}_backup_${Date.now()}`;

export async function createBackup() {
  if (NO_BACKUP || !fs.existsSync(DATA_DIR)) {
    return;
  }

  const hasFiles = fs.readdirSync(DATA_DIR);

  if (hasFiles.length === 0) {
    return;
  }

  console.log(`Existing data directory found. Creating backup: ${backupDir}`);

  await fs.copy(DATA_DIR, backupDir);

  console.log(`Backup created.\n`);
}

export async function deleteBackup() {
  if (!fs.existsSync(backupDir)) {
    return;
  }

  console.log(
    `Cleaning up backup: If anything went wrong, you'll find it in your system's trash.`
  );

  try {
    // NB: trash doesn't work on many Linux distros
    await trash(backupDir);
    return;
  } catch (error) {
    console.log('Moving backup to trash failed.');
  }

  if (!process.env['TRASH_HARDER']) {
    console.log(`Set TRASH_HARDER=1 to delete files permanently.`);
    return;
  }

  try {
    await rimraf(backupDir);
  } catch (error) {
    console.log(`Deleting backup permanently failed. Aborting here.`);
  }
}

export async function deleteOlderBackups() {
  try {
    const oldBackupNames: Array<string> = [];
    const oldBackupPaths: Array<string> = [];

    for (const entry of fs.readdirSync(OUT_DIR)) {
      const isBackup = entry.startsWith("data_backup_");
      if (!isBackup) continue;

      const dir = path.join(OUT_DIR, entry);
      const { isDirectory } = fs.statSync(dir);
      if (!isDirectory) continue;

      oldBackupPaths.push(dir);
      oldBackupNames.push(entry);
    }

    if (oldBackupPaths.length === 0) return;

    if (AUTOMATIC_MODE) {
      console.log(
        `Found existing older backups, but in automatic mode: Proceeding without deleting them.`
      );
      return;
    }

    const { del } = await prompt([
      {
        type: "confirm",
        default: true,
        name: "del",
        message: `We've found existing backups (${oldBackupNames.join(
          ", "
        )}). Do you want to delete them?`,
      },
    ]);

    if (del) {
      oldBackupPaths.forEach((v) => fs.removeSync(v));
    }
  } catch (error) {
    // noop
  }
}


================================================
FILE: src/channels.ts
================================================
import {
  ConversationsListArguments,
  ConversationsListResponse,
} from "@slack/web-api";
import ora from "ora";
import { NO_SLACK_CONNECT } from "./config.js";

import { Channel, Users } from "./interfaces.js";
import { downloadUser, getName } from "./users.js";
import { getWebClient } from "./web-client.js";

export function getChannelName(channel: Channel) {
  return (
    channel.name || channel.id || channel.purpose?.value || "Unknown channel"
  );
}

export function isPublicChannel(channel: Channel) {
  return !channel.is_private && !channel.is_mpim && !channel.is_im;
}

export function isPrivateChannel(channel: Channel) {
  return channel.is_private && !channel.is_im && !channel.is_mpim;
}

export function isDmChannel(channel: Channel, users: Users) {
  return channel.is_im && channel.user && !users[channel.user]?.is_bot;
}

export function isBotChannel(channel: Channel, users: Users) {
  return channel.user && users[channel.user]?.is_bot;
}

function isChannels(input: any): input is ConversationsListResponse {
  return !!input.channels;
}

export async function downloadChannels(
  options: ConversationsListArguments,
  users: Users
): Promise<Array<Channel>> {
  const channels: Array<Channel> = [];

  if (NO_SLACK_CONNECT) {
    return channels;
  }

  const spinner = ora("Downloading channels").start();

  for await (const page of getWebClient().paginate(
    "conversations.list",
    options
  )) {
    if (isChannels(page)) {
      spinner.text = `Found ${page.channels?.length} channels (found so far: ${
        channels.length + (page.channels?.length || 0)
      })`;

      const pageChannels = (page.channels || []).filter((c) => !!c.id);

      for (const channel of pageChannels) {
        if (channel.is_im) {
          const user = await downloadUser(channel, users);
          channel.name =
            channel.name || `${getName(user?.id, users)} (${user?.name})`;
        }

        if (channel.is_mpim) {
          channel.name = channel.purpose?.value;
        }
      }

      channels.push(...pageChannels);
    }
  }

  spinner.succeed(`Found ${channels.length} channels`);

  return channels;
}


================================================
FILE: src/cli.ts
================================================
import { uniqBy } from "lodash-es";
import inquirer from "inquirer";
import fs from "fs-extra";
import { User } from "@slack/web-api/dist/response/UsersInfoResponse";
import { Channel } from "@slack/web-api/dist/response/ConversationsListResponse";
import ora from "ora";

import {
  CHANNELS_DATA_PATH,
  USERS_DATA_PATH,
  getChannelDataFilePath,
  OUT_DIR,
  config,
  TOKEN_FILE,
  AUTOMATIC_MODE,
  USE_PREVIOUS_CHANNEL_CONFIG,
  CHANNEL_TYPES,
  DATE_FILE,
  EMOJIS_DATA_PATH,
  NO_SLACK_CONNECT,
  EXCLUDE_CHANNELS,
} from "./config.js";
import { downloadExtras } from "./messages.js";
import { downloadMessages } from "./messages.js";
import { downloadFilesForChannel } from "./download-files.js";
import {
  createHtmlForChannels,
  getChannelsToCreateFilesFor,
} from "./create-html.js";
import { createBackup, deleteBackup, deleteOlderBackups } from "./backup.js";
import { isValid, parseISO } from "date-fns";
import { createSearch } from "./search.js";
import { write, writeAndMerge } from "./data-write.js";
import { messagesCache, getUsers, getChannels } from "./data-load.js";
import { getSlackArchiveData, setSlackArchiveData } from "./archive-data.js";
import { downloadEmojiList, downloadEmojis } from "./emoji.js";
import { downloadAvatars } from "./users.js";
import { downloadChannels } from "./channels.js";
import { authTest } from "./web-client.js";
import { SlackArchiveChannelData } from "./interfaces.js";

const { prompt } = inquirer;

async function selectMergeFiles(): Promise<boolean> {
  const defaultResponse = true;

  if (!fs.existsSync(CHANNELS_DATA_PATH)) {
    return false;
  }

  // We didn't download any data. Merge.
  if (AUTOMATIC_MODE || NO_SLACK_CONNECT) {
    return defaultResponse;
  }

  const { merge } = await prompt([
    {
      type: "confirm",
      default: defaultResponse,
      name: "merge",
      message: `We've found existing archive files. Do you want to append new data (recommended)? \n If you select "No", we'll delete the existing data.`,
    },
  ]);

  if (!merge) {
    fs.emptyDirSync(OUT_DIR);
  }

  return merge;
}

async function selectChannels(
  channels: Array<Channel>,
  previouslyDownloadedChannels: Record<string, SlackArchiveChannelData>
): Promise<Array<Channel>> {
  if (USE_PREVIOUS_CHANNEL_CONFIG) {
    const selectedChannels: Array<Channel> = channels.filter(
      (channel) => channel.id && channel.id in previouslyDownloadedChannels
    );
    const selectedChannelNames = selectedChannels.map(
      (channel) => channel.name || channel.id || "Unknown"
    );
    console.log(
      `Downloading channels selected previously: ${selectedChannelNames}.`
    );

    const previousChannelIds = Object.keys(previouslyDownloadedChannels);
    if (previousChannelIds.length != selectedChannels.length) {
      console.warn(
        "WARNING: Did not find all previously selected channel IDs."
      );
      console.log(
        `Expected to find ${previousChannelIds.length} channels, but only ${selectedChannels.length} matched.`
      );
      // Consider Looking up the user-facing names of the missing channels in the saved data.
      const availableChannelIds = new Set<string>(
        channels.map((channel) => channel.id || "")
      );
      const missingChannelIds = previousChannelIds.filter(
        (cId) => !availableChannelIds.has(cId)
      );
      //console.log(availableChannelIds);
      console.log(`Missing channel ids: ${missingChannelIds}`);
    } else {
      console.log(
        `Matched all ${previousChannelIds.length} previously selected channels out of ${channels.length} total channels available.`
      );
    }

    return selectedChannels;
  }

  const choices = channels.map((channel) => ({
    name: channel.name || channel.id || "Unknown",
    value: channel,
  }));

  if (AUTOMATIC_MODE || NO_SLACK_CONNECT) {
    if (EXCLUDE_CHANNELS) {
      const excludeChannels = EXCLUDE_CHANNELS.split(',');
      return channels.filter((channel) => !excludeChannels.includes(channel.name || ''));
    }
    return channels;
  }

  const result = await prompt([
    {
      type: "checkbox",
      loop: true,
      name: "channels",
      message: "Which channels do you want to download?",
      choices,
    },
  ]);

  return result.channels;
}

async function selectChannelTypes(): Promise<Array<string>> {
  const choices = [
    {
      name: "Public Channels",
      value: "public_channel",
    },
    {
      name: "Private Channels",
      value: "private_channel",
    },
    {
      name: "Multi-Person Direct Message",
      value: "mpim",
    },
    {
      name: "Direct Messages",
      value: "im",
    },
  ];

  if (CHANNEL_TYPES) {
    return CHANNEL_TYPES.split(",");
  }

  if (AUTOMATIC_MODE || USE_PREVIOUS_CHANNEL_CONFIG || NO_SLACK_CONNECT) {
    return ["public_channel", "private_channel", "mpim", "im"];
  }

  const result = await prompt([
    {
      type: "checkbox",
      loop: true,
      name: "channel-types",
      message: `Which channel types do you want to download?`,
      choices,
    },
  ]);

  return result["channel-types"];
}

async function getToken() {
  if (NO_SLACK_CONNECT) {
    return;
  }

  if (config.token) {
    console.log(`Using token ${config.token}`);
    return;
  }

  if (fs.existsSync(TOKEN_FILE)) {
    config.token = fs.readFileSync(TOKEN_FILE, "utf-8").trim();
    return;
  }

  const result = await prompt([
    {
      name: "token",
      type: "input",
      message:
        "Please enter your Slack token (xoxp-...). See README for more details.",
    },
  ]);

  config.token = result.token;
}

async function writeLastSuccessfulArchive() {
  const now = new Date();
  write(DATE_FILE, now.toISOString());
}

function getLastSuccessfulRun() {
  if (!fs.existsSync(DATE_FILE)) {
    return "";
  }

  const lastSuccessfulArchive = fs.readFileSync(DATE_FILE, "utf-8");

  let date = null;

  try {
    date = parseISO(lastSuccessfulArchive);
  } catch (error) {
    return "";
  }

  if (date && isValid(date)) {
    return `. Last successful run: ${date.toLocaleString()}`;
  }

  return "";
}

async function getAuthTest() {
  if (NO_SLACK_CONNECT) {
    return;
  }

  const spinner = ora("Testing authentication with Slack...").start();
  const result = await authTest();

  if (!result.ok) {
    spinner.fail(`Authentication with Slack failed.`);

    console.log(
      `Authentication with Slack failed. The error was: ${result.error}`
    );
    console.log(
      `The provided token was ${config.token}. Double-check the token and try again.`
    );
    console.log(
      `For more information on the error code, see the error table at https://api.slack.com/methods/auth.test`
    );
    console.log(`This tool will now exit.`);

    await deleteBackup();
    process.exit(-1);
  } else {
    spinner.succeed(`Successfully authorized with Slack as ${result.user}\n`);
  }

  return result;
}

export async function main() {
  console.log(`Welcome to slack-archive${getLastSuccessfulRun()}`);

  if (AUTOMATIC_MODE) {
    console.log(`Running in fully automatic mode without prompts`);
  }

  if (NO_SLACK_CONNECT) {
    console.log(`Not connecting to Slack and skipping all Slack API calls`);
  }

  await getToken();
  await createBackup();

  const slackArchiveData = await getSlackArchiveData();
  const users: Record<string, User> = await getUsers();
  const channelTypes = (await selectChannelTypes()).join(",");

  slackArchiveData.auth = await getAuthTest();

  const channels = await downloadChannels({ types: channelTypes }, users);
  const selectedChannels = await selectChannels(
    channels,
    slackArchiveData.channels
  );
  const newMessages: Record<string, number> = {};

  // Emoji
  // We don't actually download the images here, we'll
  // do that as needed
  const emojis = await downloadEmojiList();
  await writeAndMerge(EMOJIS_DATA_PATH, emojis);

  // Do we want to merge data?
  await selectMergeFiles();
  await writeAndMerge(CHANNELS_DATA_PATH, selectedChannels);

  // Download messages and extras for each channel
  await downloadEachChannel();

  // Save data
  await setSlackArchiveData(slackArchiveData);

  // Create HTML, but only for channels with new messages
  // - or channels that we didn't make HTML for yet
  const channelsToCreateFilesFor = await getChannelsToCreateFilesFor(
    selectedChannels,
    newMessages
  );
  await createHtmlForChannels(channelsToCreateFilesFor);

  // Create search file
  await createSearch();

  // Cleanup and finalize
  await deleteBackup();
  await deleteOlderBackups();
  await writeLastSuccessfulArchive();

  console.log(`All done.`);

  async function downloadEachChannel() {
    if (NO_SLACK_CONNECT) return;

    for (const [i, channel] of selectedChannels.entries()) {
      if (!channel.id) {
        console.warn(`Selected channel does not have an id`, channel);
        continue;
      }

      // Do we already have everything?
      slackArchiveData.channels[channel.id] =
        slackArchiveData.channels[channel.id] || {};
      if (slackArchiveData.channels[channel.id].fullyDownloaded) {
        continue;
      }

      // Download messages & users
      let downloadData = await downloadMessages(
        channel,
        i,
        selectedChannels.length
      );
      let result = downloadData.messages;
      newMessages[channel.id] = downloadData.new;

      await downloadExtras(channel, result, users);
      await downloadEmojis(result, emojis);
      await downloadAvatars();

      // Sort messages
      const spinner = ora(
        `Saving message data for ${channel.name || channel.id} to disk`
      ).start();
      spinner.render();

      result = uniqBy(result, "ts");
      result = result.sort((a, b) => {
        return parseFloat(b.ts || "0") - parseFloat(a.ts || "0");
      });

      await writeAndMerge(USERS_DATA_PATH, users);
      fs.outputFileSync(
        getChannelDataFilePath(channel.id),
        JSON.stringify(result, undefined, 2)
      );

      // Download files. This needs to run after the messages are saved to disk
      // since it uses the message data to find which files to download.
      await downloadFilesForChannel(channel.id!, spinner);

      // Update the data load cache
      messagesCache[channel.id!] = result;

      // Update the data
      const { is_archived, is_im, is_user_deleted } = channel;
      if (is_archived || (is_im && is_user_deleted)) {
        slackArchiveData.channels[channel.id].fullyDownloaded = true;
      }
      slackArchiveData.channels[channel.id].messages = result.length;

      spinner.succeed(`Saved message data for ${channel.name || channel.id}`);
    }
  }
}

main();


================================================
FILE: src/config.ts
================================================
import path from "path";
import { dirname } from "path";
import { fileURLToPath } from "url";

const __dirname = dirname(fileURLToPath(import.meta.url));

export const config = {
  token: process.env.SLACK_TOKEN,
};

function findCliParameter(param: string) {
  const args = process.argv;

  for (const arg of args) {
    if (arg === param) {
      return true;
    }
  }

  return false;
}

function getCliParameter(param: string) {
  const args = process.argv;

  for (const [i, arg] of args.entries()) {
    if (arg === param) {
      return args[i + 1];
    }
  }

  return null;
}

export const AUTOMATIC_MODE = findCliParameter("--automatic");
export const USE_PREVIOUS_CHANNEL_CONFIG = findCliParameter(
  "--use-previous-channel-config"
);
export const CHANNEL_TYPES = getCliParameter("--channel-types");
export const NO_BACKUP = findCliParameter("--no-backup");
export const NO_SEARCH = findCliParameter("--no-search");
export const NO_FILE_DOWNLOAD = findCliParameter("--no-file-download");
export const NO_SLACK_CONNECT = findCliParameter("--no-slack-connect");
export const FORCE_HTML_GENERATION = findCliParameter(
  "--force-html-generation"
);
export const EXCLUDE_CHANNELS = getCliParameter("--exclude-channels");
export const BASE_DIR = process.cwd();
export const OUT_DIR = path.join(BASE_DIR, "slack-archive");
export const TOKEN_FILE = path.join(OUT_DIR, ".token");
export const DATE_FILE = path.join(OUT_DIR, ".last-successful-run");
export const DATA_DIR = path.join(OUT_DIR, "data");
export const HTML_DIR = path.join(OUT_DIR, "html");
export const FILES_DIR = path.join(HTML_DIR, "files");
export const AVATARS_DIR = path.join(HTML_DIR, "avatars");
export const EMOJIS_DIR = path.join(HTML_DIR, "emojis");

export const INDEX_PATH = path.join(OUT_DIR, "index.html");
export const SEARCH_PATH = path.join(OUT_DIR, "search.html");
export const MESSAGES_JS_PATH = path.join(__dirname, "../static/scroll.js");
export const SEARCH_TEMPLATE_PATH = path.join(
  __dirname,
  "../static/search.html"
);
export const CHANNELS_DATA_PATH = path.join(DATA_DIR, "channels.json");
export const USERS_DATA_PATH = path.join(DATA_DIR, "users.json");
export const EMOJIS_DATA_PATH = path.join(DATA_DIR, "emojis.json");
export const SLACK_ARCHIVE_DATA_PATH = path.join(
  DATA_DIR,
  "slack-archive.json"
);
export const SEARCH_DATA_PATH = path.join(DATA_DIR, "search.js");

export function getChannelDataFilePath(channelId: string) {
  return path.join(DATA_DIR, `${channelId}.json`);
}

export function getChannelUploadFilePath(channelId: string, fileName: string) {
  return path.join(FILES_DIR, channelId, fileName);
}

export function getHTMLFilePath(channelId: string, index: number) {
  return path.join(HTML_DIR, `${channelId}-${index}.html`);
}

export function getAvatarFilePath(userId: string, extension: string) {
  return path.join(AVATARS_DIR, `${userId}${extension}`);
}


================================================
FILE: src/create-html.tsx
================================================
import { format } from "date-fns";
import fs from "fs-extra";
import path from "path";
import React from "react";
import ReactDOMServer from "react-dom/server.js";
import ora, { Ora } from "ora";
import { chunk, sortBy } from "lodash-es";
import { dirname } from "path";
import { fileURLToPath } from "url";
import esMain from "es-main";
import slackMarkdown from "slack-markdown";

import { getChannels, getMessages, getUsers } from "./data-load.js";
import {
  ArchiveMessage,
  Channel,
  ChunksInfo,
  Message,
  Reaction,
  SlackArchiveData,
  User,
  Users,
} from "./interfaces.js";
import {
  getHTMLFilePath,
  INDEX_PATH,
  OUT_DIR,
  MESSAGES_JS_PATH,
  FORCE_HTML_GENERATION,
} from "./config.js";
import { slackTimestampToJavaScriptTimestamp } from "./timestamp.js";
import { recordPage } from "./search.js";
import { write } from "./data-write.js";
import { getSlackArchiveData } from "./archive-data.js";
import { getEmojiFilePath, getEmojiUnicode, isEmojiUnicode } from "./emoji.js";
import { getName } from "./users.js";
import {
  isBotChannel,
  isDmChannel,
  isPrivateChannel,
  isPublicChannel,
} from "./channels.js";

const _dirname = dirname(fileURLToPath(import.meta.url));
const MESSAGE_CHUNK = 1000;

// This used to be a prop on the components, but passing it around
// was surprisingly slow. Global variables are cool again!
// Set by createHtmlForChannels().
let users: Users = {};
let slackArchiveData: SlackArchiveData = { channels: {} };
let me: User | null;

// Little hack to switch between ./index.html and ./html/...
let base = "";

function formatTimestamp(message: Message, dateFormat = "PPPPpppp") {
  const jsTs = slackTimestampToJavaScriptTimestamp(message.ts);
  const ts = format(jsTs, dateFormat);

  return ts;
}

interface FilesProps {
  message: Message;
  channelId: string;
}
const Files: React.FunctionComponent<FilesProps> = (props) => {
  const { message, channelId } = props;
  const { files } = message;

  if (!files || files.length === 0) return null;

  const fileElements = files.map((file) => {
    const { thumb_1024, thumb_720, thumb_480, thumb_pdf } = file as any;
    const thumb = thumb_1024 || thumb_720 || thumb_480 || thumb_pdf;
    let src = `files/${channelId}/${file.id}.${file.filetype}`;
    let href = src;

    if (file.mimetype?.startsWith("image")) {
      return (
        <a key={file.id} href={href} target="_blank">
          <img className="file" src={src} />
        </a>
      );
    }

    if (file.mimetype?.startsWith("video")) {
      return <video key={file.id} controls src={src} />;
    }

    if (file.mimetype?.startsWith("audio")) {
      return <audio key={file.id} controls src={src} />;
    }

    if (!file.mimetype?.startsWith("image") && thumb) {
      href = file.url_private || href;
      src = src.replace(`.${file.filetype}`, ".png");

      return (
        <a key={file.id} href={href} target="_blank">
          <img className="file" src={src} />
        </a>
      );
    }

    return (
      <a key={file.id} href={href} target="_blank">
        {file.name}
      </a>
    );
  });

  return <div className="files">{...fileElements}</div>;
};

interface AvatarProps {
  userId?: string;
}
const Avatar: React.FunctionComponent<AvatarProps> = ({ userId }) => {
  if (!userId) return null;

  const user = users[userId];
  if (!user || !user.profile || !user.profile.image_512) return null;

  const ext = path.extname(user?.profile?.image_512!);
  const src = `${base}avatars/${userId}${ext}`;

  return <img className="avatar" src={src} />;
};

interface ParentMessageProps {
  message: ArchiveMessage;
  channelId: string;
}
const ParentMessage: React.FunctionComponent<ParentMessageProps> = (props) => {
  const { message, channelId } = props;
  const hasFiles = !!message.files;

  return (
    <Message message={message} channelId={channelId}>
      {hasFiles ? <Files message={message} channelId={channelId} /> : null}
      {message.reactions?.map((reaction) => (
        <Reaction key={reaction.name} reaction={reaction} />
      ))}
      {message.replies?.map((reply) => (
        <ParentMessage message={reply} channelId={channelId} key={reply.ts} />
      ))}
    </Message>
  );
};

interface ReactionProps {
  reaction: Reaction;
}
const Reaction: React.FunctionComponent<ReactionProps> = ({ reaction }) => {
  const reactors = [];

  if (reaction.users) {
    for (const userId of reaction.users) {
      reactors.push(getName(userId, users));
    }
  }

  return (
    <div className="reaction" title={reactors.join(", ")}>
      <Emoji name={reaction.name!} />
      <span>{reaction.count}</span>
    </div>
  );
};

interface EmojiProps {
  name: string;
}
const Emoji: React.FunctionComponent<EmojiProps> = ({ name }) => {
  if (isEmojiUnicode(name)) {
    return <>{getEmojiUnicode(name)}</>;
  }

  return <img src={getEmojiFilePath(name)} />;
};

interface MessageProps {
  message: ArchiveMessage;
  channelId: string;
}
const Message: React.FunctionComponent<MessageProps> = (props) => {
  const { message } = props;
  const username = getName(message.user, users);
  const slackCallbacks = {
    user: ({ id }: { id: string }) => `@${getName(id, users)}`,
  };

  return (
    <div className="message-gutter" id={message.ts}>
      <div className="" data-stringify-ignore="true">
        <Avatar userId={message.user} />
      </div>
      <div className="">
        <span className="sender">{username}</span>
        <span className="timestamp">
          <span className="c-timestamp__label">{formatTimestamp(message)}</span>
        </span>
        <br />
        <div
          className="text"
          dangerouslySetInnerHTML={{
            __html: slackMarkdown.toHTML(message.text, {
              escapeHTML: false,
              slackCallbacks,
            }),
          }}
        />
        {props.children}
      </div>
    </div>
  );
};

interface MessagesPageProps {
  messages: Array<ArchiveMessage>;
  channel: Channel;
  index: number;
  chunksInfo: ChunksInfo;
}
const MessagesPage: React.FunctionComponent<MessagesPageProps> = (props) => {
  const { channel, index, chunksInfo } = props;
  const messagesJs = fs.readFileSync(MESSAGES_JS_PATH, "utf8");

  // Newest message is first
  const messages = props.messages
    .map((m) => (
      <ParentMessage key={m.ts} message={m} channelId={channel.id!} />
    ))
    .reverse();

  if (messages.length === 0) {
    messages.push(<span key="empty">No messages were ever sent!</span>);
  }

  return (
    <HtmlPage>
      <div style={{ paddingLeft: 10 }}>
        <Header index={index} chunksInfo={chunksInfo} channel={channel} />
        <div className="messages-list">{messages}</div>
        <script dangerouslySetInnerHTML={{ __html: messagesJs }} />
      </div>
    </HtmlPage>
  );
};

interface ChannelLinkProps {
  channel: Channel;
}
const ChannelLink: React.FunctionComponent<ChannelLinkProps> = ({
  channel,
}) => {
  let name = channel.name || channel.id;
  let leadSymbol = <span># </span>;

  const channelData = slackArchiveData.channels[channel.id!];
  if (channelData && channelData.messages === 0) {
    return null;
  }

  // Remove the user's name from the group mpdm channel name
  if (me && channel.is_mpim) {
    name = name?.replace(`@${me.name}`, "").replace("  ", " ");
  }

  if (channel.is_im && (channel as any).user) {
    leadSymbol = <Avatar userId={(channel as any).user} />;
  }

  if (channel.is_mpim) {
    leadSymbol = <></>;
    name = name?.replace("Group messaging with: ", "");
  }

  return (
    <li key={name}>
      <a title={name} href={`html/${channel.id!}-0.html`} target="iframe">
        {leadSymbol}
        <span>{name}</span>
      </a>
    </li>
  );
};

interface IndexPageProps {
  channels: Array<Channel>;
}
const IndexPage: React.FunctionComponent<IndexPageProps> = (props) => {
  const { channels } = props;
  const sortedChannels = sortBy(channels, "name");

  const publicChannels = sortedChannels
    .filter((channel) => isPublicChannel(channel) && !channel.is_archived)
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  const publicArchivedChannels = sortedChannels
    .filter((channel) => isPublicChannel(channel) && channel.is_archived)
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  const privateChannels = sortedChannels
    .filter((channel) => isPrivateChannel(channel) && !channel.is_archived)
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  const privateArchivedChannels = sortedChannels
    .filter((channel) => isPrivateChannel(channel) && channel.is_archived)
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  const dmChannels = sortedChannels
    .filter(
      (channel) => isDmChannel(channel, users) && !users[channel.user!].deleted
    )
    .sort((a, b) => {
      // Self first
      if (me && a.user && a.user === me.id) {
        return -1;
      }

      // Then alphabetically
      return (a.name || "Unknown").localeCompare(b.name || "Unknown");
    })
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  const dmDeletedChannels = sortedChannels
    .filter(
      (channel) => isDmChannel(channel, users) && users[channel.user!].deleted
    )
    .sort((a, b) => (a.name || "Unknown").localeCompare(b.name || "Unknown"))
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  const groupChannels = sortedChannels
    .filter((channel) => channel.is_mpim)
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  const botChannels = sortedChannels
    .filter((channel) => isBotChannel(channel, users))
    .sort((a, b) => {
      if (a.name && b.name) {
        return a.name!.localeCompare(b.name!);
      }

      return 1;
    })
    .map((channel) => <ChannelLink key={channel.id} channel={channel} />);

  return (
    <HtmlPage>
      <div id="index">
        <div id="channels">
          <p className="section">Public Channels</p>
          <ul>{publicChannels}</ul>
          <p className="section">Private Channels</p>
          <ul>{privateChannels}</ul>
          <p className="section">DMs</p>
          <ul>{dmChannels}</ul>
          <p className="section">Group DMs</p>
          <ul>{groupChannels}</ul>
          <p className="section">Bots</p>
          <ul>{botChannels}</ul>
          <p className="section">Archived Public Channels</p>
          <ul>{publicArchivedChannels}</ul>
          <p className="section">Archived Private Channels</p>
          <ul>{privateArchivedChannels}</ul>
          <p className="section">DMs (Deleted Users)</p>
          <ul>{dmDeletedChannels}</ul>
        </div>
        <div id="messages">
          <iframe name="iframe" src={`html/${channels[0].id!}-0.html`} />
        </div>
        <script
          dangerouslySetInnerHTML={{
            __html: `
            const urlSearchParams = new URLSearchParams(window.location.search);
            const channelValue = urlSearchParams.get("c");
            const tsValue = urlSearchParams.get("ts");
            
            if (channelValue) {
              const iframe = document.getElementsByName('iframe')[0]
              iframe.src = "html/" + decodeURIComponent(channelValue) + '.html' + '#' + (tsValue || '');
            }
            `,
          }}
        />
      </div>
    </HtmlPage>
  );
};

const HtmlPage: React.FunctionComponent = (props) => {
  return (
    <html lang="en">
      <head>
        <meta httpEquiv="X-UA-Compatible" content="IE=edge" />
        <meta charSet="UTF-8" />
        <meta name="viewport" content="width=device-width, initial-scale=1.0" />
        <title>Slack</title>
        <link rel="stylesheet" href={`${base}style.css`} />
      </head>
      <body>{props.children}</body>
    </html>
  );
};

interface HeaderProps {
  index: number;
  chunksInfo: ChunksInfo;
  channel: Channel;
}
const Header: React.FunctionComponent<HeaderProps> = (props) => {
  const { channel, index, chunksInfo } = props;
  let created;

  if (!channel.is_im && !channel.is_mpim) {
    const creator = getName(channel.creator, users);
    const time = channel.created
      ? format(channel.created * 1000, "PPPP")
      : "Unknown";

    created =
      creator && time ? (
        <span className="created">
          Created by {creator} on {time}
        </span>
      ) : null;
  }

  return (
    <div className="header">
      <h1>{channel.name || channel.id}</h1>
      {created}
      <p className="topic">{channel.topic?.value}</p>
      <Pagination
        channelId={channel.id!}
        index={index}
        chunksInfo={chunksInfo}
      />
    </div>
  );
};

interface PaginationProps {
  index: number;
  chunksInfo: ChunksInfo;
  channelId: string;
}
const Pagination: React.FunctionComponent<PaginationProps> = (props) => {
  const { index, channelId, chunksInfo } = props;
  const length = chunksInfo.length;

  if (length === 1) {
    return null;
  }

  const older =
    index + 1 < length ? (
      <span>
        <a href={`${channelId}-${index + 1}.html`}>Older Messages</a>
      </span>
    ) : null;
  const newer =
    index > 0 ? (
      <span>
        <a href={`${channelId}-${index - 1}.html`}>Newer Messages </a>
      </span>
    ) : null;
  const sep1 = older && newer ? " | " : null;
  const sep2 = older || newer ? " | " : null;

  const options: Array<JSX.Element> = [];
  for (const [i, chunk] of chunksInfo.entries()) {
    const text = `${i} - ${chunk.newest} to ${chunk.oldest}`;
    const value = `${channelId}-${i}.html`;
    const selected = i === index;
    options.push(
      <option selected={selected} key={value} value={value}>
        {text}
      </option>
    );
  }

  return (
    <div className="pagination">
      {newer}
      {sep1}
      {older}
      {sep2}
      <div className="jumper">
        <select id="jumper">{options}</select>
        <script
          dangerouslySetInnerHTML={{
            __html: `
              document.getElementById("jumper").onchange = function () {
                window.location.href = this.value;
              }
            `,
          }}
        />
      </div>
    </div>
  );
};

async function renderIndexPage() {
  base = "html/";
  const channels = await getChannels();
  const page = <IndexPage channels={channels} />;

  return renderAndWrite(page, INDEX_PATH);
}

interface RenderMessagesPageOptions {
  channel: Channel;
  messages: Array<ArchiveMessage>;
  chunkIndex: number;
  chunksInfo: ChunksInfo;
}

function renderMessagesPage(options: RenderMessagesPageOptions, spinner: Ora) {
  const { channel, messages, chunkIndex: index, chunksInfo } = options;
  const page = (
    <MessagesPage
      channel={channel}
      messages={messages}
      index={index}
      chunksInfo={chunksInfo}
    />
  );

  const filePath = getHTMLFilePath(channel.id!, index);
  spinner.text = `${channel.name || channel.id}: Writing ${index + 1}/${
    chunksInfo.length
  } ${filePath}`;
  spinner.render();

  // Update the search index. In messages, the youngest message is first.
  if (messages.length > 0) {
    recordPage(channel.id, messages[messages.length - 1]?.ts);
  }

  return renderAndWrite(page, filePath);
}

async function renderAndWrite(page: JSX.Element, filePath: string) {
  const html = ReactDOMServer.renderToStaticMarkup(page);
  const htmlWDoc = "<!DOCTYPE html>" + html;

  await write(filePath, htmlWDoc);
}

export async function getChannelsToCreateFilesFor(
  channels: Array<Channel>,
  newMessages: Record<string, number>
) {
  const result: Array<Channel> = [];

  // If HTML regeneration is forced, ignore everything
  // and just return all channels
  if (FORCE_HTML_GENERATION) {
    return await getChannels();
  }

  for (const channel of channels) {
    if (channel.id) {
      // Do we have new messages?
      if (newMessages[channel.id] > 0) {
        result.push(channel);
      }

      // Did we never create a file?
      if (!fs.existsSync(getHTMLFilePath(channel.id!, 0))) {
        result.push(channel);
      }
    }
  }

  return result;
}

async function createHtmlForChannel({
  channel,
  i,
  total,
}: {
  channel: Channel;
  i: number;
  total: number;
}) {
  const messages = await getMessages(channel.id!, true);
  const chunks = chunk(messages, MESSAGE_CHUNK);
  const spinner = ora(
    `Rendering HTML for ${i + 1}/${total} ${channel.name || channel.id}`
  ).start();

  // Calculate info about all chunks
  const chunksInfo: ChunksInfo = [];
  for (const iChunk of chunks) {
    chunksInfo.push({
      oldest: formatTimestamp(iChunk[iChunk.length - 1], "Pp"),
      newest: formatTimestamp(iChunk[0], "Pp"),
      count: iChunk.length,
    });
  }

  if (chunks.length === 0) {
    await renderMessagesPage(
      {
        channel,
        messages: [],
        chunkIndex: 0,
        chunksInfo: chunksInfo,
      },
      spinner
    );
  }

  for (const [chunkI, chunk] of chunks.entries()) {
    await renderMessagesPage(
      {
        channel,
        messages: chunk,
        chunkIndex: chunkI,
        chunksInfo,
      },
      spinner
    );
  }

  spinner.succeed(
    `Rendered HTML for ${i + 1}/${total} ${channel.name || channel.id}`
  );
}

export async function createHtmlForChannels(channels: Array<Channel> = []) {
  console.log(`\n Creating HTML files for ${channels.length} channels...`);

  users = await getUsers();
  slackArchiveData = await getSlackArchiveData();
  me = slackArchiveData.auth?.user_id
    ? users[slackArchiveData.auth?.user_id]
    : null;

  for (const [i, channel] of channels.entries()) {
    if (!channel.id) {
      console.warn(`Can't create HTML for channel: No id found`, channel);
      continue;
    }

    await createHtmlForChannel({ channel, i, total: channels.length });
  }

  await renderIndexPage();

  // Copy in fonts & css
  fs.copySync(path.join(_dirname, "../static"), path.join(OUT_DIR, "html/"));
}

if (esMain(import.meta)) {
  createHtmlForChannels();
}


================================================
FILE: src/data-load.ts
================================================
import fs from "fs-extra";

import {
  ArchiveMessage,
  Channel,
  Emojis,
  SearchFile,
  Users,
} from "./interfaces.js";
import {
  CHANNELS_DATA_PATH,
  EMOJIS_DATA_PATH,
  getChannelDataFilePath,
  SEARCH_DATA_PATH,
  USERS_DATA_PATH,
} from "./config.js";
import { retry } from "./retry.js";

async function getFile<T>(filePath: string, returnIfEmpty: T): Promise<T> {
  if (!fs.existsSync(filePath)) {
    return returnIfEmpty;
  }

  const data: T = await readJSON(filePath);

  return data;
}

export const messagesCache: Record<string, Array<ArchiveMessage>> = {};

export async function getMessages(
  channelId: string,
  cachedOk: boolean = false
): Promise<Array<ArchiveMessage>> {
  if (cachedOk && messagesCache[channelId]) {
    return messagesCache[channelId];
  }

  const filePath = getChannelDataFilePath(channelId);
  messagesCache[channelId] = await getFile<Array<ArchiveMessage>>(filePath, []);

  return messagesCache[channelId];
}

export async function getUsers(): Promise<Users> {
  return getFile<Users>(USERS_DATA_PATH, {});
}

export async function getEmoji(): Promise<Emojis> {
  return getFile<Emojis>(EMOJIS_DATA_PATH, {});
}

export async function getChannels(): Promise<Array<Channel>> {
  return getFile<Array<Channel>>(CHANNELS_DATA_PATH, []);
}

export async function getSearchFile(): Promise<SearchFile> {
  const returnIfEmpty = { users: {}, channels: {}, messages: {}, pages: {} };

  if (!fs.existsSync(SEARCH_DATA_PATH)) {
    return returnIfEmpty;
  }

  const contents = await readFile(SEARCH_DATA_PATH, "utf8");

  // See search.ts, the file is actually JS (not JSON)
  return JSON.parse(contents.slice(21, contents.length - 1));
}

export async function readFile(filePath: string, encoding = "utf8") {
  return retry<string>({ name: `Reading ${filePath}` }, () => {
    return fs.readFileSync(SEARCH_DATA_PATH, "utf8");
  });
}

export async function readJSON<T>(filePath: string) {
  return retry<T>({ name: `Loading JSON from ${filePath}` }, () => {
    return fs.readJSONSync(filePath);
  });
}


================================================
FILE: src/data-write.ts
================================================
import fs from "fs-extra";
import { differenceBy } from "lodash-es";

import { retry } from "./retry.js";

export async function write(filePath: string, data: any) {
  await retry({ name: `Writing ${filePath}` }, () => {
    fs.outputFileSync(filePath, data);
  });
}

export async function writeAndMerge(filePath: string, newData: any) {
  await retry({ name: `Writing ${filePath}` }, () => {
    let dataToWrite = newData;

    if (fs.existsSync(filePath)) {
      const oldData = fs.readJSONSync(filePath);

      if (Array.isArray(oldData)) {
        if (newData && newData[0] && newData[0].id) {
          // Take the old data, exclude aything that is in the new data,
          // and then add the new data
          dataToWrite = [
            ...differenceBy(oldData, newData, (v: any) => v.id),
            ...newData,
          ];
        } else {
          dataToWrite = [...oldData, ...newData];
        }
      } else if (typeof newData === "object") {
        dataToWrite = { ...oldData, ...newData };
      } else {
        console.error(`writeAndMerge: Did not understand type of data`, {
          filePath,
          newData,
        });
      }
    }

    fs.outputFileSync(filePath, JSON.stringify(dataToWrite, undefined, 2));
  });
}


================================================
FILE: src/download-files.ts
================================================
import fetch from "node-fetch";
import fs from "fs-extra";
import esMain from "es-main";
import ora, { Ora } from "ora";
import path from "path";

import { File } from "./interfaces.js";
import {
  getChannelUploadFilePath,
  config,
  NO_FILE_DOWNLOAD,
} from "./config.js";
import { getChannels, getMessages } from "./data-load.js";
import { downloadAvatars } from "./users.js";

export interface DownloadUrlOptions {
  authorize?: boolean;
  force?: boolean;
}

export async function downloadURL(
  url: string,
  filePath: string,
  options: DownloadUrlOptions = {}
) {
  const authorize = options.authorize === undefined ? true : options.authorize;

  if (!options.force && fs.existsSync(filePath)) {
    return;
  }

  const { token } = config;
  const headers: HeadersInit = authorize
    ? {
        Authorization: `Bearer ${token}`,
      }
    : {};

  try {
    const response = await fetch(url, { headers });
    const buffer = await response.buffer();
    fs.outputFileSync(filePath, buffer);
  } catch (error) {
    console.warn(`Failed to download file ${url}`, error);
  }
}

async function downloadFile(
  file: File,
  channelId: string,
  i: number,
  total: number,
  spinner: Ora
) {
  const { url_private, id, is_external, mimetype } = file;
  const { thumb_1024, thumb_720, thumb_480, thumb_pdf } = file as any;

  const fileUrl = is_external
    ? thumb_1024 || thumb_720 || thumb_480 || thumb_pdf
    : url_private;

  if (!fileUrl) return;

  spinner.text = `Downloading ${i}/${total}: ${fileUrl}`;

  const extension = path.extname(fileUrl);
  const filePath = getChannelUploadFilePath(channelId, `${id}${extension}`);

  await downloadURL(fileUrl, filePath);

  if (mimetype === "application/pdf" && thumb_pdf) {
    spinner.text = `Downloading ${i}/${total}: ${thumb_pdf}`;
    const thumbFile = filePath.replace(extension, ".png");
    await downloadURL(thumb_pdf, thumbFile);
  }
}

export async function downloadFilesForChannel(channelId: string, spinner: Ora) {
  if (NO_FILE_DOWNLOAD) {
    return;
  }

  const messages = await getMessages(channelId);
  const channels = await getChannels();
  const channel = channels.find(({ id }) => id === channelId);
  const fileMessages = messages.filter(
    (m) => (m.files?.length || m.replies?.length || 0) > 0
  );
  const getSpinnerText = (i: number, ri?: number) => {
    let reply = "";
    if (ri !== undefined) {
      reply = ` (reply ${ri})`;
    }

    return `Downloading ${i}/${
      fileMessages.length
    }${reply} messages with files for channel ${channel?.name || channelId}...`;
  };

  spinner.text = getSpinnerText(0);

  for (const [i, fileMessage] of fileMessages.entries()) {
    if (!fileMessage.files && !fileMessage.replies) {
      continue;
    }

    if (fileMessage.files) {
      for (const file of fileMessage.files) {
        spinner.text = getSpinnerText(i);
        spinner.render();
        await downloadFile(file, channelId, i, fileMessages.length, spinner);
      }
    }

    if (fileMessage.replies) {
      for (const [ri, reply] of fileMessage.replies.entries()) {
        if (reply.files) {
          for (const file of reply.files) {
            spinner.text = getSpinnerText(i, ri);
            spinner.render();
            await downloadFile(
              file,
              channelId,
              i,
              fileMessages.length,
              spinner
            );
          }
        }
      }
    }
  }
}


================================================
FILE: src/emoji.ts
================================================
import path from "path";
import ora from "ora";
import fs from "fs";
import { createRequire } from "node:module";

import { EMOJIS_DIR, NO_SLACK_CONNECT } from "./config.js";
import { downloadURL } from "./download-files.js";
import { ArchiveMessage, Emojis } from "./interfaces.js";
import { getWebClient } from "./web-client.js";

const require = createRequire(import.meta.url);
const emojiData = require("emoji-datasource");

let _unicodeEmoji: Record<string, string>;
function getUnicodeEmoji() {
  if (_unicodeEmoji) {
    return _unicodeEmoji;
  }

  _unicodeEmoji = {};
  for (const emoji of emojiData) {
    _unicodeEmoji[emoji.short_name as string] = emoji.unified;
  }

  return _unicodeEmoji;
}

export function getEmojiFilePath(name: string, extension?: string) {
  // If we have an extension, return the correct path
  if (extension) {
    return path.join(EMOJIS_DIR, `${name}${extension}`);
  }

  // If we don't have an extension, return the first path that exists
  // regardless of extension
  const extensions = [".png", ".jpg", ".gif"];
  for (const ext of extensions) {
    if (fs.existsSync(path.join(EMOJIS_DIR, `${name}${ext}`))) {
      return path.join(EMOJIS_DIR, `${name}${ext}`);
    }
  }
}

export function isEmojiUnicode(name: string) {
  const unicodeEmoji = getUnicodeEmoji();
  return !!unicodeEmoji[name];
}

export function getEmojiUnicode(name: string) {
  const unicodeEmoji = getUnicodeEmoji();
  const unified = unicodeEmoji[name];
  const split = unified.split("-");

  return split
    .map((code) => {
      return String.fromCodePoint(parseInt(code, 16));
    })
    .join("");
}

export async function downloadEmojiList(): Promise<Emojis> {
  if (NO_SLACK_CONNECT) {
    return {};
  }

  const response = await getWebClient().emoji.list();

  if (response.ok) {
    return response.emoji!;
  } else {
    return {};
  }
}

export async function downloadEmoji(
  name: string,
  url: string,
  emojis: Emojis
): Promise<void> {
  // Alias?
  if (url.startsWith("alias:")) {
    const alias = getEmojiAlias(url);

    if (!emojis[alias]) {
      console.warn(
        `Found emoji alias ${alias}, which does not exist in master emoji list`
      );
      return;
    } else {
      return downloadEmoji(alias, emojis[alias], emojis);
    }
  }

  const extension = path.extname(url);
  const filePath = getEmojiFilePath(name, extension);

  return downloadURL(url, filePath!);
}

export function getEmojiAlias(name: string): string {
  // Ugh regex methods - this should turn "alias:hi-bob" into "hi-bob"
  const alias = [...name.matchAll(/alias:(.*)/g)][0][1]!;
  return alias!;
}

export async function downloadEmojis(
  messages: Array<ArchiveMessage>,
  emojis: Emojis
) {
  const regex = /:[^:\s]*(?:::[^:\s]*)*:/g;

  const spinner = ora(
    `Scanning 0/${messages.length} messages for emoji shortcodes...`
  ).start();
  let downloaded = 0;

  for (const [i, message] of messages.entries()) {
    spinner.text = `Scanning ${i}/${messages.length} messages for emoji shortcodes...`;

    // Reactions
    if (message.reactions && message.reactions.length > 0) {
      for (const reaction of message.reactions) {
        const reactEmoji = emojis[reaction.name!];
        if (reactEmoji) {
          downloaded++;
          await downloadEmoji(reaction.name!, reactEmoji, emojis);
        }
      }
    }
  }

  spinner.succeed(
    `Scanned ${messages.length} messages for emoji (and downloaded ${downloaded})`
  );
}


================================================
FILE: src/interfaces.ts
================================================
import { Message as SlackMessage } from "@slack/web-api/dist/response/ConversationsHistoryResponse";
import { Channel as SlackChannel } from "@slack/web-api/dist/response/ConversationsListResponse";
import { User as SlackUser } from "@slack/web-api/dist/response/UsersInfoResponse";
import { File as SlackFile } from "@slack/web-api/dist/response/FilesInfoResponse";
import { Reaction as SlackReaction } from "@slack/web-api/dist/response/ReactionsGetResponse";
import { AuthTestResponse } from "@slack/web-api";

export type User = SlackUser;

export type Users = Record<string, User>;

export type Emojis = Record<string, string>;

export interface ArchiveMessage extends SlackMessage {
  replies?: Array<SlackMessage>;
}

export type Reaction = SlackReaction;

export type Message = SlackMessage;

export type Channel = SlackChannel;

export type File = SlackFile;

export type SearchPageIndex = Record<string, Array<string>>;

export type SearchFile = {
  users: Record<string, string>; // userId -> userName
  channels: Record<string, string>; // channelId -> channelName
  messages: Record<string, Array<SearchMessage>>;
  pages: SearchPageIndex;
};

export type SearchMessage = {
  m?: string; // Message
  u?: string; // User
  t?: string; // Timestamp
  c?: string; // Channel
};

export interface SlackArchiveChannelData {
  messages: number;
  fullyDownloaded: boolean;
}

export interface SlackArchiveData {
  channels: Record<string, SlackArchiveChannelData>;
  auth?: AuthTestResponse;
}

export interface ChunkInfo {
  oldest?: string;
  newest?: string;
  count: number;
}

export type ChunksInfo = Array<ChunkInfo>;


================================================
FILE: src/messages.ts
================================================
import {
  ConversationsHistoryResponse,
  ConversationsListArguments,
  ConversationsListResponse,
} from "@slack/web-api";
import { Channel } from "@slack/web-api/dist/response/ConversationsListResponse";
import ora from "ora";

import { ArchiveMessage, Message, Users } from "./interfaces.js";
import { getMessages } from "./data-load.js";
import { isThread } from "./threads.js";
import { downloadUser, getName } from "./users.js";
import { getWebClient } from "./web-client.js";

function isConversation(input: any): input is ConversationsHistoryResponse {
  return !!input.messages;
}

interface DownloadMessagesResult {
  messages: Array<ArchiveMessage>;
  new: number;
}

export async function downloadMessages(
  channel: Channel,
  i: number,
  channelCount: number
): Promise<DownloadMessagesResult> {
  let result: DownloadMessagesResult = {
    messages: [],
    new: 0,
  };

  if (!channel.id) {
    console.warn(`Channel without id`, channel);
    return result;
  }

  for (const message of await getMessages(channel.id)) {
    result.messages.push(message);
  }

  const oldest =
    result.messages.length > 0 ? parseInt(result.messages[0].ts || "0", 10) : 0;
  const name =
    channel.name || channel.id || channel.purpose?.value || "Unknown channel";

  const spinner = ora(
    `Downloading messages for channel ${i + 1}/${channelCount} (${name})...`
  ).start();

  for await (const page of getWebClient().paginate("conversations.history", {
    channel: channel.id,
    oldest,
  })) {
    if (isConversation(page)) {
      const pageLength = page.messages?.length || 0;
      const fetched = `Fetched ${pageLength} messages`;
      const total = `(total so far: ${result.messages.length + pageLength}`;

      spinner.text = `Downloading ${
        i + 1
      }/${channelCount} ${name}: ${fetched} ${total})`;

      result.new = result.new + (page.messages || []).length;

      result.messages.unshift(...(page.messages || []));
    }
  }

  spinner.succeed(
    `Downloaded messages for channel ${i + 1}/${channelCount} (${name})`
  );

  return result;
}

export async function downloadReplies(
  channel: Channel,
  message: ArchiveMessage
): Promise<Array<Message>> {
  if (!channel.id || !message.ts) {
    console.warn("Could not find channel or message id", channel, message);
    return [];
  }

  if (!message.reply_count) {
    console.warn("Message has no reply count", message);
    return [];
  }

  // Do we already have all replies?
  if (message.replies && message.replies.length >= message.reply_count) {
    return message.replies;
  }

  const replies = message.replies || [];
  // Oldest is the last entry
  const oldest = replies.length > 0 ? replies[replies.length - 1].ts : "0";
  const result = await getWebClient().conversations.replies({
    channel: channel.id,
    ts: message.ts,
    oldest,
  });

  // First message is the parent
  return (result.messages || []).slice(1);
}

export async function downloadExtras(
  channel: Channel,
  messages: Array<ArchiveMessage>,
  users: Users
) {
  const spinner = ora(
    `Downloading threads and users for ${channel.name || channel.id}...`
  ).start();

  // Then, all messages and threads
  let processedThreads = 0;
  const totalThreads = messages.filter(isThread).length;
  for (const message of messages) {
    // Download threads
    if (isThread(message)) {
      processedThreads++;
      spinner.text = `Downloading threads (${processedThreads}/${totalThreads}) for ${
        channel.name || channel.id
      }...`;
      message.replies = await downloadReplies(channel, message);
    }

    // Download users and avatars
    if (message.user) {
      await downloadUser(message, users);
    }
  }

  spinner.succeed(
    `Downloaded ${totalThreads} threads and users for ${
      channel.name || channel.id
    }.`
  );
}


================================================
FILE: src/reactions.ts
================================================
import { Message } from "./interfaces";

export function hasReactions(message: Message) {
  return message.reactions && message.reactions.length > 0;
}


================================================
FILE: src/retry.ts
================================================
export interface RetryOptions {
  retries: number;
  name?: string;
}

const defaultOptions: RetryOptions = {
  retries: 3,
};

export async function retry<T>(
  options: Partial<RetryOptions>,
  operation: () => T,
  attempt = 0
): Promise<T> {
  let mergedOptions = { ...defaultOptions, ...options };

  try {
    return operation();
  } catch (error) {
    if (attempt >= mergedOptions.retries) {
      throw error;
    }

    const ms = 250 + attempt * 250;

    if (mergedOptions.name) {
      console.warn(`Operation "${options.name}" failed, retrying in ${ms}`);
    }

    await wait(ms);

    return retry(options, operation, attempt + 1);
  }
}

function wait(ms = 250) {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
}


================================================
FILE: src/search.ts
================================================
import fs from "fs-extra";
import ora, { Ora } from "ora";
import { getChannelName } from "./channels.js";

import {
  NO_SEARCH,
  SEARCH_DATA_PATH,
  SEARCH_PATH,
  SEARCH_TEMPLATE_PATH,
} from "./config.js";
import { SearchFile, SearchMessage, SearchPageIndex } from "./interfaces";
import {
  getChannels,
  getMessages,
  getSearchFile,
  getUsers,
} from "./data-load.js";

// Format:
// channelId: [ timestamp0, timestamp1, timestamp2, ... ]
//
// channelId: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
// pages: {
//   0: [ 10, 9, 8 ]
//   1: [ 7, 6, 5 ]
//   2: [ 4, 3, 2 ]
//   3: [ 1, 0 ]
// }
// INDEX_OF_PAGES: {
//   channelId: [8, 5, 2, 0]
// }
//
// For channelId, a message older than timestamp 0 but younger than timestamp1 is on page 1.
// In our example above, the message with timestamp 6 is older than 5 but younger than 8.
const INDEX_OF_PAGES: SearchPageIndex = {};

export function recordPage(channelId?: string, timestamp?: string) {
  if (!channelId || !timestamp) {
    console.warn(
      `Search: Cannot record page: channelId: ${channelId} timestamp: ${timestamp}`
    );
    return;
  }

  if (!INDEX_OF_PAGES[channelId]) {
    INDEX_OF_PAGES[channelId] = [];
  }

  INDEX_OF_PAGES[channelId].push(timestamp);
}

export async function createSearch() {
  if (NO_SEARCH) return;

  const spinner = ora(`Creating search file...`).start();
  spinner.render();

  await createSearchFile(spinner);
  await createSearchHTML();

  spinner.succeed(`Search file created`);
}

async function createSearchFile(spinner: Ora) {
  const existingData = await getSearchFile();
  const users = await getUsers();
  const channels = await getChannels();
  const result: SearchFile = {
    channels: {},
    users: {},
    messages: {},
    pages: { ...existingData.pages, ...INDEX_OF_PAGES },
  };

  // Users
  for (const user in users) {
    result.users[user] = users[user].name || users[user].real_name || "Unknown";
  }

  // Channels & Messages
  for (const [i, channel] of channels.entries()) {
    if (!channel.id) {
      console.warn(
        `Can't create search file for channel ${channel.name}: No id found`,
        channel
      );
      continue;
    }

    const name = getChannelName(channel);

    spinner.text = `Creating search messages for channel ${name}`;
    spinner.render();

    const messages = (await getMessages(channel.id, true)).map((message) => {
      const searchMessage: SearchMessage = {
        m: message.text,
        u: message.user,
        t: message.ts,
      };

      return searchMessage;
    });

    result.messages![channel.id] = messages;
    result.channels[channel.id] = name;
  }

  const jsContent = `window.search_data = ${JSON.stringify(result)};`;
  await fs.outputFile(SEARCH_DATA_PATH, jsContent);
}

async function createSearchHTML() {
  let template = fs.readFileSync(SEARCH_TEMPLATE_PATH, "utf8");

  template = template.replace(
    "<!-- react -->",
    getScript(`react@18.2.0/umd/react.production.min.js`)
  );
  template = template.replace(
    "<!-- react-dom -->",
    getScript(`react-dom@18.2.0/umd/react-dom.production.min.js`)
  );
  template = template.replace(
    `<!-- babel -->`,
    getScript(`babel-standalone@6.26.0/babel.min.js`)
  );
  template = template.replace(
    `<!-- minisearch -->`,
    getScript("minisearch@5.0.0/dist/umd/index.min.js")
  );

  template = template.replace(`<!-- Size -->`, getSize());

  fs.outputFileSync(SEARCH_PATH, template);
}

function getSize() {
  const mb = fs.statSync(SEARCH_DATA_PATH).size / 1048576; //MB
  return `Loading ${Math.round(mb)}MB of data`;
}

function getScript(script: string) {
  return `<script crossorigin src="https://cdn.jsdelivr.net/npm/${script}"></script>`;
}


================================================
FILE: src/threads.ts
================================================
import { Message } from "./interfaces";

export function isThread(message: Message) {
  return message.reply_count && message.reply_count > 0;
}


================================================
FILE: src/timestamp.ts
================================================
export function slackTimestampToJavaScriptTimestamp(ts?: string) {
  if (!ts) {
    return 0;
  }

  const splitTs = ts.split(".") || [];
  const jsTs = parseInt(`${splitTs[0]}${splitTs[1].slice(0, 3)}`, 10);

  return jsTs;
}


================================================
FILE: src/users.ts
================================================
import path from "path";

import { getWebClient } from "./web-client.js";
import { Message, User, Users } from "./interfaces.js";
import { getAvatarFilePath } from "./config.js";
import { getUsers } from "./data-load.js";
import { downloadURL } from "./download-files.js";
import ora from "ora";

// We'll redownload users every run, but only once per user
// To keep track, we'll keep the ids in this array
export const usersRefetchedThisRun: Array<string> = [];
export const avatarsRefetchedThisRun: Array<string> = [];

export async function downloadUser(
  item: Message | any,
  users: Users
): Promise<User | null> {
  if (!item.user) return null;

  // If we already have this user *and* downloaded them before,
  // return cached version
  if (users[item.user] && usersRefetchedThisRun.includes(item.user))
    return users[item.user];

  const spinner = ora(`Downloading info for user ${item.user}...`).start();
  const user = (item.user === 'U00') ? {} as User : (
      await getWebClient().users.info({
        user: item.user,
      })
    ).user;

  if (user) {
    usersRefetchedThisRun.push(item.user);
    spinner.succeed(`Downloaded info for user ${item.user} (${user.name})`);
    return (users[item.user] = user);
  }

  return null;
}

export async function downloadAvatars() {
  const users = await getUsers();
  const userIds = Object.keys(users);
  const spinner = ora(`Downloading avatars (0/${userIds.length})`).start();

  for (const [i, userId] of userIds.entries()) {
    spinner.text = `Downloading avatars (${i + 1}/${userIds.length})`;
    await downloadAvatarForUser(users[userId]);
  }

  spinner.stop();
}

export async function downloadAvatarForUser(user?: User | null) {
  if (!user || !user.id || avatarsRefetchedThisRun.includes(user.id)) {
    return;
  }

  const { profile } = user;

  if (!profile || !profile.image_512) {
    return;
  }

  try {
    const filePath = getAvatarFilePath(
      user.id!,
      path.extname(profile.image_512)
    );
    await downloadURL(profile.image_512, filePath, {
      authorize: false,
      force: true,
    });
    avatarsRefetchedThisRun.push(user.id!);
  } catch (error) {
    console.warn(`Failed to download avatar for user ${user.id!}`, error);
  }
}

export function getName(userId: string | undefined, users: Users) {
  if (!userId) return "Unknown";
  const user = users[userId];
  if (!user) return userId;

  return user.profile?.display_name || user.profile?.real_name || user.name;
}


================================================
FILE: src/web-client.ts
================================================
import { WebClient } from "@slack/web-api";

import { config } from "./config.js";

let _webClient: WebClient;
export function getWebClient() {
  if (_webClient) return _webClient;

  const { token } = config;
  return (_webClient = new WebClient(token));
}

export async function authTest() {
  return getWebClient().auth.test();
}


================================================
FILE: static/scroll.js
================================================
if (window.location.hash) {
  document.getElementById(window.location.hash).scrollTo();
} else {
  scrollBy({ top: 99999999 });
}


================================================
FILE: static/search.html
================================================
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Message Search</title>
    <link rel="stylesheet" href="html/style.css" />

    <!-- react -->
    <!-- react-dom -->
    <!-- babel -->
    <!-- minisearch -->
  </head>

  <body>
    <script type="text/babel" data-type="module">
      class App extends React.PureComponent {
        constructor(props) {
          super(props);

          this.handleSearchChange = this.handleSearchChange.bind(this);
          this.handleSearchClear = this.handleSearchClear.bind(this);
          this.searchMessages = this.searchMessages.bind(this);
          this.loadSearchData = this.loadSearchData.bind(this);
          this.loadSearchDataWhenReady =
            this.loadSearchDataWhenReady.bind(this);

          this.searchInputRef = React.createRef();
          this.state = {
            matchingMessages: [],
            searchValue: "",
            ready: false,
            miniSearch: null,
          };
        }

        componentDidMount() {
          this.loadSearchDataWhenReady();
        }

        loadSearchDataWhenReady() {
          if (window.search_data) {
            this.loadSearchData();
          } else {
            setTimeout(() => {
              this.loadSearchDataWhenReady();
            }, 100);
          }
        }

        loadSearchData() {
          const { channels, users, messages } = window.search_data;

          console.time(`Loading minisearch`);
          const allMessages = [];
          const miniSearch = new MiniSearch({
            idField: "t",
            fields: ["m"], // fields to index for full-text search
            storeFields: ["t", "u", "m", "c"], // fields to return with search results
          });

          for (const channel in messages) {
            for (const message of messages[channel]) {
              allMessages.push({ ...message, c: channel });
            }
          }

          miniSearch.addAll(allMessages);
          console.timeEnd(`Loading minisearch`);

          this.setState({ ready: true, miniSearch });
        }

        handleSearchChange({ target: { value } }) {
          this.setState({ searchValue: value });
          const matchingMessages =
            value.length > 1 ? this.searchMessages(value) : [];
          this.setState({ matchingMessages });
        }

        handleSearchClear() {
          this.setState({ searchValue: "", matchingMessages: [] });
        }

        searchMessages(query) {
          const { miniSearch } = this.state;
          return miniSearch.search(query).slice(0, 50);
        }

        render() {
          const { matchingMessages, searchValue, ready } = this.state;
          return (
            <div className="App">
              <article className="main">
                {ready ? (
                  <Header
                    onChange={this.handleSearchChange}
                    onKeyDown={this.handleSearchKeyDown}
                    onSearchClear={this.handleSearchClear}
                    value={searchValue}
                    searchInputRef={this.searchInputRef}
                  />
                ) : (
                  "Loading"
                )}
                {matchingMessages && matchingMessages.length > 0 ? (
                  <MessagesList messages={matchingMessages} />
                ) : (
                  <p>This search is incredibly basic, but it works.</p>
                )}
              </article>
            </div>
          );
        }
      }

      const MessagesList = ({ messages }) => (
        <ul className="MessagesList">
          {messages.map(({ t, ...props }) => (
            <Message {...props} t={t} key={t} />
          ))}
        </ul>
      );

      const Message = ({ m, u, t, c }) => {
        // Let's find the page
        const { pages } = window.search_data;
        // Returns the index of the first timestamp that's
        // smaller than the timestamp we passed in.
        const channelPages = pages[c] ? pages[c] : null;
        let href;

        if (channelPages) {
          const index = channelPages.findIndex((pageTs) => pageTs < t);
          const page = `${c}-${index}`;
          href = `index.html?c=${encodeURIComponent(page)}&ts=${t}`;
        }

        const message = (
          <li className="Message">
            <p>
              <Channel id={c} /> - <Timestamp timestamp={t} />
            </p>
            <p>
              <User id={u} /> {m}
            </p>
          </li>
        );

        if (href) {
          return (
            <a href={href} target="_blank">
              {message}
            </a>
          );
        } else {
          return message;
        }
      };

      const User = ({ id }) => (
        <strong>@{window.search_data.users[id]}: </strong>
      );

      const Channel = ({ id }) => (
        <span className="Channel">#{window.search_data.channels[id]}</span>
      );

      const Timestamp = ({ timestamp }) => {
        const splitTs = timestamp.split(".") || [];
        const jsTs = parseInt(`${splitTs[0]}${splitTs[1].slice(0, 3)}`, 10);
        const date = new Date(jsTs);

        return (
          <span class="timestamp">
            <span className="c-timestamp__label">{date.toLocaleString()}</span>
          </span>
        );
      };

      const Header = (props) => (
        <header className="Header">
          <h1>Message Search</h1>
          <SearchBox {...props} />
        </header>
      );

      const SearchBox = ({
        onChange,
        onSearchClear,
        value,
        searchInputRef,
      }) => (
        <div className="SearchBox">
          <div className="Search">
            <input
              type="text"
              value={value}
              onChange={onChange}
              ref={searchInputRef}
              autoComplete="none"
              autoCorrect="none"
              autoCapitalize="none"
              spellCheck="false"
            />
            <button
              className="clear"
              onClick={onSearchClear}
              style={{ margin: 10 }}
            >
              &times;
            </button>
          </div>
        </div>
      );

      ReactDOM.render(
        React.createElement(App),
        document.getElementById("search")
      );
    </script>
    <div id="search">
      Loading and indexing messages, please wait. If you have a large number of
      messages, this might take a minute.
      <p>
        <!-- Size -->
      </p>
    </div>
    <script defer src="data/search.js" type="text/javascript"></script>
  </body>
</html>


================================================
FILE: static/style.css
================================================
/* Reset */

/* Box sizing rules */
*,
*::before,
*::after {
  box-sizing: border-box;
}

/* Remove default margin */
body,
h1,
h2,
h3,
h4,
p,
figure,
blockquote,
dl,
dd {
  margin: 0;
}

/* Remove list styles on ul, ol elements with a list role, which suggests default styling will be removed */
ul[role='list'],
ol[role='list'] {
  list-style: none;
}

/* Set core root defaults */
html:focus-within {
  scroll-behavior: smooth;
}

/* Set core body defaults */
body {
  min-height: 100vh;
  text-rendering: optimizeSpeed;
  line-height: 1.5;
}

/* A elements that don't have a class get default styles */
a:not([class]) {
  text-decoration-skip-ink: auto;
}

/* Make images easier to work with */
img,
picture {
  max-width: 100%;
  display: block;
}

/* Inherit fonts for inputs and buttons */
input,
button,
textarea,
select {
  font: inherit;
}

/* Remove all animations, transitions and smooth scroll for people that prefer not to see them */
@media (prefers-reduced-motion: reduce) {
  html:focus-within {
   scroll-behavior: auto;
  }
  
  *,
  *::before,
  *::after {
    animation-duration: 0.01ms !important;
    animation-iteration-count: 1 !important;
    transition-duration: 0.01ms !important;
    scroll-behavior: auto !important;
  }
}

@font-face {
  font-family: "Lato";
  src: url('fonts/Lato-Regular.ttf') format('truetype');
  font-weight: normal;
  font-style: normal;
}

@font-face {
  font-family: "Lato";
  src: url('fonts/Lato-Bold.ttf') format('truetype');
  font-weight: bold;
  font-style: normal;
}

body, html {
  font-family: 'Lato', sans-serif;
  font-size: 14px;
  color: rgb(29, 28, 29);
}

a {
  color: rgb(18, 100, 163);
}

audio, video {
  max-width: 400px;
}

.messages-list {
  padding-bottom: 20px;
}

.messages-list .avatar {
  height: 36px;
  width: 36px;
  border-radius: 7px;
  margin-right: 10px;
  background: #c1c1c1;
}

.message-gutter {
  display: flex;
  margin: 10px;
  scroll-margin-top: 120px;
}

.message-gutter:target {
  background-color: #fafafa;
  border: 2px solid #39113E;
  padding: 10px;
  border-radius: 5px; 
}

.message-gutter div:first-of-type {
  flex-shrink: 0;
}

.message-gutter > .message-gutter {
  /** i.e. replies in thread. Just here to be easily findable */
}

.sender {
  font-weight: 800;
  margin-right: 10px;
}

.timestamp {
  font-weight: 200;
  font-size: 13px;
  color: rgb(97, 96, 97);
}

.header {
  position: sticky;
  background: #fff;
  color: #616061;
  top: 0;
  left: 0;
  padding: 10px;
  min-height: 70px;
  border-bottom: 1px solid #E2E2E2;
  box-sizing: border-box;
}

.header h1 {
  font-size: 16px;
  color: #1D1C1D;
  display: inline-block;
}

.header a {
  color: #616061;
}

.header a:active, .header a.current {
  color: #000;
}

.header .created {
  float: right;
}

.jumper {
  display: inline-block;
}

.jumper a {
  margin: 2px;
}

.text {
  overflow-wrap: break-word;
}

.file {
  max-height: 270px;
  margin-right: 10px;
  margin-top: 10px;
  border-radius: 4px;
  border: 1px solid #80808045;
  outline: none;
}

.reaction {
  background-color: #eaeaea;
  display: inline-block;
  border-radius: 10px;
  font-size: .7em;
  padding-left: 6px;
  padding-right: 6px;
  padding-bottom: 4px;
  margin-right: 5px;
  padding-top: 4px;
}

.reaction img {
  height: 16px;
  width: 16px;
  margin-right: 3px;
  vertical-align: middle;
  display: inline-block;
}

.reaction span {
  position: relative;
  top: 1px;
}

#index {
  display: flex;
  height: calc(100vh - 4px);
}

#channels {
  background: #39113E;
  width: 250px;
  color: #CDC3CE;
  padding-top: 10px;
  overflow: scroll;
  padding-bottom: 20px;
}

#channels ul {
  margin: 0;
  padding: 0;
  list-style: none;
}

#channels p {
  padding-left: 20px;
}

#channels .section {
  font-weight: 800;
  color: #fff;
  margin-top: 10px;
}

#channels .section:first-of-type {
  margin-top: 0;
}

#channels a {
  padding: 5px;
  display: block;
  color: #CDC3CE;
  text-decoration: none;
  padding-left: 20px;
  display: flex;
  max-height: 28px;
  white-space: pre;
  text-overflow: ellipsis;
  overflow: hidden;
}

#channels a .avatar {
  height: 20px;
  width: 20px;
  border-radius: 3px;
  margin-right: 10px;
  object-fit: contain;
}

#channels a:hover {
  background: #301034;
  color: #edeced;
}

#messages {
  flex-grow: 1;
}

#messages iframe {
  height: 100%;
  width: calc(100vw - 250px);
  border: none;
}

#search {
  margin: 10px;
  text-align: center;
}

#search ul {
  list-style: none;
  display: flex;
  flex-direction: column;
  align-items: center;
}

#search li {
  padding: 5px;
  border-bottom: 1px solid #E2E2E2;
  background: hsl(0deg 0% 98%);
  border-radius: 5px;
  width: 600px;
  text-align: left;
  margin-bottom: 5px;
}

#search a {
  text-decoration: none;
  color: unset;
}

================================================
FILE: tsconfig.json
================================================
{
  "compilerOptions": {
    "jsx": "react",
    /* Visit https://aka.ms/tsconfig.json to read more about this file */

    /* Projects */
    // "incremental": true,                              /* Enable incremental compilation */
    // "composite": true,                                /* Enable constraints that allow a TypeScript project to be used with project references. */
    // "tsBuildInfoFile": "./",                          /* Specify the folder for .tsbuildinfo incremental compilation files. */
    // "disableSourceOfProjectReferenceRedirect": true,  /* Disable preferring source files instead of declaration files when referencing composite projects */
    // "disableSolutionSearching": true,                 /* Opt a project out of multi-project reference checking when editing. */
    // "disableReferencedProjectLoad": true,             /* Reduce the number of projects loaded automatically by TypeScript. */

    /* Language and Environment */
    "target": "ES2020",                                     /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
    // "lib": [],                                        /* Specify a set of bundled library declaration files that describe the target runtime environment. */
    // "jsx": "preserve",                                /* Specify what JSX code is generated. */
    // "experimentalDecorators": true,                   /* Enable experimental support for TC39 stage 2 draft decorators. */
    // "emitDecoratorMetadata": true,                    /* Emit design-type metadata for decorated declarations in source files. */
    // "jsxFactory": "",                                 /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h' */
    // "jsxFragmentFactory": "",                         /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
    // "jsxImportSource": "",                            /* Specify module specifier used to import the JSX factory functions when using `jsx: react-jsx*`.` */
    // "reactNamespace": "",                             /* Specify the object invoked for `createElement`. This only applies when targeting `react` JSX emit. */
    // "noLib": true,                                    /* Disable including any library files, including the default lib.d.ts. */
    // "useDefineForClassFields": true,                  /* Emit ECMAScript-standard-compliant class fields. */

    /* Modules */
    "module": "ES2020",                                /* Specify what module code is generated. */
    // "rootDir": "./",                                  /* Specify the root folder within your source files. */
    "moduleResolution": "node",                       /* Specify how TypeScript looks up a file from a given module specifier. */
    // "baseUrl": "./",                                  /* Specify the base directory to resolve non-relative module names. */
    // "paths": {},                                      /* Specify a set of entries that re-map imports to additional lookup locations. */
    // "rootDirs": [],                                   /* Allow multiple folders to be treated as one when resolving modules. */
    // "typeRoots": [],                                  /* Specify multiple folders that act like `./node_modules/@types`. */
    // "types": [],                                      /* Specify type package names to be included without being referenced in a source file. */
    // "allowUmdGlobalAccess": true,                     /* Allow accessing UMD globals from modules. */
    // "resolveJsonModule": true,                        /* Enable importing .json files */
    // "noResolve": true,                                /* Disallow `import`s, `require`s or `<reference>`s from expanding the number of files TypeScript should add to a project. */

    /* JavaScript Support */
    // "allowJs": true,                                  /* Allow JavaScript files to be a part of your program. Use the `checkJS` option to get errors from these files. */
    // "checkJs": true,                                  /* Enable error reporting in type-checked JavaScript files. */
    // "maxNodeModuleJsDepth": 1,                        /* Specify the maximum folder depth used for checking JavaScript files from `node_modules`. Only applicable with `allowJs`. */

    /* Emit */
    // "declaration": true,                              /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
    // "declarationMap": true,                           /* Create sourcemaps for d.ts files. */
    // "emitDeclarationOnly": true,                      /* Only output d.ts files and not JavaScript files. */
    // "sourceMap": true,                                /* Create source map files for emitted JavaScript files. */
    // "outFile": "./",                                  /* Specify a file that bundles all outputs into one JavaScript file. If `declaration` is true, also designates a file that bundles all .d.ts output. */
    "outDir": "./lib",                                   /* Specify an output folder for all emitted files. */
    // "removeComments": true,                           /* Disable emitting comments. */
    // "noEmit": true,                                   /* Disable emitting files from a compilation. */
    // "importHelpers": true,                            /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
    // "importsNotUsedAsValues": "remove",               /* Specify emit/checking behavior for imports that are only used for types */
    // "downlevelIteration": true,                       /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
    // "sourceRoot": "",                                 /* Specify the root path for debuggers to find the reference source code. */
    // "mapRoot": "",                                    /* Specify the location where debugger should locate map files instead of generated locations. */
    // "inlineSourceMap": true,                          /* Include sourcemap files inside the emitted JavaScript. */
    // "inlineSources": true,                            /* Include source code in the sourcemaps inside the emitted JavaScript. */
    // "emitBOM": true,                                  /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
    // "newLine": "crlf",                                /* Set the newline character for emitting files. */
    // "stripInternal": true,                            /* Disable emitting declarations that have `@internal` in their JSDoc comments. */
    // "noEmitHelpers": true,                            /* Disable generating custom helper functions like `__extends` in compiled output. */
    // "noEmitOnError": true,                            /* Disable emitting files if any type checking errors are reported. */
    // "preserveConstEnums": true,                       /* Disable erasing `const enum` declarations in generated code. */
    // "declarationDir": "./",                           /* Specify the output directory for generated declaration files. */

    /* Interop Constraints */
    // "isolatedModules": true,                          /* Ensure that each file can be safely transpiled without relying on other imports. */
    "allowSyntheticDefaultImports": true,                /* Allow 'import x from y' when a module doesn't have a default export. */
    "esModuleInterop": true,                             /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables `allowSyntheticDefaultImports` for type compatibility. */
    // "preserveSymlinks": true,                         /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
    "forceConsistentCasingInFileNames": true,            /* Ensure that casing is correct in imports. */

    /* Type Checking */
    "strict": true,                                      /* Enable all strict type-checking options. */
    // "noImplicitAny": true,                            /* Enable error reporting for expressions and declarations with an implied `any` type.. */
    // "strictNullChecks": true,                         /* When type checking, take into account `null` and `undefined`. */
    // "strictFunctionTypes": true,                      /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
    // "strictBindCallApply": true,                      /* Check that the arguments for `bind`, `call`, and `apply` methods match the original function. */
    // "strictPropertyInitialization": true,             /* Check for class properties that are declared but not set in the constructor. */
    // "noImplicitThis": true,                           /* Enable error reporting when `this` is given the type `any`. */
    // "useUnknownInCatchVariables": true,               /* Type catch clause variables as 'unknown' instead of 'any'. */
    // "alwaysStrict": true,                             /* Ensure 'use strict' is always emitted. */
    // "noUnusedLocals": true,                           /* Enable error reporting when a local variables aren't read. */
    // "noUnusedParameters": true,                       /* Raise an error when a function parameter isn't read */
    // "exactOptionalPropertyTypes": true,               /* Interpret optional property types as written, rather than adding 'undefined'. */
    // "noImplicitReturns": true,                        /* Enable error reporting for codepaths that do not explicitly return in a function. */
    // "noFallthroughCasesInSwitch": true,               /* Enable error reporting for fallthrough cases in switch statements. */
    // "noUncheckedIndexedAccess": true,                 /* Include 'undefined' in index signature results */
    // "noImplicitOverride": true,                       /* Ensure overriding members in derived classes are marked with an override modifier. */
    // "noPropertyAccessFromIndexSignature": true,       /* Enforces using indexed accessors for keys declared using an indexed type */
    // "allowUnusedLabels": true,                        /* Disable error reporting for unused labels. */
    // "allowUnreachableCode": true,                     /* Disable error reporting for unreachable code. */

    /* Completeness */
    // "skipDefaultLibCheck": true,                      /* Skip type checking .d.ts files that are included with TypeScript. */
    "skipLibCheck": true                                 /* Skip type checking all .d.ts files. */
  },
  "ts-node": {
    "esm": true,
    "transpileOnly": true
  },
}