Repository: felixrieseberg/slack-archive
Branch: main
Commit: 4ce7b36fe08f
Files: 30
Total size: 87.7 KB
Directory structure:
gitextract_oionf7w6/
├── .gitignore
├── .node-version
├── .npmignore
├── README.md
├── bin/
│ └── slack-archive.js
├── package.json
├── src/
│ ├── ambient.d.ts
│ ├── archive-data.ts
│ ├── backup.ts
│ ├── channels.ts
│ ├── cli.ts
│ ├── config.ts
│ ├── create-html.tsx
│ ├── data-load.ts
│ ├── data-write.ts
│ ├── download-files.ts
│ ├── emoji.ts
│ ├── interfaces.ts
│ ├── messages.ts
│ ├── reactions.ts
│ ├── retry.ts
│ ├── search.ts
│ ├── threads.ts
│ ├── timestamp.ts
│ ├── users.ts
│ └── web-client.ts
├── static/
│ ├── scroll.js
│ ├── search.html
│ └── style.css
└── tsconfig.json
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
node_modules
out
slack-archive
.DS_Store
*.log
.token
lib
================================================
FILE: .node-version
================================================
16.4.0
================================================
FILE: .npmignore
================================================
node_modules
out
slack-archive
.DS_Store
*.log
.token
src
================================================
FILE: README.md
================================================
# Export your Slack workspace as static HTML
Alright, so you want to export all your messages on Slack. You want them in a format that you
can still enjoy in 20 years. This tool will help you do that.
* **Completely static**: The generated files are pure HTML and will still work in 50 years.
* **Everything you care about**: This tool downloads messages, files, and avatars.
* **Nothing you do not care about**: Choose exactly which channels and DMs to download.
* **All types of conversations**: We'll fetch public channels, private channels, DMs, and multi-person DMs.
* **Incremental backups**: If you already have local data, we'll extend it - no need to download existing stuff again.
* **JSON included**: All data is also stored as JSON, so you can consume it with other tools later.
* **No cloud, free**: Do all of this for free, without giving anyone your information.
* **Basic search**: Offers basic search functionality.
<img width="1151" alt="Screen Shot 2021-09-09 at 6 43 55 PM" src="https://user-images.githubusercontent.com/1426799/132776566-0f75a1b4-4b9a-4b53-8a39-e44e8a747a68.png">
## Using it
1. Do you already have a user token for your workspace? If not, read on below on how to get a token.
2. Make sure you have [`node` and `npm`](https://nodejs.org/en/) installed, ideally something newer than Node v14.
3. Run `slack-archive`, which will interactively guide you through the options.
```sh
npx slack-archive
```
### Parameters
```
--automatic: Don't prompt and automatically fetch all messages from all channels.
--use-previous-channel-config: Fetch messages from channels selected in previous run instead of prompting.
--channel-types Comma-separated list of channel types to fetch messages from.
(public_channel, private_channel, mpim, im)
--exclude-channels Comma-separated list of channels to exclude, in automatic mode
--no-backup: Don't create backups. Not recommended.
--no-search: Don't create a search file, saving disk space.
--no-file-download: Don't download files.
--no-slack-connect: Don't connect to Slack, just generate HTML from local data.
--force-html-generation: Force regeneration of HTML files. Useful after slack-archive upgrades.
```
## Getting a token
In order to download messages from private channels and direct messages, we will need a "user
token". Slack uses the token to identify what permissions it'll give this app. We used to be able
to just copy a token out of your Slack app, but now, we'll need to create a custom app and jump
through a few hoops.
This will be mostly painless, I promise.
### 1) Make a custom app
Head over to https://api.slack.com/apps and `Create New App`. Select `From scratch`.
Give it a name and choose the workspace you'd like to export.
Then, from the `Features` menu on the left, select `OAuth & Permission`.
As a redirect URL, enter something random that doesn't actually exist, or a domain you control. For instace:
```
https://notarealurl.com/
```
(Note that redirects will take a _very_ long time if using a domain that doesn't actually exist)
Then, add the following `User Token Scopes`:
* channels:history
* channels:read
* files:read
* groups:history
* groups:read
* im:history
* im:read
* mpim:history
* mpim:read
* remote_files:read
* users:read
Finally, head back to `Basic Information` and make a note of your app's `client
id` and `client secret`. We'll need both later.
### 2) Authorize
Make sure you have your Slack workspace `URL` (aka team name) and your app's `client id`.
Then, in a browser, open this URL - replacing `{your-team-name}` and `{your-client-id}`
with your values.
```
https://{your-team-name}.slack.com/oauth/authorize?client_id={your-client-id}&scope=client
```
Confirm everything until Slack sends you to the mentioned non-existent URL. Look at your
browser's address bar - it should contain an URL that looks like this:
```
https://notarealurl.com/?code={code}&state=
```
Copy everything between `?code=` and `&state`. This is your `code`. We'll need it in the
next step.
Next, we'll exchange your code for a token. To do so, we'll also need your `client secret`
from the first step when we created your app. In a browser, open this URL - replacing
`{your-team-name}`, `{your-client-id}`, `{your-code}` and `{your-client-secret}` with
your values.
```
https://{your-team-name}.slack.com/api/oauth.access?client_id={your-client-id}&client_secret={your-client-secret}&code={your-code}
```
Your browser should now be returning some JSON including a token. Make a note of it - that's what we'll use. Paste it in the command line, OR create a file called `.token` in the slack-archive directory (created when the command is first run) and paste it in there.
================================================
FILE: bin/slack-archive.js
================================================
#!/usr/bin/env node
import('../lib/cli.js')
================================================
FILE: package.json
================================================
{
"name": "slack-archive",
"version": "1.6.1",
"description": "Create static HTML archives for your Slack workspaces",
"scripts": {
"prettier": "npx prettier --write src/*",
"cli": "ts-node src/cli.ts",
"html": "ts-node src/create-html.tsx",
"test": "echo \"Error: no test specified\" && exit 1",
"compile": "tsc",
"watch": "tsc -w",
"prepublishOnly": "npm run compile"
},
"bin": {
"slack-archive": "./bin/slack-archive.js"
},
"type": "module",
"keywords": [
"slack",
"export",
"download"
],
"repository": {
"type": "git",
"url": "https://github.com/felixrieseberg/slack-archive.git"
},
"author": "Felix Rieseberg <felix@felixrieseberg.com>",
"license": "MIT",
"dependencies": {
"@slack/web-api": "^6.7.2",
"date-fns": "^2.28.0",
"emoji-datasource": "^14.0.0",
"es-main": "^1.0.2",
"fs-extra": "^10.1.0",
"inquirer": "^8.2.0",
"lodash-es": "^4.17.21",
"node-fetch": "^2.6.7",
"ora": "^6.1.0",
"react": "^17.0.2",
"react-dom": "^17.0.2",
"rimraf": "^5.0.5",
"slack-markdown": "^0.2.0",
"trash": "^8.1.0"
},
"devDependencies": {
"@types/date-fns": "^2.6.0",
"@types/fs-extra": "^9.0.13",
"@types/inquirer": "^8.1.3",
"@types/lodash-es": "^4.17.5",
"@types/node": "^17.0.5",
"@types/node-fetch": "^2.5.12",
"@types/react": "^17.0.38",
"@types/react-dom": "^17.0.11",
"ts-node": "^10.8.1",
"tslib": "^2.4.0",
"typescript": "^4.7.4"
},
"ts-node": {
"files": true
}
}
================================================
FILE: src/ambient.d.ts
================================================
declare module "slack-markdown";
declare module "es-main";
declare module "emoji-datasource";
================================================
FILE: src/archive-data.ts
================================================
import fs from "fs-extra";
import { SLACK_ARCHIVE_DATA_PATH } from "./config.js";
import { readJSON } from "./data-load.js";
import { write } from "./data-write.js";
import { SlackArchiveData, User } from "./interfaces.js";
export async function getSlackArchiveData(): Promise<SlackArchiveData> {
const returnIfEmpty: SlackArchiveData = { channels: {} };
if (!fs.existsSync(SLACK_ARCHIVE_DATA_PATH)) {
return returnIfEmpty;
}
const result = await readJSON<SlackArchiveData>(SLACK_ARCHIVE_DATA_PATH);
const merged = { channels: result.channels || {}, auth: result.auth };
return merged;
}
export async function setSlackArchiveData(
newData: SlackArchiveData
): Promise<void> {
const oldData = await getSlackArchiveData();
const dataToWrite = {
channels: { ...oldData.channels, ...newData.channels },
auth: newData.auth,
};
return write(
SLACK_ARCHIVE_DATA_PATH,
JSON.stringify(dataToWrite, undefined, 2)
);
}
================================================
FILE: src/backup.ts
================================================
import fs from "fs-extra";
import inquirer from "inquirer";
import path from "path";
import trash from "trash";
import { rimraf } from "rimraf";
import { AUTOMATIC_MODE, DATA_DIR, NO_BACKUP, OUT_DIR } from "./config.js";
const { prompt } = inquirer;
let backupDir = `${DATA_DIR}_backup_${Date.now()}`;
export async function createBackup() {
if (NO_BACKUP || !fs.existsSync(DATA_DIR)) {
return;
}
const hasFiles = fs.readdirSync(DATA_DIR);
if (hasFiles.length === 0) {
return;
}
console.log(`Existing data directory found. Creating backup: ${backupDir}`);
await fs.copy(DATA_DIR, backupDir);
console.log(`Backup created.\n`);
}
export async function deleteBackup() {
if (!fs.existsSync(backupDir)) {
return;
}
console.log(
`Cleaning up backup: If anything went wrong, you'll find it in your system's trash.`
);
try {
// NB: trash doesn't work on many Linux distros
await trash(backupDir);
return;
} catch (error) {
console.log('Moving backup to trash failed.');
}
if (!process.env['TRASH_HARDER']) {
console.log(`Set TRASH_HARDER=1 to delete files permanently.`);
return;
}
try {
await rimraf(backupDir);
} catch (error) {
console.log(`Deleting backup permanently failed. Aborting here.`);
}
}
export async function deleteOlderBackups() {
try {
const oldBackupNames: Array<string> = [];
const oldBackupPaths: Array<string> = [];
for (const entry of fs.readdirSync(OUT_DIR)) {
const isBackup = entry.startsWith("data_backup_");
if (!isBackup) continue;
const dir = path.join(OUT_DIR, entry);
const { isDirectory } = fs.statSync(dir);
if (!isDirectory) continue;
oldBackupPaths.push(dir);
oldBackupNames.push(entry);
}
if (oldBackupPaths.length === 0) return;
if (AUTOMATIC_MODE) {
console.log(
`Found existing older backups, but in automatic mode: Proceeding without deleting them.`
);
return;
}
const { del } = await prompt([
{
type: "confirm",
default: true,
name: "del",
message: `We've found existing backups (${oldBackupNames.join(
", "
)}). Do you want to delete them?`,
},
]);
if (del) {
oldBackupPaths.forEach((v) => fs.removeSync(v));
}
} catch (error) {
// noop
}
}
================================================
FILE: src/channels.ts
================================================
import {
ConversationsListArguments,
ConversationsListResponse,
} from "@slack/web-api";
import ora from "ora";
import { NO_SLACK_CONNECT } from "./config.js";
import { Channel, Users } from "./interfaces.js";
import { downloadUser, getName } from "./users.js";
import { getWebClient } from "./web-client.js";
export function getChannelName(channel: Channel) {
return (
channel.name || channel.id || channel.purpose?.value || "Unknown channel"
);
}
export function isPublicChannel(channel: Channel) {
return !channel.is_private && !channel.is_mpim && !channel.is_im;
}
export function isPrivateChannel(channel: Channel) {
return channel.is_private && !channel.is_im && !channel.is_mpim;
}
export function isDmChannel(channel: Channel, users: Users) {
return channel.is_im && channel.user && !users[channel.user]?.is_bot;
}
export function isBotChannel(channel: Channel, users: Users) {
return channel.user && users[channel.user]?.is_bot;
}
function isChannels(input: any): input is ConversationsListResponse {
return !!input.channels;
}
export async function downloadChannels(
options: ConversationsListArguments,
users: Users
): Promise<Array<Channel>> {
const channels: Array<Channel> = [];
if (NO_SLACK_CONNECT) {
return channels;
}
const spinner = ora("Downloading channels").start();
for await (const page of getWebClient().paginate(
"conversations.list",
options
)) {
if (isChannels(page)) {
spinner.text = `Found ${page.channels?.length} channels (found so far: ${
channels.length + (page.channels?.length || 0)
})`;
const pageChannels = (page.channels || []).filter((c) => !!c.id);
for (const channel of pageChannels) {
if (channel.is_im) {
const user = await downloadUser(channel, users);
channel.name =
channel.name || `${getName(user?.id, users)} (${user?.name})`;
}
if (channel.is_mpim) {
channel.name = channel.purpose?.value;
}
}
channels.push(...pageChannels);
}
}
spinner.succeed(`Found ${channels.length} channels`);
return channels;
}
================================================
FILE: src/cli.ts
================================================
import { uniqBy } from "lodash-es";
import inquirer from "inquirer";
import fs from "fs-extra";
import { User } from "@slack/web-api/dist/response/UsersInfoResponse";
import { Channel } from "@slack/web-api/dist/response/ConversationsListResponse";
import ora from "ora";
import {
CHANNELS_DATA_PATH,
USERS_DATA_PATH,
getChannelDataFilePath,
OUT_DIR,
config,
TOKEN_FILE,
AUTOMATIC_MODE,
USE_PREVIOUS_CHANNEL_CONFIG,
CHANNEL_TYPES,
DATE_FILE,
EMOJIS_DATA_PATH,
NO_SLACK_CONNECT,
EXCLUDE_CHANNELS,
} from "./config.js";
import { downloadExtras } from "./messages.js";
import { downloadMessages } from "./messages.js";
import { downloadFilesForChannel } from "./download-files.js";
import {
createHtmlForChannels,
getChannelsToCreateFilesFor,
} from "./create-html.js";
import { createBackup, deleteBackup, deleteOlderBackups } from "./backup.js";
import { isValid, parseISO } from "date-fns";
import { createSearch } from "./search.js";
import { write, writeAndMerge } from "./data-write.js";
import { messagesCache, getUsers, getChannels } from "./data-load.js";
import { getSlackArchiveData, setSlackArchiveData } from "./archive-data.js";
import { downloadEmojiList, downloadEmojis } from "./emoji.js";
import { downloadAvatars } from "./users.js";
import { downloadChannels } from "./channels.js";
import { authTest } from "./web-client.js";
import { SlackArchiveChannelData } from "./interfaces.js";
const { prompt } = inquirer;
async function selectMergeFiles(): Promise<boolean> {
const defaultResponse = true;
if (!fs.existsSync(CHANNELS_DATA_PATH)) {
return false;
}
// We didn't download any data. Merge.
if (AUTOMATIC_MODE || NO_SLACK_CONNECT) {
return defaultResponse;
}
const { merge } = await prompt([
{
type: "confirm",
default: defaultResponse,
name: "merge",
message: `We've found existing archive files. Do you want to append new data (recommended)? \n If you select "No", we'll delete the existing data.`,
},
]);
if (!merge) {
fs.emptyDirSync(OUT_DIR);
}
return merge;
}
async function selectChannels(
channels: Array<Channel>,
previouslyDownloadedChannels: Record<string, SlackArchiveChannelData>
): Promise<Array<Channel>> {
if (USE_PREVIOUS_CHANNEL_CONFIG) {
const selectedChannels: Array<Channel> = channels.filter(
(channel) => channel.id && channel.id in previouslyDownloadedChannels
);
const selectedChannelNames = selectedChannels.map(
(channel) => channel.name || channel.id || "Unknown"
);
console.log(
`Downloading channels selected previously: ${selectedChannelNames}.`
);
const previousChannelIds = Object.keys(previouslyDownloadedChannels);
if (previousChannelIds.length != selectedChannels.length) {
console.warn(
"WARNING: Did not find all previously selected channel IDs."
);
console.log(
`Expected to find ${previousChannelIds.length} channels, but only ${selectedChannels.length} matched.`
);
// Consider Looking up the user-facing names of the missing channels in the saved data.
const availableChannelIds = new Set<string>(
channels.map((channel) => channel.id || "")
);
const missingChannelIds = previousChannelIds.filter(
(cId) => !availableChannelIds.has(cId)
);
//console.log(availableChannelIds);
console.log(`Missing channel ids: ${missingChannelIds}`);
} else {
console.log(
`Matched all ${previousChannelIds.length} previously selected channels out of ${channels.length} total channels available.`
);
}
return selectedChannels;
}
const choices = channels.map((channel) => ({
name: channel.name || channel.id || "Unknown",
value: channel,
}));
if (AUTOMATIC_MODE || NO_SLACK_CONNECT) {
if (EXCLUDE_CHANNELS) {
const excludeChannels = EXCLUDE_CHANNELS.split(',');
return channels.filter((channel) => !excludeChannels.includes(channel.name || ''));
}
return channels;
}
const result = await prompt([
{
type: "checkbox",
loop: true,
name: "channels",
message: "Which channels do you want to download?",
choices,
},
]);
return result.channels;
}
async function selectChannelTypes(): Promise<Array<string>> {
const choices = [
{
name: "Public Channels",
value: "public_channel",
},
{
name: "Private Channels",
value: "private_channel",
},
{
name: "Multi-Person Direct Message",
value: "mpim",
},
{
name: "Direct Messages",
value: "im",
},
];
if (CHANNEL_TYPES) {
return CHANNEL_TYPES.split(",");
}
if (AUTOMATIC_MODE || USE_PREVIOUS_CHANNEL_CONFIG || NO_SLACK_CONNECT) {
return ["public_channel", "private_channel", "mpim", "im"];
}
const result = await prompt([
{
type: "checkbox",
loop: true,
name: "channel-types",
message: `Which channel types do you want to download?`,
choices,
},
]);
return result["channel-types"];
}
async function getToken() {
if (NO_SLACK_CONNECT) {
return;
}
if (config.token) {
console.log(`Using token ${config.token}`);
return;
}
if (fs.existsSync(TOKEN_FILE)) {
config.token = fs.readFileSync(TOKEN_FILE, "utf-8").trim();
return;
}
const result = await prompt([
{
name: "token",
type: "input",
message:
"Please enter your Slack token (xoxp-...). See README for more details.",
},
]);
config.token = result.token;
}
async function writeLastSuccessfulArchive() {
const now = new Date();
write(DATE_FILE, now.toISOString());
}
function getLastSuccessfulRun() {
if (!fs.existsSync(DATE_FILE)) {
return "";
}
const lastSuccessfulArchive = fs.readFileSync(DATE_FILE, "utf-8");
let date = null;
try {
date = parseISO(lastSuccessfulArchive);
} catch (error) {
return "";
}
if (date && isValid(date)) {
return `. Last successful run: ${date.toLocaleString()}`;
}
return "";
}
async function getAuthTest() {
if (NO_SLACK_CONNECT) {
return;
}
const spinner = ora("Testing authentication with Slack...").start();
const result = await authTest();
if (!result.ok) {
spinner.fail(`Authentication with Slack failed.`);
console.log(
`Authentication with Slack failed. The error was: ${result.error}`
);
console.log(
`The provided token was ${config.token}. Double-check the token and try again.`
);
console.log(
`For more information on the error code, see the error table at https://api.slack.com/methods/auth.test`
);
console.log(`This tool will now exit.`);
await deleteBackup();
process.exit(-1);
} else {
spinner.succeed(`Successfully authorized with Slack as ${result.user}\n`);
}
return result;
}
export async function main() {
console.log(`Welcome to slack-archive${getLastSuccessfulRun()}`);
if (AUTOMATIC_MODE) {
console.log(`Running in fully automatic mode without prompts`);
}
if (NO_SLACK_CONNECT) {
console.log(`Not connecting to Slack and skipping all Slack API calls`);
}
await getToken();
await createBackup();
const slackArchiveData = await getSlackArchiveData();
const users: Record<string, User> = await getUsers();
const channelTypes = (await selectChannelTypes()).join(",");
slackArchiveData.auth = await getAuthTest();
const channels = await downloadChannels({ types: channelTypes }, users);
const selectedChannels = await selectChannels(
channels,
slackArchiveData.channels
);
const newMessages: Record<string, number> = {};
// Emoji
// We don't actually download the images here, we'll
// do that as needed
const emojis = await downloadEmojiList();
await writeAndMerge(EMOJIS_DATA_PATH, emojis);
// Do we want to merge data?
await selectMergeFiles();
await writeAndMerge(CHANNELS_DATA_PATH, selectedChannels);
// Download messages and extras for each channel
await downloadEachChannel();
// Save data
await setSlackArchiveData(slackArchiveData);
// Create HTML, but only for channels with new messages
// - or channels that we didn't make HTML for yet
const channelsToCreateFilesFor = await getChannelsToCreateFilesFor(
selectedChannels,
newMessages
);
await createHtmlForChannels(channelsToCreateFilesFor);
// Create search file
await createSearch();
// Cleanup and finalize
await deleteBackup();
await deleteOlderBackups();
await writeLastSuccessfulArchive();
console.log(`All done.`);
async function downloadEachChannel() {
if (NO_SLACK_CONNECT) return;
for (const [i, channel] of selectedChannels.entries()) {
if (!channel.id) {
console.warn(`Selected channel does not have an id`, channel);
continue;
}
// Do we already have everything?
slackArchiveData.channels[channel.id] =
slackArchiveData.channels[channel.id] || {};
if (slackArchiveData.channels[channel.id].fullyDownloaded) {
continue;
}
// Download messages & users
let downloadData = await downloadMessages(
channel,
i,
selectedChannels.length
);
let result = downloadData.messages;
newMessages[channel.id] = downloadData.new;
await downloadExtras(channel, result, users);
await downloadEmojis(result, emojis);
await downloadAvatars();
// Sort messages
const spinner = ora(
`Saving message data for ${channel.name || channel.id} to disk`
).start();
spinner.render();
result = uniqBy(result, "ts");
result = result.sort((a, b) => {
return parseFloat(b.ts || "0") - parseFloat(a.ts || "0");
});
await writeAndMerge(USERS_DATA_PATH, users);
fs.outputFileSync(
getChannelDataFilePath(channel.id),
JSON.stringify(result, undefined, 2)
);
// Download files. This needs to run after the messages are saved to disk
// since it uses the message data to find which files to download.
await downloadFilesForChannel(channel.id!, spinner);
// Update the data load cache
messagesCache[channel.id!] = result;
// Update the data
const { is_archived, is_im, is_user_deleted } = channel;
if (is_archived || (is_im && is_user_deleted)) {
slackArchiveData.channels[channel.id].fullyDownloaded = true;
}
slackArchiveData.channels[channel.id].messages = result.length;
spinner.succeed(`Saved message data for ${channel.name || channel.id}`);
}
}
}
main();
================================================
FILE: src/config.ts
================================================
import path from "path";
import { dirname } from "path";
import { fileURLToPath } from "url";
const __dirname = dirname(fileURLToPath(import.meta.url));
export const config = {
token: process.env.SLACK_TOKEN,
};
function findCliParameter(param: string) {
const args = process.argv;
for (const arg of args) {
if (arg === param) {
return true;
}
}
return false;
}
function getCliParameter(param: string) {
const args = process.argv;
for (const [i, arg] of args.entries()) {
if (arg === param) {
return args[i + 1];
}
}
return null;
}
export const AUTOMATIC_MODE = findCliParameter("--automatic");
export const USE_PREVIOUS_CHANNEL_CONFIG = findCliParameter(
"--use-previous-channel-config"
);
export const CHANNEL_TYPES = getCliParameter("--channel-types");
export const NO_BACKUP = findCliParameter("--no-backup");
export const NO_SEARCH = findCliParameter("--no-search");
export const NO_FILE_DOWNLOAD = findCliParameter("--no-file-download");
export const NO_SLACK_CONNECT = findCliParameter("--no-slack-connect");
export const FORCE_HTML_GENERATION = findCliParameter(
"--force-html-generation"
);
export const EXCLUDE_CHANNELS = getCliParameter("--exclude-channels");
export const BASE_DIR = process.cwd();
export const OUT_DIR = path.join(BASE_DIR, "slack-archive");
export const TOKEN_FILE = path.join(OUT_DIR, ".token");
export const DATE_FILE = path.join(OUT_DIR, ".last-successful-run");
export const DATA_DIR = path.join(OUT_DIR, "data");
export const HTML_DIR = path.join(OUT_DIR, "html");
export const FILES_DIR = path.join(HTML_DIR, "files");
export const AVATARS_DIR = path.join(HTML_DIR, "avatars");
export const EMOJIS_DIR = path.join(HTML_DIR, "emojis");
export const INDEX_PATH = path.join(OUT_DIR, "index.html");
export const SEARCH_PATH = path.join(OUT_DIR, "search.html");
export const MESSAGES_JS_PATH = path.join(__dirname, "../static/scroll.js");
export const SEARCH_TEMPLATE_PATH = path.join(
__dirname,
"../static/search.html"
);
export const CHANNELS_DATA_PATH = path.join(DATA_DIR, "channels.json");
export const USERS_DATA_PATH = path.join(DATA_DIR, "users.json");
export const EMOJIS_DATA_PATH = path.join(DATA_DIR, "emojis.json");
export const SLACK_ARCHIVE_DATA_PATH = path.join(
DATA_DIR,
"slack-archive.json"
);
export const SEARCH_DATA_PATH = path.join(DATA_DIR, "search.js");
export function getChannelDataFilePath(channelId: string) {
return path.join(DATA_DIR, `${channelId}.json`);
}
export function getChannelUploadFilePath(channelId: string, fileName: string) {
return path.join(FILES_DIR, channelId, fileName);
}
export function getHTMLFilePath(channelId: string, index: number) {
return path.join(HTML_DIR, `${channelId}-${index}.html`);
}
export function getAvatarFilePath(userId: string, extension: string) {
return path.join(AVATARS_DIR, `${userId}${extension}`);
}
================================================
FILE: src/create-html.tsx
================================================
import { format } from "date-fns";
import fs from "fs-extra";
import path from "path";
import React from "react";
import ReactDOMServer from "react-dom/server.js";
import ora, { Ora } from "ora";
import { chunk, sortBy } from "lodash-es";
import { dirname } from "path";
import { fileURLToPath } from "url";
import esMain from "es-main";
import slackMarkdown from "slack-markdown";
import { getChannels, getMessages, getUsers } from "./data-load.js";
import {
ArchiveMessage,
Channel,
ChunksInfo,
Message,
Reaction,
SlackArchiveData,
User,
Users,
} from "./interfaces.js";
import {
getHTMLFilePath,
INDEX_PATH,
OUT_DIR,
MESSAGES_JS_PATH,
FORCE_HTML_GENERATION,
} from "./config.js";
import { slackTimestampToJavaScriptTimestamp } from "./timestamp.js";
import { recordPage } from "./search.js";
import { write } from "./data-write.js";
import { getSlackArchiveData } from "./archive-data.js";
import { getEmojiFilePath, getEmojiUnicode, isEmojiUnicode } from "./emoji.js";
import { getName } from "./users.js";
import {
isBotChannel,
isDmChannel,
isPrivateChannel,
isPublicChannel,
} from "./channels.js";
const _dirname = dirname(fileURLToPath(import.meta.url));
const MESSAGE_CHUNK = 1000;
// This used to be a prop on the components, but passing it around
// was surprisingly slow. Global variables are cool again!
// Set by createHtmlForChannels().
let users: Users = {};
let slackArchiveData: SlackArchiveData = { channels: {} };
let me: User | null;
// Little hack to switch between ./index.html and ./html/...
let base = "";
function formatTimestamp(message: Message, dateFormat = "PPPPpppp") {
const jsTs = slackTimestampToJavaScriptTimestamp(message.ts);
const ts = format(jsTs, dateFormat);
return ts;
}
interface FilesProps {
message: Message;
channelId: string;
}
const Files: React.FunctionComponent<FilesProps> = (props) => {
const { message, channelId } = props;
const { files } = message;
if (!files || files.length === 0) return null;
const fileElements = files.map((file) => {
const { thumb_1024, thumb_720, thumb_480, thumb_pdf } = file as any;
const thumb = thumb_1024 || thumb_720 || thumb_480 || thumb_pdf;
let src = `files/${channelId}/${file.id}.${file.filetype}`;
let href = src;
if (file.mimetype?.startsWith("image")) {
return (
<a key={file.id} href={href} target="_blank">
<img className="file" src={src} />
</a>
);
}
if (file.mimetype?.startsWith("video")) {
return <video key={file.id} controls src={src} />;
}
if (file.mimetype?.startsWith("audio")) {
return <audio key={file.id} controls src={src} />;
}
if (!file.mimetype?.startsWith("image") && thumb) {
href = file.url_private || href;
src = src.replace(`.${file.filetype}`, ".png");
return (
<a key={file.id} href={href} target="_blank">
<img className="file" src={src} />
</a>
);
}
return (
<a key={file.id} href={href} target="_blank">
{file.name}
</a>
);
});
return <div className="files">{...fileElements}</div>;
};
interface AvatarProps {
userId?: string;
}
const Avatar: React.FunctionComponent<AvatarProps> = ({ userId }) => {
if (!userId) return null;
const user = users[userId];
if (!user || !user.profile || !user.profile.image_512) return null;
const ext = path.extname(user?.profile?.image_512!);
const src = `${base}avatars/${userId}${ext}`;
return <img className="avatar" src={src} />;
};
interface ParentMessageProps {
message: ArchiveMessage;
channelId: string;
}
const ParentMessage: React.FunctionComponent<ParentMessageProps> = (props) => {
const { message, channelId } = props;
const hasFiles = !!message.files;
return (
<Message message={message} channelId={channelId}>
{hasFiles ? <Files message={message} channelId={channelId} /> : null}
{message.reactions?.map((reaction) => (
<Reaction key={reaction.name} reaction={reaction} />
))}
{message.replies?.map((reply) => (
<ParentMessage message={reply} channelId={channelId} key={reply.ts} />
))}
</Message>
);
};
interface ReactionProps {
reaction: Reaction;
}
const Reaction: React.FunctionComponent<ReactionProps> = ({ reaction }) => {
const reactors = [];
if (reaction.users) {
for (const userId of reaction.users) {
reactors.push(getName(userId, users));
}
}
return (
<div className="reaction" title={reactors.join(", ")}>
<Emoji name={reaction.name!} />
<span>{reaction.count}</span>
</div>
);
};
interface EmojiProps {
name: string;
}
const Emoji: React.FunctionComponent<EmojiProps> = ({ name }) => {
if (isEmojiUnicode(name)) {
return <>{getEmojiUnicode(name)}</>;
}
return <img src={getEmojiFilePath(name)} />;
};
interface MessageProps {
message: ArchiveMessage;
channelId: string;
}
const Message: React.FunctionComponent<MessageProps> = (props) => {
const { message } = props;
const username = getName(message.user, users);
const slackCallbacks = {
user: ({ id }: { id: string }) => `@${getName(id, users)}`,
};
return (
<div className="message-gutter" id={message.ts}>
<div className="" data-stringify-ignore="true">
<Avatar userId={message.user} />
</div>
<div className="">
<span className="sender">{username}</span>
<span className="timestamp">
<span className="c-timestamp__label">{formatTimestamp(message)}</span>
</span>
<br />
<div
className="text"
dangerouslySetInnerHTML={{
__html: slackMarkdown.toHTML(message.text, {
escapeHTML: false,
slackCallbacks,
}),
}}
/>
{props.children}
</div>
</div>
);
};
interface MessagesPageProps {
messages: Array<ArchiveMessage>;
channel: Channel;
index: number;
chunksInfo: ChunksInfo;
}
const MessagesPage: React.FunctionComponent<MessagesPageProps> = (props) => {
const { channel, index, chunksInfo } = props;
const messagesJs = fs.readFileSync(MESSAGES_JS_PATH, "utf8");
// Newest message is first
const messages = props.messages
.map((m) => (
<ParentMessage key={m.ts} message={m} channelId={channel.id!} />
))
.reverse();
if (messages.length === 0) {
messages.push(<span key="empty">No messages were ever sent!</span>);
}
return (
<HtmlPage>
<div style={{ paddingLeft: 10 }}>
<Header index={index} chunksInfo={chunksInfo} channel={channel} />
<div className="messages-list">{messages}</div>
<script dangerouslySetInnerHTML={{ __html: messagesJs }} />
</div>
</HtmlPage>
);
};
interface ChannelLinkProps {
channel: Channel;
}
const ChannelLink: React.FunctionComponent<ChannelLinkProps> = ({
channel,
}) => {
let name = channel.name || channel.id;
let leadSymbol = <span># </span>;
const channelData = slackArchiveData.channels[channel.id!];
if (channelData && channelData.messages === 0) {
return null;
}
// Remove the user's name from the group mpdm channel name
if (me && channel.is_mpim) {
name = name?.replace(`@${me.name}`, "").replace(" ", " ");
}
if (channel.is_im && (channel as any).user) {
leadSymbol = <Avatar userId={(channel as any).user} />;
}
if (channel.is_mpim) {
leadSymbol = <></>;
name = name?.replace("Group messaging with: ", "");
}
return (
<li key={name}>
<a title={name} href={`html/${channel.id!}-0.html`} target="iframe">
{leadSymbol}
<span>{name}</span>
</a>
</li>
);
};
interface IndexPageProps {
channels: Array<Channel>;
}
const IndexPage: React.FunctionComponent<IndexPageProps> = (props) => {
const { channels } = props;
const sortedChannels = sortBy(channels, "name");
const publicChannels = sortedChannels
.filter((channel) => isPublicChannel(channel) && !channel.is_archived)
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
const publicArchivedChannels = sortedChannels
.filter((channel) => isPublicChannel(channel) && channel.is_archived)
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
const privateChannels = sortedChannels
.filter((channel) => isPrivateChannel(channel) && !channel.is_archived)
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
const privateArchivedChannels = sortedChannels
.filter((channel) => isPrivateChannel(channel) && channel.is_archived)
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
const dmChannels = sortedChannels
.filter(
(channel) => isDmChannel(channel, users) && !users[channel.user!].deleted
)
.sort((a, b) => {
// Self first
if (me && a.user && a.user === me.id) {
return -1;
}
// Then alphabetically
return (a.name || "Unknown").localeCompare(b.name || "Unknown");
})
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
const dmDeletedChannels = sortedChannels
.filter(
(channel) => isDmChannel(channel, users) && users[channel.user!].deleted
)
.sort((a, b) => (a.name || "Unknown").localeCompare(b.name || "Unknown"))
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
const groupChannels = sortedChannels
.filter((channel) => channel.is_mpim)
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
const botChannels = sortedChannels
.filter((channel) => isBotChannel(channel, users))
.sort((a, b) => {
if (a.name && b.name) {
return a.name!.localeCompare(b.name!);
}
return 1;
})
.map((channel) => <ChannelLink key={channel.id} channel={channel} />);
return (
<HtmlPage>
<div id="index">
<div id="channels">
<p className="section">Public Channels</p>
<ul>{publicChannels}</ul>
<p className="section">Private Channels</p>
<ul>{privateChannels}</ul>
<p className="section">DMs</p>
<ul>{dmChannels}</ul>
<p className="section">Group DMs</p>
<ul>{groupChannels}</ul>
<p className="section">Bots</p>
<ul>{botChannels}</ul>
<p className="section">Archived Public Channels</p>
<ul>{publicArchivedChannels}</ul>
<p className="section">Archived Private Channels</p>
<ul>{privateArchivedChannels}</ul>
<p className="section">DMs (Deleted Users)</p>
<ul>{dmDeletedChannels}</ul>
</div>
<div id="messages">
<iframe name="iframe" src={`html/${channels[0].id!}-0.html`} />
</div>
<script
dangerouslySetInnerHTML={{
__html: `
const urlSearchParams = new URLSearchParams(window.location.search);
const channelValue = urlSearchParams.get("c");
const tsValue = urlSearchParams.get("ts");
if (channelValue) {
const iframe = document.getElementsByName('iframe')[0]
iframe.src = "html/" + decodeURIComponent(channelValue) + '.html' + '#' + (tsValue || '');
}
`,
}}
/>
</div>
</HtmlPage>
);
};
const HtmlPage: React.FunctionComponent = (props) => {
return (
<html lang="en">
<head>
<meta httpEquiv="X-UA-Compatible" content="IE=edge" />
<meta charSet="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Slack</title>
<link rel="stylesheet" href={`${base}style.css`} />
</head>
<body>{props.children}</body>
</html>
);
};
interface HeaderProps {
index: number;
chunksInfo: ChunksInfo;
channel: Channel;
}
const Header: React.FunctionComponent<HeaderProps> = (props) => {
const { channel, index, chunksInfo } = props;
let created;
if (!channel.is_im && !channel.is_mpim) {
const creator = getName(channel.creator, users);
const time = channel.created
? format(channel.created * 1000, "PPPP")
: "Unknown";
created =
creator && time ? (
<span className="created">
Created by {creator} on {time}
</span>
) : null;
}
return (
<div className="header">
<h1>{channel.name || channel.id}</h1>
{created}
<p className="topic">{channel.topic?.value}</p>
<Pagination
channelId={channel.id!}
index={index}
chunksInfo={chunksInfo}
/>
</div>
);
};
interface PaginationProps {
index: number;
chunksInfo: ChunksInfo;
channelId: string;
}
const Pagination: React.FunctionComponent<PaginationProps> = (props) => {
const { index, channelId, chunksInfo } = props;
const length = chunksInfo.length;
if (length === 1) {
return null;
}
const older =
index + 1 < length ? (
<span>
<a href={`${channelId}-${index + 1}.html`}>Older Messages</a>
</span>
) : null;
const newer =
index > 0 ? (
<span>
<a href={`${channelId}-${index - 1}.html`}>Newer Messages </a>
</span>
) : null;
const sep1 = older && newer ? " | " : null;
const sep2 = older || newer ? " | " : null;
const options: Array<JSX.Element> = [];
for (const [i, chunk] of chunksInfo.entries()) {
const text = `${i} - ${chunk.newest} to ${chunk.oldest}`;
const value = `${channelId}-${i}.html`;
const selected = i === index;
options.push(
<option selected={selected} key={value} value={value}>
{text}
</option>
);
}
return (
<div className="pagination">
{newer}
{sep1}
{older}
{sep2}
<div className="jumper">
<select id="jumper">{options}</select>
<script
dangerouslySetInnerHTML={{
__html: `
document.getElementById("jumper").onchange = function () {
window.location.href = this.value;
}
`,
}}
/>
</div>
</div>
);
};
async function renderIndexPage() {
base = "html/";
const channels = await getChannels();
const page = <IndexPage channels={channels} />;
return renderAndWrite(page, INDEX_PATH);
}
interface RenderMessagesPageOptions {
channel: Channel;
messages: Array<ArchiveMessage>;
chunkIndex: number;
chunksInfo: ChunksInfo;
}
function renderMessagesPage(options: RenderMessagesPageOptions, spinner: Ora) {
const { channel, messages, chunkIndex: index, chunksInfo } = options;
const page = (
<MessagesPage
channel={channel}
messages={messages}
index={index}
chunksInfo={chunksInfo}
/>
);
const filePath = getHTMLFilePath(channel.id!, index);
spinner.text = `${channel.name || channel.id}: Writing ${index + 1}/${
chunksInfo.length
} ${filePath}`;
spinner.render();
// Update the search index. In messages, the youngest message is first.
if (messages.length > 0) {
recordPage(channel.id, messages[messages.length - 1]?.ts);
}
return renderAndWrite(page, filePath);
}
async function renderAndWrite(page: JSX.Element, filePath: string) {
const html = ReactDOMServer.renderToStaticMarkup(page);
const htmlWDoc = "<!DOCTYPE html>" + html;
await write(filePath, htmlWDoc);
}
export async function getChannelsToCreateFilesFor(
channels: Array<Channel>,
newMessages: Record<string, number>
) {
const result: Array<Channel> = [];
// If HTML regeneration is forced, ignore everything
// and just return all channels
if (FORCE_HTML_GENERATION) {
return await getChannels();
}
for (const channel of channels) {
if (channel.id) {
// Do we have new messages?
if (newMessages[channel.id] > 0) {
result.push(channel);
}
// Did we never create a file?
if (!fs.existsSync(getHTMLFilePath(channel.id!, 0))) {
result.push(channel);
}
}
}
return result;
}
async function createHtmlForChannel({
channel,
i,
total,
}: {
channel: Channel;
i: number;
total: number;
}) {
const messages = await getMessages(channel.id!, true);
const chunks = chunk(messages, MESSAGE_CHUNK);
const spinner = ora(
`Rendering HTML for ${i + 1}/${total} ${channel.name || channel.id}`
).start();
// Calculate info about all chunks
const chunksInfo: ChunksInfo = [];
for (const iChunk of chunks) {
chunksInfo.push({
oldest: formatTimestamp(iChunk[iChunk.length - 1], "Pp"),
newest: formatTimestamp(iChunk[0], "Pp"),
count: iChunk.length,
});
}
if (chunks.length === 0) {
await renderMessagesPage(
{
channel,
messages: [],
chunkIndex: 0,
chunksInfo: chunksInfo,
},
spinner
);
}
for (const [chunkI, chunk] of chunks.entries()) {
await renderMessagesPage(
{
channel,
messages: chunk,
chunkIndex: chunkI,
chunksInfo,
},
spinner
);
}
spinner.succeed(
`Rendered HTML for ${i + 1}/${total} ${channel.name || channel.id}`
);
}
export async function createHtmlForChannels(channels: Array<Channel> = []) {
console.log(`\n Creating HTML files for ${channels.length} channels...`);
users = await getUsers();
slackArchiveData = await getSlackArchiveData();
me = slackArchiveData.auth?.user_id
? users[slackArchiveData.auth?.user_id]
: null;
for (const [i, channel] of channels.entries()) {
if (!channel.id) {
console.warn(`Can't create HTML for channel: No id found`, channel);
continue;
}
await createHtmlForChannel({ channel, i, total: channels.length });
}
await renderIndexPage();
// Copy in fonts & css
fs.copySync(path.join(_dirname, "../static"), path.join(OUT_DIR, "html/"));
}
if (esMain(import.meta)) {
createHtmlForChannels();
}
================================================
FILE: src/data-load.ts
================================================
import fs from "fs-extra";
import {
ArchiveMessage,
Channel,
Emojis,
SearchFile,
Users,
} from "./interfaces.js";
import {
CHANNELS_DATA_PATH,
EMOJIS_DATA_PATH,
getChannelDataFilePath,
SEARCH_DATA_PATH,
USERS_DATA_PATH,
} from "./config.js";
import { retry } from "./retry.js";
async function getFile<T>(filePath: string, returnIfEmpty: T): Promise<T> {
if (!fs.existsSync(filePath)) {
return returnIfEmpty;
}
const data: T = await readJSON(filePath);
return data;
}
export const messagesCache: Record<string, Array<ArchiveMessage>> = {};
export async function getMessages(
channelId: string,
cachedOk: boolean = false
): Promise<Array<ArchiveMessage>> {
if (cachedOk && messagesCache[channelId]) {
return messagesCache[channelId];
}
const filePath = getChannelDataFilePath(channelId);
messagesCache[channelId] = await getFile<Array<ArchiveMessage>>(filePath, []);
return messagesCache[channelId];
}
export async function getUsers(): Promise<Users> {
return getFile<Users>(USERS_DATA_PATH, {});
}
export async function getEmoji(): Promise<Emojis> {
return getFile<Emojis>(EMOJIS_DATA_PATH, {});
}
export async function getChannels(): Promise<Array<Channel>> {
return getFile<Array<Channel>>(CHANNELS_DATA_PATH, []);
}
export async function getSearchFile(): Promise<SearchFile> {
const returnIfEmpty = { users: {}, channels: {}, messages: {}, pages: {} };
if (!fs.existsSync(SEARCH_DATA_PATH)) {
return returnIfEmpty;
}
const contents = await readFile(SEARCH_DATA_PATH, "utf8");
// See search.ts, the file is actually JS (not JSON)
return JSON.parse(contents.slice(21, contents.length - 1));
}
export async function readFile(filePath: string, encoding = "utf8") {
return retry<string>({ name: `Reading ${filePath}` }, () => {
return fs.readFileSync(SEARCH_DATA_PATH, "utf8");
});
}
export async function readJSON<T>(filePath: string) {
return retry<T>({ name: `Loading JSON from ${filePath}` }, () => {
return fs.readJSONSync(filePath);
});
}
================================================
FILE: src/data-write.ts
================================================
import fs from "fs-extra";
import { differenceBy } from "lodash-es";
import { retry } from "./retry.js";
export async function write(filePath: string, data: any) {
await retry({ name: `Writing ${filePath}` }, () => {
fs.outputFileSync(filePath, data);
});
}
export async function writeAndMerge(filePath: string, newData: any) {
await retry({ name: `Writing ${filePath}` }, () => {
let dataToWrite = newData;
if (fs.existsSync(filePath)) {
const oldData = fs.readJSONSync(filePath);
if (Array.isArray(oldData)) {
if (newData && newData[0] && newData[0].id) {
// Take the old data, exclude aything that is in the new data,
// and then add the new data
dataToWrite = [
...differenceBy(oldData, newData, (v: any) => v.id),
...newData,
];
} else {
dataToWrite = [...oldData, ...newData];
}
} else if (typeof newData === "object") {
dataToWrite = { ...oldData, ...newData };
} else {
console.error(`writeAndMerge: Did not understand type of data`, {
filePath,
newData,
});
}
}
fs.outputFileSync(filePath, JSON.stringify(dataToWrite, undefined, 2));
});
}
================================================
FILE: src/download-files.ts
================================================
import fetch from "node-fetch";
import fs from "fs-extra";
import esMain from "es-main";
import ora, { Ora } from "ora";
import path from "path";
import { File } from "./interfaces.js";
import {
getChannelUploadFilePath,
config,
NO_FILE_DOWNLOAD,
} from "./config.js";
import { getChannels, getMessages } from "./data-load.js";
import { downloadAvatars } from "./users.js";
export interface DownloadUrlOptions {
authorize?: boolean;
force?: boolean;
}
export async function downloadURL(
url: string,
filePath: string,
options: DownloadUrlOptions = {}
) {
const authorize = options.authorize === undefined ? true : options.authorize;
if (!options.force && fs.existsSync(filePath)) {
return;
}
const { token } = config;
const headers: HeadersInit = authorize
? {
Authorization: `Bearer ${token}`,
}
: {};
try {
const response = await fetch(url, { headers });
const buffer = await response.buffer();
fs.outputFileSync(filePath, buffer);
} catch (error) {
console.warn(`Failed to download file ${url}`, error);
}
}
async function downloadFile(
file: File,
channelId: string,
i: number,
total: number,
spinner: Ora
) {
const { url_private, id, is_external, mimetype } = file;
const { thumb_1024, thumb_720, thumb_480, thumb_pdf } = file as any;
const fileUrl = is_external
? thumb_1024 || thumb_720 || thumb_480 || thumb_pdf
: url_private;
if (!fileUrl) return;
spinner.text = `Downloading ${i}/${total}: ${fileUrl}`;
const extension = path.extname(fileUrl);
const filePath = getChannelUploadFilePath(channelId, `${id}${extension}`);
await downloadURL(fileUrl, filePath);
if (mimetype === "application/pdf" && thumb_pdf) {
spinner.text = `Downloading ${i}/${total}: ${thumb_pdf}`;
const thumbFile = filePath.replace(extension, ".png");
await downloadURL(thumb_pdf, thumbFile);
}
}
export async function downloadFilesForChannel(channelId: string, spinner: Ora) {
if (NO_FILE_DOWNLOAD) {
return;
}
const messages = await getMessages(channelId);
const channels = await getChannels();
const channel = channels.find(({ id }) => id === channelId);
const fileMessages = messages.filter(
(m) => (m.files?.length || m.replies?.length || 0) > 0
);
const getSpinnerText = (i: number, ri?: number) => {
let reply = "";
if (ri !== undefined) {
reply = ` (reply ${ri})`;
}
return `Downloading ${i}/${
fileMessages.length
}${reply} messages with files for channel ${channel?.name || channelId}...`;
};
spinner.text = getSpinnerText(0);
for (const [i, fileMessage] of fileMessages.entries()) {
if (!fileMessage.files && !fileMessage.replies) {
continue;
}
if (fileMessage.files) {
for (const file of fileMessage.files) {
spinner.text = getSpinnerText(i);
spinner.render();
await downloadFile(file, channelId, i, fileMessages.length, spinner);
}
}
if (fileMessage.replies) {
for (const [ri, reply] of fileMessage.replies.entries()) {
if (reply.files) {
for (const file of reply.files) {
spinner.text = getSpinnerText(i, ri);
spinner.render();
await downloadFile(
file,
channelId,
i,
fileMessages.length,
spinner
);
}
}
}
}
}
}
================================================
FILE: src/emoji.ts
================================================
import path from "path";
import ora from "ora";
import fs from "fs";
import { createRequire } from "node:module";
import { EMOJIS_DIR, NO_SLACK_CONNECT } from "./config.js";
import { downloadURL } from "./download-files.js";
import { ArchiveMessage, Emojis } from "./interfaces.js";
import { getWebClient } from "./web-client.js";
const require = createRequire(import.meta.url);
const emojiData = require("emoji-datasource");
let _unicodeEmoji: Record<string, string>;
function getUnicodeEmoji() {
if (_unicodeEmoji) {
return _unicodeEmoji;
}
_unicodeEmoji = {};
for (const emoji of emojiData) {
_unicodeEmoji[emoji.short_name as string] = emoji.unified;
}
return _unicodeEmoji;
}
export function getEmojiFilePath(name: string, extension?: string) {
// If we have an extension, return the correct path
if (extension) {
return path.join(EMOJIS_DIR, `${name}${extension}`);
}
// If we don't have an extension, return the first path that exists
// regardless of extension
const extensions = [".png", ".jpg", ".gif"];
for (const ext of extensions) {
if (fs.existsSync(path.join(EMOJIS_DIR, `${name}${ext}`))) {
return path.join(EMOJIS_DIR, `${name}${ext}`);
}
}
}
export function isEmojiUnicode(name: string) {
const unicodeEmoji = getUnicodeEmoji();
return !!unicodeEmoji[name];
}
export function getEmojiUnicode(name: string) {
const unicodeEmoji = getUnicodeEmoji();
const unified = unicodeEmoji[name];
const split = unified.split("-");
return split
.map((code) => {
return String.fromCodePoint(parseInt(code, 16));
})
.join("");
}
export async function downloadEmojiList(): Promise<Emojis> {
if (NO_SLACK_CONNECT) {
return {};
}
const response = await getWebClient().emoji.list();
if (response.ok) {
return response.emoji!;
} else {
return {};
}
}
export async function downloadEmoji(
name: string,
url: string,
emojis: Emojis
): Promise<void> {
// Alias?
if (url.startsWith("alias:")) {
const alias = getEmojiAlias(url);
if (!emojis[alias]) {
console.warn(
`Found emoji alias ${alias}, which does not exist in master emoji list`
);
return;
} else {
return downloadEmoji(alias, emojis[alias], emojis);
}
}
const extension = path.extname(url);
const filePath = getEmojiFilePath(name, extension);
return downloadURL(url, filePath!);
}
export function getEmojiAlias(name: string): string {
// Ugh regex methods - this should turn "alias:hi-bob" into "hi-bob"
const alias = [...name.matchAll(/alias:(.*)/g)][0][1]!;
return alias!;
}
export async function downloadEmojis(
messages: Array<ArchiveMessage>,
emojis: Emojis
) {
const regex = /:[^:\s]*(?:::[^:\s]*)*:/g;
const spinner = ora(
`Scanning 0/${messages.length} messages for emoji shortcodes...`
).start();
let downloaded = 0;
for (const [i, message] of messages.entries()) {
spinner.text = `Scanning ${i}/${messages.length} messages for emoji shortcodes...`;
// Reactions
if (message.reactions && message.reactions.length > 0) {
for (const reaction of message.reactions) {
const reactEmoji = emojis[reaction.name!];
if (reactEmoji) {
downloaded++;
await downloadEmoji(reaction.name!, reactEmoji, emojis);
}
}
}
}
spinner.succeed(
`Scanned ${messages.length} messages for emoji (and downloaded ${downloaded})`
);
}
================================================
FILE: src/interfaces.ts
================================================
import { Message as SlackMessage } from "@slack/web-api/dist/response/ConversationsHistoryResponse";
import { Channel as SlackChannel } from "@slack/web-api/dist/response/ConversationsListResponse";
import { User as SlackUser } from "@slack/web-api/dist/response/UsersInfoResponse";
import { File as SlackFile } from "@slack/web-api/dist/response/FilesInfoResponse";
import { Reaction as SlackReaction } from "@slack/web-api/dist/response/ReactionsGetResponse";
import { AuthTestResponse } from "@slack/web-api";
export type User = SlackUser;
export type Users = Record<string, User>;
export type Emojis = Record<string, string>;
export interface ArchiveMessage extends SlackMessage {
replies?: Array<SlackMessage>;
}
export type Reaction = SlackReaction;
export type Message = SlackMessage;
export type Channel = SlackChannel;
export type File = SlackFile;
export type SearchPageIndex = Record<string, Array<string>>;
export type SearchFile = {
users: Record<string, string>; // userId -> userName
channels: Record<string, string>; // channelId -> channelName
messages: Record<string, Array<SearchMessage>>;
pages: SearchPageIndex;
};
export type SearchMessage = {
m?: string; // Message
u?: string; // User
t?: string; // Timestamp
c?: string; // Channel
};
export interface SlackArchiveChannelData {
messages: number;
fullyDownloaded: boolean;
}
export interface SlackArchiveData {
channels: Record<string, SlackArchiveChannelData>;
auth?: AuthTestResponse;
}
export interface ChunkInfo {
oldest?: string;
newest?: string;
count: number;
}
export type ChunksInfo = Array<ChunkInfo>;
================================================
FILE: src/messages.ts
================================================
import {
ConversationsHistoryResponse,
ConversationsListArguments,
ConversationsListResponse,
} from "@slack/web-api";
import { Channel } from "@slack/web-api/dist/response/ConversationsListResponse";
import ora from "ora";
import { ArchiveMessage, Message, Users } from "./interfaces.js";
import { getMessages } from "./data-load.js";
import { isThread } from "./threads.js";
import { downloadUser, getName } from "./users.js";
import { getWebClient } from "./web-client.js";
function isConversation(input: any): input is ConversationsHistoryResponse {
return !!input.messages;
}
interface DownloadMessagesResult {
messages: Array<ArchiveMessage>;
new: number;
}
export async function downloadMessages(
channel: Channel,
i: number,
channelCount: number
): Promise<DownloadMessagesResult> {
let result: DownloadMessagesResult = {
messages: [],
new: 0,
};
if (!channel.id) {
console.warn(`Channel without id`, channel);
return result;
}
for (const message of await getMessages(channel.id)) {
result.messages.push(message);
}
const oldest =
result.messages.length > 0 ? parseInt(result.messages[0].ts || "0", 10) : 0;
const name =
channel.name || channel.id || channel.purpose?.value || "Unknown channel";
const spinner = ora(
`Downloading messages for channel ${i + 1}/${channelCount} (${name})...`
).start();
for await (const page of getWebClient().paginate("conversations.history", {
channel: channel.id,
oldest,
})) {
if (isConversation(page)) {
const pageLength = page.messages?.length || 0;
const fetched = `Fetched ${pageLength} messages`;
const total = `(total so far: ${result.messages.length + pageLength}`;
spinner.text = `Downloading ${
i + 1
}/${channelCount} ${name}: ${fetched} ${total})`;
result.new = result.new + (page.messages || []).length;
result.messages.unshift(...(page.messages || []));
}
}
spinner.succeed(
`Downloaded messages for channel ${i + 1}/${channelCount} (${name})`
);
return result;
}
export async function downloadReplies(
channel: Channel,
message: ArchiveMessage
): Promise<Array<Message>> {
if (!channel.id || !message.ts) {
console.warn("Could not find channel or message id", channel, message);
return [];
}
if (!message.reply_count) {
console.warn("Message has no reply count", message);
return [];
}
// Do we already have all replies?
if (message.replies && message.replies.length >= message.reply_count) {
return message.replies;
}
const replies = message.replies || [];
// Oldest is the last entry
const oldest = replies.length > 0 ? replies[replies.length - 1].ts : "0";
const result = await getWebClient().conversations.replies({
channel: channel.id,
ts: message.ts,
oldest,
});
// First message is the parent
return (result.messages || []).slice(1);
}
export async function downloadExtras(
channel: Channel,
messages: Array<ArchiveMessage>,
users: Users
) {
const spinner = ora(
`Downloading threads and users for ${channel.name || channel.id}...`
).start();
// Then, all messages and threads
let processedThreads = 0;
const totalThreads = messages.filter(isThread).length;
for (const message of messages) {
// Download threads
if (isThread(message)) {
processedThreads++;
spinner.text = `Downloading threads (${processedThreads}/${totalThreads}) for ${
channel.name || channel.id
}...`;
message.replies = await downloadReplies(channel, message);
}
// Download users and avatars
if (message.user) {
await downloadUser(message, users);
}
}
spinner.succeed(
`Downloaded ${totalThreads} threads and users for ${
channel.name || channel.id
}.`
);
}
================================================
FILE: src/reactions.ts
================================================
import { Message } from "./interfaces";
export function hasReactions(message: Message) {
return message.reactions && message.reactions.length > 0;
}
================================================
FILE: src/retry.ts
================================================
export interface RetryOptions {
retries: number;
name?: string;
}
const defaultOptions: RetryOptions = {
retries: 3,
};
export async function retry<T>(
options: Partial<RetryOptions>,
operation: () => T,
attempt = 0
): Promise<T> {
let mergedOptions = { ...defaultOptions, ...options };
try {
return operation();
} catch (error) {
if (attempt >= mergedOptions.retries) {
throw error;
}
const ms = 250 + attempt * 250;
if (mergedOptions.name) {
console.warn(`Operation "${options.name}" failed, retrying in ${ms}`);
}
await wait(ms);
return retry(options, operation, attempt + 1);
}
}
function wait(ms = 250) {
return new Promise((resolve) => {
setTimeout(resolve, ms);
});
}
================================================
FILE: src/search.ts
================================================
import fs from "fs-extra";
import ora, { Ora } from "ora";
import { getChannelName } from "./channels.js";
import {
NO_SEARCH,
SEARCH_DATA_PATH,
SEARCH_PATH,
SEARCH_TEMPLATE_PATH,
} from "./config.js";
import { SearchFile, SearchMessage, SearchPageIndex } from "./interfaces";
import {
getChannels,
getMessages,
getSearchFile,
getUsers,
} from "./data-load.js";
// Format:
// channelId: [ timestamp0, timestamp1, timestamp2, ... ]
//
// channelId: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]
// pages: {
// 0: [ 10, 9, 8 ]
// 1: [ 7, 6, 5 ]
// 2: [ 4, 3, 2 ]
// 3: [ 1, 0 ]
// }
// INDEX_OF_PAGES: {
// channelId: [8, 5, 2, 0]
// }
//
// For channelId, a message older than timestamp 0 but younger than timestamp1 is on page 1.
// In our example above, the message with timestamp 6 is older than 5 but younger than 8.
const INDEX_OF_PAGES: SearchPageIndex = {};
export function recordPage(channelId?: string, timestamp?: string) {
if (!channelId || !timestamp) {
console.warn(
`Search: Cannot record page: channelId: ${channelId} timestamp: ${timestamp}`
);
return;
}
if (!INDEX_OF_PAGES[channelId]) {
INDEX_OF_PAGES[channelId] = [];
}
INDEX_OF_PAGES[channelId].push(timestamp);
}
export async function createSearch() {
if (NO_SEARCH) return;
const spinner = ora(`Creating search file...`).start();
spinner.render();
await createSearchFile(spinner);
await createSearchHTML();
spinner.succeed(`Search file created`);
}
async function createSearchFile(spinner: Ora) {
const existingData = await getSearchFile();
const users = await getUsers();
const channels = await getChannels();
const result: SearchFile = {
channels: {},
users: {},
messages: {},
pages: { ...existingData.pages, ...INDEX_OF_PAGES },
};
// Users
for (const user in users) {
result.users[user] = users[user].name || users[user].real_name || "Unknown";
}
// Channels & Messages
for (const [i, channel] of channels.entries()) {
if (!channel.id) {
console.warn(
`Can't create search file for channel ${channel.name}: No id found`,
channel
);
continue;
}
const name = getChannelName(channel);
spinner.text = `Creating search messages for channel ${name}`;
spinner.render();
const messages = (await getMessages(channel.id, true)).map((message) => {
const searchMessage: SearchMessage = {
m: message.text,
u: message.user,
t: message.ts,
};
return searchMessage;
});
result.messages![channel.id] = messages;
result.channels[channel.id] = name;
}
const jsContent = `window.search_data = ${JSON.stringify(result)};`;
await fs.outputFile(SEARCH_DATA_PATH, jsContent);
}
async function createSearchHTML() {
let template = fs.readFileSync(SEARCH_TEMPLATE_PATH, "utf8");
template = template.replace(
"<!-- react -->",
getScript(`react@18.2.0/umd/react.production.min.js`)
);
template = template.replace(
"<!-- react-dom -->",
getScript(`react-dom@18.2.0/umd/react-dom.production.min.js`)
);
template = template.replace(
`<!-- babel -->`,
getScript(`babel-standalone@6.26.0/babel.min.js`)
);
template = template.replace(
`<!-- minisearch -->`,
getScript("minisearch@5.0.0/dist/umd/index.min.js")
);
template = template.replace(`<!-- Size -->`, getSize());
fs.outputFileSync(SEARCH_PATH, template);
}
function getSize() {
const mb = fs.statSync(SEARCH_DATA_PATH).size / 1048576; //MB
return `Loading ${Math.round(mb)}MB of data`;
}
function getScript(script: string) {
return `<script crossorigin src="https://cdn.jsdelivr.net/npm/${script}"></script>`;
}
================================================
FILE: src/threads.ts
================================================
import { Message } from "./interfaces";
export function isThread(message: Message) {
return message.reply_count && message.reply_count > 0;
}
================================================
FILE: src/timestamp.ts
================================================
export function slackTimestampToJavaScriptTimestamp(ts?: string) {
if (!ts) {
return 0;
}
const splitTs = ts.split(".") || [];
const jsTs = parseInt(`${splitTs[0]}${splitTs[1].slice(0, 3)}`, 10);
return jsTs;
}
================================================
FILE: src/users.ts
================================================
import path from "path";
import { getWebClient } from "./web-client.js";
import { Message, User, Users } from "./interfaces.js";
import { getAvatarFilePath } from "./config.js";
import { getUsers } from "./data-load.js";
import { downloadURL } from "./download-files.js";
import ora from "ora";
// We'll redownload users every run, but only once per user
// To keep track, we'll keep the ids in this array
export const usersRefetchedThisRun: Array<string> = [];
export const avatarsRefetchedThisRun: Array<string> = [];
export async function downloadUser(
item: Message | any,
users: Users
): Promise<User | null> {
if (!item.user) return null;
// If we already have this user *and* downloaded them before,
// return cached version
if (users[item.user] && usersRefetchedThisRun.includes(item.user))
return users[item.user];
const spinner = ora(`Downloading info for user ${item.user}...`).start();
const user = (item.user === 'U00') ? {} as User : (
await getWebClient().users.info({
user: item.user,
})
).user;
if (user) {
usersRefetchedThisRun.push(item.user);
spinner.succeed(`Downloaded info for user ${item.user} (${user.name})`);
return (users[item.user] = user);
}
return null;
}
export async function downloadAvatars() {
const users = await getUsers();
const userIds = Object.keys(users);
const spinner = ora(`Downloading avatars (0/${userIds.length})`).start();
for (const [i, userId] of userIds.entries()) {
spinner.text = `Downloading avatars (${i + 1}/${userIds.length})`;
await downloadAvatarForUser(users[userId]);
}
spinner.stop();
}
export async function downloadAvatarForUser(user?: User | null) {
if (!user || !user.id || avatarsRefetchedThisRun.includes(user.id)) {
return;
}
const { profile } = user;
if (!profile || !profile.image_512) {
return;
}
try {
const filePath = getAvatarFilePath(
user.id!,
path.extname(profile.image_512)
);
await downloadURL(profile.image_512, filePath, {
authorize: false,
force: true,
});
avatarsRefetchedThisRun.push(user.id!);
} catch (error) {
console.warn(`Failed to download avatar for user ${user.id!}`, error);
}
}
export function getName(userId: string | undefined, users: Users) {
if (!userId) return "Unknown";
const user = users[userId];
if (!user) return userId;
return user.profile?.display_name || user.profile?.real_name || user.name;
}
================================================
FILE: src/web-client.ts
================================================
import { WebClient } from "@slack/web-api";
import { config } from "./config.js";
let _webClient: WebClient;
export function getWebClient() {
if (_webClient) return _webClient;
const { token } = config;
return (_webClient = new WebClient(token));
}
export async function authTest() {
return getWebClient().auth.test();
}
================================================
FILE: static/scroll.js
================================================
if (window.location.hash) {
document.getElementById(window.location.hash).scrollTo();
} else {
scrollBy({ top: 99999999 });
}
================================================
FILE: static/search.html
================================================
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<title>Message Search</title>
<link rel="stylesheet" href="html/style.css" />
<!-- react -->
<!-- react-dom -->
<!-- babel -->
<!-- minisearch -->
</head>
<body>
<script type="text/babel" data-type="module">
class App extends React.PureComponent {
constructor(props) {
super(props);
this.handleSearchChange = this.handleSearchChange.bind(this);
this.handleSearchClear = this.handleSearchClear.bind(this);
this.searchMessages = this.searchMessages.bind(this);
this.loadSearchData = this.loadSearchData.bind(this);
this.loadSearchDataWhenReady =
this.loadSearchDataWhenReady.bind(this);
this.searchInputRef = React.createRef();
this.state = {
matchingMessages: [],
searchValue: "",
ready: false,
miniSearch: null,
};
}
componentDidMount() {
this.loadSearchDataWhenReady();
}
loadSearchDataWhenReady() {
if (window.search_data) {
this.loadSearchData();
} else {
setTimeout(() => {
this.loadSearchDataWhenReady();
}, 100);
}
}
loadSearchData() {
const { channels, users, messages } = window.search_data;
console.time(`Loading minisearch`);
const allMessages = [];
const miniSearch = new MiniSearch({
idField: "t",
fields: ["m"], // fields to index for full-text search
storeFields: ["t", "u", "m", "c"], // fields to return with search results
});
for (const channel in messages) {
for (const message of messages[channel]) {
allMessages.push({ ...message, c: channel });
}
}
miniSearch.addAll(allMessages);
console.timeEnd(`Loading minisearch`);
this.setState({ ready: true, miniSearch });
}
handleSearchChange({ target: { value } }) {
this.setState({ searchValue: value });
const matchingMessages =
value.length > 1 ? this.searchMessages(value) : [];
this.setState({ matchingMessages });
}
handleSearchClear() {
this.setState({ searchValue: "", matchingMessages: [] });
}
searchMessages(query) {
const { miniSearch } = this.state;
return miniSearch.search(query).slice(0, 50);
}
render() {
const { matchingMessages, searchValue, ready } = this.state;
return (
<div className="App">
<article className="main">
{ready ? (
<Header
onChange={this.handleSearchChange}
onKeyDown={this.handleSearchKeyDown}
onSearchClear={this.handleSearchClear}
value={searchValue}
searchInputRef={this.searchInputRef}
/>
) : (
"Loading"
)}
{matchingMessages && matchingMessages.length > 0 ? (
<MessagesList messages={matchingMessages} />
) : (
<p>This search is incredibly basic, but it works.</p>
)}
</article>
</div>
);
}
}
const MessagesList = ({ messages }) => (
<ul className="MessagesList">
{messages.map(({ t, ...props }) => (
<Message {...props} t={t} key={t} />
))}
</ul>
);
const Message = ({ m, u, t, c }) => {
// Let's find the page
const { pages } = window.search_data;
// Returns the index of the first timestamp that's
// smaller than the timestamp we passed in.
const channelPages = pages[c] ? pages[c] : null;
let href;
if (channelPages) {
const index = channelPages.findIndex((pageTs) => pageTs < t);
const page = `${c}-${index}`;
href = `index.html?c=${encodeURIComponent(page)}&ts=${t}`;
}
const message = (
<li className="Message">
<p>
<Channel id={c} /> - <Timestamp timestamp={t} />
</p>
<p>
<User id={u} /> {m}
</p>
</li>
);
if (href) {
return (
<a href={href} target="_blank">
{message}
</a>
);
} else {
return message;
}
};
const User = ({ id }) => (
<strong>@{window.search_data.users[id]}: </strong>
);
const Channel = ({ id }) => (
<span className="Channel">#{window.search_data.channels[id]}</span>
);
const Timestamp = ({ timestamp }) => {
const splitTs = timestamp.split(".") || [];
const jsTs = parseInt(`${splitTs[0]}${splitTs[1].slice(0, 3)}`, 10);
const date = new Date(jsTs);
return (
<span class="timestamp">
<span className="c-timestamp__label">{date.toLocaleString()}</span>
</span>
);
};
const Header = (props) => (
<header className="Header">
<h1>Message Search</h1>
<SearchBox {...props} />
</header>
);
const SearchBox = ({
onChange,
onSearchClear,
value,
searchInputRef,
}) => (
<div className="SearchBox">
<div className="Search">
<input
type="text"
value={value}
onChange={onChange}
ref={searchInputRef}
autoComplete="none"
autoCorrect="none"
autoCapitalize="none"
spellCheck="false"
/>
<button
className="clear"
onClick={onSearchClear}
style={{ margin: 10 }}
>
×
</button>
</div>
</div>
);
ReactDOM.render(
React.createElement(App),
document.getElementById("search")
);
</script>
<div id="search">
Loading and indexing messages, please wait. If you have a large number of
messages, this might take a minute.
<p>
<!-- Size -->
</p>
</div>
<script defer src="data/search.js" type="text/javascript"></script>
</body>
</html>
================================================
FILE: static/style.css
================================================
/* Reset */
/* Box sizing rules */
*,
*::before,
*::after {
box-sizing: border-box;
}
/* Remove default margin */
body,
h1,
h2,
h3,
h4,
p,
figure,
blockquote,
dl,
dd {
margin: 0;
}
/* Remove list styles on ul, ol elements with a list role, which suggests default styling will be removed */
ul[role='list'],
ol[role='list'] {
list-style: none;
}
/* Set core root defaults */
html:focus-within {
scroll-behavior: smooth;
}
/* Set core body defaults */
body {
min-height: 100vh;
text-rendering: optimizeSpeed;
line-height: 1.5;
}
/* A elements that don't have a class get default styles */
a:not([class]) {
text-decoration-skip-ink: auto;
}
/* Make images easier to work with */
img,
picture {
max-width: 100%;
display: block;
}
/* Inherit fonts for inputs and buttons */
input,
button,
textarea,
select {
font: inherit;
}
/* Remove all animations, transitions and smooth scroll for people that prefer not to see them */
@media (prefers-reduced-motion: reduce) {
html:focus-within {
scroll-behavior: auto;
}
*,
*::before,
*::after {
animation-duration: 0.01ms !important;
animation-iteration-count: 1 !important;
transition-duration: 0.01ms !important;
scroll-behavior: auto !important;
}
}
@font-face {
font-family: "Lato";
src: url('fonts/Lato-Regular.ttf') format('truetype');
font-weight: normal;
font-style: normal;
}
@font-face {
font-family: "Lato";
src: url('fonts/Lato-Bold.ttf') format('truetype');
font-weight: bold;
font-style: normal;
}
body, html {
font-family: 'Lato', sans-serif;
font-size: 14px;
color: rgb(29, 28, 29);
}
a {
color: rgb(18, 100, 163);
}
audio, video {
max-width: 400px;
}
.messages-list {
padding-bottom: 20px;
}
.messages-list .avatar {
height: 36px;
width: 36px;
border-radius: 7px;
margin-right: 10px;
background: #c1c1c1;
}
.message-gutter {
display: flex;
margin: 10px;
scroll-margin-top: 120px;
}
.message-gutter:target {
background-color: #fafafa;
border: 2px solid #39113E;
padding: 10px;
border-radius: 5px;
}
.message-gutter div:first-of-type {
flex-shrink: 0;
}
.message-gutter > .message-gutter {
/** i.e. replies in thread. Just here to be easily findable */
}
.sender {
font-weight: 800;
margin-right: 10px;
}
.timestamp {
font-weight: 200;
font-size: 13px;
color: rgb(97, 96, 97);
}
.header {
position: sticky;
background: #fff;
color: #616061;
top: 0;
left: 0;
padding: 10px;
min-height: 70px;
border-bottom: 1px solid #E2E2E2;
box-sizing: border-box;
}
.header h1 {
font-size: 16px;
color: #1D1C1D;
display: inline-block;
}
.header a {
color: #616061;
}
.header a:active, .header a.current {
color: #000;
}
.header .created {
float: right;
}
.jumper {
display: inline-block;
}
.jumper a {
margin: 2px;
}
.text {
overflow-wrap: break-word;
}
.file {
max-height: 270px;
margin-right: 10px;
margin-top: 10px;
border-radius: 4px;
border: 1px solid #80808045;
outline: none;
}
.reaction {
background-color: #eaeaea;
display: inline-block;
border-radius: 10px;
font-size: .7em;
padding-left: 6px;
padding-right: 6px;
padding-bottom: 4px;
margin-right: 5px;
padding-top: 4px;
}
.reaction img {
height: 16px;
width: 16px;
margin-right: 3px;
vertical-align: middle;
display: inline-block;
}
.reaction span {
position: relative;
top: 1px;
}
#index {
display: flex;
height: calc(100vh - 4px);
}
#channels {
background: #39113E;
width: 250px;
color: #CDC3CE;
padding-top: 10px;
overflow: scroll;
padding-bottom: 20px;
}
#channels ul {
margin: 0;
padding: 0;
list-style: none;
}
#channels p {
padding-left: 20px;
}
#channels .section {
font-weight: 800;
color: #fff;
margin-top: 10px;
}
#channels .section:first-of-type {
margin-top: 0;
}
#channels a {
padding: 5px;
display: block;
color: #CDC3CE;
text-decoration: none;
padding-left: 20px;
display: flex;
max-height: 28px;
white-space: pre;
text-overflow: ellipsis;
overflow: hidden;
}
#channels a .avatar {
height: 20px;
width: 20px;
border-radius: 3px;
margin-right: 10px;
object-fit: contain;
}
#channels a:hover {
background: #301034;
color: #edeced;
}
#messages {
flex-grow: 1;
}
#messages iframe {
height: 100%;
width: calc(100vw - 250px);
border: none;
}
#search {
margin: 10px;
text-align: center;
}
#search ul {
list-style: none;
display: flex;
flex-direction: column;
align-items: center;
}
#search li {
padding: 5px;
border-bottom: 1px solid #E2E2E2;
background: hsl(0deg 0% 98%);
border-radius: 5px;
width: 600px;
text-align: left;
margin-bottom: 5px;
}
#search a {
text-decoration: none;
color: unset;
}
================================================
FILE: tsconfig.json
================================================
{
"compilerOptions": {
"jsx": "react",
/* Visit https://aka.ms/tsconfig.json to read more about this file */
/* Projects */
// "incremental": true, /* Enable incremental compilation */
// "composite": true, /* Enable constraints that allow a TypeScript project to be used with project references. */
// "tsBuildInfoFile": "./", /* Specify the folder for .tsbuildinfo incremental compilation files. */
// "disableSourceOfProjectReferenceRedirect": true, /* Disable preferring source files instead of declaration files when referencing composite projects */
// "disableSolutionSearching": true, /* Opt a project out of multi-project reference checking when editing. */
// "disableReferencedProjectLoad": true, /* Reduce the number of projects loaded automatically by TypeScript. */
/* Language and Environment */
"target": "ES2020", /* Set the JavaScript language version for emitted JavaScript and include compatible library declarations. */
// "lib": [], /* Specify a set of bundled library declaration files that describe the target runtime environment. */
// "jsx": "preserve", /* Specify what JSX code is generated. */
// "experimentalDecorators": true, /* Enable experimental support for TC39 stage 2 draft decorators. */
// "emitDecoratorMetadata": true, /* Emit design-type metadata for decorated declarations in source files. */
// "jsxFactory": "", /* Specify the JSX factory function used when targeting React JSX emit, e.g. 'React.createElement' or 'h' */
// "jsxFragmentFactory": "", /* Specify the JSX Fragment reference used for fragments when targeting React JSX emit e.g. 'React.Fragment' or 'Fragment'. */
// "jsxImportSource": "", /* Specify module specifier used to import the JSX factory functions when using `jsx: react-jsx*`.` */
// "reactNamespace": "", /* Specify the object invoked for `createElement`. This only applies when targeting `react` JSX emit. */
// "noLib": true, /* Disable including any library files, including the default lib.d.ts. */
// "useDefineForClassFields": true, /* Emit ECMAScript-standard-compliant class fields. */
/* Modules */
"module": "ES2020", /* Specify what module code is generated. */
// "rootDir": "./", /* Specify the root folder within your source files. */
"moduleResolution": "node", /* Specify how TypeScript looks up a file from a given module specifier. */
// "baseUrl": "./", /* Specify the base directory to resolve non-relative module names. */
// "paths": {}, /* Specify a set of entries that re-map imports to additional lookup locations. */
// "rootDirs": [], /* Allow multiple folders to be treated as one when resolving modules. */
// "typeRoots": [], /* Specify multiple folders that act like `./node_modules/@types`. */
// "types": [], /* Specify type package names to be included without being referenced in a source file. */
// "allowUmdGlobalAccess": true, /* Allow accessing UMD globals from modules. */
// "resolveJsonModule": true, /* Enable importing .json files */
// "noResolve": true, /* Disallow `import`s, `require`s or `<reference>`s from expanding the number of files TypeScript should add to a project. */
/* JavaScript Support */
// "allowJs": true, /* Allow JavaScript files to be a part of your program. Use the `checkJS` option to get errors from these files. */
// "checkJs": true, /* Enable error reporting in type-checked JavaScript files. */
// "maxNodeModuleJsDepth": 1, /* Specify the maximum folder depth used for checking JavaScript files from `node_modules`. Only applicable with `allowJs`. */
/* Emit */
// "declaration": true, /* Generate .d.ts files from TypeScript and JavaScript files in your project. */
// "declarationMap": true, /* Create sourcemaps for d.ts files. */
// "emitDeclarationOnly": true, /* Only output d.ts files and not JavaScript files. */
// "sourceMap": true, /* Create source map files for emitted JavaScript files. */
// "outFile": "./", /* Specify a file that bundles all outputs into one JavaScript file. If `declaration` is true, also designates a file that bundles all .d.ts output. */
"outDir": "./lib", /* Specify an output folder for all emitted files. */
// "removeComments": true, /* Disable emitting comments. */
// "noEmit": true, /* Disable emitting files from a compilation. */
// "importHelpers": true, /* Allow importing helper functions from tslib once per project, instead of including them per-file. */
// "importsNotUsedAsValues": "remove", /* Specify emit/checking behavior for imports that are only used for types */
// "downlevelIteration": true, /* Emit more compliant, but verbose and less performant JavaScript for iteration. */
// "sourceRoot": "", /* Specify the root path for debuggers to find the reference source code. */
// "mapRoot": "", /* Specify the location where debugger should locate map files instead of generated locations. */
// "inlineSourceMap": true, /* Include sourcemap files inside the emitted JavaScript. */
// "inlineSources": true, /* Include source code in the sourcemaps inside the emitted JavaScript. */
// "emitBOM": true, /* Emit a UTF-8 Byte Order Mark (BOM) in the beginning of output files. */
// "newLine": "crlf", /* Set the newline character for emitting files. */
// "stripInternal": true, /* Disable emitting declarations that have `@internal` in their JSDoc comments. */
// "noEmitHelpers": true, /* Disable generating custom helper functions like `__extends` in compiled output. */
// "noEmitOnError": true, /* Disable emitting files if any type checking errors are reported. */
// "preserveConstEnums": true, /* Disable erasing `const enum` declarations in generated code. */
// "declarationDir": "./", /* Specify the output directory for generated declaration files. */
/* Interop Constraints */
// "isolatedModules": true, /* Ensure that each file can be safely transpiled without relying on other imports. */
"allowSyntheticDefaultImports": true, /* Allow 'import x from y' when a module doesn't have a default export. */
"esModuleInterop": true, /* Emit additional JavaScript to ease support for importing CommonJS modules. This enables `allowSyntheticDefaultImports` for type compatibility. */
// "preserveSymlinks": true, /* Disable resolving symlinks to their realpath. This correlates to the same flag in node. */
"forceConsistentCasingInFileNames": true, /* Ensure that casing is correct in imports. */
/* Type Checking */
"strict": true, /* Enable all strict type-checking options. */
// "noImplicitAny": true, /* Enable error reporting for expressions and declarations with an implied `any` type.. */
// "strictNullChecks": true, /* When type checking, take into account `null` and `undefined`. */
// "strictFunctionTypes": true, /* When assigning functions, check to ensure parameters and the return values are subtype-compatible. */
// "strictBindCallApply": true, /* Check that the arguments for `bind`, `call`, and `apply` methods match the original function. */
// "strictPropertyInitialization": true, /* Check for class properties that are declared but not set in the constructor. */
// "noImplicitThis": true, /* Enable error reporting when `this` is given the type `any`. */
// "useUnknownInCatchVariables": true, /* Type catch clause variables as 'unknown' instead of 'any'. */
// "alwaysStrict": true, /* Ensure 'use strict' is always emitted. */
// "noUnusedLocals": true, /* Enable error reporting when a local variables aren't read. */
// "noUnusedParameters": true, /* Raise an error when a function parameter isn't read */
// "exactOptionalPropertyTypes": true, /* Interpret optional property types as written, rather than adding 'undefined'. */
// "noImplicitReturns": true, /* Enable error reporting for codepaths that do not explicitly return in a function. */
// "noFallthroughCasesInSwitch": true, /* Enable error reporting for fallthrough cases in switch statements. */
// "noUncheckedIndexedAccess": true, /* Include 'undefined' in index signature results */
// "noImplicitOverride": true, /* Ensure overriding members in derived classes are marked with an override modifier. */
// "noPropertyAccessFromIndexSignature": true, /* Enforces using indexed accessors for keys declared using an indexed type */
// "allowUnusedLabels": true, /* Disable error reporting for unused labels. */
// "allowUnreachableCode": true, /* Disable error reporting for unreachable code. */
/* Completeness */
// "skipDefaultLibCheck": true, /* Skip type checking .d.ts files that are included with TypeScript. */
"skipLibCheck": true /* Skip type checking all .d.ts files. */
},
"ts-node": {
"esm": true,
"transpileOnly": true
},
}
gitextract_oionf7w6/ ├── .gitignore ├── .node-version ├── .npmignore ├── README.md ├── bin/ │ └── slack-archive.js ├── package.json ├── src/ │ ├── ambient.d.ts │ ├── archive-data.ts │ ├── backup.ts │ ├── channels.ts │ ├── cli.ts │ ├── config.ts │ ├── create-html.tsx │ ├── data-load.ts │ ├── data-write.ts │ ├── download-files.ts │ ├── emoji.ts │ ├── interfaces.ts │ ├── messages.ts │ ├── reactions.ts │ ├── retry.ts │ ├── search.ts │ ├── threads.ts │ ├── timestamp.ts │ ├── users.ts │ └── web-client.ts ├── static/ │ ├── scroll.js │ ├── search.html │ └── style.css └── tsconfig.json
SYMBOL INDEX (134 symbols across 19 files)
FILE: src/archive-data.ts
function getSlackArchiveData (line 8) | async function getSlackArchiveData(): Promise<SlackArchiveData> {
function setSlackArchiveData (line 21) | async function setSlackArchiveData(
FILE: src/backup.ts
function createBackup (line 13) | async function createBackup() {
function deleteBackup (line 31) | async function deleteBackup() {
function deleteOlderBackups (line 60) | async function deleteOlderBackups() {
FILE: src/channels.ts
function getChannelName (line 12) | function getChannelName(channel: Channel) {
function isPublicChannel (line 18) | function isPublicChannel(channel: Channel) {
function isPrivateChannel (line 22) | function isPrivateChannel(channel: Channel) {
function isDmChannel (line 26) | function isDmChannel(channel: Channel, users: Users) {
function isBotChannel (line 30) | function isBotChannel(channel: Channel, users: Users) {
function isChannels (line 34) | function isChannels(input: any): input is ConversationsListResponse {
function downloadChannels (line 38) | async function downloadChannels(
FILE: src/cli.ts
function selectMergeFiles (line 44) | async function selectMergeFiles(): Promise<boolean> {
function selectChannels (line 72) | async function selectChannels(
function selectChannelTypes (line 139) | async function selectChannelTypes(): Promise<Array<string>> {
function getToken (line 180) | async function getToken() {
function writeLastSuccessfulArchive (line 207) | async function writeLastSuccessfulArchive() {
function getLastSuccessfulRun (line 212) | function getLastSuccessfulRun() {
function getAuthTest (line 234) | async function getAuthTest() {
function main (line 265) | async function main() {
FILE: src/config.ts
function findCliParameter (line 11) | function findCliParameter(param: string) {
function getCliParameter (line 23) | function getCliParameter(param: string) {
constant AUTOMATIC_MODE (line 35) | const AUTOMATIC_MODE = findCliParameter("--automatic");
constant USE_PREVIOUS_CHANNEL_CONFIG (line 36) | const USE_PREVIOUS_CHANNEL_CONFIG = findCliParameter(
constant CHANNEL_TYPES (line 39) | const CHANNEL_TYPES = getCliParameter("--channel-types");
constant NO_BACKUP (line 40) | const NO_BACKUP = findCliParameter("--no-backup");
constant NO_SEARCH (line 41) | const NO_SEARCH = findCliParameter("--no-search");
constant NO_FILE_DOWNLOAD (line 42) | const NO_FILE_DOWNLOAD = findCliParameter("--no-file-download");
constant NO_SLACK_CONNECT (line 43) | const NO_SLACK_CONNECT = findCliParameter("--no-slack-connect");
constant FORCE_HTML_GENERATION (line 44) | const FORCE_HTML_GENERATION = findCliParameter(
constant EXCLUDE_CHANNELS (line 47) | const EXCLUDE_CHANNELS = getCliParameter("--exclude-channels");
constant BASE_DIR (line 48) | const BASE_DIR = process.cwd();
constant OUT_DIR (line 49) | const OUT_DIR = path.join(BASE_DIR, "slack-archive");
constant TOKEN_FILE (line 50) | const TOKEN_FILE = path.join(OUT_DIR, ".token");
constant DATE_FILE (line 51) | const DATE_FILE = path.join(OUT_DIR, ".last-successful-run");
constant DATA_DIR (line 52) | const DATA_DIR = path.join(OUT_DIR, "data");
constant HTML_DIR (line 53) | const HTML_DIR = path.join(OUT_DIR, "html");
constant FILES_DIR (line 54) | const FILES_DIR = path.join(HTML_DIR, "files");
constant AVATARS_DIR (line 55) | const AVATARS_DIR = path.join(HTML_DIR, "avatars");
constant EMOJIS_DIR (line 56) | const EMOJIS_DIR = path.join(HTML_DIR, "emojis");
constant INDEX_PATH (line 58) | const INDEX_PATH = path.join(OUT_DIR, "index.html");
constant SEARCH_PATH (line 59) | const SEARCH_PATH = path.join(OUT_DIR, "search.html");
constant MESSAGES_JS_PATH (line 60) | const MESSAGES_JS_PATH = path.join(__dirname, "../static/scroll.js");
constant SEARCH_TEMPLATE_PATH (line 61) | const SEARCH_TEMPLATE_PATH = path.join(
constant CHANNELS_DATA_PATH (line 65) | const CHANNELS_DATA_PATH = path.join(DATA_DIR, "channels.json");
constant USERS_DATA_PATH (line 66) | const USERS_DATA_PATH = path.join(DATA_DIR, "users.json");
constant EMOJIS_DATA_PATH (line 67) | const EMOJIS_DATA_PATH = path.join(DATA_DIR, "emojis.json");
constant SLACK_ARCHIVE_DATA_PATH (line 68) | const SLACK_ARCHIVE_DATA_PATH = path.join(
constant SEARCH_DATA_PATH (line 72) | const SEARCH_DATA_PATH = path.join(DATA_DIR, "search.js");
function getChannelDataFilePath (line 74) | function getChannelDataFilePath(channelId: string) {
function getChannelUploadFilePath (line 78) | function getChannelUploadFilePath(channelId: string, fileName: string) {
function getHTMLFilePath (line 82) | function getHTMLFilePath(channelId: string, index: number) {
function getAvatarFilePath (line 86) | function getAvatarFilePath(userId: string, extension: string) {
FILE: src/create-html.tsx
constant MESSAGE_CHUNK (line 45) | const MESSAGE_CHUNK = 1000;
function formatTimestamp (line 57) | function formatTimestamp(message: Message, dateFormat = "PPPPpppp") {
type FilesProps (line 64) | interface FilesProps {
type AvatarProps (line 117) | interface AvatarProps {
type ParentMessageProps (line 132) | interface ParentMessageProps {
type ReactionProps (line 153) | interface ReactionProps {
type EmojiProps (line 173) | interface EmojiProps {
type MessageProps (line 184) | interface MessageProps {
type MessagesPageProps (line 221) | interface MessagesPageProps {
type ChannelLinkProps (line 253) | interface ChannelLinkProps {
type IndexPageProps (line 291) | interface IndexPageProps {
type HeaderProps (line 409) | interface HeaderProps {
type PaginationProps (line 446) | interface PaginationProps {
function renderIndexPage (line 508) | async function renderIndexPage() {
type RenderMessagesPageOptions (line 516) | interface RenderMessagesPageOptions {
function renderMessagesPage (line 523) | function renderMessagesPage(options: RenderMessagesPageOptions, spinner:...
function renderAndWrite (line 548) | async function renderAndWrite(page: JSX.Element, filePath: string) {
function getChannelsToCreateFilesFor (line 555) | async function getChannelsToCreateFilesFor(
function createHtmlForChannel (line 584) | async function createHtmlForChannel({
function createHtmlForChannels (line 638) | async function createHtmlForChannels(channels: Array<Channel> = []) {
FILE: src/data-load.ts
function getFile (line 19) | async function getFile<T>(filePath: string, returnIfEmpty: T): Promise<T> {
function getMessages (line 31) | async function getMessages(
function getUsers (line 45) | async function getUsers(): Promise<Users> {
function getEmoji (line 49) | async function getEmoji(): Promise<Emojis> {
function getChannels (line 53) | async function getChannels(): Promise<Array<Channel>> {
function getSearchFile (line 57) | async function getSearchFile(): Promise<SearchFile> {
function readFile (line 70) | async function readFile(filePath: string, encoding = "utf8") {
function readJSON (line 76) | async function readJSON<T>(filePath: string) {
FILE: src/data-write.ts
function write (line 6) | async function write(filePath: string, data: any) {
function writeAndMerge (line 12) | async function writeAndMerge(filePath: string, newData: any) {
FILE: src/download-files.ts
type DownloadUrlOptions (line 16) | interface DownloadUrlOptions {
function downloadURL (line 21) | async function downloadURL(
function downloadFile (line 48) | async function downloadFile(
function downloadFilesForChannel (line 78) | async function downloadFilesForChannel(channelId: string, spinner: Ora) {
FILE: src/emoji.ts
function getUnicodeEmoji (line 15) | function getUnicodeEmoji() {
function getEmojiFilePath (line 28) | function getEmojiFilePath(name: string, extension?: string) {
function isEmojiUnicode (line 44) | function isEmojiUnicode(name: string) {
function getEmojiUnicode (line 49) | function getEmojiUnicode(name: string) {
function downloadEmojiList (line 61) | async function downloadEmojiList(): Promise<Emojis> {
function downloadEmoji (line 75) | async function downloadEmoji(
function getEmojiAlias (line 100) | function getEmojiAlias(name: string): string {
function downloadEmojis (line 106) | async function downloadEmojis(
FILE: src/interfaces.ts
type User (line 8) | type User = SlackUser;
type Users (line 10) | type Users = Record<string, User>;
type Emojis (line 12) | type Emojis = Record<string, string>;
type ArchiveMessage (line 14) | interface ArchiveMessage extends SlackMessage {
type Reaction (line 18) | type Reaction = SlackReaction;
type Message (line 20) | type Message = SlackMessage;
type Channel (line 22) | type Channel = SlackChannel;
type File (line 24) | type File = SlackFile;
type SearchPageIndex (line 26) | type SearchPageIndex = Record<string, Array<string>>;
type SearchFile (line 28) | type SearchFile = {
type SearchMessage (line 35) | type SearchMessage = {
type SlackArchiveChannelData (line 42) | interface SlackArchiveChannelData {
type SlackArchiveData (line 47) | interface SlackArchiveData {
type ChunkInfo (line 52) | interface ChunkInfo {
type ChunksInfo (line 58) | type ChunksInfo = Array<ChunkInfo>;
FILE: src/messages.ts
function isConversation (line 15) | function isConversation(input: any): input is ConversationsHistoryRespon...
type DownloadMessagesResult (line 19) | interface DownloadMessagesResult {
function downloadMessages (line 24) | async function downloadMessages(
function downloadReplies (line 78) | async function downloadReplies(
function downloadExtras (line 110) | async function downloadExtras(
FILE: src/reactions.ts
function hasReactions (line 3) | function hasReactions(message: Message) {
FILE: src/retry.ts
type RetryOptions (line 1) | interface RetryOptions {
function retry (line 10) | async function retry<T>(
function wait (line 36) | function wait(ms = 250) {
FILE: src/search.ts
constant INDEX_OF_PAGES (line 35) | const INDEX_OF_PAGES: SearchPageIndex = {};
function recordPage (line 37) | function recordPage(channelId?: string, timestamp?: string) {
function createSearch (line 52) | async function createSearch() {
function createSearchFile (line 64) | async function createSearchFile(spinner: Ora) {
function createSearchHTML (line 113) | async function createSearchHTML() {
function getSize (line 138) | function getSize() {
function getScript (line 143) | function getScript(script: string) {
FILE: src/threads.ts
function isThread (line 3) | function isThread(message: Message) {
FILE: src/timestamp.ts
function slackTimestampToJavaScriptTimestamp (line 1) | function slackTimestampToJavaScriptTimestamp(ts?: string) {
FILE: src/users.ts
function downloadUser (line 15) | async function downloadUser(
function downloadAvatars (line 42) | async function downloadAvatars() {
function downloadAvatarForUser (line 55) | async function downloadAvatarForUser(user?: User | null) {
function getName (line 81) | function getName(userId: string | undefined, users: Users) {
FILE: src/web-client.ts
function getWebClient (line 6) | function getWebClient() {
function authTest (line 13) | async function authTest() {
Condensed preview — 30 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (95K chars).
[
{
"path": ".gitignore",
"chars": 57,
"preview": "node_modules\nout\nslack-archive\n.DS_Store\n*.log\n.token\nlib"
},
{
"path": ".node-version",
"chars": 7,
"preview": "16.4.0\n"
},
{
"path": ".npmignore",
"chars": 57,
"preview": "node_modules\nout\nslack-archive\n.DS_Store\n*.log\n.token\nsrc"
},
{
"path": "README.md",
"chars": 4842,
"preview": "# Export your Slack workspace as static HTML\n\nAlright, so you want to export all your messages on Slack. You want them i"
},
{
"path": "bin/slack-archive.js",
"chars": 45,
"preview": "#!/usr/bin/env node\n\nimport('../lib/cli.js')\n"
},
{
"path": "package.json",
"chars": 1558,
"preview": "{\n \"name\": \"slack-archive\",\n \"version\": \"1.6.1\",\n \"description\": \"Create static HTML archives for your Slack workspac"
},
{
"path": "src/ambient.d.ts",
"chars": 94,
"preview": "declare module \"slack-markdown\";\ndeclare module \"es-main\";\ndeclare module \"emoji-datasource\";\n"
},
{
"path": "src/archive-data.ts",
"chars": 959,
"preview": "import fs from \"fs-extra\";\n\nimport { SLACK_ARCHIVE_DATA_PATH } from \"./config.js\";\nimport { readJSON } from \"./data-load"
},
{
"path": "src/backup.ts",
"chars": 2374,
"preview": "import fs from \"fs-extra\";\nimport inquirer from \"inquirer\";\nimport path from \"path\";\nimport trash from \"trash\";\nimport {"
},
{
"path": "src/channels.ts",
"chars": 2152,
"preview": "import {\n ConversationsListArguments,\n ConversationsListResponse,\n} from \"@slack/web-api\";\nimport ora from \"ora\";\nimpo"
},
{
"path": "src/cli.ts",
"chars": 10674,
"preview": "import { uniqBy } from \"lodash-es\";\nimport inquirer from \"inquirer\";\nimport fs from \"fs-extra\";\nimport { User } from \"@s"
},
{
"path": "src/config.ts",
"chars": 2891,
"preview": "import path from \"path\";\nimport { dirname } from \"path\";\nimport { fileURLToPath } from \"url\";\n\nconst __dirname = dirname"
},
{
"path": "src/create-html.tsx",
"chars": 18009,
"preview": "import { format } from \"date-fns\";\nimport fs from \"fs-extra\";\nimport path from \"path\";\nimport React from \"react\";\nimport"
},
{
"path": "src/data-load.ts",
"chars": 2047,
"preview": "import fs from \"fs-extra\";\n\nimport {\n ArchiveMessage,\n Channel,\n Emojis,\n SearchFile,\n Users,\n} from \"./interfaces."
},
{
"path": "src/data-write.ts",
"chars": 1255,
"preview": "import fs from \"fs-extra\";\nimport { differenceBy } from \"lodash-es\";\n\nimport { retry } from \"./retry.js\";\n\nexport async "
},
{
"path": "src/download-files.ts",
"chars": 3445,
"preview": "import fetch from \"node-fetch\";\nimport fs from \"fs-extra\";\nimport esMain from \"es-main\";\nimport ora, { Ora } from \"ora\";"
},
{
"path": "src/emoji.ts",
"chars": 3469,
"preview": "import path from \"path\";\nimport ora from \"ora\";\nimport fs from \"fs\";\nimport { createRequire } from \"node:module\";\n\nimpor"
},
{
"path": "src/interfaces.ts",
"chars": 1633,
"preview": "import { Message as SlackMessage } from \"@slack/web-api/dist/response/ConversationsHistoryResponse\";\nimport { Channel as"
},
{
"path": "src/messages.ts",
"chars": 3836,
"preview": "import {\n ConversationsHistoryResponse,\n ConversationsListArguments,\n ConversationsListResponse,\n} from \"@slack/web-a"
},
{
"path": "src/reactions.ts",
"chars": 152,
"preview": "import { Message } from \"./interfaces\";\n\nexport function hasReactions(message: Message) {\n return message.reactions && "
},
{
"path": "src/retry.ts",
"chars": 755,
"preview": "export interface RetryOptions {\n retries: number;\n name?: string;\n}\n\nconst defaultOptions: RetryOptions = {\n retries:"
},
{
"path": "src/search.ts",
"chars": 3713,
"preview": "import fs from \"fs-extra\";\nimport ora, { Ora } from \"ora\";\nimport { getChannelName } from \"./channels.js\";\n\nimport {\n N"
},
{
"path": "src/threads.ts",
"chars": 145,
"preview": "import { Message } from \"./interfaces\";\n\nexport function isThread(message: Message) {\n return message.reply_count && me"
},
{
"path": "src/timestamp.ts",
"chars": 227,
"preview": "export function slackTimestampToJavaScriptTimestamp(ts?: string) {\n if (!ts) {\n return 0;\n }\n\n const splitTs = ts."
},
{
"path": "src/users.ts",
"chars": 2481,
"preview": "import path from \"path\";\n\nimport { getWebClient } from \"./web-client.js\";\nimport { Message, User, Users } from \"./interf"
},
{
"path": "src/web-client.ts",
"chars": 333,
"preview": "import { WebClient } from \"@slack/web-api\";\n\nimport { config } from \"./config.js\";\n\nlet _webClient: WebClient;\nexport fu"
},
{
"path": "static/scroll.js",
"chars": 130,
"preview": "if (window.location.hash) {\n document.getElementById(window.location.hash).scrollTo();\n} else {\n scrollBy({ top: 99999"
},
{
"path": "static/search.html",
"chars": 6746,
"preview": "<!DOCTYPE html>\n<html lang=\"en\">\n <head>\n <meta charset=\"UTF-8\" />\n <meta http-equiv=\"X-UA-Compatible\" content=\"I"
},
{
"path": "static/style.css",
"chars": 4759,
"preview": "/* Reset */\n\n/* Box sizing rules */\n*,\n*::before,\n*::after {\n box-sizing: border-box;\n}\n\n/* Remove default margin */\nbo"
},
{
"path": "tsconfig.json",
"chars": 10929,
"preview": "{\n \"compilerOptions\": {\n \"jsx\": \"react\",\n /* Visit https://aka.ms/tsconfig.json to read more about this file */\n\n"
}
]
About this extraction
This page contains the full source code of the felixrieseberg/slack-archive GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 30 files (87.7 KB), approximately 22.6k tokens, and a symbol index with 134 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.