Repository: typeonce-dev/sync-engine-web Branch: main Commit: 956aaa9cec0f Files: 68 Total size: 101.0 KB Directory structure: gitextract_usv87bki/ ├── .gitignore ├── .npmrc ├── .vscode/ │ └── settings.json ├── README.md ├── apps/ │ ├── client/ │ │ ├── .gitignore │ │ ├── index.html │ │ ├── package.json │ │ ├── src/ │ │ │ ├── lib/ │ │ │ │ ├── constants.ts │ │ │ │ ├── hooks/ │ │ │ │ │ ├── use-food.ts │ │ │ │ │ └── use-meal.ts │ │ │ │ ├── runtime-client.ts │ │ │ │ └── services/ │ │ │ │ └── storage.ts │ │ │ ├── main.tsx │ │ │ ├── routeTree.gen.ts │ │ │ ├── routes/ │ │ │ │ ├── $workspaceId/ │ │ │ │ │ ├── index.tsx │ │ │ │ │ ├── join.tsx │ │ │ │ │ └── token.tsx │ │ │ │ ├── __root.tsx │ │ │ │ └── index.tsx │ │ │ └── workers/ │ │ │ ├── bootstrap.ts │ │ │ └── live.ts │ │ ├── tsconfig.json │ │ └── vite.config.ts │ └── server/ │ ├── drizzle/ │ │ ├── 0000_supreme_bedlam.sql │ │ └── meta/ │ │ ├── 0000_snapshot.json │ │ └── _journal.json │ ├── drizzle.config.ts │ ├── package.json │ ├── src/ │ │ ├── database.ts │ │ ├── db/ │ │ │ └── schema.ts │ │ ├── group/ │ │ │ ├── sync-auth.ts │ │ │ └── sync-data.ts │ │ ├── main.ts │ │ ├── middleware/ │ │ │ ├── authorization.ts │ │ │ ├── master-authorization.ts │ │ │ └── version-check.ts │ │ └── services/ │ │ ├── drizzle.ts │ │ └── jwt.ts │ └── tsconfig.json ├── docker-compose.yaml ├── package.json ├── packages/ │ ├── client-lib/ │ │ ├── package.json │ │ ├── src/ │ │ │ ├── main.ts │ │ │ ├── runtime-layer.ts │ │ │ ├── schema.ts │ │ │ ├── services/ │ │ │ │ ├── api-client.ts │ │ │ │ ├── dexie.ts │ │ │ │ ├── index.ts │ │ │ │ ├── loro-storage.ts │ │ │ │ ├── migration.ts │ │ │ │ ├── sync.ts │ │ │ │ ├── temp-workspace.ts │ │ │ │ └── workspace-manager.ts │ │ │ ├── sync-worker.ts │ │ │ ├── use-action-effect.ts │ │ │ └── use-dexie-query.ts │ │ └── tsconfig.json │ ├── schema/ │ │ ├── package.json │ │ ├── src/ │ │ │ ├── main.ts │ │ │ ├── migrations.ts │ │ │ ├── schema.ts │ │ │ └── versioning.ts │ │ └── tsconfig.json │ └── sync/ │ ├── package.json │ ├── src/ │ │ └── main.ts │ └── tsconfig.json ├── pnpm-workspace.yaml └── turbo.json ================================================ FILE CONTENTS ================================================ ================================================ FILE: .gitignore ================================================ # See https://help.github.com/articles/ignoring-files/ for more about ignoring files. # Dependencies node_modules .pnp .pnp.js # Local env files .env .env.local .env.development.local .env.test.local .env.production.local # Testing coverage # Turbo .turbo # Vercel .vercel # Build Outputs .next/ out/ build dist # Debug npm-debug.log* yarn-debug.log* yarn-error.log* # Misc .DS_Store *.pem ================================================ FILE: .npmrc ================================================ ================================================ FILE: .vscode/settings.json ================================================ { "files.watcherExclude": { "**/routeTree.gen.ts": true }, "search.exclude": { "**/routeTree.gen.ts": true }, "files.readonlyInclude": { "**/routeTree.gen.ts": true } } ================================================ FILE: README.md ================================================ # Sync Engine Web A local-first, offline-capable web sync engine implementation with CRDT-based synchronization. The project provides a complete solution for data synchronization between multiple devices while maintaining data consistency and offline capabilities. ## Architecture The project is structured as a monorepo with three main components: 1. **Client** - Web-based UI implementation - Local-first storage using IndexedDB ([`dexie`](https://dexie.org/)) - Background sync using Web Workers - Client-side schema migrations - Offline-first data management 2. **Server** - REST API for data synchronization - JWT-based authentication and authorization - Byte-level data storage (client-agnostic) - Workspace management and backup - Token-based access control 3. **Shared Packages** - Schema definitions - Type definitions - Shared utilities - Migration utilities ## Key Features - **Local-First Architecture**: Data is primarily stored and managed locally, with server acting as sync + backup - **CRDT-Based Sync**: Uses [`loro-crdt`](https://loro.dev/) for conflict-free data synchronization - **Offline Support**: Full offline capability with background sync - **Multi-Device Access**: Share workspaces across devices using access tokens - **Type-Safe Migrations**: Version-controlled schema migrations - **Secure Authentication**: JWT-based auth with master/access token system ## Implementation Details - Client stores and syncs data using CRDT (Conflict-free Replicated Data Type) - Server is client-agnostic, storing only byte-level data - Master client controls workspace access through token generation - Schema migrations are performed client-side - Background syncing handled by Web Workers - No server-side querying - all data operations happen client-side ## Authentication Flow 1. Client creates local workspace 2. Server stores workspace and designates client as "master" 3. Master client receives master token 4. Master client generates access tokens for other devices 5. Access tokens can be shared via links 6. Server handles token verification and authorization ## Data Flow ``` Client (Local Storage) <-> Web Worker (Background Sync) <-> Server (Byte Storage) ``` - UI reads directly from local storage - Sync operations happen in background - Server stores encoded CRDT data as bytes - Migrations happen during client initialization ## Technologies - CRDT: `loro-crdt` for data synchronization - Storage: IndexedDB (client), Database (server) - Authentication: JWT tokens - Background Processing: Web Workers ## Dependencies ### Client - **UI & Routing** - `react` - UI framework - `@tanstack/react-router` - Type-safe routing - **Storage & Sync** - `loro-crdt` - CRDT implementation - `dexie` - IndexedDB wrapper - `dexie-react-hooks` - React hooks for Dexie - **Core** - `effect` - Functional programming toolkit - `@effect/platform-browser` - Browser-specific Effect utilities ### Server - **Database** - `drizzle-orm` - TypeScript ORM - `@effect/sql` - SQL integration for Effect - `@effect/sql-pg` - PostgreSQL driver - `pg` - PostgreSQL client - **Authentication** - `jsonwebtoken` - JWT implementation - **Core** - `effect` - Functional programming toolkit - `@effect/platform-node` - Node.js-specific Effect utilities ### Build Tools - `turbo` - Monorepo build system - `vite` - Frontend build tool - `typescript` - Type system ## Project Status This is a functional implementation of a web sync engine, optimized for single-user scenarios (not real-time collaboration). The focus is on providing reliable data synchronization while maintaining offline capabilities. ================================================ FILE: apps/client/.gitignore ================================================ # Local .DS_Store *.local *.log* # Dist node_modules dist/ .vinxi .output .vercel .netlify .wrangler # IDE .vscode/* !.vscode/extensions.json .idea ================================================ FILE: apps/client/index.html ================================================ TanStack Router
================================================ FILE: apps/client/package.json ================================================ { "name": "client", "version": "0.0.0", "private": true, "type": "module", "scripts": { "typecheck": "tsc --noEmit", "dev": "vite --port=3001", "build": "vite build", "serve": "vite preview", "start": "vite" }, "devDependencies": { "@tanstack/router-plugin": "^1.105.0", "@types/react": "^19.0.8", "@types/react-dom": "^19.0.3", "@vitejs/plugin-react": "^4.3.2", "vite": "^6.0.3", "vite-plugin-top-level-await": "^1.5.0", "vite-plugin-wasm": "^3.4.1" }, "dependencies": { "@effect/platform": "^0.77.2", "@effect/platform-browser": "^0.56.2", "@local/sync": "workspace:*", "@local/schema": "workspace:*", "@local/client-lib": "workspace:*", "@tanstack/react-router": "^1.105.0", "dexie": "^4.0.11", "dexie-react-hooks": "^1.1.7", "effect": "^3.13.2", "loro-crdt": "^1.4.2", "react": "^19.0.0", "react-dom": "^19.0.0" } } ================================================ FILE: apps/client/src/lib/constants.ts ================================================ export const WEBSITE_URL = "http://localhost:3001"; ================================================ FILE: apps/client/src/lib/hooks/use-food.ts ================================================ import { Service, useDexieQuery } from "@local/client-lib"; import { SnapshotSchema } from "@local/schema"; import { RuntimeClient } from "../runtime-client"; export const useFood = ({ workspaceId }: { workspaceId: string }) => { return useDexieQuery(() => RuntimeClient.runPromise( Service.LoroStorage.use(({ query }) => query((doc) => doc.getList("food"), SnapshotSchema.fields.food.value, { workspaceId, }) ) ) ); }; ================================================ FILE: apps/client/src/lib/hooks/use-meal.ts ================================================ import { Service, useDexieQuery } from "@local/client-lib"; import { SnapshotSchema } from "@local/schema"; import { Effect } from "effect"; import { RuntimeClient } from "../runtime-client"; export const useMeal = ({ workspaceId }: { workspaceId: string }) => { return useDexieQuery(() => RuntimeClient.runPromise( Effect.gen(function* () { const { query } = yield* Service.LoroStorage; const meals = yield* query( (doc) => doc.getList("meal"), SnapshotSchema.fields.meal.value, { workspaceId } ); const foods = yield* query( (doc) => doc.getList("food"), SnapshotSchema.fields.food.value, { workspaceId } ); return meals.map(({ foodId, id, quantity }) => { const food = foods.find((food) => food.id === foodId); return { id, quantity, food }; }); }) ) ); }; ================================================ FILE: apps/client/src/lib/runtime-client.ts ================================================ import { RuntimeLayer } from "@local/client-lib"; import { Layer, ManagedRuntime } from "effect"; import { Storage } from "./services/storage"; const MainLayer = Layer.mergeAll(RuntimeLayer, Storage.Default); export const RuntimeClient = ManagedRuntime.make(MainLayer); ================================================ FILE: apps/client/src/lib/services/storage.ts ================================================ import { Service } from "@local/client-lib"; import { CurrentSchema, SnapshotSchema } from "@local/schema"; import { Effect, Schema } from "effect"; import { LoroMap, VersionVector } from "loro-crdt"; export class Storage extends Effect.Service()("Storage", { dependencies: [Service.TempWorkspace.Default, Service.LoroStorage.Default], effect: Effect.gen(function* () { const temp = yield* Service.TempWorkspace; const { load } = yield* Service.LoroStorage; const insert = (table: T) => ({ workspaceId, value, }: { workspaceId: string; value: Schema.Schema.Encoded<(typeof CurrentSchema.fields)[T]>[number]; }) => Effect.gen(function* () { const { doc, workspace } = yield* load({ workspaceId }); const list = doc.getList(table); const container = list.insertContainer(list.length, new LoroMap()); const data = yield* Schema.encode( CurrentSchema.fields[table].value as Schema.Schema )(value); Object.entries(data).forEach(([key, val]) => { container.set(key, val); }); const snapshotExport = workspace === undefined ? doc.export({ mode: "snapshot" }) : doc.export({ mode: "update", from: new VersionVector(workspace.version), }); return yield* temp.put({ workspaceId, snapshot: snapshotExport, snapshotId: crypto.randomUUID(), }); }); return { insertFood: insert("food"), insertMeal: insert("meal"), } as const; }), }) {} ================================================ FILE: apps/client/src/main.tsx ================================================ import { RouterProvider, createRouter } from "@tanstack/react-router"; import ReactDOM from "react-dom/client"; import { routeTree } from "./routeTree.gen"; // Set up a Router instance const router = createRouter({ routeTree, defaultPreload: "intent", }); // Register things for typesafety declare module "@tanstack/react-router" { interface Register { router: typeof router; } } const rootElement = document.getElementById("app")!; if (!rootElement.innerHTML) { const root = ReactDOM.createRoot(rootElement); root.render(); } ================================================ FILE: apps/client/src/routeTree.gen.ts ================================================ /* eslint-disable */ // @ts-nocheck // noinspection JSUnusedGlobalSymbols // This file was automatically generated by TanStack Router. // You should NOT make any changes in this file as it will be overwritten. // Additionally, you should also exclude this file from your linter and/or formatter to prevent it from being checked or modified. // Import Routes import { Route as rootRoute } from './routes/__root' import { Route as IndexImport } from './routes/index' import { Route as WorkspaceIdIndexImport } from './routes/$workspaceId/index' import { Route as WorkspaceIdTokenImport } from './routes/$workspaceId/token' import { Route as WorkspaceIdJoinImport } from './routes/$workspaceId/join' // Create/Update Routes const IndexRoute = IndexImport.update({ id: '/', path: '/', getParentRoute: () => rootRoute, } as any) const WorkspaceIdIndexRoute = WorkspaceIdIndexImport.update({ id: '/$workspaceId/', path: '/$workspaceId/', getParentRoute: () => rootRoute, } as any) const WorkspaceIdTokenRoute = WorkspaceIdTokenImport.update({ id: '/$workspaceId/token', path: '/$workspaceId/token', getParentRoute: () => rootRoute, } as any) const WorkspaceIdJoinRoute = WorkspaceIdJoinImport.update({ id: '/$workspaceId/join', path: '/$workspaceId/join', getParentRoute: () => rootRoute, } as any) // Populate the FileRoutesByPath interface declare module '@tanstack/react-router' { interface FileRoutesByPath { '/': { id: '/' path: '/' fullPath: '/' preLoaderRoute: typeof IndexImport parentRoute: typeof rootRoute } '/$workspaceId/join': { id: '/$workspaceId/join' path: '/$workspaceId/join' fullPath: '/$workspaceId/join' preLoaderRoute: typeof WorkspaceIdJoinImport parentRoute: typeof rootRoute } '/$workspaceId/token': { id: '/$workspaceId/token' path: '/$workspaceId/token' fullPath: '/$workspaceId/token' preLoaderRoute: typeof WorkspaceIdTokenImport parentRoute: typeof rootRoute } '/$workspaceId/': { id: '/$workspaceId/' path: '/$workspaceId' fullPath: '/$workspaceId' preLoaderRoute: typeof WorkspaceIdIndexImport parentRoute: typeof rootRoute } } } // Create and export the route tree export interface FileRoutesByFullPath { '/': typeof IndexRoute '/$workspaceId/join': typeof WorkspaceIdJoinRoute '/$workspaceId/token': typeof WorkspaceIdTokenRoute '/$workspaceId': typeof WorkspaceIdIndexRoute } export interface FileRoutesByTo { '/': typeof IndexRoute '/$workspaceId/join': typeof WorkspaceIdJoinRoute '/$workspaceId/token': typeof WorkspaceIdTokenRoute '/$workspaceId': typeof WorkspaceIdIndexRoute } export interface FileRoutesById { __root__: typeof rootRoute '/': typeof IndexRoute '/$workspaceId/join': typeof WorkspaceIdJoinRoute '/$workspaceId/token': typeof WorkspaceIdTokenRoute '/$workspaceId/': typeof WorkspaceIdIndexRoute } export interface FileRouteTypes { fileRoutesByFullPath: FileRoutesByFullPath fullPaths: | '/' | '/$workspaceId/join' | '/$workspaceId/token' | '/$workspaceId' fileRoutesByTo: FileRoutesByTo to: '/' | '/$workspaceId/join' | '/$workspaceId/token' | '/$workspaceId' id: | '__root__' | '/' | '/$workspaceId/join' | '/$workspaceId/token' | '/$workspaceId/' fileRoutesById: FileRoutesById } export interface RootRouteChildren { IndexRoute: typeof IndexRoute WorkspaceIdJoinRoute: typeof WorkspaceIdJoinRoute WorkspaceIdTokenRoute: typeof WorkspaceIdTokenRoute WorkspaceIdIndexRoute: typeof WorkspaceIdIndexRoute } const rootRouteChildren: RootRouteChildren = { IndexRoute: IndexRoute, WorkspaceIdJoinRoute: WorkspaceIdJoinRoute, WorkspaceIdTokenRoute: WorkspaceIdTokenRoute, WorkspaceIdIndexRoute: WorkspaceIdIndexRoute, } export const routeTree = rootRoute ._addFileChildren(rootRouteChildren) ._addFileTypes() /* ROUTE_MANIFEST_START { "routes": { "__root__": { "filePath": "__root.tsx", "children": [ "/", "/$workspaceId/join", "/$workspaceId/token", "/$workspaceId/" ] }, "/": { "filePath": "index.tsx" }, "/$workspaceId/join": { "filePath": "$workspaceId/join.tsx" }, "/$workspaceId/token": { "filePath": "$workspaceId/token.tsx" }, "/$workspaceId/": { "filePath": "$workspaceId/index.tsx" } } } ROUTE_MANIFEST_END */ ================================================ FILE: apps/client/src/routes/$workspaceId/index.tsx ================================================ import { Worker } from "@effect/platform"; import { BrowserWorker } from "@effect/platform-browser"; import { Service, SyncWorker, useActionEffect } from "@local/client-lib"; import { createFileRoute, Link } from "@tanstack/react-router"; import { Effect } from "effect"; import { startTransition, useEffect } from "react"; import { useFood } from "../../lib/hooks/use-food"; import { useMeal } from "../../lib/hooks/use-meal"; import { RuntimeClient } from "../../lib/runtime-client"; import { Storage } from "../../lib/services/storage"; const bootstrap = ({ workspaceId }: { workspaceId: string }) => Effect.gen(function* () { const pool = yield* Worker.makePoolSerialized({ size: 1 }); return yield* pool.broadcast(new SyncWorker.Bootstrap({ workspaceId })); }).pipe( Effect.scoped, Effect.provide( BrowserWorker.layer( () => new globalThis.Worker( new URL("./src/workers/bootstrap.ts", globalThis.origin), { type: "module" } ) ) ), Effect.catchAll((error) => Effect.logError("Bootstrap error", error)) ); export const Route = createFileRoute("/$workspaceId/")({ component: RouteComponent, loader: ({ params: { workspaceId } }) => RuntimeClient.runPromise( Service.WorkspaceManager.getById({ workspaceId }).pipe( Effect.flatMap(Effect.fromNullable), Effect.tap(({ workspaceId }) => bootstrap({ workspaceId })) ) ), }); function RouteComponent() { const workspace = Route.useLoaderData(); const { data, error, loading } = useFood({ workspaceId: workspace.workspaceId, }); const { data: meals } = useMeal({ workspaceId: workspace.workspaceId, }); const [, onBootstrap, bootstrapping] = useActionEffect( RuntimeClient, bootstrap ); const [, onAddFood] = useActionEffect(RuntimeClient, (formData: FormData) => Effect.gen(function* () { const loroStorage = yield* Storage; const name = formData.get("name") as string; const calories = formData.get("calories") as string; yield* loroStorage.insertFood({ workspaceId: workspace.workspaceId, value: { id: crypto.randomUUID(), name, calories: parseInt(calories, 10), }, }); }) ); const [, onAddMeal] = useActionEffect(RuntimeClient, (formData: FormData) => Effect.gen(function* () { const loroStorage = yield* Storage; const foodId = formData.get("foodId") as string; const quantity = formData.get("quantity") as string; yield* loroStorage.insertMeal({ workspaceId: workspace.workspaceId, value: { id: crypto.randomUUID(), foodId, quantity: parseInt(quantity, 10), }, }); }) ); useEffect(() => { const url = new URL("./src/workers/live.ts", globalThis.origin); const newWorker = new globalThis.Worker(url, { type: "module" }); void RuntimeClient.runPromise( Effect.gen(function* () { const pool = yield* Worker.makePoolSerialized({ size: 1 }); return yield* pool.broadcast( new SyncWorker.LiveQuery({ workspaceId: workspace.workspaceId }) ); }).pipe( Effect.scoped, Effect.provide(BrowserWorker.layer(() => newWorker)) ) ); newWorker.onerror = (error) => { console.error("Live query worker error", error); }; return () => { newWorker.terminate(); }; }, []); return (
Tokens

{workspace.workspaceId}

{loading &&

Loading...

} {error && 
{JSON.stringify(error, null, 2)}
} {(data ?? []).map((food) => (

Name: {food.name}

Calories: {food.calories}

))}
{(data ?? []).map((food) => (
))}
{(meals ?? []).map((meal) => (

Food: {meal.food?.name}

Quantity: {meal.quantity}

))}
); } ================================================ FILE: apps/client/src/routes/$workspaceId/join.tsx ================================================ import { Service } from "@local/client-lib"; import { createFileRoute, redirect } from "@tanstack/react-router"; import { Effect } from "effect"; import { RuntimeClient } from "../../lib/runtime-client"; export const Route = createFileRoute("/$workspaceId/join")({ component: RouteComponent, loader: ({ params }) => RuntimeClient.runPromise( Effect.gen(function* () { const { join } = yield* Service.Sync; yield* join({ workspaceId: params.workspaceId }); return redirect({ to: `/$workspaceId`, params: { workspaceId: params.workspaceId }, }); }) ), }); function RouteComponent() { return null; } ================================================ FILE: apps/client/src/routes/$workspaceId/token.tsx ================================================ import { Service, useActionEffect } from "@local/client-lib"; import { createFileRoute, Link, useRouter } from "@tanstack/react-router"; import { Duration, Effect } from "effect"; import { WEBSITE_URL } from "../../lib/constants"; import { RuntimeClient } from "../../lib/runtime-client"; export const Route = createFileRoute("/$workspaceId/token")({ component: RouteComponent, loader: ({ params: { workspaceId } }) => RuntimeClient.runPromise( Effect.gen(function* () { const api = yield* Service.ApiClient; const token = yield* Service.WorkspaceManager.getById({ workspaceId, }).pipe( Effect.flatMap((workspace) => Effect.fromNullable(workspace?.token)) ); const tokens = yield* api.client.syncAuth.listTokens({ path: { workspaceId }, headers: { "x-api-key": token }, }); return { tokens, token }; }) ), }); function RouteComponent() { const { workspaceId } = Route.useParams(); const { tokens, token } = Route.useLoaderData(); const router = useRouter(); const [, onIssueToken, issuing] = useActionEffect( RuntimeClient, (formData: FormData) => Effect.gen(function* () { const api = yield* Service.ApiClient; const clientId = formData.get("clientId") as string; yield* api.client.syncAuth.issueToken({ path: { workspaceId }, headers: { "x-api-key": token }, payload: { clientId, expiresIn: Duration.days(30), scope: "read_write", }, }); yield* Effect.promise(() => router.invalidate({ sync: true })); }) ); const [, onRevoke, revoking] = useActionEffect( RuntimeClient, (formData: FormData) => Effect.gen(function* () { const api = yield* Service.ApiClient; const clientId = formData.get("clientId") as string; yield* api.client.syncAuth.revokeToken({ path: { workspaceId, clientId }, headers: { "x-api-key": token }, }); yield* Effect.promise(() => router.invalidate({ sync: true })); }) ); return (
Back

Tokens

{tokens.map((token, index) => ( ))}
Index clientId isMaster scope issuedAt expiresAt revokedAt Share
{index} {token.clientId} {token.isMaster ? "✔️" : "❌"} {token.scope} {new Date(token.issuedAt).toLocaleDateString(undefined, { year: "numeric", month: "long", day: "numeric", })} {token.expiresAt ? new Date(token.expiresAt).toLocaleDateString(undefined, { year: "numeric", month: "long", day: "numeric", }) : "N/A"} {token.revokedAt ? ( {new Date(token.revokedAt).toLocaleDateString(undefined, { year: "numeric", month: "long", day: "numeric", })} ) : (
)}
); } ================================================ FILE: apps/client/src/routes/__root.tsx ================================================ import { Service } from "@local/client-lib"; import { Outlet, createRootRoute } from "@tanstack/react-router"; import { Effect } from "effect"; import { RuntimeClient } from "../lib/runtime-client"; export const Route = createRootRoute({ component: RootComponent, loader: () => RuntimeClient.runPromise( Service.Migration.pipe( Effect.flatMap((migration) => migration.migrate), Effect.catchAll((error) => Effect.logError("Migration error", error)), Effect.andThen( Effect.gen(function* () { const { initClient } = yield* Service.Dexie; return yield* initClient; }) ) ) ), }); function RootComponent() { const clientId = Route.useLoaderData(); return ( <> ); } ================================================ FILE: apps/client/src/routes/index.tsx ================================================ import { Service, useActionEffect } from "@local/client-lib"; import { createFileRoute, Link, useNavigate } from "@tanstack/react-router"; import { Effect } from "effect"; import { RuntimeClient } from "../lib/runtime-client"; export const Route = createFileRoute("/")({ component: HomeComponent, loader: () => RuntimeClient.runPromise(Service.WorkspaceManager.getAll), }); function HomeComponent() { const allWorkspaces = Route.useLoaderData(); const navigate = useNavigate(); const [, joinWorkspace] = useActionEffect(RuntimeClient, () => Effect.gen(function* () { const workspace = yield* Service.WorkspaceManager.create; yield* Effect.sync(() => navigate({ to: `/$workspaceId`, params: { workspaceId: workspace.workspaceId }, }) ); }) ); return (

Select workspace

{allWorkspaces.map((workspace) => ( {workspace.workspaceId} ))}
); } ================================================ FILE: apps/client/src/workers/bootstrap.ts ================================================ import { WorkerRunner } from "@effect/platform"; import { BrowserWorkerRunner } from "@effect/platform-browser"; import { SyncWorker } from "@local/client-lib"; import { Effect, Layer } from "effect"; import { RuntimeClient } from "../lib/runtime-client"; const WorkerLive = WorkerRunner.layerSerialized(SyncWorker.WorkerMessage, { Bootstrap: (params) => Effect.gen(function* () { const worker = yield* SyncWorker.SyncWorker; return yield* worker.bootstrap(params); }), }).pipe(Layer.provide(BrowserWorkerRunner.layer)); RuntimeClient.runFork(WorkerRunner.launch(WorkerLive)); ================================================ FILE: apps/client/src/workers/live.ts ================================================ import { WorkerRunner } from "@effect/platform"; import { BrowserWorkerRunner } from "@effect/platform-browser"; import { SyncWorker } from "@local/client-lib"; import { Effect, Layer } from "effect"; import { RuntimeClient } from "../lib/runtime-client"; const WorkerLive = WorkerRunner.layer((params: SyncWorker.LiveQuery) => Effect.scoped( Effect.gen(function* () { const worker = yield* SyncWorker.SyncWorker; yield* Effect.fork(worker.liveSync({ workspaceId: params.workspaceId })); yield* Effect.never; }) ) ).pipe(Layer.provide(BrowserWorkerRunner.layer)); RuntimeClient.runFork(WorkerRunner.launch(WorkerLive)); ================================================ FILE: apps/client/tsconfig.json ================================================ { "compilerOptions": { "strict": true, "esModuleInterop": true, "jsx": "react-jsx", "target": "ESNext", "module": "ESNext", "moduleResolution": "Bundler", "skipLibCheck": true, "allowJs": true, "resolveJsonModule": true, "moduleDetection": "force", "isolatedModules": true, "verbatimModuleSyntax": true, "noUncheckedIndexedAccess": true, // "exactOptionalPropertyTypes": true, "noImplicitOverride": true, "noEmit": true, "lib": ["es2022", "dom", "dom.iterable"], "types": ["vite/client"] }, "include": ["**/*.ts", "**/*.tsx"], "exclude": ["node_modules"] } ================================================ FILE: apps/client/vite.config.ts ================================================ import { TanStackRouterVite } from "@tanstack/router-plugin/vite"; import react from "@vitejs/plugin-react"; import { defineConfig } from "vite"; import topLevelAwait from "vite-plugin-top-level-await"; import wasm from "vite-plugin-wasm"; // https://vitejs.dev/config/ export default defineConfig({ plugins: [TanStackRouterVite({}), react(), wasm(), topLevelAwait()], worker: { format: "es" }, }); ================================================ FILE: apps/server/drizzle/0000_supreme_bedlam.sql ================================================ CREATE TYPE "public"."scope" AS ENUM('read', 'read_write');--> statement-breakpoint CREATE TABLE "client" ( "clientId" uuid NOT NULL, "createdAt" timestamp DEFAULT now() NOT NULL ); --> statement-breakpoint CREATE TABLE "token" ( "tokenId" integer PRIMARY KEY GENERATED ALWAYS AS IDENTITY (sequence name "token_tokenId_seq" INCREMENT BY 1 MINVALUE 1 MAXVALUE 2147483647 START WITH 1 CACHE 1), "tokenValue" varchar NOT NULL, "clientId" uuid NOT NULL, "workspaceId" uuid NOT NULL, "isMaster" boolean DEFAULT false NOT NULL, "scope" "scope" NOT NULL, "issuedAt" timestamp DEFAULT now() NOT NULL, "expiresAt" timestamp, "revokedAt" timestamp ); --> statement-breakpoint CREATE TABLE "workspace" ( "workspaceId" uuid NOT NULL, "ownerClientId" uuid NOT NULL, "clientId" uuid NOT NULL, "snapshotId" uuid NOT NULL, "createdAt" timestamp DEFAULT now() NOT NULL, "snapshot" "bytea" NOT NULL, CONSTRAINT "workspace_snapshotId_unique" UNIQUE("snapshotId") ); ================================================ FILE: apps/server/drizzle/meta/0000_snapshot.json ================================================ { "id": "b089a447-bbae-4d52-b83e-a0ce1cc4b359", "prevId": "00000000-0000-0000-0000-000000000000", "version": "7", "dialect": "postgresql", "tables": { "public.client": { "name": "client", "schema": "", "columns": { "clientId": { "name": "clientId", "type": "uuid", "primaryKey": false, "notNull": true }, "createdAt": { "name": "createdAt", "type": "timestamp", "primaryKey": false, "notNull": true, "default": "now()" } }, "indexes": {}, "foreignKeys": {}, "compositePrimaryKeys": {}, "uniqueConstraints": {}, "policies": {}, "checkConstraints": {}, "isRLSEnabled": false }, "public.token": { "name": "token", "schema": "", "columns": { "tokenId": { "name": "tokenId", "type": "integer", "primaryKey": true, "notNull": true, "identity": { "type": "always", "name": "token_tokenId_seq", "schema": "public", "increment": "1", "startWith": "1", "minValue": "1", "maxValue": "2147483647", "cache": "1", "cycle": false } }, "tokenValue": { "name": "tokenValue", "type": "varchar", "primaryKey": false, "notNull": true }, "clientId": { "name": "clientId", "type": "uuid", "primaryKey": false, "notNull": true }, "workspaceId": { "name": "workspaceId", "type": "uuid", "primaryKey": false, "notNull": true }, "isMaster": { "name": "isMaster", "type": "boolean", "primaryKey": false, "notNull": true, "default": false }, "scope": { "name": "scope", "type": "scope", "typeSchema": "public", "primaryKey": false, "notNull": true }, "issuedAt": { "name": "issuedAt", "type": "timestamp", "primaryKey": false, "notNull": true, "default": "now()" }, "expiresAt": { "name": "expiresAt", "type": "timestamp", "primaryKey": false, "notNull": false }, "revokedAt": { "name": "revokedAt", "type": "timestamp", "primaryKey": false, "notNull": false } }, "indexes": {}, "foreignKeys": {}, "compositePrimaryKeys": {}, "uniqueConstraints": {}, "policies": {}, "checkConstraints": {}, "isRLSEnabled": false }, "public.workspace": { "name": "workspace", "schema": "", "columns": { "workspaceId": { "name": "workspaceId", "type": "uuid", "primaryKey": false, "notNull": true }, "ownerClientId": { "name": "ownerClientId", "type": "uuid", "primaryKey": false, "notNull": true }, "clientId": { "name": "clientId", "type": "uuid", "primaryKey": false, "notNull": true }, "snapshotId": { "name": "snapshotId", "type": "uuid", "primaryKey": false, "notNull": true }, "createdAt": { "name": "createdAt", "type": "timestamp", "primaryKey": false, "notNull": true, "default": "now()" }, "snapshot": { "name": "snapshot", "type": "bytea", "primaryKey": false, "notNull": true } }, "indexes": {}, "foreignKeys": {}, "compositePrimaryKeys": {}, "uniqueConstraints": { "workspace_snapshotId_unique": { "name": "workspace_snapshotId_unique", "nullsNotDistinct": false, "columns": [ "snapshotId" ] } }, "policies": {}, "checkConstraints": {}, "isRLSEnabled": false } }, "enums": { "public.scope": { "name": "scope", "schema": "public", "values": [ "read", "read_write" ] } }, "schemas": {}, "sequences": {}, "roles": {}, "policies": {}, "views": {}, "_meta": { "columns": {}, "schemas": {}, "tables": {} } } ================================================ FILE: apps/server/drizzle/meta/_journal.json ================================================ { "version": "7", "dialect": "postgresql", "entries": [ { "idx": 0, "version": "7", "when": 1740889946454, "tag": "0000_supreme_bedlam", "breakpoints": true } ] } ================================================ FILE: apps/server/drizzle.config.ts ================================================ import { defineConfig } from "drizzle-kit"; export default defineConfig({ out: "./drizzle", schema: "./src/db/schema.ts", dialect: "postgresql", }); ================================================ FILE: apps/server/package.json ================================================ { "name": "@local/server", "version": "1.0.0", "description": "", "main": "main.js", "scripts": { "typecheck": "tsc", "dev": "tsx watch src/main.ts", "generate": "pnpm drizzle-kit generate" }, "keywords": [], "author": "", "license": "ISC", "devDependencies": { "@types/jsonwebtoken": "^9.0.9", "@types/node": "^22.5.1", "drizzle-kit": "^0.30.4", "tsx": "^4.19.2" }, "dependencies": { "@effect/platform": "^0.77.2", "@effect/platform-node": "^0.73.1", "@effect/sql": "^0.30.2", "@effect/sql-pg": "^0.31.2", "@local/sync": "workspace:*", "@local/schema": "workspace:*", "drizzle-orm": "^0.39.3", "effect": "^3.13.2", "jsonwebtoken": "^9.0.2", "pg": "^8.13.3" } } ================================================ FILE: apps/server/src/database.ts ================================================ import { PgClient } from "@effect/sql-pg"; import { Config, Effect, Layer, Redacted } from "effect"; const password = Config.redacted("POSTGRES_PASSWORD"); const username = Config.string("POSTGRES_USERNAME"); const database = Config.string("POSTGRES_DATABASE"); const host = Config.string("POSTGRES_HOST"); const port = Config.number("POSTGRES_PORT"); export const DatabaseUrl = Config.all({ username, password, host, port, database, }).pipe( Config.map(({ username, password, host, port, database }) => Redacted.make( `postgresql://${username}:${Redacted.value(password)}@${host}:${port}/${database}` ) ) ); export const DatabaseLive = DatabaseUrl.pipe( Effect.tap((url) => Effect.log(`Connecting to database: ${Redacted.value(url)}`) ), Effect.map((url) => PgClient.layer({ url })), Layer.unwrapEffect ); ================================================ FILE: apps/server/src/db/schema.ts ================================================ import { boolean, customType, integer, pgEnum, pgTable, timestamp, uuid, varchar, } from "drizzle-orm/pg-core"; export const scope = pgEnum("scope", ["read", "read_write"]); export const bytea = customType<{ data: Uint8Array }>({ dataType: () => "bytea", }); export const workspaceTable = pgTable("workspace", { workspaceId: uuid().notNull(), ownerClientId: uuid().notNull(), clientId: uuid().notNull(), snapshotId: uuid().notNull().unique(), createdAt: timestamp().notNull().defaultNow(), snapshot: bytea().notNull(), }); export const clientTable = pgTable("client", { clientId: uuid().notNull(), createdAt: timestamp().notNull().defaultNow(), }); export const tokenTable = pgTable("token", { tokenId: integer().primaryKey().generatedAlwaysAsIdentity(), tokenValue: varchar().notNull(), clientId: uuid().notNull(), workspaceId: uuid().notNull(), isMaster: boolean().notNull().default(false), scope: scope().notNull(), issuedAt: timestamp().notNull().defaultNow(), expiresAt: timestamp(), revokedAt: timestamp(), }); ================================================ FILE: apps/server/src/group/sync-auth.ts ================================================ import { HttpApiBuilder } from "@effect/platform"; import { AuthWorkspace, Scope, SyncApi } from "@local/sync"; import { and, eq, not } from "drizzle-orm"; import { DateTime, Effect, Layer, Schema } from "effect"; import { tokenTable, workspaceTable } from "../db/schema"; import { AuthorizationLive } from "../middleware/authorization"; import { MasterAuthorizationLive } from "../middleware/master-authorization"; import { VersionCheckLive } from "../middleware/version-check"; import { Drizzle } from "../services/drizzle"; import { Jwt } from "../services/jwt"; export const SyncAuthGroupLive = HttpApiBuilder.group( SyncApi, "syncAuth", (handlers) => Effect.gen(function* () { const jwt = yield* Jwt; const { query } = yield* Drizzle; return handlers .handle("generateToken", ({ payload }) => Effect.gen(function* () { yield* Effect.log( `Generating token for workspace ${payload.workspaceId}` ); const scope: typeof Scope.Type = "read_write"; const isMaster = true; const issuedAt = DateTime.toDate(yield* DateTime.now); yield* query({ Request: Schema.Struct({ workspaceId: Schema.String }), execute: (db, { workspaceId }) => db .select() .from(workspaceTable) .where(eq(workspaceTable.workspaceId, workspaceId)), })({ workspaceId: payload.workspaceId }).pipe( Effect.flatMap((rows) => rows.length === 0 ? Effect.void : Effect.fail({ message: "Workspace already exists" }) ) ); const token = yield* jwt.sign({ clientId: payload.clientId, workspaceId: payload.workspaceId, }); yield* Effect.all([ query({ Request: Schema.Struct({ clientId: Schema.String, workspaceId: Schema.String, snapshot: Schema.Uint8ArrayFromSelf, }), execute: (db, { clientId, snapshot, workspaceId }) => db.insert(workspaceTable).values({ snapshot, clientId, workspaceId, ownerClientId: clientId, snapshotId: payload.snapshotId, }), })({ clientId: payload.clientId, snapshot: payload.snapshot, workspaceId: payload.workspaceId, }), query({ Request: Schema.Struct({ clientId: Schema.String, workspaceId: Schema.String, tokenValue: Schema.String, }), execute: (db, { clientId, tokenValue, workspaceId }) => db.insert(tokenTable).values({ clientId, scope, tokenValue, workspaceId, isMaster, issuedAt, expiresAt: null, revokedAt: null, }), })({ clientId: payload.clientId, tokenValue: token, workspaceId: payload.workspaceId, }), ]); return { token, workspaceId: payload.workspaceId, snapshot: payload.snapshot, createdAt: issuedAt, }; }).pipe( Effect.tapErrorCause(Effect.logError), Effect.mapError((error) => error.message) ) ) .handle("listTokens", ({ path: { workspaceId } }) => Effect.gen(function* () { const workspace = yield* AuthWorkspace; const tokens = yield* query({ Request: Schema.Struct({ workspaceId: Schema.String, clientId: Schema.String, }), execute: (db, { workspaceId, clientId }) => db .select() .from(tokenTable) .where( and( eq(tokenTable.workspaceId, workspaceId), not(eq(tokenTable.clientId, clientId)) ) ), })({ workspaceId, clientId: workspace.clientId }); return tokens.map((token) => ({ clientId: token.clientId, tokenValue: token.tokenValue, scope: token.scope, isMaster: token.isMaster, issuedAt: token.issuedAt, expiresAt: token.expiresAt, revokedAt: token.revokedAt, })); }).pipe( Effect.tapErrorCause(Effect.logError), Effect.mapError((error) => error.message) ) ) .handle("issueToken", ({ path: { workspaceId }, payload }) => Effect.gen(function* () { yield* Effect.log(`Issuing token for workspace ${workspaceId}`); const issuedAt = yield* DateTime.now; const expiresAt = issuedAt.pipe( DateTime.addDuration(payload.expiresIn), DateTime.toDate ); const token = yield* jwt.sign({ clientId: payload.clientId, workspaceId, }); yield* query({ Request: Schema.Struct({ clientId: Schema.String, workspaceId: Schema.String, tokenValue: Schema.String, scope: Scope, expiresAt: Schema.DateFromSelf, issuedAt: Schema.DateFromSelf, }), execute: ( db, { clientId, tokenValue, workspaceId, scope, expiresAt, issuedAt, } ) => db.insert(tokenTable).values({ clientId, scope, tokenValue, workspaceId, issuedAt, expiresAt, isMaster: false, revokedAt: null, }), })({ expiresAt, workspaceId, tokenValue: token, scope: payload.scope, clientId: payload.clientId, issuedAt: DateTime.toDate(issuedAt), }); return { token, expiresAt, scope: payload.scope, }; }).pipe( Effect.tapErrorCause(Effect.logError), Effect.mapError((error) => error.message) ) ) .handle("revokeToken", ({ path: { workspaceId, clientId } }) => Effect.gen(function* () { yield* Effect.log(`Revoking token for workspace ${workspaceId}`); const revokedAt = yield* DateTime.now; yield* query({ Request: Schema.Struct({ workspaceId: Schema.String, clientId: Schema.String, }), execute: (db, { workspaceId, clientId }) => db .update(tokenTable) .set({ revokedAt: DateTime.toDate(revokedAt) }) .where( and( eq(tokenTable.workspaceId, workspaceId), eq(tokenTable.clientId, clientId) ) ), })({ workspaceId, clientId }); return true; }).pipe( Effect.tapErrorCause(Effect.logError), Effect.mapError((error) => error.message) ) ); }) ).pipe( Layer.provide([ Drizzle.Default, AuthorizationLive, MasterAuthorizationLive, VersionCheckLive, Jwt.Default, ]) ); ================================================ FILE: apps/server/src/group/sync-data.ts ================================================ import { HttpApiBuilder } from "@effect/platform"; import { SnapshotToLoroDoc } from "@local/schema"; import { AuthWorkspace, SyncApi } from "@local/sync"; import { and, desc, eq, gt, isNull, or } from "drizzle-orm"; import { Array, Data, Effect, Layer, Schema } from "effect"; import { tokenTable, workspaceTable } from "../db/schema"; import { AuthorizationLive } from "../middleware/authorization"; import { VersionCheckLive } from "../middleware/version-check"; import { Drizzle } from "../services/drizzle"; class InvalidVersionError extends Data.TaggedError("InvalidVersionError")<{ reason: "missing" | "outdated"; }> {} export const SyncDataGroupLive = HttpApiBuilder.group( SyncApi, "syncData", (handlers) => Effect.gen(function* () { const { query } = yield* Drizzle; return handlers .handle( "push", ({ payload: { snapshot, snapshotId }, path: { workspaceId } }) => Effect.gen(function* () { const workspace = yield* AuthWorkspace; const doc = yield* Schema.decode(SnapshotToLoroDoc)(snapshot); yield* Effect.log(`Pushing workspace ${workspaceId}`); doc.import(workspace.snapshot); const newSnapshot = yield* Schema.encode(SnapshotToLoroDoc)(doc); const newWorkspace = yield* query({ Request: Schema.Struct({ newSnapshot: Schema.Uint8ArrayFromSelf, }), execute: (db, { newSnapshot: snapshot }) => db .insert(workspaceTable) .values({ snapshot, snapshotId, clientId: workspace.clientId, workspaceId: workspace.workspaceId, ownerClientId: workspace.ownerClientId, }) .returning(), })({ newSnapshot }).pipe(Effect.flatMap(Array.head)); return { workspaceId: newWorkspace.workspaceId, createdAt: newWorkspace.createdAt, snapshot: newWorkspace.snapshot, }; }).pipe( Effect.tapErrorCause(Effect.logError), Effect.mapError((error) => error.message) ) ) .handle("pull", ({ path: { workspaceId } }) => Effect.gen(function* () { const workspace = yield* query({ Request: Schema.Struct({ workspaceId: Schema.String }), execute: (db, { workspaceId }) => db .select() .from(workspaceTable) .where(eq(workspaceTable.workspaceId, workspaceId)) .orderBy(desc(workspaceTable.createdAt)) .limit(1), })({ workspaceId }).pipe( Effect.flatMap(Array.head), Effect.mapError(() => ({ message: "Missing workspace" })) ); return { snapshot: workspace.snapshot }; }).pipe( Effect.tapErrorCause(Effect.logError), Effect.mapError((error) => error.message) ) ) .handle("join", ({ path: { workspaceId, clientId } }) => Effect.gen(function* () { const { tokenValue } = yield* query({ Request: Schema.Struct({ clientId: Schema.String, workspaceId: Schema.String, }), execute: (db, { clientId, workspaceId }) => db .select() .from(tokenTable) .where( and( eq(tokenTable.workspaceId, workspaceId), eq(tokenTable.clientId, clientId), or( isNull(tokenTable.revokedAt), gt(tokenTable.revokedAt, new Date()) ), or( isNull(tokenTable.expiresAt), gt(tokenTable.expiresAt, new Date()) ) ) ) .orderBy(desc(tokenTable.issuedAt)) .limit(1), })({ clientId, workspaceId }).pipe( Effect.flatMap(Array.head), Effect.mapError(() => ({ message: "Missing token" })) ); const workspace = yield* query({ Request: Schema.Struct({ workspaceId: Schema.String }), execute: (db, { workspaceId }) => db .select() .from(workspaceTable) .where(eq(workspaceTable.workspaceId, workspaceId)) .orderBy(desc(workspaceTable.createdAt)) .limit(1), })({ workspaceId }).pipe( Effect.flatMap(Array.head), Effect.mapError(() => ({ message: "Missing workspace" })) ); return { token: tokenValue, snapshot: workspace.snapshot, }; }).pipe( Effect.tapErrorCause(Effect.logError), Effect.mapError((error) => error.message) ) ); }) ).pipe(Layer.provide([Drizzle.Default, AuthorizationLive, VersionCheckLive])); ================================================ FILE: apps/server/src/main.ts ================================================ import { HttpApiBuilder, HttpMiddleware, HttpServer, PlatformConfigProvider, } from "@effect/platform"; import { NodeFileSystem, NodeHttpServer, NodeRuntime, } from "@effect/platform-node"; import { SyncApi } from "@local/sync"; import { Effect, Layer } from "effect"; import { createServer } from "node:http"; import { SyncAuthGroupLive } from "./group/sync-auth"; import { SyncDataGroupLive } from "./group/sync-data"; import { Drizzle } from "./services/drizzle"; const EnvProviderLayer = Layer.unwrapEffect( PlatformConfigProvider.fromDotEnv(".env").pipe( Effect.map(Layer.setConfigProvider), Effect.provide(NodeFileSystem.layer) ) ); const MainApiLive = HttpApiBuilder.api(SyncApi).pipe( Layer.provide([Drizzle.Default, SyncDataGroupLive, SyncAuthGroupLive]), Layer.provide(EnvProviderLayer) ); const HttpLive = HttpApiBuilder.serve(HttpMiddleware.logger).pipe( Layer.provide(HttpApiBuilder.middlewareCors()), Layer.provide(MainApiLive), HttpServer.withLogAddress, Layer.provide(NodeHttpServer.layer(createServer, { port: 3000 })) ); NodeRuntime.runMain(Layer.launch(HttpLive)); ================================================ FILE: apps/server/src/middleware/authorization.ts ================================================ import { Authorization, ClientId, DatabaseError, MissingWorkspace, Unauthorized, WorkspaceId, } from "@local/sync"; import { and, desc, eq, gt, isNull, or } from "drizzle-orm"; import { Array, Effect, Layer, Match, Redacted, Schema } from "effect"; import { tokenTable, workspaceTable } from "../db/schema"; import { Drizzle } from "../services/drizzle"; import { Jwt } from "../services/jwt"; export const AuthorizationLive = Layer.effect( Authorization, Effect.gen(function* () { const jwt = yield* Jwt; const { query } = yield* Drizzle; yield* Effect.log("Creating Authorization middleware"); return { apiKey: (apiKey) => Effect.gen(function* () { yield* Effect.log("Api key", Redacted.value(apiKey)); const tokenPayload = yield* jwt.decode({ apiKey }); yield* Effect.log(`Valid auth ${tokenPayload.workspaceId}`); yield* query({ Request: Schema.Struct({ workspaceId: WorkspaceId, clientId: ClientId, }), execute: (db, { workspaceId, clientId }) => db .select() .from(tokenTable) .where( and( eq(tokenTable.workspaceId, workspaceId), eq(tokenTable.clientId, clientId), or( isNull(tokenTable.revokedAt), gt(tokenTable.revokedAt, new Date()) ), or( isNull(tokenTable.expiresAt), gt(tokenTable.expiresAt, new Date()) ) ) ) .orderBy(desc(tokenTable.issuedAt)) .limit(1), })({ workspaceId: tokenPayload.workspaceId, clientId: tokenPayload.sub, }).pipe( Effect.flatMap(Array.head), Effect.tapError(Effect.logError), Effect.mapError( () => new Unauthorized({ message: "Missing, expired, or revoked token", }) ) ); return yield* query({ Request: Schema.Struct({ workspaceId: Schema.UUID, }), execute: (db, { workspaceId }) => db .select() .from(workspaceTable) .where(eq(workspaceTable.workspaceId, workspaceId)) .orderBy(desc(workspaceTable.createdAt)) .limit(1), })({ workspaceId: tokenPayload.workspaceId }).pipe( Effect.flatMap(Array.head) ); }).pipe( Effect.mapError((error) => Match.value(error).pipe( Match.tagsExhaustive({ NoSuchElementException: () => new MissingWorkspace(), Unauthorized: (error) => error, JwtError: () => new Unauthorized({ message: "Invalid token" }), ParseError: () => new Unauthorized({ message: "Invalid parameters" }), QueryError: () => new DatabaseError(), }) ) ) ), }; }) ).pipe(Layer.provide([Drizzle.Default, Jwt.Default])); ================================================ FILE: apps/server/src/middleware/master-authorization.ts ================================================ import { DatabaseError, MasterAuthorization, MissingWorkspace, Unauthorized, } from "@local/sync"; import { and, desc, eq } from "drizzle-orm"; import { Array, Effect, Layer, Match, Redacted, Schema } from "effect"; import { workspaceTable } from "../db/schema"; import { Drizzle } from "../services/drizzle"; import { Jwt } from "../services/jwt"; export const MasterAuthorizationLive = Layer.effect( MasterAuthorization, Effect.gen(function* () { const jwt = yield* Jwt; const { query } = yield* Drizzle; yield* Effect.log("Creating Master Authorization middleware"); return { apiKey: (apiKey) => Effect.gen(function* () { yield* Effect.log("Api key", Redacted.value(apiKey)); const tokenPayload = yield* jwt.decode({ apiKey }); if (tokenPayload.isMaster) { return yield* query({ Request: Schema.Struct({ workspaceId: Schema.UUID, clientId: Schema.UUID, }), execute: (db, { workspaceId, clientId }) => db .select() .from(workspaceTable) .where( and( eq(workspaceTable.workspaceId, workspaceId), eq(workspaceTable.clientId, clientId) ) ) .orderBy(desc(workspaceTable.createdAt)) .limit(1), })({ clientId: tokenPayload.sub, workspaceId: tokenPayload.workspaceId, }).pipe(Effect.flatMap(Array.head)); } return yield* new Unauthorized({ message: "Not master access" }); }).pipe( Effect.mapError((error) => Match.value(error).pipe( Match.tagsExhaustive({ NoSuchElementException: () => new MissingWorkspace(), Unauthorized: (error) => error, JwtError: () => new Unauthorized({ message: "Invalid token" }), ParseError: () => new Unauthorized({ message: "Invalid parameters" }), QueryError: () => new DatabaseError(), }) ) ) ), }; }) ).pipe(Layer.provide([Drizzle.Default, Jwt.Default])); ================================================ FILE: apps/server/src/middleware/version-check.ts ================================================ import { HttpServerRequest } from "@effect/platform"; import { SnapshotToLoroDoc, VERSION } from "@local/schema"; import { Snapshot, VersionCheck, VersionError } from "@local/sync"; import { Effect, Layer, Schema } from "effect"; export const VersionCheckLive = Layer.effect( VersionCheck, Effect.gen(function* () { yield* Effect.log("Creating version check middleware"); return Effect.gen(function* () { const request = yield* HttpServerRequest.HttpServerRequest; yield* Effect.log("Checking version"); const { snapshot } = yield* request.json.pipe( Effect.flatMap( Schema.decodeUnknown( Schema.Struct({ snapshot: Snapshot, }) ) ), Effect.mapError(() => new VersionError({ reason: "missing-snapshot" })) ); const doc = yield* Schema.decode(SnapshotToLoroDoc)(snapshot).pipe( Effect.mapError(() => new VersionError({ reason: "invalid-doc" })) ); yield* Effect.log("Checking doc", doc.toJSON()); const currentVersion = doc.getMap("metadata").get("version"); yield* Effect.log("Current version", currentVersion); if (typeof currentVersion !== "number") { return yield* new VersionError({ reason: "missing-version" }); } else if (currentVersion !== VERSION) { return yield* new VersionError({ reason: "outdated-version" }); } return snapshot; }); }) ); ================================================ FILE: apps/server/src/services/drizzle.ts ================================================ import { Data, Effect, Redacted, Schema } from "effect"; import { DatabaseUrl } from "../database"; import { drizzle } from "drizzle-orm/node-postgres"; import { migrate } from "drizzle-orm/node-postgres/migrator"; import { fileURLToPath } from "node:url"; class MigrationError extends Data.TaggedError("MigrationError")<{ cause: unknown; }> {} class QueryError extends Data.TaggedError("QueryError")<{ cause: unknown; }> {} export class Drizzle extends Effect.Service()("Drizzle", { effect: Effect.gen(function* () { const databaseUrl = yield* DatabaseUrl; const db = drizzle(Redacted.value(databaseUrl)); yield* Effect.log("Applying migrations").pipe( Effect.andThen( Effect.tryPromise({ try: () => migrate(db, { migrationsFolder: fileURLToPath( new URL("../../drizzle", import.meta.url) ), }), catch: (error) => new MigrationError({ cause: error }), }) ), Effect.tap(() => Effect.log("Migrations applied")) ); const query = ({ Request, // Result, execute, }: { Request: Schema.Schema; // Result: Schema.Schema; execute: (_: typeof db, __: RQA) => Promise; }) => (params: RQI) => Schema.decode(Request)(params).pipe( Effect.flatMap((_) => Effect.tryPromise({ try: () => execute(db, _), catch: (error) => new QueryError({ cause: error }), }) ) // Effect.flatMap(Schema.decode(Result)) ); return { db, query }; }), }) {} ================================================ FILE: apps/server/src/services/jwt.ts ================================================ import { Scope } from "@local/sync"; import { Config, Data, Effect, Redacted, Schema } from "effect"; import * as jwt from "jsonwebtoken"; class TokenPayload extends Schema.Class("TokenPayload")({ iat: Schema.Number, exp: Schema.optional(Schema.Number), sub: Schema.String, workspaceId: Schema.String, scope: Scope, isMaster: Schema.Boolean, }) {} class JwtError extends Data.TaggedError("JwtError")<{ reason: "missing" | "invalid"; }> {} export class Jwt extends Effect.Service()("Jwt", { effect: Effect.gen(function* () { const secretKey = yield* Config.redacted("JWT_SECRET"); return { sign: ({ clientId, workspaceId, }: { clientId: string; workspaceId: string; }) => Schema.encode(TokenPayload)( new TokenPayload({ iat: Math.floor(Date.now() / 1000), sub: clientId, workspaceId, scope: "read_write", isMaster: true, }) ).pipe( Effect.flatMap((payload) => Effect.try({ try: () => jwt.sign(payload, Redacted.value(secretKey), { algorithm: "HS256", }), catch: () => new JwtError({ reason: "invalid" }), }) ) ), decode: ({ apiKey }: { apiKey: Redacted.Redacted }) => Effect.gen(function* () { const decoded = jwt.decode(Redacted.value(apiKey)); if (decoded === null) { return yield* new JwtError({ reason: "missing" }); } return yield* Schema.decodeUnknown(TokenPayload)(decoded).pipe( Effect.mapError(() => new JwtError({ reason: "invalid" })) ); }), }; }), }) {} ================================================ FILE: apps/server/tsconfig.json ================================================ { "compilerOptions": { "esModuleInterop": true, "skipLibCheck": true, "target": "es2022", "allowJs": true, "resolveJsonModule": true, "moduleDetection": "force", "isolatedModules": true, "verbatimModuleSyntax": true, "strict": true, "noUncheckedIndexedAccess": true, "noImplicitOverride": true, "module": "preserve", "noEmit": true, "lib": ["es2022"] }, "include": ["**/*.ts"], "exclude": ["node_modules"] } ================================================ FILE: docker-compose.yaml ================================================ version: "0.2" name: app_docker services: postgres: env_file: .env container_name: postgres image: postgres:16-alpine environment: - POSTGRES_USER=${POSTGRES_USER} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} - POSTGRES_DB=${POSTGRES_DATABASE} ports: - 5435:5432 pgadmin: env_file: .env container_name: pgadmin image: dpage/pgadmin4:latest environment: - PGADMIN_DEFAULT_EMAIL=${PGADMIN_MAIL} - PGADMIN_DEFAULT_PASSWORD=${PGADMIN_PW} ports: - 5050:80 ================================================ FILE: package.json ================================================ { "name": "sql-database-local-config", "private": true, "scripts": { "build": "turbo run build", "dev": "turbo run dev", "typecheck": "turbo run typecheck", "generate": "turbo run generate", "format": "prettier --write \"**/*.{ts,tsx,md}\"" }, "devDependencies": { "prettier": "^3.5.0", "turbo": "^2.4.2", "typescript": "5.7.3" }, "packageManager": "pnpm@9.0.0", "engines": { "node": ">=18" } } ================================================ FILE: packages/client-lib/package.json ================================================ { "name": "@local/client-lib", "type": "module", "scripts": { "typecheck": "tsc" }, "exports": { ".": "./src/main.ts" }, "devDependencies": { "@types/react": "^19.0.8", "@types/react-dom": "^19.0.3" }, "peerDependencies": { "@effect/platform": "^0.77.2", "effect": "^3.13.2", "@local/sync": "workspace:*", "@local/schema": "workspace:*", "dexie": "^4.0.11", "dexie-react-hooks": "^1.1.7", "loro-crdt": "^1.4.2", "react": "^19.0.0", "react-dom": "^19.0.0" }, "dependencies": {} } ================================================ FILE: packages/client-lib/src/main.ts ================================================ import { RuntimeLayer } from "./runtime-layer"; import * as Service from "./services"; import * as SyncWorker from "./sync-worker"; import { useActionEffect } from "./use-action-effect"; import { useDexieQuery } from "./use-dexie-query"; export { RuntimeLayer, Service, SyncWorker, useActionEffect as useActionEffect, useDexieQuery, }; ================================================ FILE: packages/client-lib/src/runtime-layer.ts ================================================ import { Layer } from "effect"; import { ApiClient } from "./services/api-client"; import { Dexie } from "./services/dexie"; import { LoroStorage } from "./services/loro-storage"; import { Migration } from "./services/migration"; import { Sync } from "./services/sync"; import { TempWorkspace } from "./services/temp-workspace"; import { WorkspaceManager } from "./services/workspace-manager"; import { SyncWorker } from "./sync-worker"; export const RuntimeLayer = Layer.mergeAll( Dexie.Default, ApiClient.Default, WorkspaceManager.Default, TempWorkspace.Default, LoroStorage.Default, Sync.Default, Migration.Default, SyncWorker.Default ); ================================================ FILE: packages/client-lib/src/schema.ts ================================================ import { ClientId, Snapshot, SnapshotId, WorkspaceId } from "@local/sync"; import { Schema } from "effect"; export class ClientTable extends Schema.Class("ClientTable")({ clientId: ClientId, }) {} export class WorkspaceTable extends Schema.Class( "WorkspaceTable" )({ workspaceId: WorkspaceId, snapshot: Snapshot, token: Schema.NullOr(Schema.String), version: Schema.NullOr(Schema.Uint8Array), }) {} export class TempWorkspaceTable extends Schema.Class( "TempWorkspaceTable" )({ workspaceId: WorkspaceId, snapshot: Snapshot, snapshotId: SnapshotId, }) {} ================================================ FILE: packages/client-lib/src/services/api-client.ts ================================================ import { FetchHttpClient, HttpApiClient } from "@effect/platform"; import { SyncApi } from "@local/sync"; import { Effect } from "effect"; export class ApiClient extends Effect.Service()("ApiClient", { dependencies: [FetchHttpClient.layer], effect: Effect.gen(function* () { const client = yield* HttpApiClient.make(SyncApi, { baseUrl: "http://localhost:3000", }); return { client }; }), }) {} ================================================ FILE: packages/client-lib/src/services/dexie.ts ================================================ import * as _Dexie from "dexie"; import { Data, Effect, Schema } from "effect"; import { type ClientTable, type TempWorkspaceTable, type WorkspaceTable, } from "../schema"; class QueryApiError extends Data.TaggedError("QueryApiError")<{ cause: unknown; }> {} class WriteApiError extends Data.TaggedError("WriteApiError")<{ cause: unknown; }> {} const formDataToRecord = (formData: FormData): Record => { const record: Record = {}; for (const [key, value] of formData.entries()) { if (typeof value === "string") { record[key] = value; } } return record; }; export class Dexie extends Effect.Service()("Dexie", { accessors: true, effect: Effect.gen(function* () { const db = new _Dexie.Dexie("_db") as _Dexie.Dexie & { client: _Dexie.EntityTable; workspace: _Dexie.EntityTable< typeof WorkspaceTable.Encoded, "workspaceId" >; temp_workspace: _Dexie.EntityTable< typeof TempWorkspaceTable.Encoded, "workspaceId" >; }; db.version(1).stores({ client: "clientId", workspace: "workspaceId", temp_workspace: "workspaceId", }); const query = (execute: (_: typeof db) => Promise) => Effect.tryPromise({ try: () => execute(db), catch: (error) => new QueryApiError({ cause: error }), }).pipe(Effect.tapErrorCause(Effect.logError)); const initClient = query((_) => _.client.toCollection().last()).pipe( Effect.map((client) => client?.clientId), Effect.flatMap(Effect.fromNullable), Effect.orElse(() => query((_) => _.client.add({ clientId: crypto.randomUUID() })) ) ); const formAction = ( source: Schema.Schema>, exec: (values: Readonly) => Promise ) => (formData: FormData) => Schema.decodeUnknown(source)(formDataToRecord(formData)).pipe( Effect.mapError((error) => new WriteApiError({ cause: error })), Effect.flatMap((values) => Effect.tryPromise({ try: () => exec(values), catch: (error) => new WriteApiError({ cause: error }), }) ) ); const changeAction = ( source: Schema.Schema, exec: (values: Readonly) => Promise ) => (params: I) => Schema.decode(source)(params).pipe( Effect.tap(Effect.log), Effect.mapError((error) => new WriteApiError({ cause: error })), Effect.flatMap((values) => Effect.tryPromise({ try: () => exec(values), catch: (error) => new WriteApiError({ cause: error }), }) ) ); return { db, initClient, query }; }), }) {} ================================================ FILE: packages/client-lib/src/services/index.ts ================================================ import { ApiClient } from "./api-client"; import { Dexie } from "./dexie"; import { LoroStorage } from "./loro-storage"; import { Migration } from "./migration"; import { Sync } from "./sync"; import { TempWorkspace } from "./temp-workspace"; import { WorkspaceManager } from "./workspace-manager"; export { ApiClient, Dexie, LoroStorage, Migration, Sync, TempWorkspace, WorkspaceManager, }; ================================================ FILE: packages/client-lib/src/services/loro-storage.ts ================================================ import { SnapshotSchema, type LoroSchema } from "@local/schema"; import { Effect, Schema } from "effect"; import { LoroDoc, type LoroList, type LoroMap } from "loro-crdt"; import { TempWorkspace } from "./temp-workspace"; import { WorkspaceManager } from "./workspace-manager"; export class LoroStorage extends Effect.Service()("LoroStorage", { accessors: true, dependencies: [TempWorkspace.Default, WorkspaceManager.Default], effect: Effect.gen(function* () { const manager = yield* WorkspaceManager; const temp = yield* TempWorkspace; const load = ({ workspaceId }: { workspaceId: string }) => Effect.all( { workspace: manager.getById({ workspaceId }), tempWorkspace: temp.getById({ workspaceId }), }, { concurrency: "unbounded" } ).pipe( Effect.map(({ workspace, tempWorkspace }) => { const doc = SnapshotSchema.EmptyDoc(); if (workspace !== undefined) { doc.import(workspace.snapshot); } if (tempWorkspace !== undefined) { doc.import(tempWorkspace.snapshot); } return { doc, workspace }; }) ); const query = >( extract: (doc: LoroDoc) => LoroList>, schema: Schema.Schema, { workspaceId }: { workspaceId: string } ) => Effect.gen(function* () { const { doc } = yield* load({ workspaceId }); const data = extract(doc); const list = data.toArray(); return yield* Effect.all( list.map((item) => Schema.decode(schema)(item.toJSON() as A)), { concurrency: 10 } ); }); return { load, query } as const; }), }) {} ================================================ FILE: packages/client-lib/src/services/migration.ts ================================================ import { SnapshotToLoroDoc } from "@local/schema"; import { LoroDocMigration } from "@local/schema/migrations"; import { Data, Effect, Schema, type ParseResult } from "effect"; import { TempWorkspace } from "./temp-workspace"; class MigrationError extends Data.TaggedError("MigrationError")<{ parseError: ParseResult.ParseError; }> {} export class Migration extends Effect.Service()("Migration", { dependencies: [TempWorkspace.Default], effect: Effect.gen(function* () { const temp = yield* TempWorkspace; return { migrate: temp.getAll.pipe( Effect.tap((workspaces) => Effect.log(`Migrating ${workspaces.length} workspaces`) ), Effect.flatMap((workspaces) => Effect.all( workspaces.map((workspace) => Effect.gen(function* () { const doc = yield* Schema.decode(SnapshotToLoroDoc)( workspace.snapshot ); yield* Effect.log(doc.toJSON()); const newDoc = yield* Schema.decode(LoroDocMigration)(doc).pipe( Effect.catchTag( "ParseError", (parseError) => new MigrationError({ parseError }) ) ); const newSnapshot = yield* Schema.encode(SnapshotToLoroDoc)(newDoc); yield* temp.put({ workspaceId: workspace.workspaceId, snapshot: newSnapshot, snapshotId: workspace.snapshotId, }); }) ) ) ), Effect.tap(() => Effect.log("Migration completed successfully for all workspaces") ) ), }; }), }) {} ================================================ FILE: packages/client-lib/src/services/sync.ts ================================================ import type { LoroSchema } from "@local/schema"; import { Effect, flow, Option } from "effect"; import { LoroDoc } from "loro-crdt"; import { ApiClient } from "./api-client"; import { Dexie } from "./dexie"; import { TempWorkspace } from "./temp-workspace"; import { WorkspaceManager } from "./workspace-manager"; export class Sync extends Effect.Service()("Sync", { dependencies: [ TempWorkspace.Default, WorkspaceManager.Default, ApiClient.Default, Dexie.Default, ], effect: Effect.gen(function* () { const { client } = yield* ApiClient; const { initClient } = yield* Dexie; const manager = yield* WorkspaceManager; const temp = yield* TempWorkspace; return { push: ({ snapshot, workspaceId, snapshotId, }: { workspaceId: string; snapshotId: string; snapshot: globalThis.Uint8Array; }) => manager.getById({ workspaceId }).pipe( Effect.flatMap( flow( Option.fromNullable, Option.match({ onNone: () => Effect.log("No workspace found"), onSome: (workspace) => Effect.gen(function* () { const clientId = yield* initClient; yield* Effect.log(`Pushing snapshot ${snapshotId}`); const response = yield* Effect.fromNullable( workspace.token ).pipe( Effect.flatMap((token) => client.syncData .push({ headers: { "x-api-key": token }, path: { workspaceId: workspace.workspaceId }, payload: { snapshot, snapshotId }, }) .pipe( Effect.map((response) => ({ ...response, token, })) ) ), Effect.catchTag("NoSuchElementException", () => client.syncAuth .generateToken({ payload: { clientId, snapshot, snapshotId, workspaceId: workspace.workspaceId, }, }) .pipe( Effect.tap(({ token }) => manager.setToken({ workspaceId: workspace.workspaceId, token, }) ) ) ) ); const doc = new LoroDoc(); doc.import(response.snapshot); yield* Effect.all([ manager.put({ workspaceId: response.workspaceId, snapshot: response.snapshot, token: response.token, version: doc.version().encode(), }), temp.clean({ workspaceId: workspace.workspaceId, }), ]); }), }) ) ) ), pull: ({ workspaceId }: { workspaceId: string }) => manager.getById({ workspaceId }).pipe( Effect.flatMap( flow( Option.fromNullable, Option.flatMap((workspace) => Option.fromNullable(workspace.token) ), Option.match({ onNone: () => Effect.log("No token found").pipe(Effect.map(() => null)), onSome: (token) => Effect.gen(function* () { yield* Effect.log(`Pulling from ${workspaceId}`); const response = yield* client.syncData.pull({ headers: { "x-api-key": token }, path: { workspaceId }, }); const doc = new LoroDoc(); doc.import(response.snapshot); yield* manager.put({ token, workspaceId, snapshot: response.snapshot, version: doc.version().encode(), }); return response; }), }) ) ) ), join: ({ workspaceId }: { workspaceId: string }) => Effect.gen(function* () { const clientId = yield* initClient; const response = yield* client.syncData.join({ path: { clientId, workspaceId }, }); const doc = new LoroDoc(); doc.import(response.snapshot); yield* manager.put({ workspaceId, token: response.token, snapshot: response.snapshot, version: doc.version().encode(), }); return response; }), }; }), }) {} ================================================ FILE: packages/client-lib/src/services/temp-workspace.ts ================================================ import { Effect, Schema } from "effect"; import { TempWorkspaceTable } from "../schema"; import { Dexie } from "./dexie"; export class TempWorkspace extends Effect.Service()( "TempWorkspace", { dependencies: [Dexie.Default], effect: Effect.gen(function* () { const { query } = yield* Dexie; return { getAll: query((_) => _.temp_workspace.toArray()).pipe( Effect.flatMap(Schema.decode(Schema.Array(TempWorkspaceTable))) ), put: (params: typeof TempWorkspaceTable.Type) => Schema.encode(TempWorkspaceTable)(params).pipe( Effect.flatMap((data) => query((_) => _.temp_workspace.put(data))) ), getById: ({ workspaceId }: { workspaceId: string }) => query((_) => _.temp_workspace .where("workspaceId") .equals(workspaceId) .limit(1) .first() ).pipe( Effect.flatMap((workspace) => workspace === undefined ? Effect.succeed(undefined) : Schema.decode(TempWorkspaceTable)(workspace) ) ), clean: ({ workspaceId }: { workspaceId: string }) => query((_) => _.temp_workspace.where("workspaceId").equals(workspaceId).delete() ), }; }), } ) {} ================================================ FILE: packages/client-lib/src/services/workspace-manager.ts ================================================ import { SnapshotSchema } from "@local/schema"; import { Snapshot } from "@local/sync"; import { Effect, Schema } from "effect"; import { WorkspaceTable } from "../schema"; import { Dexie } from "./dexie"; export class WorkspaceManager extends Effect.Service()( "WorkspaceManager", { accessors: true, dependencies: [Dexie.Default], effect: Effect.gen(function* () { const { query } = yield* Dexie; return { setToken: ({ token, workspaceId, }: { workspaceId: typeof WorkspaceTable.Type.workspaceId; token: NonNullable; }) => query((_) => _.workspace.update(workspaceId, { token })), put: (update: typeof WorkspaceTable.Type) => Schema.encode(WorkspaceTable)(update).pipe( Effect.flatMap((data) => query((_) => _.workspace.put(data))) ), getAll: query((_) => _.workspace.toArray()), getById: ({ workspaceId }: { workspaceId: string }) => query((_) => _.workspace .where("workspaceId") .equals(workspaceId) .limit(1) .first() ).pipe( Effect.flatMap((workspace) => workspace === undefined ? Effect.succeed(undefined) : Schema.decode(WorkspaceTable)(workspace) ) ), create: query((_) => _.workspace.toCollection().modify({ current: false }) ).pipe( Effect.andThen( Schema.encode(Snapshot)( SnapshotSchema.EmptyDoc().export({ mode: "snapshot", }) ) ), Effect.flatMap((snapshot) => query((_) => _.workspace.put({ snapshot, token: null, version: null, workspaceId: crypto.randomUUID(), }) ) ), Effect.map((workspaceId) => ({ workspaceId })) ), }; }), } ) {} ================================================ FILE: packages/client-lib/src/sync-worker.ts ================================================ import { Snapshot } from "@local/sync"; import { liveQuery } from "dexie"; import { Array, Effect, Number, Schema, Stream, SynchronizedRef } from "effect"; import { Dexie, Sync, TempWorkspace, WorkspaceManager } from "./services"; export class LiveQuery extends Schema.TaggedRequest()("LiveQuery", { failure: Schema.String, payload: { workspaceId: Schema.String }, success: Schema.Boolean, }) {} export class Bootstrap extends Schema.TaggedRequest()("Bootstrap", { failure: Schema.String, payload: { workspaceId: Schema.String }, success: Schema.Boolean, }) {} export const WorkerMessage = Schema.Union(Bootstrap); export class SyncWorker extends Effect.Service()("SyncWorker", { effect: Effect.gen(function* () { return { liveSync: (params: { workspaceId: string }) => Effect.gen(function* () { const manager = yield* WorkspaceManager; const { db } = yield* Dexie; const { push } = yield* Sync; const snapshotEq = Array.getEquivalence(Number.Equivalence); yield* Effect.log(`Live query workspace '${params.workspaceId}'`); const workspace = yield* manager .getById({ workspaceId: params.workspaceId }) .pipe(Effect.flatMap(Effect.fromNullable)); const live = liveQuery(() => db.temp_workspace .where("workspaceId") .equals(params.workspaceId) .toArray() ); yield* Effect.forkScoped( Effect.acquireRelease( Effect.gen(function* () { yield* Effect.log("Subscribing"); const ref = yield* SynchronizedRef.make(0); return live.subscribe((payload) => Effect.runPromise( Effect.gen(function* () { yield* Effect.log(`Change detected`); const id = yield* ref.pipe( SynchronizedRef.updateAndGet((n) => n + 1) ); yield* Stream.runDrain( Stream.make(...payload).pipe( Stream.changesWith((a, b) => snapshotEq(a.snapshot, b.snapshot) ), Stream.debounce("3 seconds"), Stream.tap((message) => Effect.gen(function* () { const streamId = yield* ref.get; if (streamId === id) { yield* Effect.log( `Syncing ${payload.length} changes` ); const snapshot = yield* Schema.decode(Snapshot)( message.snapshot ); yield* push({ snapshot, snapshotId: message.snapshotId, workspaceId: workspace.workspaceId, }); } }) ) ) ); }) ) ); }), (subscription) => Effect.gen(function* () { yield* Effect.log("Live query unsubscribing"); return subscription.unsubscribe(); }) ) ); return true; }), bootstrap: (params: Bootstrap) => Effect.gen(function* () { const { push, pull } = yield* Sync; const manager = yield* WorkspaceManager; const temp = yield* TempWorkspace; yield* Effect.log(`Running workspace '${params.workspaceId}'`); const workspace = yield* manager .getById({ workspaceId: params.workspaceId }) .pipe(Effect.flatMap(Effect.fromNullable)); const tempUpdates = yield* temp.getById({ workspaceId: workspace.workspaceId, }); if (tempUpdates !== undefined) { yield* push({ workspaceId: workspace.workspaceId, snapshot: tempUpdates.snapshot, snapshotId: tempUpdates.snapshotId, }); yield* Effect.log("Push sync completed"); } else { yield* pull({ workspaceId: workspace.workspaceId }); yield* Effect.log("Pull sync completed"); } return true; }).pipe( Effect.mapError( (error) => `Bootstrap error: ${JSON.stringify(error)}` ) ), }; }), }) {} ================================================ FILE: packages/client-lib/src/use-action-effect.ts ================================================ import { Effect, type ManagedRuntime } from "effect"; import { useActionState } from "react"; export const useActionEffect = ( runtime: ManagedRuntime.ManagedRuntime, effect: (payload: Payload) => Effect.Effect ) => { return useActionState< | { error: E; data: null } | { error: null; data: A } | { error: null; data: null }, Payload >( (_, payload) => runtime.runPromise( effect(payload).pipe( Effect.match({ onFailure: (error) => ({ error, data: null }), onSuccess: (data) => ({ error: null, data }), }) ) ), { error: null, data: null } ); }; ================================================ FILE: packages/client-lib/src/use-dexie-query.ts ================================================ import { useLiveQuery } from "dexie-react-hooks"; import { Data, Effect, Either, Function, Match, pipe } from "effect"; import { Dexie } from "./services/dexie"; class MissingData extends Data.TaggedError("MissingData")<{}> {} class DexieError extends Data.TaggedError("DexieError")<{ reason: "invalid-data" | "query-error"; cause: unknown; }> {} export const useDexieQuery = ( query: (db: (typeof Dexie.Service)["db"]) => Promise, deps: unknown[] = [] ) => { const results = useLiveQuery( () => Effect.runPromise( Effect.gen(function* () { const { db } = yield* Dexie; return yield* Effect.tryPromise({ try: () => query(db), catch: (cause) => new DexieError({ reason: "query-error", cause }), }); }).pipe(Effect.either, Effect.provide(Dexie.Default)) ), deps ); return pipe( results, Either.fromNullable(() => new MissingData()), Either.flatMap(Function.identity), Either.match({ onLeft: (_) => Match.value(_).pipe( Match.tagsExhaustive({ DexieError: (error) => ({ error, loading: false as const, data: undefined, }), MissingData: (_) => ({ loading: true as const, data: undefined, error: undefined, }), }) ), onRight: (rows) => ({ data: rows, loading: false as const, error: undefined, }), }) ); }; ================================================ FILE: packages/client-lib/tsconfig.json ================================================ { "compilerOptions": { "esModuleInterop": true, "skipLibCheck": true, "target": "es2022", "allowJs": true, "resolveJsonModule": true, "moduleDetection": "force", "isolatedModules": true, "verbatimModuleSyntax": true, "strict": true, "noUncheckedIndexedAccess": true, "noImplicitOverride": true, "module": "preserve", "noEmit": true, "lib": ["es2022", "DOM", "DOM.Iterable"] }, "include": ["**/*.ts"], "exclude": ["node_modules"] } ================================================ FILE: packages/schema/package.json ================================================ { "name": "@local/schema", "type": "module", "scripts": { "typecheck": "tsc" }, "exports": { ".": "./src/main.ts", "./migrations": "./src/migrations.ts" }, "devDependencies": {}, "peerDependencies": { "loro-crdt": "^1.4.2", "effect": "^3.13.2" }, "dependencies": {} } ================================================ FILE: packages/schema/src/main.ts ================================================ import { Schema } from "effect"; import { LoroDoc, LoroList, LoroMap } from "loro-crdt"; import { AnyLoroDocSchema, Table, VersioningSchema } from "./schema"; import type { Version } from "./versioning"; export const VERSION = 1 satisfies Version; export const CurrentSchema = VersioningSchema[VERSION]; const Metadata = Schema.Struct({ version: Schema.Number }); export type LoroSchema = { metadata: LoroMap>; food: LoroList>; meal: LoroList>; }; export class SnapshotSchema extends Schema.Class( "SnapshotSchema" )({ metadata: Metadata, ...CurrentSchema.fields, }) { static readonly Table = Table; static readonly EmptyDoc = () => { const doc = new LoroDoc(); doc.getMap("metadata").set("version", VERSION); Table.literals.map((key) => { doc.getList(key); }); return doc; }; } export const SnapshotToLoroDoc = Schema.Uint8ArrayFromSelf.pipe( Schema.transform(AnyLoroDocSchema, { decode: (from) => { const doc = new LoroDoc(); doc.import(from); return doc; }, encode: (to) => to.export({ mode: "snapshot" }), }) ); ================================================ FILE: packages/schema/src/migrations.ts ================================================ import { ParseResult, Schema } from "effect"; import { LoroDoc } from "loro-crdt"; import { SnapshotSchema } from "./main"; import { AnyLoroDocSchema } from "./schema"; import { Version } from "./versioning"; const migrations = { 1: (_) => SnapshotSchema.EmptyDoc(), } satisfies Record LoroDoc>; export const LoroDocMigration = AnyLoroDocSchema.pipe( Schema.transformOrFail(AnyLoroDocSchema, { decode: (from, _, ast) => { const doc = new LoroDoc(); doc.import(from.export({ mode: "snapshot" })); const currentVersion = doc.getMap("metadata").get("version"); if (typeof currentVersion === "number") { Version.forEach((version) => { doc.import(migrations[version](doc).export({ mode: "snapshot" })); }); } else { return ParseResult.fail( new ParseResult.Type(ast, from, "Invalid version number in metadata") ); } return ParseResult.succeed(doc); }, encode: (to, _, ast) => ParseResult.fail( new ParseResult.Forbidden( ast, to, "Encoding LoroDoc migration is not allowed (should not happen...)" ) ), }) ); ================================================ FILE: packages/schema/src/schema.ts ================================================ import { Schema } from "effect"; import { LoroDoc } from "loro-crdt"; import { type Version } from "./versioning"; export const AnyLoroDocSchema = Schema.instanceOf(LoroDoc); export class FoodV1 extends Schema.Class("FoodV1")({ id: Schema.UUID, name: Schema.String, calories: Schema.Number.pipe(Schema.positive()), }) {} export class MealV1 extends Schema.Class("MealV1")({ id: Schema.UUID, foodId: FoodV1.fields.id, quantity: Schema.Number.pipe(Schema.positive()), }) {} export const Table = Schema.Literal("food", "meal"); export const VersioningSchema = { 1: Schema.Struct({ [Table.literals[0]]: Schema.Array(FoodV1), [Table.literals[1]]: Schema.Array(MealV1), }), } as const satisfies Record; ================================================ FILE: packages/schema/src/versioning.ts ================================================ export const Version = [1] as const; export type Version = (typeof Version)[number]; ================================================ FILE: packages/schema/tsconfig.json ================================================ { "compilerOptions": { "esModuleInterop": true, "skipLibCheck": true, "target": "es2022", "allowJs": true, "resolveJsonModule": true, "moduleDetection": "force", "isolatedModules": true, "verbatimModuleSyntax": true, "strict": true, "noUncheckedIndexedAccess": true, "noImplicitOverride": true, "module": "preserve", "noEmit": true, "lib": ["es2022"] }, "include": ["**/*.ts"], "exclude": ["node_modules"] } ================================================ FILE: packages/sync/package.json ================================================ { "name": "@local/sync", "type": "module", "scripts": { "typecheck": "tsc" }, "exports": { ".": "./src/main.ts" }, "devDependencies": {}, "peerDependencies": { "@effect/platform": "^0.77.2", "effect": "^3.13.2" }, "dependencies": {} } ================================================ FILE: packages/sync/src/main.ts ================================================ import { HttpApi, HttpApiEndpoint, HttpApiGroup, HttpApiMiddleware, HttpApiSchema, HttpApiSecurity, } from "@effect/platform"; import { Context, Schema } from "effect"; export class Unauthorized extends Schema.TaggedError()( "Unauthorized", { message: Schema.String }, HttpApiSchema.annotations({ status: 401 }) ) {} export class MissingWorkspace extends Schema.TaggedError()( "MissingWorkspace", {}, HttpApiSchema.annotations({ status: 404 }) ) {} export class DatabaseError extends Schema.TaggedError()( "DatabaseError", {}, HttpApiSchema.annotations({ status: 500 }) ) {} export class VersionError extends Schema.TaggedError()( "VersionError", { reason: Schema.Literal( "missing-snapshot", "invalid-doc", "missing-version", "outdated-version" ), }, HttpApiSchema.annotations({ status: 400 }) ) {} export const ClientId = Schema.UUID; export const WorkspaceId = Schema.UUID; export const Snapshot = Schema.Uint8Array; export const SnapshotId = Schema.UUID; export const Scope = Schema.Literal("read", "read_write"); export class ClientTable extends Schema.Class("ClientTable")({ clientId: ClientId, createdAt: Schema.DateFromString, }) {} export class WorkspaceTable extends Schema.Class( "WorkspaceTable" )({ workspaceId: WorkspaceId, ownerClientId: ClientId, createdAt: Schema.DateFromString, clientId: ClientId, snapshotId: SnapshotId, snapshot: Snapshot, }) {} export class TokenTable extends Schema.Class("TokenTable")({ tokenId: Schema.Number, tokenValue: Schema.String, clientId: ClientId, workspaceId: WorkspaceId, isMaster: Schema.Boolean, scope: Scope, issuedAt: Schema.DateFromString, expiresAt: Schema.NullOr(Schema.DateFromString), revokedAt: Schema.NullOr(Schema.DateFromString), }) {} export class AuthWorkspace extends Context.Tag("AuthWorkspace")< AuthWorkspace, WorkspaceTable >() {} export class ValidDoc extends Context.Tag("ValidDoc")< ValidDoc, typeof Snapshot.Type >() {} const authKey = "x-api-key"; export const ApiKey = HttpApiSecurity.apiKey({ in: "header", key: authKey, }); const ApiKeyHeader = Schema.Struct({ [authKey]: Schema.String, }); export class VersionCheck extends HttpApiMiddleware.Tag()( "VersionCheck", { failure: VersionError, provides: ValidDoc, } ) {} export class Authorization extends HttpApiMiddleware.Tag()( "Authorization", { failure: Schema.Union(Unauthorized, MissingWorkspace, DatabaseError), provides: AuthWorkspace, security: { apiKey: ApiKey }, } ) {} export class MasterAuthorization extends HttpApiMiddleware.Tag()( "MasterAuthorization", { failure: Schema.Union(Unauthorized, MissingWorkspace, DatabaseError), provides: AuthWorkspace, security: { apiKey: ApiKey }, } ) {} export class SyncAuthGroup extends HttpApiGroup.make("syncAuth") .add( /** Allows a client to create a new workshop and upload its initial data to the server. The server marks the client as the owner and issues a master token for full control. */ HttpApiEndpoint.post("generateToken")`/` .setPayload( Schema.Struct({ clientId: ClientId, workspaceId: WorkspaceId, snapshotId: SnapshotId, snapshot: WorkspaceTable.fields.snapshot, }) ) .addError(Schema.String) .addSuccess( Schema.Struct({ token: Schema.String, workspaceId: WorkspaceTable.fields.workspaceId, createdAt: WorkspaceTable.fields.createdAt, snapshot: WorkspaceTable.fields.snapshot, }) ) .middleware(VersionCheck) ) .add( /** Allows the owner (via master token) to generate an access token for another client, specifying permissions and expiration. The owner shares this token with the client securely. */ HttpApiEndpoint.post( "issueToken" )`/${HttpApiSchema.param("workspaceId", Schema.UUID)}/token` .setPayload( Schema.Struct({ clientId: ClientId, scope: Scope, expiresIn: Schema.Duration, }) ) .addError(Schema.String) .addSuccess( Schema.Struct({ token: Schema.String, scope: Scope, expiresAt: Schema.DateFromString, }) ) .setHeaders(ApiKeyHeader) .middleware(MasterAuthorization) ) .add( /** Lets the owner revoke access for a specific client by invalidating their access token. Requires the master token and targets the `clientId` tied to the token. */ HttpApiEndpoint.del( "revokeToken" )`/${HttpApiSchema.param("workspaceId", Schema.UUID)}/token/${HttpApiSchema.param("clientId", Schema.UUID)}` .addError(Schema.String) .addSuccess(Schema.Boolean) .setHeaders(ApiKeyHeader) .middleware(MasterAuthorization) ) .add( /** Provides the owner with a list of all active tokens (master and access) for a workshop, showing their status. Useful for managing access. */ HttpApiEndpoint.get( "listTokens" )`/${HttpApiSchema.param("workspaceId", Schema.UUID)}/tokens` .addError(Schema.String) .addSuccess( Schema.Array( TokenTable.pipe( Schema.pick( "clientId", "tokenValue", "scope", "isMaster", "issuedAt", "expiresAt", "revokedAt" ) ) ) ) .setHeaders(ApiKeyHeader) .middleware(MasterAuthorization) ) .prefix("/workspaces") {} export class SyncDataGroup extends HttpApiGroup.make("syncData") .add( /** Updates the workshop data on the server with changes from a client. Requires a valid token with `read_write` scope. */ HttpApiEndpoint.put( "push" )`/${HttpApiSchema.param("workspaceId", Schema.UUID)}/push` .setPayload(WorkspaceTable.pipe(Schema.pick("snapshot", "snapshotId"))) .addError(Schema.String) .addSuccess( WorkspaceTable.pipe(Schema.pick("workspaceId", "createdAt", "snapshot")) ) .setHeaders(ApiKeyHeader) .middleware(Authorization) .middleware(VersionCheck) ) .add( /** Retrieves the current workshop data for a client (owner or authorized user). Requires a valid token (master or access) with at least `read` scope. Used for initial download or sync verification. */ HttpApiEndpoint.get( "pull" )`/${HttpApiSchema.param("workspaceId", Schema.UUID)}/pull` .addError(Schema.String) .addSuccess(Schema.Struct({ snapshot: WorkspaceTable.fields.snapshot })) .setHeaders(ApiKeyHeader) .middleware(Authorization) ) .add( /** Client opens link to join another workspace. The server issues a token for the workspace and returns the workspace data. */ HttpApiEndpoint.get( "join" )`/${HttpApiSchema.param("workspaceId", Schema.UUID)}/join/${HttpApiSchema.param("clientId", Schema.UUID)}` .addError(Schema.String) .addSuccess( Schema.Struct({ snapshot: WorkspaceTable.fields.snapshot, token: TokenTable.fields.tokenValue, }) ) ) .prefix("/workspaces") {} export class SyncApi extends HttpApi.make("SyncApi") .add(SyncAuthGroup) .add(SyncDataGroup) {} ================================================ FILE: packages/sync/tsconfig.json ================================================ { "compilerOptions": { "esModuleInterop": true, "skipLibCheck": true, "target": "es2022", "allowJs": true, "resolveJsonModule": true, "moduleDetection": "force", "isolatedModules": true, "verbatimModuleSyntax": true, "strict": true, "noUncheckedIndexedAccess": true, "noImplicitOverride": true, "module": "preserve", "noEmit": true, "lib": ["es2022"] }, "include": ["**/*.ts"], "exclude": ["node_modules"] } ================================================ FILE: pnpm-workspace.yaml ================================================ packages: - "apps/*" - "packages/*" ================================================ FILE: turbo.json ================================================ { "$schema": "https://turbo.build/schema.json", "ui": "tui", "tasks": { "build": { "dependsOn": ["^build"], "inputs": ["$TURBO_DEFAULT$", ".env*"], "outputs": [".next/**", "!.next/cache/**"] }, "typecheck": { "dependsOn": ["^typecheck"] }, "generate": { "dependsOn": ["^generate"] }, "dev": { "cache": false, "persistent": true } } }