Tonk Overview

Welcome to Tonk!

Thank you for taking the time to explore our documentation. We're incredibly grateful that you're here building the future with us.

If you have any questions, run into issues, or just want to chat about Tonk, please don't hesitate to reach out to us in our Discord community. We'd love to hear from you and help however we can!

What is Tonk?

Tonk is a new category of software that creates portable, multiplayer, user-owned digital artifacts. It's essentially a file that contains both an application and its data, designed to work anywhere, last forever, and remain under user control.

Product Philosophy

Tonk represents a "credibly neutral platform" for building and sharing local software - positioned as an alternative to industrial-scale complexity. It embodies principles of:

Malleable software - as easy to change as it is to use
User ownership - people retain control of their tools and data
Infinite software in the age of AI - human-scale creation without corporate intermediaries

Core Tech Goals

A tonk has instructions for how to connect to its peers.
A tonk works offline.
A tonk can theoretically run on any device and be served as a web application.
A tonk can be shared just like a file.
A tonk is encrypted and only accessed by its group. (coming soon)
A tonk will synchronize the state of its filesystem with any connected peer.
A tonk can be forked or remixed by changing its network or membership group and updating its state.

Key Features

Virtual File System (VFS)

A document-based storage system backed by Automerge CRDTs that provides:

Real-time synchronization across peers
File and directory operations
Watch capabilities for reactive updates
Binary and text file support

Bundle Format

Self-contained application packages (.tonk files) that include:

Application code and assets
Serialized state documents
Metadata and configuration
Network sync endpoints

Host-Web Runtime Environment

Complete browser runtime for loading and executing .tonk applications:

Drag-and-Drop Loading: Simply drag .tonk files onto the interface
Service Worker Architecture: Intercepts requests and serves content from VFS
Multi-Bundle Support: Load and run multiple applications simultaneously
Offline-First Operation: Applications work without network connectivity
URL-based Access: Applications accessible at localhost:3000/${project-name}/
Automatic Sync: Connects to relay servers specified in bundle manifest

WASM Core

High-performance Rust implementation compiled to WebAssembly:

Runs in browsers and Node.js
Consistent behavior across platforms
Safe memory management
Efficient CRDT operations

Real-time Object-level Sync

Automatic synchronization of JSON-like objects powered by Automerge:

WebSocket-based relay transport
Peer discovery and room-based connection
Conflict-free merge semantics
Offline queue and replay

Use Cases

Living Business Experiments

Make your excel spreadsheet interactive then share it with a client.

Multiplayer Digital Gardens

Shared wikis, blogs, or notebooks that feel cosy like a private chat.

Canvas Jam

Put TLDR draw in a tonk!

Collaborative Research Notebooks

Put a Jupyter notebook inside your tonk!

Next Steps

Quickstart Guide - Get up and running quickly
Architecture - Deep dive into Tonk's design
Virtual File System - Learn about the VFS layer
Bundle Format - Understand bundle packaging

Quickstart Guide

⚠️ Important: Tonk is under heavy development and APIs are changing rapidly. Getting started requires manual setup and isn't for the faint of heart. We're working on making this easier!

Prerequisites

Before you begin, you'll need to set up the development environment:

Build core-js:

cd packages/core-js
pnpm install
pnpm build

Try the Example

The most complete example is latergram. Here's how to run it:

Start the relay server (required for sync):

cd packages/relay
pnpm dev

Bundle the latergram example:

cd examples/latergram
pnpm install
pnpm bundle create # Creates a .tonk file
touch .env #create .env file, see .env.example for required API_KEY, latergram uses anthropic claude

Load it in host-web:

cd packages/host-web
pnpm dev
# Then upload the .tonk file created in step 2

Note on Templates

The create package has templates, but they're still in flux and may not work reliably. For now, we recommend starting from the latergram example and modifying it to suit your needs.

Examples in the Repository

Explore these working examples:

latergram - Advanced application with dynamic components

Next Steps

Architecture - Deep dive into Tonk's design
Virtual File System - Learn about the VFS layer
Bundle Format - Understand bundle packaging

Tonk Architecture

Tonk is built on a local-first, peer-to-peer architecture that enables real-time collaboration without centralized servers. This document provides a detailed overview of Tonk's core components and how they work together.

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Host Web Environment                     │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │   Host-Web   │  │   .tonk      │  │   Relay Server   │  │
│  │   Runtime    │  │   Bundle     │  │   (WebSocket)    │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                     JavaScript/TypeScript Layer              │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │   Core-JS    │  │   Keepsync   │  │   Application    │  │
│  │   Wrapper    │  │  Middleware  │  │     Layer        │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    WASM Core (Rust)                          │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │   TonkCore   │  │     VFS      │  │     Bundle       │  │
│  │              │  │              │  │     System       │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │   Automerge  │  │   WebSocket  │  │    Storage       │  │
│  │     CRDT     │  │     Sync     │  │    Backend       │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
└─────────────────────────────────────────────────────────────┘

Core Components

1. TonkCore (Rust/WASM)

The heart of Tonk - a Rust library compiled to WebAssembly that provides the foundation for all Tonk operations.

Key Responsibilities:

Managing the Automerge document store
Coordinating VFS operations
Handling peer connections
Bundle import/export
Storage persistence

Implementation Details:

#![allow(unused)]
fn main() {
pub struct TonkCore {
    repo: Arc<Mutex<Repo>>,
    vfs: Arc<VirtualFileSystem>,
    connections: HashMap<String, Connection>,
    storage: Box<dyn StorageBackend>,
}
}

2. Virtual File System (VFS)

A document-based file system abstraction that provides familiar file operations while leveraging Automerge for synchronization.

Features:

Hierarchical directory structure
File read/write operations
Directory listing and creation
File watching for reactive updates
Support for text and binary data

Document Structure: Each file or directory is represented as an Automerge document with:

Metadata (name, type, timestamps)
Content (text or binary data)
References to child documents (for directories)

3. Bundle System

A packaging format for distributing Tonk applications as self-contained units.

Bundle Structure:

tonk-app.tonk (ZIP archive)
├── manifest.json           # Metadata and configuration
├── storage/               # Serialized Automerge documents
│   ├── root.automerge
│   └── [document-id].automerge

Manifest Format:

This format hasn't been fully integrated yet.

{
  "manifest_version": 1,
  "version": { "major": 1, "minor": 0 },
  "root_id": "automerge-document-id",
  "entrypoints": ["app_folder"],
  "network_uris": ["wss://sync.example.com"]
}

4. Host-Web Environment

The host-web package assists in loading and executing .tonk applications in browsers via a simple index.html bootloader screen and service worker:

Key Components:

WASM Runtime Integration: Bundled Tonk WASM core for local execution (no server dependency)
Service Worker Architecture: Intercepts requests and serves content from VFS
Bundle Loading System: Supports drag-and-drop and remote URL loading of .tonk files
Multi-Bundle Support: Can load and run multiple applications simultaneously
Offline-First Design: Applications work without network connectivity

Bundle Loading Flow:

Bundle Parsing: Extracts .tonk ZIP archives and reads manifest.json
VFS Population: Loads application files into the virtual file system
Service Worker Registration: Takes control of HTTP requests for the application
Request Mapping: Maps URL paths to VFS file locations

URL Structure: Applications are accessible at ${hostname}/${project-name}/ where the project name corresponds to the bundle's application namespace.

Network Integration: Bundles specify relay server endpoints in their manifest for real-time peer synchronization. The host-web environment automatically connects to these endpoints using the bundle's rootId as the sync room identifier.

5. Automerge

Tonk very intentionally does not want to veer too far away from Automerge. This is in order to encourage maximum interoperability and leave open optionality for a broader standard. If you are interested in certain low-level features about the protocol and about how network and storage adapters work, please see the Automerge site.

Virtual File System (VFS)

The Virtual File System is a core abstraction in Tonk that provides a familiar file system interface while leveraging Automerge CRDTs for distributed, conflict-free synchronization.

Overview

The VFS allows you to:

Work with files and directories using familiar APIs
Automatically sync changes across connected peers
Watch for real-time updates
Store both JSON data and binary content
Maintain offline-first functionality

All VFS methods are available directly on the TonkCore instance.

File Operations

Creating Files

// Create a file with JSON content
await tonk.createFile('/app/config.json', {
  theme: 'dark',
  fontSize: 16,
});

// Create a file with string content
await tonk.createFile('/app/notes.txt', 'Hello, World!');

// Create a file with array content
await tonk.createFile('/data/items.json', [1, 2, 3, 4, 5]);

Creating Files with Binary Data

// Create an image file with metadata and bytes
await tonk.createFileWithBytes(
  '/images/tree.png',
  { mime: 'image/png', alt: 'picture of a tree' },
  imageBytes // Uint8Array or base64 string
);

Reading Files

// Read a file
const doc = await tonk.readFile('/app/config.json');
console.log(doc.content); // JSON content
console.log(doc.name); // 'config.json'
console.log(doc.type); // 'document'
console.log(doc.timestamps.created); // timestamp
console.log(doc.timestamps.modified); // timestamp

// If file has binary data
if (doc.bytes) {
  console.log(doc.bytes); // base64-encoded binary data
}

Updating Files

// Update file content
await tonk.updateFile('/app/config.json', {
  theme: 'light',
  fontSize: 18,
});

// Update file with binary data
await tonk.updateFileWithBytes(
  '/images/tree.png',
  { mime: 'image/png', alt: 'updated picture' },
  newImageBytes
);

// Returns false if file doesn't exist
const updated = await tonk.updateFile('/nonexistent.txt', 'content');
console.log(updated); // false

Deleting Files

// Delete a file
const deleted = await tonk.deleteFile('/temp/cache.txt');
console.log(deleted); // true if deleted, false if didn't exist

// Check before deleting
if (await tonk.exists('/temp/cache.txt')) {
  await tonk.deleteFile('/temp/cache.txt');
}

Checking Existence

// Check if file or directory exists
const exists = await tonk.exists('/app/config.json');
if (!exists) {
  await tonk.createFile('/app/config.json', {});
}

Renaming Files

// Rename a file
await tonk.rename('/old-name.txt', '/new-name.txt');

// Rename a directory
await tonk.rename('/old-folder', '/new-folder');

// Returns false if source doesn't exist
const renamed = await tonk.rename('/nonexistent.txt', '/new.txt');
console.log(renamed); // false

Directory Operations

Creating Directories

// Create a single directory
await tonk.createDirectory('/app');

// Create nested directory structure
await tonk.createDirectory('/app');
await tonk.createDirectory('/app/components');
await tonk.createDirectory('/app/components/ui');

Listing Directory Contents

// List directory contents
const entries = await tonk.listDirectory('/app');

// Process entries
for (const entry of entries) {
  console.log(`Name: ${entry.name}`);
  console.log(`Type: ${entry.type}`); // "document" or "directory"
  console.log(`Pointer: ${entry.pointer}`); // Automerge document ID
  console.log(`Created: ${entry.timestamps.created}`);
  console.log(`Modified: ${entry.timestamps.modified}`);
}

// Filter by type
const files = entries.filter(e => e.type === 'document');
const dirs = entries.filter(e => e.type === 'directory');

Getting Metadata

// Get metadata for a file or directory
const metadata = await tonk.getMetadata('/app/config.json');
console.log(`Type: ${metadata.type}`);
console.log(`Created: ${metadata.timestamps.created}`);
console.log(`Modified: ${metadata.timestamps.modified}`);
console.log(`Pointer: ${metadata.pointer}`);

File Watching

Watching Files

// Watch a file for changes
const watcher = await tonk.watchFile('/app/config.json', doc => {
  console.log('File changed:', doc.content);
  console.log('Modified at:', doc.timestamps.modified);
});

// Get the document ID being watched
console.log(watcher.documentId());

// Stop watching when done
await watcher.stop();

Watching Directories

// Watch a directory for changes (direct descendants only)
const dirWatcher = await tonk.watchDirectory('/app/data', dirNode => {
  console.log('Directory changed:', dirNode.name);
  console.log('Children:', dirNode.children);

  // Check timestamps to see what changed
  if (dirNode.children) {
    for (const child of dirNode.children) {
      console.log(`${child.name} modified: ${child.timestamps.modified}`);
    }
  }
});

// Stop watching
await dirWatcher.stop();

Watch Patterns

// React component example
import { useEffect, useState } from 'react';

function ConfigViewer({ tonk }) {
  const [config, setConfig] = useState(null);

  useEffect(() => {
    let watcher: DocumentWatcher | null = null;

    tonk.watchFile('/config.json', (doc) => {
      setConfig(doc.content);
    }).then(w => watcher = w);

    return () => {
      if (watcher) {
        watcher.stop();
      }
    };
  }, [tonk]);

  return <div>{JSON.stringify(config)}</div>;
}

// Multi-file watcher class
class MultiFileWatcher {
  private watchers: Map<string, DocumentWatcher> = new Map();

  async watchFiles(tonk: TonkCore, paths: string[]) {
    for (const path of paths) {
      const watcher = await tonk.watchFile(path, (doc) => {
        this.handleFileChange(path, doc);
      });
      this.watchers.set(path, watcher);
    }
  }

  async cleanup() {
    for (const [path, watcher] of this.watchers) {
      await watcher.stop();
    }
    this.watchers.clear();
  }

  private handleFileChange(path: string, doc: DocumentData) {
    console.log(`File ${path} changed:`, doc.content);
  }
}

Path Resolution

Path Rules

Absolute Paths Required: All paths must start with /
No Trailing Slashes: Paths should not end with / (except root)
Case Sensitive: /App and /app are different
Forward Slashes Only: Use / even on Windows

Valid Path Examples

// ✅ Valid paths
'/app/config.json';
'/data/users/profile.json';
'/assets/images/logo.png';
'/'; // Root directory

// ❌ Invalid paths
'app/config.json'; // No leading slash
'/app/config.json/'; // Trailing slash
'\\app\\config.json'; // Wrong slash type
'./app/config.json'; // Relative path

Binary File Support

Writing Binary Files

// Browser: Convert file to bytes
async function uploadImage(file: File, tonk: TonkCore) {
  const arrayBuffer = await file.arrayBuffer();
  const bytes = new Uint8Array(arrayBuffer);

  await tonk.createFileWithBytes(
    `/images/${file.name}`,
    {
      mime: file.type,
      size: file.size,
      uploadedAt: Date.now(),
    },
    bytes
  );
}

// Node.js: Read file as bytes
import { readFile } from 'fs/promises';

const imageBytes = await readFile('./tree.png');
await tonk.createFileWithBytes('/images/tree.png', { mime: 'image/png' }, imageBytes);

Reading Binary Files

// Read and display image (browser)
async function displayImage(path: string, tonk: TonkCore) {
  const doc = await tonk.readFile(path);

  if (doc.bytes) {
    const img = document.createElement('img');
    img.src = `data:${doc.content.mime};base64,${doc.bytes}`;
    document.body.appendChild(img);
  }
}

Performance Considerations

Efficient File Operations

// ❌ Inefficient: Sequential operations
for (const file of files) {
  await tonk.createFile(`/data/${file.name}`, file.content);
}

// ✅ Efficient: Parallel operations
await Promise.all(files.map(file => tonk.createFile(`/data/${file.name}`, file.content)));

Error Handling

import { FileSystemError } from '@tonk/core';

async function safeReadFile(tonk: TonkCore, path: string) {
  try {
    return await tonk.readFile(path);
  } catch (error) {
    if (error instanceof FileSystemError) {
      console.error(`File error: ${error.message}`);
      return null;
    }
    throw error;
  }
}

// Check existence before operations
async function updateFileIfExists(tonk: TonkCore, path: string, content: any) {
  if (await tonk.exists(path)) {
    await tonk.updateFile(path, content);
  } else {
    await tonk.createFile(path, content);
  }
}

Best Practices

1. Organize with Directories

// Good: Clear hierarchy
await tonk.createDirectory('/app');
await tonk.createDirectory('/app/components');
await tonk.createDirectory('/app/stores');
await tonk.createDirectory('/app/utils');

2. Use Descriptive Paths

// Good: Self-documenting
'/config/app-settings.json';
'/data/user-profiles/john-doe.json';
'/cache/api-responses/weather-2024-01-15.json';

// Avoid: Cryptic names
'/c.json';
'/data/u1.json';
'/tmp/x.json';

3. Clean Up Watchers

Always call stop() on watchers when done to prevent memory leaks.

const watcher = await tonk.watchFile('/config.json', handleUpdate);

// Later...
await watcher.stop();

4. Handle Non-Existent Files

// Use exists() for cleaner code
if (await tonk.exists(path)) {
  const doc = await tonk.readFile(path);
  // ... process doc
} else {
  // ... handle missing file
}

// Or use updateFile's return value
const updated = await tonk.updateFile(path, content);
if (!updated) {
  await tonk.createFile(path, content);
}

Type Definitions

interface DocumentData {
  content: JsonValue;
  name: string;
  timestamps: DocumentTimestamps;
  type: 'document' | 'directory';
  bytes?: string; // Base64-encoded when file has binary data
}

interface RefNode {
  name: string;
  type: 'directory' | 'document';
  pointer: string; // Automerge document ID
  timestamps: DocumentTimestamps;
}

interface DirectoryNode {
  name: string;
  type: 'directory';
  pointer: string;
  timestamps: DocumentTimestamps;
  children?: RefNode[];
}

interface DocumentTimestamps {
  created: number;
  modified: number;
}

interface DocumentWatcher {
  documentId(): string;
  stop(): Promise<void>;
}

type JsonValue =
  | { [key: string]: JsonValue | null | boolean | number | string }
  | (JsonValue | null | boolean | number | string)[];

Synchronization Behavior

Automatic Sync

When connected to peers, VFS operations automatically synchronize:

// Peer A writes
await tonkA.createFile('/shared/doc.txt', 'Hello from A');

// Peer B receives update automatically (if watching)
await tonkB.watchFile('/shared/doc.txt', doc => {
  console.log(doc.content); // "Hello from A"
});

Conflict Resolution

The VFS uses Automerge's CRDT algorithms for automatic conflict resolution. Different types of changes merge in different ways:

Object merges: Properties from both sides are preserved
Array operations: Insertions and deletions are merged
Primitive overwrites: Last-write-wins based on Lamport timestamps

Current Limitations

Maximum recommended file size: 10MB
No native symlink support
No file permissions/ownership model
Directory watches only track direct descendants
Binary data is base64-encoded (size overhead)

Bundle Format

Bundles are self-contained packages that encapsulate Tonk applications, their data, and metadata into a single distributable file with the .tonk extension.

Overview

A bundle is essentially a ZIP archive containing:

Application code and assets as serialized automerge documents
Manifest with metadata and configuration

Bundle Structure

my-app.tonk (ZIP archive)
├── manifest.json           # Bundle metadata and configuration
├── storage/               # Serialized Automerge documents
    ├── root.automerge    # Root document (always present)
    ├── [doc-id-1].automerge
    ├── [doc-id-2].automerge
    └── ...

## Manifest Format

The `manifest.json` file contains essential metadata about the bundle:

```json
{
  "manifest_version": 1,
  "version": {
    "major": 1,
    "minor": 0
  },
  "root_id": "automerge:3qF8x9...",
  "entrypoints": ["index.html"],
  "network_uris": ["wss://sync.example.com"],
  "x_notes": "Optional human-readable notes",
  "x_vendor": {
    "app_name": "My Tonk App",
    "author": "John Doe",
    "custom_field": "any value"
  }
}

Manifest Fields

Field	Type	Required	Description
`manifest_version`	number	Yes	Bundle format version (currently 1)
`version`	object	Yes	Bundle version with major/minor
`root_id`	string	Yes	Automerge document ID of root directory
`entrypoints`	string[]	No	Entry files for the application
`network_uris`	string[]	No	WebSocket URIs for sync
`x_notes`	string	No	Human-readable notes
`x_vendor`	object	No	Custom vendor-specific metadata

Tonk Docs