plum

#treesitter#compiler#wasm

git clone https://git.pyrossh.dev/plum

A statically typed, imperative programming language inspired by rust, python

b8d0694 — Peter John 2026-11-04T17:36:17+05:30 
docs: self-hosting design and groundwork review plan
Files changed (3) hide show
  1. docs/backlog.md +4 -0
  2. docs/plans/2026-11-04-self-hosting-groundwork.review.md +117 -0
  3. docs/self-hosting.md +105 -0
docs/backlog.md CHANGED
@@ -13,6 +13,10 @@ Open work, in the order the user queued it, plus smaller items found during buil
13
13
  5. **One product type, `struct`** — after FBIP, merge `record` into an immutable `struct` with compiler-chosen boxing (optional hint, shown by `--explain`). (Decision record §9b.)
14
14
  6. **Deterministic games** — fixed timestep, input recording and replay, seeded `Random` in game state, state checksums; port the design from `../kam_macroquad` (`main.rs` tick accumulator, `game/replay.rs`). The `Io` capability makes recording and replaying host input straightforward. (Decision record §10.)
15
15
 
16
+ ## Planned, not yet placed in the queue
17
+
18
+ - **Self-hosting groundwork** (user 2026-11-04: plan now, build later) — `docs/plans/2026-11-04-self-hosting-groundwork.review.md`, design in `docs/self-hosting.md`. All host I/O onto `Io` (`Os.plum` deleted, `io.now()`, `io.httpGet`), then a Plum lexer, a winnow-style combinator library with chumsky-style errors, an AST and a parser in `compiler/` that parses every `.plum` file. AST comparison with the Rust parser and new host calls (args, exit, env, stderr, binary files) wait for cleanup phase 2a. The `Io` move should land before the `Io`-primitive plan above.
19
+
16
20
  ## FBIP worklist: `mut` fields left after language-restrictions
17
21
 
18
22
  | Type | `mut` fields | Why it is mutable today |
docs/plans/2026-11-04-self-hosting-groundwork.review.md ADDED
@@ -0,0 +1,117 @@
1
+ # Self-hosting groundwork: review plan
2
+
3
+ Design: [../self-hosting.md](../self-hosting.md). Nothing is built until you approve.
4
+
5
+ ## Summary
6
+ The first steps toward a Plum compiler written in Plum. Every host call moves onto the `Io` capability (`Os.plum`
7
+ goes away), then a lexer, a parser-combinator library and a parser are written in Plum under a new `compiler/`
8
+ folder. Done means the Plum parser parses every `.plum` file in the repo without errors and reports clear errors with
9
+ positions for broken input. Comparing its output with the Rust parser, and new host calls (arguments, exit code), come
10
+ after compiler cleanup phase 2a merges, because that phase owns the CLI and host code.
11
+
12
+ ## What you will get
13
+ - **All host I/O on `Io`:** `io.readFile(path)`, `io.writeFile`, `io.readDir`, `io.mkdtemp`, `io.rename`, ... (was
14
+ `Os.readFile(io, path)`); `io.now()` (was `Time.now(io)`); `io.httpGet(url, headers)` and friends (was
15
+ `Http.get(io, ..)`). `plum-std/Os.plum` deleted, with its `todo` stubs and duplicate names (`cp`, `access`,
16
+ `lchmod`). Clock and HTTP methods are declared in `std/Time` and `std/Http` as `Io` extensions, so a game that does
17
+ not import them does not compile them (measured: +20% wasm otherwise; the Io move itself adds 0.8%).
18
+ - **`compiler/Token.plum`, `compiler/Lexer.plum`:** tokens with line, column, "starts a line" and "space before";
19
+ numbers (`_`, `0x`, `0b`, floats, `f`), every operator, keywords incl. `fun*`, strings with `{expr}` interpolation
20
+ and escapes, comments kept aside.
21
+ - **`compiler/Parse.plum`:** the combinator library: cursor with save/rewind, `many`, `sepBy`, `between`, `choice`,
22
+ `optional`, `label`, `block` (indentation rule), `pratt` (operator precedence), error collection with the furthest
23
+ failure and expected tokens, and recovery.
24
+ - **`compiler/Ast.plum`:** the syntax tree, following the Rust AST's shapes, with a node id and a span on every node.
25
+ - **`compiler/Parser.plum`:** Plum's grammar, built up in steps: types and literals, expressions, statements and
26
+ blocks, declarations, patterns and `match`, string interpolation.
27
+ - **Tests:** small cases per part, plus one test that lexes and parses all ~75 `.plum` files in the repo.
28
+ - `docs/self-hosting.md` (already written) and a backlog entry.
29
+
30
+ ## What will be built
31
+ | Part | What it does | Size | Built by |
32
+ |---|---|---|---|
33
+ | Io move | File, clock and HTTP calls become `Io` methods; Os.plum deleted; README, LSP index and tests, game fixtures updated | M | a careful helper |
34
+ | Tokens | The token kinds and the token record | S | a fast helper |
35
+ | Lexer | Turns source text into tokens and positioned errors | M | a careful helper |
36
+ | Combinator core | Cursor, combinators, errors and recovery, tested on a toy grammar | M | a careful helper |
37
+ | Pratt + block | Operator precedence and the indentation rule | M | a careful helper |
38
+ | AST | The tree types and a printer for tests | S | a fast helper |
39
+ | Parser: expressions | Literals, names, calls, fields, operators, closures, `if then else`, `catch`, `?:` | M | a careful helper |
40
+ | Parser: statements | Assignments, `if`, `for`, `while`, `return`, `match` with patterns, blocks | M | a careful helper |
41
+ | Parser: declarations | `module`, `import`, `fun`, `enum`, `struct`, `record`, `trait`, `error`, `primitive`, `test` | M | a careful helper |
42
+ | Whole-repo check | Lex and parse every `.plum` file; fix what it finds | S | a careful helper |
43
+
44
+ ## How it fits together
45
+ ```mermaid
46
+ flowchart LR
47
+ IO["Io: files, clock, HTTP"] --> RT["repo-wide test reads files"]
48
+ SRC[".plum source"] --> LX[Lexer]
49
+ LX --> TK[Tokens]
50
+ TK --> PC["Combinator core + Pratt + block"]
51
+ PC --> PR[Parser]
52
+ PR --> AST["AST + errors"]
53
+ RT --> LX
54
+ ```
55
+
56
+ ## How it gets built
57
+ ```mermaid
58
+ flowchart TB
59
+ subgraph W1["Wave 1, in parallel"]
60
+ A[Io move]
61
+ B[Tokens]
62
+ C[Combinator core]
63
+ D[AST]
64
+ end
65
+ B --> E[Lexer]
66
+ C --> F[Pratt + block]
67
+ E --> G[Parser: expressions]
68
+ F --> G
69
+ D --> G
70
+ G --> H[Parser: statements]
71
+ H --> I[Parser: declarations]
72
+ I --> J[Whole-repo check]
73
+ A --> J
74
+ ```
75
+ The parser parts run one after another: each builds on the rules of the one before.
76
+
77
+ ## Decisions you can change
78
+ - Folder: `compiler/` (default; import paths can't contain `-`, so `plum-compiler/` is not possible).
79
+ - Parser style: winnow-style rules as functions over a cursor, chumsky-style errors (default) | chumsky-style parser
80
+ values (closures stored in records; not optimised by the compiler today).
81
+ - Layout: indentation rule in the parser (default) | INDENT/DEDENT tokens from the lexer (cannot handle indented
82
+ closure bodies inside call brackets without parser feedback).
83
+ - Os removal: delete `Os` outright (default) | keep `Os` as forwarding functions for a while.
84
+ - HTTP method names: `io.httpGet`, `io.httpPost`, ... (default) | `io.fetch(method, url, ..)` only.
85
+ - String forms: `"..."` and `"""..."""` (default; the only forms used in the repo) | also `'...'` and backticks,
86
+ which the tree-sitter scanner accepts.
87
+
88
+ ## Risks
89
+ - **Compiler bugs surface.** Generic combinators that take function arguments are the hardest test yet of Plum's
90
+ generics and closure specialisation. Each bug is fixed in the Rust compiler with a regression test, or worked
91
+ around and listed in the backlog if the fix is large.
92
+ - **Conflict with cleanup phase 2a,** which runs in another worktree and owns the CLI, runtime and game host files.
93
+ This plan touches none of those: only std `.plum` files, the LSP index, two game test fixtures, one game test
94
+ string, README and new `compiler/` files. New host imports and the AST dump wait for 2a.
95
+ - **Deep recursion.** Deeply nested input could overflow the wasm stack; the Pratt loop keeps operator chains flat,
96
+ and a nesting test covers 200 levels.
97
+ - **Speed.** The whole-repo test must stay quick in `plum test`; if it is slow it is moved to a separate, slow test.
98
+ - **The `Io` primitive plan** (another session, runs last) builds on this `Io` move; nothing here blocks it.
99
+
100
+ ## Out of scope
101
+ - Comparing the AST with the Rust parser (needs a CLI dump command after phase 2a).
102
+ - New host calls: arguments, exit code, environment, stderr, binary files (after phase 2a's `plum-host` crate).
103
+ - Checker, passes, wasm encoder, formatter, LSP in Plum (later stages in `docs/self-hosting.md`).
104
+ - `Io` as a built-in primitive with extern methods (its own plan).
105
+
106
+ ## How we will know it works
107
+ - `plum test plum-std plum-examples plum-game/web-test/fixtures compiler` all pass; `cargo test --workspace`, the
108
+ LSP tests and the plum-game host tests pass.
109
+ - No `Os.` or `std/Os` left in the repo outside old plan files.
110
+ - The whole-repo test lexes and parses every `.plum` file with zero errors.
111
+ - Broken input gives positioned errors, e.g. `fun f(a: Int` reports "expected `)`" at the end of the line, and a
112
+ second error later in the file is still reported.
113
+ - A game's wasm grows by at most 1% from the Io move.
114
+
115
+ ## Rough time and cost
116
+ Rough, until the machine plan is linted: 10 parts, about 4-6 hours of helper time, most of it the parser steps,
117
+ which run one after another.
docs/self-hosting.md ADDED
@@ -0,0 +1,105 @@
1
+ # Self-hosting Plum
2
+
3
+ Status: design, 2026-11-04. The first slices are planned in
4
+ [plans/2026-11-04-self-hosting-groundwork.review.md](plans/2026-11-04-self-hosting-groundwork.review.md); nothing here is built yet.
5
+
6
+ Goal: the lexer, parser, checker, compiler passes, wasm emitter and CLI written in Plum, compiled by a Rust "stage 0"
7
+ compiler until the Plum compiler can compile itself.
8
+
9
+ ## 1. Is it possible?
10
+
11
+ Yes. The Rust compiler is about 37k lines (plum-core 17k, plum-wasm-codegen 9k, plum-checker 6.7k, plum-cli 2k,
12
+ plum-lsp 1.9k), and nearly all of it is the kind of code Plum already handles: enums with payloads + `match`, `Map`,
13
+ `Slice`, `Buffer`, string building, recursion with tail calls, error unions. The hard part is what the Rust code gets
14
+ from libraries, and a few host gaps.
15
+
16
+ | Rust dependency | Used for | In Plum |
17
+ |---|---|---|
18
+ | `tree-sitter` + `tree-sitter-plum` | the parser (the compiler walks a tree-sitter tree) | own lexer + parser, the biggest new piece (about 3-4k lines) |
19
+ | `topiary` | `plum format` | own pretty printer, Wadler-style document combinators (about 500 lines) |
20
+ | `wasm-encoder`, `wasmparser` | writing wasm | own encoder: LEB128, sections, GC types (about 1-1.5k lines) |
21
+ | `wasmtime` | `plum run`, `plum test` | stays outside Plum: a compiler emits wasm, a host runs it |
22
+ | `clap` | CLI flags | a small `Args` module (about 150 lines) |
23
+
24
+ The tree-sitter grammar stays for editors (Helix, VS Code).
25
+
26
+ ## 2. What Plum needs first
27
+
28
+ **Host (std `Io`).** Today: printing, files as `Str`, clock, random seed, HTTP. Missing: command-line arguments,
29
+ exit code, environment variables, stderr, binary file read/write (`.wasm` output). All host calls belong on `Io`
30
+ (`plum-std/Io.plum` is the only file allowed to call an `extern`); the first step is moving `Os`/`Time.now`/`Http`
31
+ there (groundwork plan, part 1). New host imports wait for the `plum-host` crate (compiler cleanup phase 2a, T10).
32
+
33
+ **Compiler and language.**
34
+ - Deterministic output: the bootstrap check (stage 2 == stage 3, byte for byte) needs it. Cleanup phase 2a T1.
35
+ - Deep structural `==` for records (backlog): ASTs and types are compared all the time.
36
+ - Explicit function generics (`fun map[B]`, queued after cleanup phase 2): a compiler has many one-letter names.
37
+ - FBIP (backlog): without in-place updates a functional compiler copies nodes on every rewrite. Correct, but slow.
38
+ - Dead-code elimination: an unused import adds its whole module to the wasm today (+20% measured for `std/Http` +
39
+ `std/Json` in a game). Not a blocker; std works around it by declaring `Io` methods next to the types they return.
40
+
41
+ **Not needed.** Regex (lexers are hand-written byte loops) and a parser-generator.
42
+
43
+ ## 3. Parsing approach
44
+
45
+ **Lexer: hand-written.** A loop over bytes with `match` on the current byte. Every token carries `line`, `col`,
46
+ `startsLine` (first token on its line) and `spaceBefore` (needed for postfix `x!`). Strings come out as a token
47
+ sequence: `StrStart`, `StrText`, `InterpOpen`, the tokens of the interpolated expression, `InterpClose`, `StrEnd`,
48
+ with `{{`/`}}` and `\n \t \r \\ \" \b \f \xNN \u0000` decoded. Comments are kept aside as trivia for a future
49
+ formatter. No INDENT/DEDENT tokens (see layout).
50
+
51
+ **Layout: an offside rule in the parser, not the lexer.** Plum has indented blocks inside brackets
52
+ (`each(|v|` + an indented body + `)`, `plum-examples/closures.plum`) and also multi-line argument lists, which a
53
+ Python-style lexer that ignores newlines inside brackets cannot tell apart. Instead the parser keeps the current block
54
+ column: a token that starts a line at or left of it ends the expression or block; inside brackets the rule is off until
55
+ a nested block (a closure body) turns it on again. This is the Haskell / F# approach and keeps the lexer context-free.
56
+
57
+ **Parser: combinators, winnow style, with chumsky-style errors.**
58
+ - A rule is a plain function over a mutable cursor: `fun expr(p: Parser) -> Expr?` (null = no match; the cursor can
59
+ `save()`/`rewind(m)`). Combinators are generic functions that take rules: `many`, `sepBy`, `between`, `choice`,
60
+ `optional`, `label`, `block` (offside), `pratt` (operator precedence table).
61
+ - Why not chumsky's "parsers are values" style: in Plum that means closures stored in generic records, which the
62
+ closure specialiser does not optimise (it specialises closures passed straight into a call), and recursive rules
63
+ would need a lazy wrapper. Rules passed by name into generic combinators get specialised copies, so they cost
64
+ nothing extra.
65
+ - Errors as in chumsky: the furthest position any alternative reached plus the set of tokens expected there;
66
+ `label("expression")` replaces a long expected-set with one word; recovery with `recoverUntil(syncTokens)` and
67
+ skipping balanced brackets, so one error does not hide the next. Diagnostics are data (`Diag(span, message)`)
68
+ collected in the parser state, never a crash.
69
+
70
+ ## 4. Patterns for the compiler in Plum
71
+
72
+ - **ADTs + `match`, not visitors.** `enum Expr = | Var(..) | Call(..) | Match(..) ...`. A visitor is what OOP
73
+ languages use for missing sum types.
74
+ - **One "map over children" function** (`mapChildren(e, f)`) instead of a visitor; a pass handles the cases it cares
75
+ about and sends everything else there. Cleanup phase 2's shared walker (`plum-core/src/visit.rs`) is the Rust twin.
76
+ - **Node ids + side tables.** Each expression and statement has an `Int` id; types, resolved names and spans live in
77
+ `Slice`s indexed by id (cleanup phase 4). The AST stays immutable.
78
+ - **An explicit state record** for the checker (`record Checker(mut types: .., mut errors: ..)`), like the game's
79
+ `World` shell: mutation in one place (cleanup phase 3).
80
+ - **A pass list:** `Module -> Module` functions run in a fixed order (cleanup phase 3).
81
+ - **Diagnostics as data,** never stop at the first error (cleanup phase 3).
82
+ - **Document combinators for the formatter:** `text`, `line`, `group`, `nest`.
83
+
84
+ The compiler cleanup phases 2-4 turn the Rust code into this shape, so porting after them is translation, not
85
+ redesign.
86
+
87
+ ## 5. Porting stages
88
+
89
+ 1. Groundwork (planned): `Io` holds all host calls; lexer; combinator core; AST; parser that parses every `.plum`
90
+ file in the repo.
91
+ 2. Same output as Rust: an S-expression dump of the AST from both parsers, compared over every file. Needs a dump
92
+ command in `plum-cli`, so it waits for cleanup phase 2a (which owns `plum-cli/src/main.rs`).
93
+ 3. Checker in Plum: compare the inferred type of every node and the error list against Rust.
94
+ 4. Codegen + wasm encoder, no optimisations: the existing `plum test` suite (687 tests) must pass on the Plum
95
+ compiler's output. Then port fusion, TRMC, reuse.
96
+ 5. Bootstrap: Rust stage 0 builds stage 1; stage 1 builds stage 2; stage 2 builds stage 3; stage 2 == stage 3 byte
97
+ for byte, checked in CI. Keep stage 0 frozen until this holds.
98
+ 6. CLI, formatter, LSP (JSON-RPC over stdin; `std/Json` exists) in Plum. Rust shrinks to a thin wasmtime host.
99
+
100
+ ## 6. Related plans
101
+
102
+ - Compiler cleanup: `plans/2026-11-04-compiler-cleanup.review.md` (phases 2-6 shape the Rust code like the Plum target).
103
+ - `Io` as a built-in primitive with extern methods: `plans/2026-11-04-io-primitive-extern-methods.review.md`
104
+ (runs last, after the cleanup; builds on the `Io` move in the groundwork plan).
105
+ - Deterministic games (backlog): shares the "all input through `Io`" step.