plum

#treesitter#compiler#wasm

git clone https://git.pyrossh.dev/plum

A statically typed, imperative programming language inspired by rust, python

188c69e — Peter John 2026-11-04T13:30:14+05:30 
docs(plans): compiler cleanup and architecture review plan (written by another session)
Files changed (1) hide show
  1. docs/plans/2026-11-04-compiler-cleanup.review.md +116 -0
docs/plans/2026-11-04-compiler-cleanup.review.md ADDED
@@ -0,0 +1,116 @@
1
+ # Compiler cleanup and architecture: review plan
2
+
3
+ ## Summary
4
+ The compiler works, but most of its size comes from repetition. About 15 passes each walk the program tree in their own hand-written way. Type information is stored by memory address, so any copy of the program loses it, and the checker runs up to 6–7 times per compile to get it back. State lives in hidden globals. The order of passes is spread over the loader, the checker and codegen. The tests copy the same helpers into 20-odd files.
5
+ This plan removes the repetition and dead code first, then adds shared building blocks, then fixes the four root causes. Each phase is its own build and ends with every test green, so we can stop after any phase. The findings behind it are in the 2026-11-04 compiler review.
6
+
7
+ ## What you will get
8
+ - About 4–6k fewer lines of Rust and about 60–100 fewer duplicate tests, with roughly 25 fewer test programs to link.
9
+ - One shared way to walk the program tree. A new language construct then needs one edit, not fifteen.
10
+ - One shared host crate for the CLI, the runtime, the game host and the tests, replacing three copies of host functions such as print, files and http.
11
+ - One compile call with an options object, in place of six entry points.
12
+ - The order of passes written down in one place, as a list you can read and test.
13
+ - The type checker runs once per compile instead of 6–7 times. Codegen no longer guesses missing types; a missing type becomes a clear error.
14
+ - Deterministic output: the same program always compiles to the same names, so snapshot tests become possible.
15
+ - Errors that carry file, line and column, which also fixes std errors being blamed on the entry file.
16
+ - Checker and codegen split out of their 5.7k- and 9k-line files into about two dozen modules with clear jobs.
17
+ - Four latent bugs fixed:
18
+ - text-searching the debug output of code (a string containing `Try(` can block a rewrite)
19
+ - the silent fallback to Int when codegen can't find a type
20
+ - the LSP dropping generic arguments
21
+ - clashing generated names when two `return for` sit on the same line
22
+
23
+ ## What will be built
24
+ | Phase | Part | What it does | Size | Built by |
25
+ |---|---|---|---|---|
26
+ | 1 | Dead code out | Deletes unused functions and types, re-enables the compiler's dead-code warnings, removes the no-op parenthesis node | S | a fast helper |
27
+ | 1 | Small duplicates | Merges the copied re-check step, the copied receiver-name helper and the 7 copies of the primitive-name table | S | a fast helper |
28
+ | 1 | Shared test helpers | One test-support crate (parse that fails on syntax errors, load, compile, run, engine) used by every test file | M | a careful helper |
29
+ | 1 | Fewer test programs | Merges the small end-to-end test files into one per crate, removes duplicate tail-call and closure tests, makes the examples sweep look inside sub-folders | M | a careful helper |
30
+ | 2 | Tree walker | One shared walker and renamer; every pass switches to it one at a time | L | careful helpers, one per pass |
31
+ | 2 | Real tree queries | Replaces the debug-text searches in tail-recursion and closure passes with walker queries | S | a careful helper |
32
+ | 2 | Host crate | One crate with the engine setup, string passing and all host imports; CLI, runtime and game use it | M | a careful helper |
33
+ | 2 | One front end | One lowering entry used by both the loader and single-file parsing; `test` blocks are turned into functions there instead of in codegen | M | a careful helper |
34
+ | 2 | One compile call | Six entry points become one call with options; fusion settings move into those options | M | a careful helper |
35
+ | 3 | State into structs | The checker's 17 and codegen's 7 hidden globals move into a Checker and a Codegen value; generated-name counters restart each compile | L | careful helpers |
36
+ | 3 | Pass list | A small driver owns the order of passes and the "keep a rewrite only if it still type-checks" rule; the checker goes back to only checking | M | a careful helper |
37
+ | 3 | Positioned errors | One error type with file, line, column and kind, replacing plain strings and message-text matching; parser placeholders become real errors | M | a careful helper |
38
+ | 4 | Node ids | Every expression and statement gets a stable id; type tables, union plans and error coercions use it instead of memory addresses | L | careful helpers |
39
+ | 4 | Typed codegen | Codegen's own synthetic nodes are created before checking, so codegen re-infers nothing and the Int fallback goes away | M | a careful helper |
40
+ | 5 | File splits | Checker into about 13 modules, codegen into about 11, the 1,000-line codegen driver broken into its phases | M | fast helpers (pure moves) |
41
+ | 5 | Enum kinds | One "kind" field replaces the record/trait/value/inline flags and name-prefix tests; the record, embed and trait lowering steps merge | M | a careful helper |
42
+ | 6 | Tests in Plum | About 90 Rust tests that only run a program and compare the answer become `.plum` test blocks | M | a fast helper |
43
+
44
+ ## How it fits together
45
+ ```mermaid
46
+ flowchart LR
47
+ P[Parser] --> F[One front end]
48
+ F --> C[Checker, once]
49
+ C --> D[Pass list driver]
50
+ D --> G[Codegen]
51
+ G --> H[Host crate]
52
+ W[Shared tree walker] -.used by.-> F
53
+ W -.-> D
54
+ W -.-> G
55
+ N[Node ids + positioned errors] -.used by.-> C
56
+ N -.-> G
57
+ T[Test-support crate] -.tests.-> H
58
+ ```
59
+
60
+ ## How it gets built
61
+ ```mermaid
62
+ flowchart TB
63
+ subgraph P1["Phase 1: deletions and test helpers, in parallel"]
64
+ A1[Dead code] --- A2[Small duplicates] --- A3[Test helpers then merged test files]
65
+ end
66
+ subgraph P2["Phase 2: shared building blocks, in parallel"]
67
+ B1[Tree walker then each pass] --- B2[Host crate] --- B3[One front end] --- B4[One compile call]
68
+ end
69
+ subgraph P3["Phase 3: state, pass list, errors"]
70
+ C1[State into structs] --> C2[Pass list] --- C3[Positioned errors]
71
+ end
72
+ P1 --> P2 --> P3 --> P4["Phase 4: node ids, typed codegen"] --> P5["Phase 5: file splits, enum kinds"] --> P6["Phase 6: tests in Plum"]
73
+ ```
74
+ Each phase gets its own machine plan and its own commit series, and ends with the full workspace test suite and the examples green. I come back to you only between phases if something changed from this plan.
75
+
76
+ ## Decisions you can change
77
+ - When to start: after the language-restrictions branch is merged to master (decided by you, 2026-11-04). Then on a fresh branch from master.
78
+ - Where it sits in your queue: before native inline variants, because those add more AST variants and walkers (default) | after the benchmark / FBIP work.
79
+ - Enum kinds (phase 5): do it here (default) | fold it into the queued record-into-struct merge.
80
+ - Pass driver: a new small driver crate (default) | a module inside codegen.
81
+ - Tests in Plum (phase 6): only once `plum test` runs tail calls the same way (default) | skip phase 6.
82
+ - Duplicate tests: delete the lower-value copy (default) | keep both, only merge the files.
83
+
84
+ ## Risks
85
+ - The copied walkers behave slightly differently in places: pattern names, field assignments, where shadowing stops. Before each pass switches over, a test pins its current behaviour, and passes move one at a time.
86
+ - Node ids touch every crate. They come only after the walker and the state structs exist, which makes the change much smaller. If it proves too disruptive, phase 4 can stop after the error-coercion table.
87
+ - Turning the Int fallback into an error may expose real type gaps that are hidden today. Each gap gets fixed or listed, not papered over.
88
+ - A pure behaviour-preserving refactor can still shift generated names. Phases 1–2 check that the examples compile to identical wasm, and phase 3 onward uses snapshot tests.
89
+ - The open run's uncommitted work touches many of the same files. That is why the default is to start after it lands.
90
+
91
+ ## Out of scope
92
+ - No language changes: no new syntax, no new features, no changes to the queued language work.
93
+ - No performance work beyond removing the repeated type checks.
94
+ - The LSP gets only the shared helpers and the generic-argument fix, not a redesign.
95
+
96
+ ## How we will know it works
97
+ - The full workspace tests, `plum test` over std and examples, and the browser game check all pass after every phase.
98
+ - Phases 1–2: every example compiles to byte-identical wasm compared with before.
99
+ - Lines of Rust and the number of test programs go down by the amounts above (counted at each phase end).
100
+ - The checker runs once per compile, shown by a counter test.
101
+ - Compiling the same program twice gives identical output.
102
+ - A string literal containing `Try(` no longer blocks the tail-recursion rewrite; a regression test proves it.
103
+
104
+ ## Rough time and cost
105
+ Rough, until each phase's machine plan is linted:
106
+
107
+ | Phase | Tasks | Time |
108
+ |---|---|---|
109
+ | 1 | 4–5 | 15–25 min |
110
+ | 2 | 8–12 | 40–70 min |
111
+ | 3 | 5–7 | 30–50 min |
112
+ | 4 | 3–5 | 30–60 min |
113
+ | 5 | 4–6 | 20–35 min |
114
+ | 6 | 2–3 | 10–20 min |
115
+
116
+ Altogether about 2.5–4.5 hours of build time, roughly $40–90.