plum

#treesitter#compiler#wasm

git clone https://git.pyrossh.dev/plum

A statically typed, imperative programming language inspired by rust, python

f198426 — Peter John 2026-11-04T17:03:00+05:30 
docs: compiler cleanup phase 2a plan
Files changed (1) hide show
  1. docs/plans/2026-11-04-compiler-cleanup-phase2a.md +220 -0
docs/plans/2026-11-04-compiler-cleanup-phase2a.md ADDED
@@ -0,0 +1,220 @@
1
+ # Compiler cleanup, phase 2a: determinism, shared walker, generics by scope, host crate
2
+
3
+ **Review plan:** docs/plans/2026-11-04-compiler-cleanup.review.md
4
+ **Goal:** Deterministic compiler output; one shared AST walker (`plum-core/src/visit.rs`) replacing the copied walkers; generic names decided in one place; one `plum-host` crate for the CLI, runtime and game host. No behaviour change.
5
+ **Architecture:** Rust workspace (plum-core, plum-checker, plum-wasm-codegen, plum-cli, plum-lsp, plum-test-support); plum-runtime and plum-game are separate cargo projects with path deps.
6
+ **Tech stack:** Rust 2021, cargo, wasmtime 47, tree-sitter-plum.
7
+ **Detail:** contracts
8
+ **Target-min:** 25
9
+
10
+ ## Global constraints
11
+ - Agents cannot run `cargo test` (hook). Build with `cargo build -p <crate> --tests`, then run the binaries in `target/debug/deps` directly. `touch` edited .rs files first.
12
+ - No behaviour change. After T1, compiler output is deterministic, so every later task checks `bash scripts/wasm-hashes.sh <T1-sha> A` vs `bash scripts/wasm-hashes.sh <your stash-create sha> B` → `diff A B` empty (the coordinator gives T1's sha in your task message).
13
+ - camelCase fns, `#![allow(non_snake_case)]`. Never run `plum format` on real files. Don't commit.
14
+ - Phase 2b (separate plan) does the front-end lowering entry, the single compile call and `Expr::Paren` removal: don't start them here.
15
+
16
+ ## Main diagram
17
+ ```mermaid
18
+ flowchart LR
19
+ D["T1 determinism"] --> C["T3-T7 passes use visit.rs"]
20
+ V["T2 visit.rs"] --> C
21
+ C --> G["T8 generics by scope"]
22
+ H["T10 plum-host"] --> X["T9 verify"]
23
+ S["T13 test helpers + LSP fix"] --> X
24
+ G --> X
25
+ ```
26
+
27
+ ## File structure
28
+ - `plum-core/src/visit.rs` — child lists, mutable/read-only walks, scope-aware rename, type substitution, small shared queries.
29
+ - `plum-core/src/generics.rs` — the only rule for "is this name a generic", and a fn's generic set.
30
+ - `plum-host/` — wasmtime engine, Str passing, host imports, run helpers.
31
+
32
+ ## Task graph
33
+ ```mermaid
34
+ flowchart LR
35
+ T1[determinism]
36
+ T2[visit module]
37
+ T3[drive simplify]
38
+ T4[closure_spec]
39
+ T5[forms union_lower]
40
+ T6[uniqueness reuse tail]
41
+ T7[trmc queries]
42
+ T8[generics by scope]
43
+ T10[plum-host crate]
44
+ T13[test helpers LSP]
45
+ T9[verify 2a]
46
+ %% taskgraph
47
+ T1 --> T3
48
+ T1 --> T4
49
+ T1 --> T5
50
+ T1 --> T6
51
+ T1 --> T7
52
+ T2 --> T3
53
+ T2 --> T4
54
+ T2 --> T5
55
+ T2 --> T6
56
+ T2 --> T7
57
+ T3 --> T8
58
+ T5 --> T8
59
+ T7 --> T8
60
+ T4 --> T9
61
+ T6 --> T9
62
+ T8 --> T9
63
+ T10 --> T9
64
+ T13 --> T9
65
+ ```
66
+ Ruling: sequential - T8 edits drive/forms/trmc after their conversions; same files - cost: longer chain.
67
+
68
+ ## Package A: compiler core
69
+ **Probe entry:** run=bash scripts/wasm-hashes.sh
70
+
71
+ ### T1 deterministic output · Complexity 5 · Est. context 50k
72
+ **Owns:** plum-wasm-codegen/src/lib.rs, plum-checker/src/lib.rs, plum-checker/src/unions.rs, plum-core/src/{drive,simplify,closure_spec,forms,union_lower,uniqueness,reuse,tail,trmc,purity,unions,ctors,embeds,traits,loader}.rs
73
+ **Components:** 1
74
+ **LOC owned:** 60
75
+ **Tests:** slow
76
+ **Model:** sonnet
77
+ **Produces:** none
78
+ **Review Focus:** `same_bytes_twice`: `bash scripts/wasm-hashes.sh HEAD A; bash scripts/wasm-hashes.sh HEAD B; diff A B` empty for all 25 examples (run twice more to be sure); `plum test plum-std plum-examples` still 678 passed.
79
+ **Test code:** `same_bytes_twice` above via the hash script (report the diff output); all workspace test binaries pass.
80
+ **Steps:** find every HashMap/HashSet whose iteration order reaches emitted wasm or generated names (type section order, function order, globals, data, fresh names); switch to BTreeMap/BTreeSet or sort at the emit point. Don't change lookup-only maps.
81
+
82
+ ### T2 visit module · Complexity 5 · Est. context 40k
83
+ **Owns:** plum-core/src/visit.rs, plum-core/src/lib.rs, plum-core/tests/visit_test.rs
84
+ **Components:** 1
85
+ **LOC owned:** 260
86
+ **Tests:** fast
87
+ **Model:** sonnet
88
+ **Produces:**
89
+ - visit::childExprs(e: &mut Expr) -> Vec<&mut Expr>
90
+ - visit::childRefs(e: &Expr) -> Vec<&Expr>
91
+ - visit::stmtExprs(s: &mut Stmt) -> Vec<&mut Expr>
92
+ - visit::stmtBlocks(s: &mut Stmt) -> Vec<&mut Block>
93
+ - visit::visitBlock(b: &mut Block, f: &mut dyn FnMut(&mut Expr))
94
+ - visit::visitExpr(e: &mut Expr, f: &mut dyn FnMut(&mut Expr))
95
+ - visit::anyExpr(e: &Expr, pred: &mut dyn FnMut(&Expr) -> bool) -> bool
96
+ - visit::Name enum { Decl, Pat, Plain, Ref }
97
+ - visit::renameStmts(stmts: &mut [Stmt], rn: &mut dyn FnMut(&mut String, Name))
98
+ - visit::substType(t: &Type, map: &HashMap<String, Type>) -> Type
99
+ - visit::patternBinders(p: &CasePattern, out: &mut Vec<String>)
100
+ - visit::unparen(e: &Expr) -> &Expr
101
+ - visit::countVar(body: &FnBody, name: &str) -> usize
102
+ e.g. anyExpr(&parse_expr("f(Try(1))"), &mut |e| matches!(e, Expr::Call{..})) -> true
103
+ e.g. anyExpr(&parse_expr("\"Try(\""), &mut |e| matches!(e, Expr::Call{..})) -> false
104
+ e.g. substType(&Type{name:"List", generics:[T]}, {T: Int}) -> List[Int]
105
+ **Review Focus:** `rename_shadowing`: a closure param shadowing `x` is renamed as Decl and stops outer renames in its body (copy drive.rs behaviour exactly); `pattern_names`: constructor names in patterns are not binders.
106
+ **Test code:** `plum-core/tests/visit_test.rs`: the e.g. lines, one rename test per Name kind, patternBinders on `Cons(head: h, tail: t)` → ["h","t"].
107
+ **Notes:** copy `drive.rs` 1884-2260 (walkers + rename) as the reference behaviour; add only `anyExpr`. Generic `substType` must not decide what is generic — it substitutes exactly the map's keys.
108
+
109
+ ### T3 drive and simplify use visit · Complexity 4 · Est. context 45k
110
+ **Owns:** plum-core/src/drive.rs, plum-core/src/simplify.rs
111
+ **Components:** 1
112
+ **LOC owned:** 60
113
+ **Tests:** slow
114
+ **Model:** sonnet
115
+ **Consumes:** visit::childExprs(e: &mut Expr) -> Vec<&mut Expr> (T2) · visit::renameStmts(stmts: &mut [Stmt], rn: &mut dyn FnMut(&mut String, Name)) (T2) · visit::childRefs(e: &Expr) -> Vec<&Expr> (T2) · visit::unparen(e: &Expr) -> &Expr (T2)
116
+ **Produces:** none
117
+ **Test code:** plum-core drive_test + simplify_test binaries pass; wasm hashes identical to T1.
118
+ **Steps:** delete the local walker/rename copies and `isTag`/`isTagName` duplicates, call visit; keep `isGen`/`substType` for T8.
119
+
120
+ ### T4 closure_spec uses visit · Complexity 4 · Est. context 45k
121
+ **Owns:** plum-core/src/closure_spec.rs
122
+ **Components:** 1
123
+ **LOC owned:** 50
124
+ **Tests:** slow
125
+ **Model:** sonnet
126
+ **Consumes:** visit::childExprs(e: &mut Expr) -> Vec<&mut Expr> (T2) · visit::renameStmts(stmts: &mut [Stmt], rn: &mut dyn FnMut(&mut String, Name)) (T2) · visit::countVar(body: &FnBody, name: &str) -> usize (T2)
127
+ **Produces:** none
128
+ **Review Focus:** `pattern_decl`: closure_spec's own Name has no `Pat` and renames pattern names as Decl — map its call sites so behaviour is unchanged (pin with a test in `plum-core/tests/closure_spec_test.rs` first); `debug_key`: the `format!("{:?}")` key (≈322) becomes a structural `Expr` equality (ast derives PartialEq).
129
+ **Test code:** closure_spec_test binary passes plus the pinning test; hashes identical.
130
+
131
+ ### T5 forms and union_lower use visit · Complexity 4 · Est. context 45k
132
+ **Owns:** plum-core/src/forms.rs, plum-core/src/union_lower.rs
133
+ **Components:** 2
134
+ **LOC owned:** 60
135
+ **Tests:** slow
136
+ **Model:** sonnet
137
+ **Consumes:** visit::childExprs(e: &mut Expr) -> Vec<&mut Expr> (T2) · visit::stmtExprs(s: &mut Stmt) -> Vec<&mut Expr> (T2) · visit::stmtBlocks(s: &mut Stmt) -> Vec<&mut Block> (T2) · visit::substType(t: &Type, map: &HashMap<String, Type>) -> Type (T2) · visit::patternBinders(p: &CasePattern, out: &mut Vec<String>) (T2)
138
+ **Produces:** none
139
+ **Test code:** forms_* and union_lower_test binaries pass; hashes identical.
140
+ **Notes:** forms' rename uses a bool, union_lower's `rn*` renames pairs: express both through `visit::renameStmts`; `forms::mapExprsIn`/`walkStmts` stay public as thin wrappers.
141
+
142
+ ### T6 uniqueness, reuse, tail share helpers · Complexity 3 · Est. context 40k
143
+ **Owns:** plum-core/src/uniqueness.rs, plum-core/src/reuse.rs, plum-core/src/tail.rs
144
+ **Components:** 1
145
+ **LOC owned:** 50
146
+ **Tests:** slow
147
+ **Model:** sonnet
148
+ **Consumes:** visit::unparen(e: &Expr) -> &Expr (T2) · visit::patternBinders(p: &CasePattern, out: &mut Vec<String>) (T2) · visit::countVar(body: &FnBody, name: &str) -> usize (T2) · visit::childRefs(e: &Expr) -> Vec<&Expr> (T2)
149
+ **Produces:** none
150
+ **Test code:** uniqueness_test, tail_test binaries and codegen e2e_mem pass; hashes identical.
151
+ **Notes:** tail's walker tracks tail position — keep its own recursion, use visit only for child lists.
152
+
153
+ ### T7 trmc real tree queries · Complexity 3 · Est. context 35k
154
+ **Owns:** plum-core/src/trmc.rs, plum-core/tests/trmc_test.rs
155
+ **Components:** 1
156
+ **LOC owned:** 50
157
+ **Tests:** slow
158
+ **Model:** sonnet
159
+ **Consumes:** visit::anyExpr(e: &Expr, pred: &mut dyn FnMut(&Expr) -> bool) -> bool (T2)
160
+ **Produces:** none
161
+ **Review Focus:** `try_in_string`: a body containing the string literal `"Try("` (or `"Closure("`) is still rewritten; `mentions_field_not_var`: a field or method named `x` does not count as a use of variable `x`.
162
+ **Test code:** those two tests in trmc_test.rs (fail before, pass after); all trmc_test pass; hashes identical.
163
+ **Steps:** replace `bannedText(format!("{:?}"..))` (≈127, 188) and `mentions` (≈255) with `anyExpr` matches on the real variants.
164
+
165
+ ### T8 generics by scope · Complexity 5 · Est. context 50k
166
+ **Owns:** plum-core/src/generics.rs, plum-core/src/lib.rs, plum-core/src/drive.rs, plum-core/src/forms.rs, plum-core/src/trmc.rs, plum-core/src/union_lower.rs, plum-checker/src/lib.rs, plum-core/tests/generics_test.rs
167
+ **Components:** 2
168
+ **LOC owned:** 120
169
+ **Tests:** slow
170
+ **Model:** sonnet
171
+ **Produces:** generics::isGenericName(n: &str) -> bool · generics::fnGenerics(f: &Fn, owner: &[String]) -> Vec<String>
172
+ e.g. fnGenerics(`fun map(self, f: fn(T) -> B) -> List[B]` in `List[T]`, &["T"]) -> ["T","B"]
173
+ e.g. fnGenerics(`fun len(self) -> Int`, &[]) -> []
174
+ **Review Focus:** `no_spelling_checks_left`: `grep -rn "is_ascii_uppercase" plum-core/src plum-checker/src` shows only generics.rs (and unrelated casing checks, each justified in the report); `enum_fallback_gone`: checker `enumGenericParams` uses only declared generics.
175
+ **Test code:** generics_test.rs for the e.g. lines; all crate tests pass; hashes identical.
176
+ **Steps:** drive `isGen`, forms `isGenericName`, trmc `open`, checker `isGenericName` → call `generics::isGenericName`, or better, check membership in the fn/owner generic set where the set is at hand; delete the checker's single-letter-field fallback in `enumGenericParams`; leave the grammar alone.
177
+
178
+ ## Package B: hosts and tests
179
+ **Probe entry:** run=cargo build --workspace --tests
180
+
181
+ ### T10 plum-host crate · Complexity 5 · Est. context 55k
182
+ **Owns:** plum-host/Cargo.toml, plum-host/src/lib.rs, Cargo.toml, plum-cli/Cargo.toml, plum-cli/src/main.rs, plum-runtime/Cargo.toml, plum-runtime/src/main.rs, plum-game/Cargo.toml, plum-game/src/host.rs
183
+ **Components:** 2
184
+ **LOC owned:** 200
185
+ **Tests:** slow
186
+ **Model:** sonnet
187
+ **Produces:** plum_host::engine() -> wasmtime::Engine · plum_host::strBytes(store, s) -> Result<Vec<u8>> · plum_host::newStr(store, bytes) -> Result<Val> · plum_host::hostImports(store: &mut Store<()>, module: &Module, caps: Caps) -> Result<Vec<Extern>> · plum_host::Caps { fs: bool, http: bool, clock: bool, rng: bool }
188
+ **Review Focus:** `runtime_build`: `plum build` (cmdBuild ≈1120) still produces a working runtime binary with the new path dep; `game_host`: plum-game host tests pass (`cargo build --tests` in plum-game, run its binaries).
189
+ **Test code:** unit tests in plum-host for strBytes/newStr round trip; cli compile_tests/crash_test/host_stub_test pass; plum-game host_test passes.
190
+ **Notes:** the CLI copy (main.rs 284-835) is the superset; runtime keeps only what it enables via Caps. Keep the 64 MiB GC heap setting where it is used today.
191
+
192
+ ### T13 test helpers and LSP generic fix · Complexity 3 · Est. context 40k
193
+ **Owns:** plum-test-support/src/lib.rs, plum-core/tests/{drive,simplify,uniqueness,record_embed,record_init,record_init_parse,trait_parse,traits,closure_types_parse,match_body_parse,override_parse,whistle,purity,forms_list_range,parser}_test.rs, plum-checker/tests/traits_tests.rs, plum-checker/tests/trait_fields_tests.rs, plum-checker/tests/features/mutability.rs, plum-lsp/src/completion.rs
194
+ **Components:** 2
195
+ **LOC owned:** 90
196
+ **Tests:** fast
197
+ **Model:** sonnet
198
+ **Produces:** plum_test_support::parseRawSource(src: &str) -> plum_core::ast::Source (parseSourceRaw, panics on syntax errors)
199
+ **Review Focus:** `lsp_generics`: LSP `paramPlumType` (completion.rs ≈345) keeps generic args (use the generic-aware converter) — add a test there.
200
+ **Test code:** every listed test binary passes, same counts; new LSP unit test.
201
+
202
+ ## Package C: verify
203
+ **Probe entry:** run=bash scripts/wasm-hashes.sh
204
+
205
+ ### T9 verify 2a · Complexity 2 · Est. context 30k
206
+ **Owns:** docs/plans/2026-11-04-compiler-cleanup-phase2a.results.md
207
+ **Components:** 1
208
+ **LOC owned:** 40
209
+ **Tests:** slow
210
+ **Model:** sonnet
211
+ **Produces:** none
212
+ **Test code:** all workspace test binaries + plum-game + plum-runtime build pass; `plum test plum-std plum-examples` 678 passed; hashes at T1's sha == current tree.
213
+ **Steps:** results file: hash diff, test totals, Rust LOC and test counts vs phase 1 (8445080), lines removed per file in plum-core.
214
+
215
+ ## Review Focus
216
+ - Package A (compiler): deep, state — risks: walker behaviour drift (shadowing, pattern names, field-assign targets), determinism fix changing semantics, generic decision moved.
217
+ - Package B (host/tests): protocol, file — risks: host import names/signatures, Str ABI, runtime build path.
218
+
219
+ ## Self-review
220
+ Covers phase 2 rows: determinism (T1), tree walker (T2-T6), real tree queries (T4, T7), generics by scope (T8), host crate (T10), test helpers + LSP fix (T13). Front end, compile call and Paren are phase 2b.