plum

#treesitter#compiler#wasm

git clone https://git.pyrossh.dev/plum

A statically typed, imperative programming language inspired by rust, python

c576196 — Peter John 2026-11-03T17:45:53+05:30 
docs(plans): tail calls round 2 plans; post-tail-call benchmark rows
Files changed (3) hide show
  1. docs/plans/2026-11-03-own-record.benchmark.md +11 -0
  2. docs/plans/2026-11-03-tail-calls-2.md +118 -0
  3. docs/plans/2026-11-03-tail-calls-2.review.md +66 -0
docs/plans/2026-11-03-own-record.benchmark.md CHANGED
@@ -23,6 +23,17 @@ The recursive version is faster than the List-based state/view (avg 1.91 vs 2.07
23
23
  Against the mutable scene: avg +10%, p95 +21%. The remaining gap is the per-frame tree rebuild in draw (+0.2 ms).
24
24
  Tail calls and, later, a supercompiler (fusing `view` into `draw`, so no tree is built) are the next levers.
25
25
 
26
+ After tail calls (self loops, return_call, TRMC, 4 MiB stack) and the rebase onto master 59d923d, quiet machine on charge, same session:
27
+
28
+ | fixture | avg ms | p50 | p95 | p99 | update ms | draw ms |
29
+ |---|---|---|---|---|---|---|
30
+ | stress_scene (mutable) | 1.699 / 1.683 | 1.70 | 1.80 | 1.90 | 0.038 | 1.27 |
31
+ | stress_pure_scene (own) | 1.895 | 1.80 | 2.30 | 2.50 | 0.320 | 1.22 |
32
+ | stress_state_view (recursive) | 1.927 | 1.90 | 2.30 | 2.50 | 0.086 | 1.47 |
33
+ | stress_state_view_deep (5,000-deep list) | 1.906 | 1.90 | 2.20 | 2.30 | 0.064 | 1.49 |
34
+
35
+ Own scene and recursive state/view both land at +12% to +14% average against the mutable scene. The 5,000-deep single-list variant costs nothing extra.
36
+
26
37
  Bar (review plan): average within 10% and p95 within 15% of the mutable scene, same session.
27
38
 
28
39
  - Phase B: avg +11% to +21%, p95 +21% to +42%: **missed**. Update time is unchanged from phase A (about 0.32 ms).
docs/plans/2026-11-03-tail-calls-2.md ADDED
@@ -0,0 +1,118 @@
1
+ # Tail calls, round 2, and GC speed
2
+
3
+ **Review plan:** docs/plans/2026-11-03-tail-calls-2.review.md
4
+ **Goal:** tail calls in closure bodies and recursive local closures, error-union tail calls, worker/wrapper for converted results, single-pass tail recursion modulo cons, and a measured GC investigation.
5
+ **Architecture:** round 1 marks `FnCall.tail` in plum-core/src/tail.rs (run in plum-checker typedSource); plum-core/src/trmc.rs rewrites constructor-wrapped self calls; plum-wasm-codegen/src/lib.rs turns SelfCall into loop+br (TailDepth counter) and Other/Closure into return_call / return_call_indirect. This round extends codegen and trmc; GC work touches only host config and a report.
6
+ **Tech stack:** Rust (plum-core, plum-checker, plum-wasm-codegen, wasmtime 47), Plum.
7
+ **Spec:** docs/plans/2026-11-03-tail-calls-2.review.md
8
+ **Detail:** contracts
9
+ **Target-min:** 45
10
+
11
+ ## Global constraints
12
+ - Work only in /Users/pyrossh/Code/plum-pure-scene-values (branch pure-scene-values, base 184c054). Never touch /Users/pyrossh/Code/plum.
13
+ - Rust tests: `cargo test --manifest-path <crate>/Cargo.toml --test <name>` (raw `cargo test -p` may be hook-blocked). Plum: `touch plum-cli/src/main.rs && cargo build --quiet -p plum-cli && ./target/debug/plum test <file>`; sweep `./target/debug/plum test plum-std plum-examples` (607 green). `touch` edited .rs before building.
14
+ - Depth tests: allocation-free loops at 1,000,000; list-building at 200,000 (1M list build ~35 s, GC).
15
+ - Known pre-existing failure: plum-checker `generic_funstar_wired` reads an untracked repro file; ignore it.
16
+ - Codegen tasks run one after another (one shared file); keep tail_call_e2e_test, tail_calls_e2e_test, loop_break_e2e_test, own_inplace_e2e_test green.
17
+
18
+ ```mermaid
19
+ flowchart LR
20
+ TP["tail marks"] --> CL["closure return_call + recursive closures"]
21
+ CL --> EU["error-union tail calls"] --> WW["worker/wrapper"] --> SP["single-pass TRMC"]
22
+ GC["GC probes + report"]
23
+ ```
24
+
25
+ File structure: each task's Owns. Spec coverage: What you will get (T1-T5), What will be built (T1-T6), How it fits together (diagram), Decisions you can change (defaults in tasks), Risks (R1 T1, R2 T3, R3 T4, R4 T5), Out of scope (not built), How we will know it works (T1-T6), Rough time and cost (schedule).
26
+
27
+ ## Task graph
28
+ ```mermaid
29
+ flowchart LR
30
+ T1[closure tails]
31
+ T2[error-union tails]
32
+ T3[worker/wrapper]
33
+ T4[single-pass trmc]
34
+ T5[GC investigation]
35
+ T6[depth tests]
36
+ %% taskgraph
37
+ T1 --> T2
38
+ T2 --> T3
39
+ T3 --> T4
40
+ T4 --> T6
41
+ ```
42
+ Ruling: sequential [codegen] - T1-T4 all edit plum-wasm-codegen/src/lib.rs - cost: longer critical path.
43
+
44
+ ## Codegen
45
+
46
+ ### T1 closure tail calls and recursive local closures · Complexity 5 · Est. context 50k
47
+ **Owns:** plum-wasm-codegen/src/lib.rs, plum-checker/src/lib.rs, plum-core/src/tail.rs, plum-wasm-codegen/tests/closure_tail_e2e_test.rs
48
+ **Components:** 2
49
+ **LOC owned:** 160
50
+ **Tests:** fast
51
+ **Model:** sonnet
52
+ **Consumes:** none
53
+ **Produces:** none
54
+ **Review Focus:** state R1: a `:=`-bound closure that names itself resolves to itself (env self slot patched before first call); a self call inside it is a tail call. Closure-body tail calls emit return_call / return_call_indirect.
55
+ **Test code:** closure_tail_e2e_test.rs: `closure_body_tail_call_million` (`dec := |n| if n == 0 then 0 else countDown(n - 1)` style through a top-level fn), `recursive_local_closure_million` (`go := |n, acc| if n == 0 then acc else go(n - 1, acc + n)`, go(1_000_000, 0) == 500000500000), `recursive_closure_captures_outer` (captured value visible every iteration), `closure_not_recursive_unchanged`, `wasm_closure_has_return_call` (wasmparser). tail.rs: closure-body self reference marked SelfCall relative to the closure.
56
+
57
+ ### T2 error-union tail calls · Complexity 3 · Est. context 40k
58
+ **Owns:** plum-wasm-codegen/src/lib.rs, plum-wasm-codegen/tests/error_tail_e2e_test.rs
59
+ **Components:** 1
60
+ **LOC owned:** 90
61
+ **Tests:** fast
62
+ **Model:** sonnet
63
+ **Consumes:** none
64
+ **Produces:** none
65
+ **Review Focus:** state: return_call / self loop only when caller and callee result types are the identical `T!E`; `try`/`?` and error-set remapping stay normal calls.
66
+ **Test code:** error_tail_e2e_test.rs: `error_union_self_loop_million` (counts down, errors at a sentinel never reached), `error_union_self_loop_returns_error` (error from deep inside surfaces unchanged), `error_union_mutual_million`, `try_in_tail_not_tail_call` (wasmparser: no return_call), `different_error_sets_normal_call`.
67
+
68
+ ### T3 worker/wrapper for converted results · Complexity 5 · Est. context 50k
69
+ **Owns:** plum-wasm-codegen/src/lib.rs, plum-wasm-codegen/tests/convert_tail_e2e_test.rs
70
+ **Components:** 1
71
+ **LOC owned:** 150
72
+ **Tests:** fast
73
+ **Model:** sonnet
74
+ **Consumes:** none
75
+ **Produces:** none
76
+ **Review Focus:** state R2: a worker per (callee, conversion) generated on demand and deduplicated; self loops carry the raw value and convert once at exit; results identical to before.
77
+ **Test code:** convert_tail_e2e_test.rs: `int_to_number_tail_million`, `t_to_optional_tail_million`, `self_loop_converts_once` (value equal; wasmparser: one conversion site outside the loop), `worker_dedup` (two callers, one worker), `no_worker_without_conversion`.
78
+
79
+ ### T4 single-pass tail recursion modulo cons · Complexity 5 · Est. context 50k
80
+ **Owns:** plum-core/src/trmc.rs, plum-core/tests/trmc_test.rs, plum-wasm-codegen/src/lib.rs
81
+ **Components:** 2
82
+ **LOC owned:** 170
83
+ **Tests:** fast
84
+ **Model:** sonnet
85
+ **Consumes:** none
86
+ **Produces:** none
87
+ **Review Focus:** state R3: the rewrite allocates each cell once with a placeholder recursive field and fills it next iteration through a codegen-only field write (an AST marker only the rewrite can produce; user code still cannot assign inline-variant fields); no unfilled cell escapes.
88
+ **Test code:** trmc_test.rs keeps all existing tests; add `map_single_pass_one_alloc_per_cell` (wasmparser: one struct.new in the loop), `user_code_cannot_assign_variant_field` (checker error unchanged), `every_cell_filled_200k` (walk the result, no placeholder), `fromTo_single_pass_200k`.
89
+
90
+ ## GC
91
+
92
+ ### T5 GC investigation · Complexity 3 · Est. context 40k
93
+ **Owns:** docs/plans/2026-11-03-gc-report.md, plum-cli/src/main.rs, plum-runtime/src/main.rs, plum-game/src/host.rs
94
+ **Components:** 1
95
+ **LOC owned:** 60
96
+ **Tests:** fast
97
+ **Model:** sonnet
98
+ **Consumes:** none
99
+ **Produces:** none
100
+ **Review Focus:** numeric R4: timings for building and summing 100k / 1M-cell `Link[Int]` lists under wasmtime 47 config variants (collector choice, GC heap size/growth, any nursery/threshold knob that exists in 47), and in Chrome (headless via plum-game/web-test runner or a tiny page); change host config only where a variant is a clear win (≥2x) with no test regressions.
101
+ **Test code:** report with a table (variant, 100k ms, 1M ms); probes under /private/tmp/claude-501/ only; the full Plum sweep stays green after any config change.
102
+
103
+ ## Verification
104
+
105
+ ### T6 depth tests round 2 · Complexity 2 · Est. context 30k
106
+ **Owns:** plum-examples/tail_calls.plum, plum-examples/functional.plum
107
+ **Components:** 1
108
+ **LOC owned:** 80
109
+ **Tests:** fast
110
+ **Model:** sonnet
111
+ **Consumes:** none
112
+ **Produces:** none
113
+ **Review Focus:** deep: one test per new case; functional.plum header no longer says recursion is not tail call optimised.
114
+ **Test code:** add to tail_calls.plum: `recursive_closure_million`, `closure_body_tail_million`, `error_union_loop_million`, `number_conversion_tail_million`, `map_single_pass_200k`; fix functional.plum's stale header comment. Both files green; full sweep green.
115
+
116
+ ## Review Focus
117
+ - compiler (library): deep, state. Risks: closure self-binding, wrong tail classification with error unions, worker duplication, unfilled TRMC cells.
118
+ **Probe entry:** run=./target/debug/plum test
docs/plans/2026-11-03-tail-calls-2.review.md ADDED
@@ -0,0 +1,66 @@
1
+ # Tail calls, round 2, and GC speed: review plan
2
+
3
+ **Builds on:** branch `pure-scene-values`, after the rebase onto master 59d923d. It has the tail-position pass, self tail calls as loops, `return_call`, two-pass tail recursion modulo cons, and the 4 MiB stack.
4
+
5
+ ## Summary
6
+ Round 1 left four kinds of tail call as normal calls, and modulo-cons lists allocate every cell twice. This plan closes the three gaps worth closing, makes modulo cons single-pass, and investigates why building a 1,000,000-cell list takes about 35 s.
7
+
8
+ ## What you will get
9
+ - **Tail calls inside closure bodies.** A call in tail position of a closure body becomes `return_call` / `return_call_indirect` from the closure's wasm function. Example: `dec := |n| if n == 0 then 0 else dec2(n - 1)`. The tail-position pass already marks these calls (relative to the closure); codegen starts using the marks.
10
+ - **Recursive local closures.** A closure bound with `:=` may call itself by name: `go := |n, acc| if n == 0 then acc else go(n - 1, acc + n)`. The name resolves to the closure being defined (a recursive binding), and the call is a tail call.
11
+ - **Error-union functions.** When the caller and callee return the same `T!E` type (same error set), `return g(x)` is a `return_call`, and a self call is a loop. `try`/`?` propagation and calls that need error-set remapping stay normal calls.
12
+ - **Results that need converting.** Example: a callee returning `Int` into a `Number` result, or `T` into `T?`. A worker/wrapper split handles it: the compiler emits a variant of the callee that returns the converted type, then tail-calls that variant. For self calls, the loop carries the raw value and converts once on exit.
13
+ - **Single-pass modulo cons.** Codegen may write the recursive field of a freshly allocated inline-variant cell with `struct.set`, but only inside code made by the TRMC rewrite. User code still can't assign those fields. `map`, `fromTo` and `append` then allocate each cell once.
14
+ - **GC investigation, measured.** Find out why a 1,000,000-cell list takes about 35 s under wasmtime 47. Check the collector choice, heap and nursery settings, and whether our structs or arrays defeat the fast path. Compare against Chrome on the same Plum program. Deliverables: a short report with numbers, and any config change that is a clear win. Any compiler-side fix found gets planned separately.
15
+
16
+ ## What will be built
17
+ | Part | What it does | Size |
18
+ |---|---|---|
19
+ | Closure tail calls | Codegen emits return_call / return_call_indirect in closure functions where marks say tail | S |
20
+ | Recursive local closures | Checker and codegen: a `:=`-bound closure can name itself; env holds a self ref | M |
21
+ | Error-union tail calls | Allow return_call / loops when result error unions are identical | S |
22
+ | Worker/wrapper conversions | Generate a converted-result worker per (callee, conversion) on demand; self-loop converts on exit | M |
23
+ | Single-pass TRMC | trmc rewrite emits a placeholder cell and a codegen-only field write; drop the second pass | M |
24
+ | GC investigation | Probes under wasmtime config variants and Chrome; report plus config change | S–M |
25
+ | Tests | Depth tests per new case at 1,000,000 (allocation-free) or 200,000 (lists), and every suite green | S |
26
+
27
+ ## How it fits together
28
+ ```mermaid
29
+ flowchart LR
30
+ TP["tail marks"] --> CL["closure fns: return_call"]
31
+ TP --> EU["error union: same type -> return_call / loop"]
32
+ TP --> WW["conversion: worker/wrapper"]
33
+ TR["trmc rewrite"] --> SP["single-pass: codegen-only struct.set"]
34
+ GC["GC probes"] --> RP["report + config"]
35
+ ```
36
+
37
+ ## Decisions you can change
38
+ - **Recursive closures.** Default: automatic. A `:=` closure that mentions its own name is recursive. Alternative: explicit `rec go := |..| ..`.
39
+ - **Worker/wrapper.** Default: generate workers only for conversions that actually occur in tail position (bounded code growth). Alternative: skip this item; convert-at-exit for self loops covers the common case.
40
+ - **Mutating hidden cells.** Default: only code made by the TRMC rewrite, with the cell never visible before it is filled. Alternative: keep two-pass TRMC.
41
+ - **GC scope.** Default: investigate and change config only. Alternative: also prototype a compiler-side fix (for example fewer boxed values) in this plan.
42
+
43
+ ## Risks
44
+ - **R1: recursive closure env.** The closure must see itself, not a stale copy. Codegen creates the env first and patches the self slot before the first call.
45
+ - **R2: worker explosion.** Each (callee, conversion) pair adds a function. They're deduplicated, and generated only on demand.
46
+ - **R3: single-pass TRMC soundness.** A cell must never escape before its recursive field is written. The rewrite never exposes the placeholder, and a test checks every cell is filled.
47
+ - **R4: GC finding is engine-side.** If wasmtime's collector is simply slow for this pattern, the outcome is a report and maybe a config flag, not a fix.
48
+
49
+ ## Out of scope
50
+ - Imported functions in tail position. No gain: imports go through boundary wrappers and don't recurse back into Plum.
51
+ - The supercompiler. It's still the next plan.
52
+ - Mutual recursion between local closures beyond self-recursion.
53
+
54
+ ## How we will know it works
55
+ - New depth tests pass:
56
+ - a closure tail call (1,000,000);
57
+ - a recursive local closure (1,000,000);
58
+ - an error-union self loop and an error-union mutual call (1,000,000);
59
+ - an `Int`→`Number` tail conversion (1,000,000);
60
+ - single-pass `map` at 200,000, with allocation counted: one `struct.new` per cell.
61
+ - The tail-mark test asserts the newly covered cases.
62
+ - Every existing suite stays green: 1,819 or more tests, the host parity test and the browser checks.
63
+ - The GC report has timings for 100k / 1M list builds under each wasmtime config tried, and in Chrome.
64
+
65
+ ## Rough time and cost
66
+ About 6 parts. The closure and error-union items are quick; worker/wrapper and single-pass TRMC are medium; the GC investigation runs in parallel. Rough agent time is 50–80 minutes.