plum

#treesitter#compiler#wasm

git clone https://git.pyrossh.dev/plum

A statically typed, imperative programming language inspired by rust, python

05b282c — Peter John 2026-11-04T17:50:21+05:30 
docs: Elm/Roc-style error reporting for the self-hosted compiler
Files changed (2) hide show
  1. docs/plans/2026-11-04-self-hosting-groundwork.review.md +29 -4
  2. docs/self-hosting.md +104 -2
docs/plans/2026-11-04-self-hosting-groundwork.review.md CHANGED
@@ -18,9 +18,15 @@ after compiler cleanup phase 2a merges, because that phase owns the CLI and host
18
18
  - **`compiler/Token.plum`, `compiler/Lexer.plum`:** tokens with line, column, "starts a line" and "space before";
19
19
  numbers (`_`, `0x`, `0b`, floats, `f`), every operator, keywords incl. `fun*`, strings with `{expr}` interpolation
20
20
  and escapes, comments kept aside.
21
+ - **`compiler/Report.plum`:** error reporting in the Elm/Roc style (design doc §3b): a report is a title plus a
22
+ document (text, code references, types, source snippets with underlined regions, notes, hints), rendered to
23
+ terminal with colours, plain text, Markdown and LSP JSON by one walker; a canonical S-expression form for tests;
24
+ did-you-mean suggestions; a table of common mistakes from other languages. The checker will use it later.
21
25
  - **`compiler/Parse.plum`:** the combinator library: cursor with save/rewind, `many`, `sepBy`, `between`, `choice`,
22
26
  `optional`, `label`, `block` (indentation rule), `pratt` (operator precedence), error collection with the furthest
27
+ failure and expected tokens, commit points, a context stack ("in the parameters of `f`"), and recovery. Two passes as in marser: a fast pass with no error tracking, and only on
23
- failure and expected tokens, and recovery.
28
+ failure a second pass with recovery and diagnostics. A parse always returns an AST (with error nodes) and the list
29
+ of diagnostics.
24
30
  - **`compiler/Ast.plum`:** the syntax tree, following the Rust AST's shapes, with a node id and a span on every node.
25
31
  - **`compiler/Parser.plum`:** Plum's grammar, built up in steps: types and literals, expressions, statements and
26
32
  blocks, declarations, patterns and `match`, string interpolation.
@@ -32,6 +38,8 @@ after compiler cleanup phase 2a merges, because that phase owns the CLI and host
32
38
  |---|---|---|---|
33
39
  | Io move | File, clock and HTTP calls become `Io` methods; Os.plum deleted; README, LSP index and tests, game fixtures updated | M | a careful helper |
34
40
  | Tokens | The token kinds and the token record | S | a fast helper |
41
+ | Reports | Report documents, the four renderers, S-expression form, did-you-mean, common-mistakes table | M | a careful helper |
42
+ | Parse error messages | One message per parser error variant (title, sentence, snippet, example, hint), with golden tests | M | a careful helper |
35
43
  | Lexer | Turns source text into tokens and positioned errors | M | a careful helper |
36
44
  | Combinator core | Cursor, combinators, errors and recovery, tested on a toy grammar | M | a careful helper |
37
45
  | Pratt + block | Operator precedence and the indentation rule | M | a careful helper |
@@ -61,14 +69,19 @@ flowchart TB
61
69
  B[Tokens]
62
70
  C[Combinator core]
63
71
  D[AST]
72
+ K[Reports]
64
73
  end
65
74
  B --> E[Lexer]
75
+ K --> E
76
+ K --> C
66
77
  C --> F[Pratt + block]
67
78
  E --> G[Parser: expressions]
68
79
  F --> G
69
80
  D --> G
70
81
  G --> H[Parser: statements]
71
82
  H --> I[Parser: declarations]
83
+ I --> M[Parse error messages]
84
+ K --> M
72
85
  I --> J[Whole-repo check]
73
86
  A --> J
74
87
  ```
@@ -82,6 +95,11 @@ The parser parts run one after another: each builds on the rules of the one befo
82
95
  closure bodies inside call brackets without parser feedback).
83
96
  - Os removal: delete `Os` outright (default) | keep `Os` as forwarding functions for a while.
84
97
  - HTTP method names: `io.httpGet`, `io.httpPost`, ... (default) | `io.fetch(method, url, ..)` only.
98
+ - Error passes: two passes, recovery only after a failed fast pass (default, as marser) | always track errors (simpler,
99
+ slower on correct files).
100
+ - Message style: Elm/Roc (first person, capitals title, snippet, expectation, hint, example) (default) | terse
101
+ rustc style.
102
+ - Error codes shown to users: none, titles only like Elm and Roc (default) | codes like `E0102` as well.
85
103
  - String forms: `"..."` and `"""..."""` (default; the only forms used in the repo) | also `'...'` and backticks,
86
104
  which the tree-sitter scanner accepts.
87
105
 
@@ -94,6 +112,10 @@ The parser parts run one after another: each builds on the rules of the one befo
94
112
  string, README and new `compiler/` files. New host imports and the AST dump wait for 2a.
95
113
  - **Deep recursion.** Deeply nested input could overflow the wasm stack; the Pratt loop keeps operator chains flat,
96
114
  and a nesting test covers 200 levels.
115
+ - **No Rust changes.** The Rust compiler keeps today's errors (user decision); only the Plum compiler gets the new
116
+ reporting.
117
+ - **Message quality is judged by reading.** Golden files make every message visible in review; each parser error
118
+ variant needs one written message, so the count of variants is kept small and grows with real mistakes.
97
119
  - **Speed.** The whole-repo test must stay quick in `plum test`; if it is slow it is moved to a separate, slow test.
98
120
  - **The `Io` primitive plan** (another session, runs last) builds on this `Io` move; nothing here blocks it.
99
121
 
@@ -108,10 +130,13 @@ The parser parts run one after another: each builds on the rules of the one befo
108
130
  LSP tests and the plum-game host tests pass.
109
131
  - No `Os.` or `std/Os` left in the repo outside old plan files.
110
132
  - The whole-repo test lexes and parses every `.plum` file with zero errors.
111
- - Broken input gives positioned errors, e.g. `fun f(a: Int` reports "expected `)`" at the end of the line, and a
133
+ - Broken input gives the reports shown in design doc §3b, e.g. `fun f(a: Int -> Int =` gives UNFINISHED PARAMETER
134
+ LIST with `->` underlined, the `(` that opened the list, and a correct example; a second error later in the same
112
- second error later in the file is still reported.
135
+ file is still reported; a correct file produces no reports and no error nodes.
136
+ - A misspelt keyword or name gets one "did you mean" suggestion; `for x in xs` gets the Plum form as a tip.
137
+ - Golden tests: a set of broken files under `compiler/testdata/` with their expected rendered output.
113
138
  - A game's wasm grows by at most 1% from the Io move.
114
139
 
115
140
  ## Rough time and cost
116
- Rough, until the machine plan is linted: 10 parts, about 4-6 hours of helper time, most of it the parser steps,
141
+ Rough, until the machine plan is linted: 12 parts, about 6-8 hours of helper time, most of it the parser steps,
117
142
  which run one after another.
docs/self-hosting.md CHANGED
@@ -64,8 +64,110 @@ a nested block (a closure body) turns it on again. This is the Haskell / F# appr
64
64
  nothing extra.
65
65
  - Errors as in chumsky: the furthest position any alternative reached plus the set of tokens expected there;
66
66
  `label("expression")` replaces a long expected-set with one word; recovery with `recoverUntil(syncTokens)` and
67
- skipping balanced brackets, so one error does not hide the next. Diagnostics are data (`Diag(span, message)`)
67
+ skipping balanced brackets, so one error does not hide the next. Problems are data (a `Report`, see §3b)
68
68
  collected in the parser state, never a crash.
69
+ - From [marser](https://docs.rs/marser/latest/marser/) (a Rust PEG combinator library built for "useful errors,
70
+ error recovery and good performance"):
71
+ - **Two passes.** Parse first with recovery and expected-set tracking switched off; only if that fails, parse
72
+ again with them on. Correct files (nearly all of them) pay nothing for error handling.
73
+ - **Errors returned with a result.** A parse always returns an AST (with error nodes where it recovered) plus the
74
+ list of diagnostics, so the LSP and checker can keep working on broken files.
75
+ - **Labels** on rules, and snippet rendering (see §3b).
76
+ - **Memoising** a rule at a position, only where backtracking proves slow.
77
+ - **A parse trace** (which rule tried what, where) as a debugging aid, later.
78
+
79
+ marser's grammars are parser values built with `capture!`/`bind!` macros. Plum has no macros, and parser values
80
+ have the closure cost described above, so we keep winnow-style rule functions and take marser's error design.
81
+
82
+ ## 3b. Error reporting: parse and compile errors
83
+
84
+ Goal: error messages as good as Elm's and Roc's. Only the Plum compiler gets this: the Rust compiler keeps its
85
+ current errors (user, 2026-11-04: no Rust changes, we are self-hosting).
86
+
87
+ **Today, for comparison.** Parsing stops at the first error and says only `unexpected '='` (no "expected `)`").
88
+ Type errors carry a line and column but no snippet, the column often points at the start of the statement instead
89
+ of the bad operand (`a + "x"` reports column 3, not `"x"`), and messages repeat themselves
90
+ (`binary op: type mismatch: expected Int, found Str`).
91
+
92
+ ### What we take from Elm
93
+ (from `elm/compiler`, `Reporting/Error/Syntax.hs` and `Reporting/Error/Type.hs`)
94
+ - **A report is a small story,** written in the first person: a title in capitals with the file name, one sentence
95
+ saying where the compiler got stuck, the code with the problem underlined, then what it expected, then a hint.
96
+ ("I got to the end of the line without seeing the closing double quote:")
97
+ - **One error variant per construct, plus a context stack.** The parser knows it is "in the parameter list of `f`",
98
+ "in a record field", "in a `match` arm", and each construct has its own failure variants (Elm: `RecordOpen`,
99
+ `RecordEnd`, `LetIndentIn`, ...). That is what lets a message say *what* it was parsing, not just which token was
100
+ unexpected.
101
+ - **A correct example of the construct** in the report ("parameters look like `fun f(a: Int, b: Str) -> Int =`").
102
+ - **Type mismatches say where the expectation came from.** "The 2nd argument to `add` is not what I expect", "The 2nd
103
+ branch of this `if` does not match the 1st", an annotation, an operator's left or right side. Then "It is: X /
104
+ But `add` needs its 2nd argument to be: Y", with only the differing parts of the two types highlighted.
105
+ - **Hints from the kind of mismatch,** not from the message text: Int vs Float ("no implicit conversion, use
106
+ `Float(x)`"), Str vs Int, a non-Bool condition (no truthiness), too many or too few arguments (how many), missing
107
+ record fields (which), a field typo (the closest name).
108
+ - **Dedicated reports for common slips:** tabs, a reserved word used as a name, too much indentation at the top level,
109
+ an unclosed string.
110
+
111
+ ### What we take from Roc
112
+ (from `roc-lang/roc`, `src/reporting/`)
113
+ - **A report is a document, not a string.** Every stage builds `Report(title, severity, doc)` where `doc` is a tree of
114
+ elements (text, a code reference, a type, a source snippet with underlined regions, a hint) with semantic
115
+ annotations. One walker renders it to several targets: terminal (with colours), plain text, Markdown, and the LSP.
116
+ Stages never format output themselves.
117
+ - **Tests compare a canonical S-expression of the report,** separate from layout, plus golden rendered output for a few
118
+ fixtures. Rewording a message then changes only the golden files, not every test.
119
+ - **Did-you-mean:** optimal string alignment distance (insert, delete, substitute, swap neighbours); a name qualifies
120
+ at a distance of at most a third of its length (at least 1); one suggestion only, ties to the earlier name.
121
+ - **A table of common mistakes** for tokens from other languages, with a tip each. For Plum: `(` used for grouping
122
+ (Plum uses `{ }`), `for x in xs` (Plum: `for x := range xs`), `..` ranges, `let`/`var`, `elif`, `===`, `??` vs
123
+ `?:`, `Option`/`Result` (Plum: `T?`, `E!T`), `null` checks with `==` on optionals.
124
+ - **Shared snippet layout:** tabs expanded, the snippet dedented, carets kept aligned for primary and secondary
125
+ regions.
126
+
127
+ ### Parser techniques that make this possible
128
+ - **Commit points (cut).** Once a construct is certain (after `fun name(`), a failure inside it is reported there
129
+ instead of backtracking to try other alternatives, which is how "expected `)`" beats "unexpected `fun`".
130
+ - **Context stack:** `p.inContext(FunParams(name), rule)`; reports read the innermost contexts.
131
+ - **Typed errors at important places,** generic expected-token sets elsewhere, always named by `label`.
132
+ - **Two passes** (marser): fast pass with none of this switched on; only on failure, a second pass with contexts,
133
+ expected sets and recovery.
134
+ - **Recovery and no cascades:** error nodes in the AST; the checker reports nothing about an error node or anything
135
+ whose type depends on one; at most one parse error per statement; output sorted by position and capped (default
136
+ 20 per file, then "N more problems").
137
+
138
+ ### Example
139
+
140
+ ```
141
+ -- UNFINISHED PARAMETER LIST ------------------------------------ game.plum
142
+
143
+ I was partway through the parameters of `f` when I ran into `->`:
144
+
145
+ 4| fun f(a: Int -> Int =
146
+ ^^
147
+ I was expecting a `,` or a `)` here. The list started here:
148
+
149
+ 4| fun f(a: Int -> Int =
150
+ ^
151
+ Note: parameters look like this:
152
+
153
+ fun f(a: Int, b: Str) -> Int =
154
+ ```
155
+
156
+ ```
157
+ -- TYPE MISMATCH ------------------------------------------------ math.plum
158
+
159
+ The right side of `+` is not what I expect:
160
+
161
+ 6| a + "x"
162
+ ^^^
163
+ It is a `Str`, but `+` needs its right side to be an `Int` because the left side is.
164
+
165
+ Hint: Plum never converts a `Str` to a number by itself. To read a number from text,
166
+ use `Int.parse("x")`.
167
+ ```
168
+
169
+ Possible later (from Roc): let `plum run`/`plum test` run a program that still has errors, turning each error into a
170
+ crash at the place it occurs, so unrelated tests keep running.
69
171
 
70
172
  ## 4. Patterns for the compiler in Plum
71
173
 
@@ -90,7 +192,7 @@ redesign.
90
192
  file in the repo.
91
193
  2. Same output as Rust: an S-expression dump of the AST from both parsers, compared over every file. Needs a dump
92
194
  command in `plum-cli`, so it waits for cleanup phase 2a (which owns `plum-cli/src/main.rs`).
93
- 3. Checker in Plum: compare the inferred type of every node and the error list against Rust.
195
+ 3. Checker in Plum: compare the inferred type of every node, and that both compilers reject the same files at the same positions (the messages differ: only Plum's are the new reports).
94
196
  4. Codegen + wasm encoder, no optimisations: the existing `plum test` suite (687 tests) must pass on the Plum
95
197
  compiler's output. Then port fusion, TRMC, reuse.
96
198
  5. Bootstrap: Rust stage 0 builds stage 1; stage 1 builds stage 2; stage 2 builds stage 3; stage 2 == stage 3 byte