Chapter 29: Tech Debt Cleanuplinear-chapter29linear

Co-authored-by: Hsiang-Ying Fu <66062782+fuhsnn@users.noreply.github.com>
Co-authored-by: fuhsnn <66062782+fuhsnn@users.noreply.github.com>
Co-authored-by: Symious <14933944+symious@users.noreply.github.com>
Co-authored-by: Jun Kawasaki <jun784@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

4 files changed, 501 insertions, 0 deletions

diff --git a/docs/chapter29/29-grammar.ja.md b/docs/chapter29/29-grammar.ja.md
new file mode 100644
index 00000000..48251d9b
--- /dev/null
+++ b/docs/chapter29/29-grammar.ja.md
@@ -0,0 +1,175 @@
+# 第29章の文法
+
+[English](29-grammar.md) | 日本語
+
+```antlrv4
+grammar SimpleLanguage;
+
+// Name lookup, assignable operands, type compatibility, and access permissions
+// are checked by the compiler in addition to these syntax rules.
+// A file is a compilation unit; its path supplies its namespace.
+// Qualified names resolve nested types and other units without an import keyword.
+program : statement* EOF ;
+
+block : '{' statement* '}' ;
+
+statement
+    : block
+    | returnStatement
+    | ifStatement
+    | whileStatement
+    | forStatement
+    | breakStatement
+    | continueStatement
+    | structDeclaration
+    | declaration ';'
+    | assignment ';'
+    | ';'
+    ;
+
+returnStatement   : 'return' assignment ';' ;
+ifStatement       : 'if' '(' assignment ')' statement ('else' statement)? ;
+whileStatement    : 'while' '(' assignment ')' statement ;
+breakStatement    : 'break' ';' ;
+continueStatement : 'continue' ';' ;
+
+forStatement
+    : 'for' '(' (declaration | assignment)? ';' assignment? ';' assignment? ')' statement
+    ;
+
+structDeclaration : 'struct' IDENTIFIER '{' structMember* '}' ';' ;
+structMember : constructorDeclaration | methodDeclaration | statement ;
+
+// The constructor name must match the enclosing struct's name.
+constructorDeclaration : 'new' IDENTIFIER '=' functionExpression ';' ;
+
+// A method is an ordinary fixed function binding in a struct definition.
+// The compiler supplies self; it is not written in parameterList.
+// Function types and expressions both receive their implicit receiver here.
+methodDeclaration : 'val' bindingModifier? IDENTIFIER '=' functionExpression ';' ;
+
+declaration : declarationType binding (',' binding)* ;
+declarationType : type | 'var' | 'val' ;
+binding : bindingModifier? IDENTIFIER ('=' initializer)? ;
+bindingModifier : '!' | '~' ;
+// A declaration initializer consisting of the string "C" declares an extern
+// symbol: a function, or explicit numeric native storage. Ordinary strings
+// elsewhere are array-valued expressions.
+initializer : assignment ;
+
+// The prefix qualifies the named struct reference, not an enclosing array.
+// Each [] / [~] suffix qualifies its own array layer. ? requires a reference.
+type
+    : functionType
+    | primitiveType typeSuffix*
+    | accessModifier? typeName typeSuffix*
+    ;
+accessModifier : '!' | '~' ;
+typeSuffix : '?' | '[]' | '[~]' ;
+typeName : IDENTIFIER ('.' IDENTIFIER)* ;
+
+primitiveType
+    : 'int' | 'i8' | 'i16' | 'i32' | 'i64'
+    | 'u1' | 'u8' | 'u16' | 'u32' | 'byte' | 'bool'
+    | 'flt' | 'f32' | 'f64'
+    ;
+
+assignment : expression (assignmentOperator assignment)? ;
+assignmentOperator
+    : '=' | '+=' | '-=' | '*=' | '/=' | '&=' | '|=' | '^='
+    | '<<=' | '>>=' | '>>>='
+    ;
+
+expression : logicalExpression ('?' assignment (':' assignment)?)? ;
+
+// Both logical operators short-circuit and associate to the right here.
+logicalExpression : bitwiseExpression (('&&' | '||') logicalExpression)? ;
+
+bitwiseExpression
+    : equalityExpression (('&' | '|' | '^') equalityExpression)*
+    ;
+
+equalityExpression : comparisonExpression (('==' | '!=') comparisonExpression)* ;
+
+// Chained comparisons test adjacent operands and short-circuit. A chain must
+// keep its direction; equality has lower precedence and does not join a chain.
+comparisonExpression
+    : shiftExpression
+      ( (('<' | '<=') shiftExpression)+ | (('>' | '>=') shiftExpression)+ )?
+    ;
+
+shiftExpression
+    : additiveExpression (('<<' | '>>' | '>>>') additiveExpression)*
+    ;
+
+additiveExpression
+    : multiplicativeExpression (('+' | '-') multiplicativeExpression)*
+    ;
+
+multiplicativeExpression
+    : unaryExpression (('*' | '/') unaryExpression)*
+    ;
+
+unaryExpression
+    : ('-' | '!') unaryExpression
+    | updateOperator IDENTIFIER
+    | postfixExpression
+    ;
+
+// Prefix updates require a local name; postfix updates also allow fields
+// and array elements. All updates require an assignable operand.
+updateOperator : '++' | '--' ;
+postfixExpression : primaryExpression postfixSuffix* updateOperator? ;
+postfixSuffix : '.' IDENTIFIER | '[' assignment ']' | '#' | callSuffix | methodCallSuffix ;
+
+// For a fixed function-valued member, object.name(args) passes object as self.
+// This overlaps ordinary field selection followed by a call syntactically;
+// name/type resolution determines which call form applies.
+methodCallSuffix : '.' IDENTIFIER callSuffix ;
+
+primaryExpression
+    : INTEGER_LITERAL
+    | FLOAT_LITERAL
+    | STRING_LITERAL
+    | CHARACTER_LITERAL
+    | 'true'
+    | 'false'
+    | 'null'
+    | IDENTIFIER
+    | '(' assignment ')'
+    | newExpression
+    | functionExpression
+    ;
+
+// A function is an expression bound with an ordinary declaration.
+// Its last statement supplies the implicit result; return can exit earlier.
+functionExpression : '{' parameterList? '->' statement* '}' ;
+parameterList : (parameter ','?)+ ;
+parameter : type bindingModifier? IDENTIFIER ;
+
+// Function types have unnamed, space-separated argument types.
+// {int} is the zero-argument type; its function expression still uses ->.
+functionType
+    : '{' type+ '->' type '}' '?'?
+    | '{' type '}' '?'?
+    ;
+
+callSuffix : '(' argumentList? ')' ;
+argumentList : assignment (',' assignment)* ','? ;
+
+// Allocation invokes the declared constructor; inline initializer blocks from
+// earlier chapters are replaced by constructor arguments.
+newExpression : 'new' type ('[' assignment ']' | callSuffix)? ;
+
+INTEGER_LITERAL : '0' | [1-9] DIGIT* ;
+FLOAT_LITERAL : DIGIT+ ('.' DIGIT* ('e' DIGIT+)? | 'e' '-'? DIGIT+) ;
+STRING_LITERAL : '"' (ESCAPE | ~["\\])* '"' ;
+CHARACTER_LITERAL : '\'' (ESCAPE | ~['\\]) '\'' ;
+fragment ESCAPE : '\\' [0ntr\\'"] ;
+IDENTIFIER : NON_DIGIT (NON_DIGIT | DIGIT)* ;
+fragment NON_DIGIT : [a-zA-Z_] ;
+fragment DIGIT : [0-9] ;
+WHITESPACE : [ \t\r\n]+ -> skip ;
+LINE_COMMENT : '//' ~[\r\n]* -> skip ;
+BLOCK_COMMENT : '/*' .*? '*/' -> skip ;
+```
diff --git a/docs/chapter29/29-grammar.md b/docs/chapter29/29-grammar.md
new file mode 100644
index 00000000..4261715d
--- /dev/null
+++ b/docs/chapter29/29-grammar.md
@@ -0,0 +1,233 @@
+# Grammar for Chapter 29
+
+English | [日本語](29-grammar.ja.md)
+
+The handwritten recursive-descent implementation lives in
+[`com.seaofnodes.simple.parser`](../../src/main/java/com/seaofnodes/simple/parser/).
+The production-to-code map below the grammar describes where syntax ends and
+name resolution or graph construction begins. This grammar documents the
+language; it is not input to a parser generator.
+
+```antlrv4
+grammar SimpleLanguage;
+
+// Name lookup, assignable operands, type compatibility, and access permissions
+// are checked by the compiler in addition to these syntax rules.
+// A file is a compilation unit; its path supplies its namespace.
+// Qualified names resolve nested types and other units without an import keyword.
+program : statement* EOF ;
+
+block : '{' statement* '}' ;
+
+statement
+    : block
+    | returnStatement
+    | ifStatement
+    | whileStatement
+    | forStatement
+    | breakStatement
+    | continueStatement
+    | structDeclaration
+    | declaration ';'
+    | assignment ';'
+    | ';'
+    ;
+
+returnStatement   : 'return' assignment ';' ;
+ifStatement       : 'if' '(' assignment ')' statement ('else' statement)? ;
+whileStatement    : 'while' '(' assignment ')' statement ;
+breakStatement    : 'break' ';' ;
+continueStatement : 'continue' ';' ;
+
+forStatement
+    : 'for' '(' (declaration | assignment)? ';' assignment? ';' assignment? ')' statement
+    ;
+
+structDeclaration : 'struct' IDENTIFIER '{' structMember* '}' ';' ;
+structMember : constructorDeclaration | methodDeclaration | statement ;
+
+// The constructor name must match the enclosing struct's name.
+constructorDeclaration : 'new' IDENTIFIER '=' functionExpression ';' ;
+
+// A method is an ordinary fixed function binding in a struct definition.
+// The compiler supplies self; it is not written in parameterList.
+// Function types and expressions both receive their implicit receiver here.
+methodDeclaration : 'val' bindingModifier? IDENTIFIER '=' functionExpression ';' ;
+
+declaration : declarationType binding (',' binding)* ;
+declarationType : type | 'var' | 'val' ;
+binding : bindingModifier? IDENTIFIER ('=' initializer)? ;
+bindingModifier : '!' | '~' ;
+// A declaration initializer consisting of the string "C" declares an extern
+// symbol: a function, or explicit numeric native storage. Ordinary strings
+// elsewhere are array-valued expressions.
+initializer : assignment ;
+
+// The prefix qualifies the named struct reference, not an enclosing array.
+// Each [] / [~] suffix qualifies its own array layer. ? requires a reference.
+type
+    : functionType
+    | primitiveType typeSuffix*
+    | accessModifier? typeName typeSuffix*
+    ;
+accessModifier : '!' | '~' ;
+typeSuffix : '?' | '[]' | '[~]' ;
+typeName : IDENTIFIER ('.' IDENTIFIER)* ;
+
+primitiveType
+    : 'int' | 'i8' | 'i16' | 'i32' | 'i64'
+    | 'u1' | 'u8' | 'u16' | 'u32' | 'byte' | 'bool'
+    | 'flt' | 'f32' | 'f64'
+    ;
+
+assignment : expression (assignmentOperator assignment)? ;
+assignmentOperator
+    : '=' | '+=' | '-=' | '*=' | '/=' | '&=' | '|=' | '^='
+    | '<<=' | '>>=' | '>>>='
+    ;
+
+expression : logicalExpression ('?' assignment (':' assignment)?)? ;
+
+// Both logical operators short-circuit and associate to the right here.
+logicalExpression : bitwiseExpression (('&&' | '||') logicalExpression)? ;
+
+bitwiseExpression
+    : equalityExpression (('&' | '|' | '^') equalityExpression)*
+    ;
+
+equalityExpression : comparisonExpression (('==' | '!=') comparisonExpression)* ;
+
+// Chained comparisons test adjacent operands and short-circuit. A chain must
+// keep its direction; equality has lower precedence and does not join a chain.
+comparisonExpression
+    : shiftExpression
+      ( (('<' | '<=') shiftExpression)+ | (('>' | '>=') shiftExpression)+ )?
+    ;
+
+shiftExpression
+    : additiveExpression (('<<' | '>>' | '>>>') additiveExpression)*
+    ;
+
+additiveExpression
+    : multiplicativeExpression (('+' | '-') multiplicativeExpression)*
+    ;
+
+multiplicativeExpression
+    : unaryExpression (('*' | '/') unaryExpression)*
+    ;
+
+unaryExpression
+    : ('-' | '!') unaryExpression
+    | updateOperator IDENTIFIER
+    | postfixExpression
+    ;
+
+// Prefix updates require a local name; postfix updates also allow fields
+// and array elements. All updates require an assignable operand.
+updateOperator : '++' | '--' ;
+postfixExpression : primaryExpression postfixSuffix* updateOperator? ;
+postfixSuffix : '.' IDENTIFIER | '[' assignment ']' | '#' | callSuffix | methodCallSuffix ;
+
+// For a fixed function-valued member, object.name(args) passes object as self.
+// This overlaps ordinary field selection followed by a call syntactically;
+// name/type resolution determines which call form applies.
+methodCallSuffix : '.' IDENTIFIER callSuffix ;
+
+primaryExpression
+    : INTEGER_LITERAL
+    | FLOAT_LITERAL
+    | STRING_LITERAL
+    | CHARACTER_LITERAL
+    | 'true'
+    | 'false'
+    | 'null'
+    | IDENTIFIER
+    | '(' assignment ')'
+    | newExpression
+    | functionExpression
+    ;
+
+// A function is an expression bound with an ordinary declaration.
+// Its last statement supplies the implicit result; return can exit earlier.
+functionExpression : '{' parameterList? '->' statement* '}' ;
+parameterList : (parameter ','?)+ ;
+parameter : type bindingModifier? IDENTIFIER ;
+
+// Function types have unnamed, space-separated argument types.
+// {int} is the zero-argument type; its function expression still uses ->.
+functionType
+    : '{' type+ '->' type '}' '?'?
+    | '{' type '}' '?'?
+    ;
+
+callSuffix : '(' argumentList? ')' ;
+argumentList : assignment (',' assignment)* ','? ;
+
+// Allocation invokes the declared constructor; inline initializer blocks from
+// earlier chapters are replaced by constructor arguments.
+newExpression : 'new' type ('[' assignment ']' | callSuffix)? ;
+
+INTEGER_LITERAL : '0' | [1-9] DIGIT* ;
+// The f suffix also accepts an integer-shaped token, e.g. 12f.
+FLOAT_LITERAL : DIGIT+ ('.' DIGIT* ('e' DIGIT+)? | 'e' '-'? DIGIT+) 'f'?
+              | DIGIT+ 'f' ;
+STRING_LITERAL : '"' (ESCAPE | ~["\\])* '"' ;
+CHARACTER_LITERAL : '\'' (ESCAPE | ~['\\]) '\'' ;
+fragment ESCAPE : '\\' [0ntr\\'"] ;
+IDENTIFIER : NON_DIGIT (NON_DIGIT | DIGIT)* ;
+fragment NON_DIGIT : [a-zA-Z_] ;
+fragment DIGIT : [0-9] ;
+WHITESPACE : [ \t\r\n]+ -> skip ;
+LINE_COMMENT : '//' ~[\r\n]* -> skip ;
+BLOCK_COMMENT : '/*' .*? '*/' -> skip ;
+```
+
+## Reading the parser alongside the grammar
+
+All paths below are within `src/main/java/com/seaofnodes/simple/parser/`.
+The components share one `Parser` instance containing the cursor, active scope,
+compilation unit, and graph utilities. They are package-private collaborators;
+there is no inheritance chain, AST, or additional compiler pass.
+
+| Productions | Implementation |
+| --- | --- |
+| `program` | [`Parser.parse`](../../src/main/java/com/seaofnodes/simple/parser/Parser.java): establish the source-unit scope, parse its initializer, resolve remaining file names, publish its class |
+| `statement`, `block`, loops, jumps, `returnStatement` | [`Statements`](../../src/main/java/com/seaofnodes/simple/parser/Statements.java), starting at `parseStatement` |
+| `declaration`, `binding`, `bindingModifier` | [`Declarations`](../../src/main/java/com/seaofnodes/simple/parser/Declarations.java): declaration/expression lookahead, then each binding and initializer |
+| `structDeclaration`, `constructorDeclaration` | [`Structs`](../../src/main/java/com/seaofnodes/simple/parser/Structs.java), starting at `parseStruct` and `parseConstructorDeclaration`; field discovery and initialization checks follow these entry points |
+| `type`, `typeName`, `functionType` | [`Types`](../../src/main/java/com/seaofnodes/simple/parser/Types.java): `parseType` and `parseFunctionType`; qualified-name lookup lives in `typeName`, `dottedName`, and `rootedName` |
+| `assignment` through `primaryExpression` | [`Expressions`](../../src/main/java/com/seaofnodes/simple/parser/Expressions.java): `parseAssignment`, `parseExpression`, `parseLogical`, `parseBitwise`, `parseEquality`, `parseComparison`, `parseShift`, `parseAddition`, `parseMultiplication`, `parseUnary`, `parsePrimary` |
+| `postfixExpression`, member selection and updates | `Expressions.parsePostfix` / `parsePostfixMethod` dispatch to [`References.parseField`](../../src/main/java/com/seaofnodes/simple/parser/References.java); `References.parseName` handles bare names, implicit receivers, and forward references |
+| `functionExpression`, `parameterList`, calls, `argumentList` | [`Functions`](../../src/main/java/com/seaofnodes/simple/parser/Functions.java): `parseFunction`, `parseConstructor`, shared `parameters` and `arguments`, then `parseBody` |
+| `newExpression` | [`Allocations.parseNew`](../../src/main/java/com/seaofnodes/simple/parser/Allocations.java); the allocation-helper graph builder follows the expression routines |
+| Literals, identifiers, comments, token matching | [`Lexer`](../../src/main/java/com/seaofnodes/simple/parser/Lexer.java); it also supplies source-location snapshots used by diagnostics and serialized IR |
+
+`ControlFlow` shares guarded scope merging between `if`, `?:`, `&&`, and `||`.
+`Conversions` shares binding, field, and result conversions. These
+operations are compiler semantics rather than additional grammar productions.
+`Parser` retains the type registry and the small control/memory/peephole helpers
+used by several productions and by the rest of the compiler.
+
+There are a few deliberate differences between a syntax-only parser and this
+one-pass IR builder:
+
+- A statement beginning with `{` might be a block or a function-type binding.
+  `Declarations` and `Types` use speculative parsing and cursor restore;
+  qualified type lookup also stops before a suffix that belongs to an expression.
+- Bare assignment, field assignment, and compound updates share the grammar's
+  precedence but use different IR paths. `References` decides between an SSA
+  local, a constructor field's local value, and an actual memory access. A
+  function-valued field may also supply an implicit receiver to a call.
+- `for` updates are scanned first and parsed after the body, where their effects
+  belong. Explicit constructor bodies are scanned first and parsed after field
+  discovery, so later fields have local slots. Both use `Lexer.skipTo`, which
+  ignores delimiters inside comments and quoted text.
+- `Structs.parseInitializer` parses a file or struct body as a function while
+  discovering fields. `Functions.parseBody` still owns function scope/return
+  construction; moving it does not change constructor or memory semantics.
+
+The syntax chart does not describe all name/type restrictions. In particular,
+prefix updates require an already-known non-pointer binding, and a private-class
+privacy policy is separate from recognizing a qualified type name. The existing
+same-named function binding inside a struct is also recognized as a constructor;
+`new Name = { ... };` is the explicit constructor spelling above.
diff --git a/docs/chapter29/README.ja.md b/docs/chapter29/README.ja.md
new file mode 100644
index 00000000..9bed7e12
--- /dev/null
+++ b/docs/chapter29/README.ja.md
@@ -0,0 +1,27 @@
+# 第29章: 技術的負債の整理
+
+[English](README.md) | 日本語
+
+[前の章: 第28b章 — 分離コンパイル](../chapter28b/README.ja.md)
+
+この章では、第28b章の言語と分離コンパイルの動作を保ちながら、コンパイラを
+読みやすく、変更しやすく整理します。最初の変更は、3,000行を超えたパーサを
+文法のまとまりに沿って `parser/` パッケージへ分割することです。
+
+`Parser` がカーソル、スコープ、型の登録情報を持ち、各部品は `Parser P` を
+共有します。構文解析は引き続き Sea-of-Nodes IR を直接生成します。
+`Var` は `parser/` に、反復ピープホール最適化の `IterPeeps` は `codegen/` に移ります。
+
+[パーサの構成（英語）](../../src/main/java/com/seaofnodes/simple/parser/README.md)と
+[文法と実装の対応表（英語）](29-grammar.md)からコードをたどれます。
+コンストラクタの初期化完了チェックの整理は今後の課題です。
+ソース上のプライバシー制約は今回は追加せず、[ロードマップ](../roadmap.md)に残します。
+
+リポジトリのルートからビルドとテストを実行します。
+
+```sh
+make lib tests release CHAPTERS=chapter29 CTAGS=
+```
+
+IntelliJ では `chapter29` モジュールと **chapter29 tests** 実行設定を使用します。
+ネイティブ実行を使うテストの前に、Make でランタイムと `sys` ライブラリをビルドしてください。
diff --git a/docs/chapter29/README.md b/docs/chapter29/README.md
new file mode 100644
index 00000000..cdd0e5da
--- /dev/null
+++ b/docs/chapter29/README.md
@@ -0,0 +1,66 @@
+# Chapter 29: Tech Debt Cleanup
+
+English | [日本語](README.ja.md)
+
+[Previous: Chapter 28b — Separate Compilation](../chapter28b/README.md)
+
+This chapter makes the compiler easier to read and change while preserving
+Chapter 28b's language and separate-compilation behavior. It starts by breaking
+the parser into grammar-sized pieces. Further cleanup belongs here; earlier
+chapters retain their teaching snapshots.
+
+## Following the grammar
+
+The old `Parser.java` had grown beyond 3,000 lines, mixing recursive descent,
+name resolution, scope management, and IR construction. The
+[`parser/` package](../../src/main/java/com/seaofnodes/simple/parser/README.md) separates
+those responsibilities without introducing an AST: parsing still builds
+Sea-of-Nodes IR directly.
+
+`Parser` owns the source cursor, active scope, type registry, and common token
+and IR helpers. Each component shares that state through `Parser P`.
+`Statements`, `Expressions`, `Declarations`, `Types`, `Functions`, and `Structs`
+follow the corresponding grammar productions. `References` handles name and
+field resolution; `Allocations`, `Conversions`, and `ControlFlow` hold shared
+graph-building operations. `Lexer` owns tokenization and source locations, and
+`Var` records lexical bindings.
+
+The [grammar and code map](29-grammar.md) connects productions to methods.
+Start with `Statements.parseStatement` or `Expressions.parseAssignment` and
+follow the calls alongside it. Deferred parsing remains explicit: `for` updates
+are parsed after their bodies, and declared constructors after field discovery.
+
+The [source layout](../../src/main/java/com/seaofnodes/simple/README.md) describes the
+other packages. `IterPeeps` now lives in `codegen/` beside the compilation passes.
+
+## Cleanup still to do
+
+Constructor completion is a useful next target. `addReturn` already merges field
+values across exit paths, but `_ctorExits` saves those types until all fields
+are known. Checking the final `Return` alone is insufficient today: field
+conversions can erase uninitialized state, and the declaration initializer is
+separate from the user constructor. Simplify that bookkeeping without weakening
+the initialization checks.
+
+Source privacy is deferred. Leading underscores retain the existing export
+filtering behavior; this chapter does not yet enforce access restrictions on
+source uses. The [roadmap](../roadmap.md) retains that proposal and the other
+language candidates.
+
+## Build and test
+
+From the repository root:
+
+```sh
+make lib tests release CHAPTERS=chapter29 CTAGS=
+```
+
+The suite includes the inherited compiler, evaluator, native execution,
+serialization, and separate-compilation tests, plus parser regression tests for
+deferred fragments. Historical test class names retain their original identities.
+The [Chapter 28b library/client example](../chapter28b/README.md) also runs from
+this chapter directory.
+
+IntelliJ has a `chapter29` module and a **chapter29 tests** run configuration.
+Build the native runtime and `sys` library with Make before debugging tests
+that need them; see [the IntelliJ setup](../intellij.md).