diff options
Diffstat (limited to 'skills')
| -rw-r--r-- | skills/chapter-split.md | 220 | ||||
| -rw-r--r-- | skills/chapter25-codex-notes.md | 146 | ||||
| -rw-r--r-- | skills/current-spill-matrix.md | 14 | ||||
| -rw-r--r-- | skills/graph-viewer-codex-notes.md | 45 | ||||
| -rw-r--r-- | skills/incremental-builds.md | 76 | ||||
| -rw-r--r-- | skills/japanese-translation.md | 22 | ||||
| -rw-r--r-- | skills/stage-audit.md | 113 |
7 files changed, 592 insertions, 44 deletions
diff --git a/skills/chapter-split.md b/skills/chapter-split.md new file mode 100644 index 00000000..5321eca0 --- /dev/null +++ b/skills/chapter-split.md @@ -0,0 +1,220 @@ +# Approved chapter split + +Cliff approved the seven-part split on 2026-10-19 and the final numbering on +2026-10-24. The old combined Chapters 24 and 25 are retired; their history is +available in Git. Every new directory is a complete compiler snapshot. + +| Previous directory | Final chapter | +|---|---| +| chapter-sccp | 24a: SCCP within a function | +| chapter-interprocedural-sccp | 24b: SCCP across functions | +| chapter-incomplete-types | 25: SSA before types are known | +| chapter-construction | 26: Construction and private memory | +| chapter-escape | 27: Escape analysis and unknown callers | +| chapter-modules | 28a: Modules and lazy class initialization | +| chapter-separate-compilation | 28b: Separate compilation | + +Keep 17a/17b. The other paired chapters extend one idea: 24a/24b goes from +local to interprocedural SCCP; 28a/28b goes from compiling modules together to +compiling them independently. Construction and escape remain separate topics; +neither is presented as requiring the other. + +Inherited Java test names such as `Chapter24Test` and `Chapter25Test` retain +their historical identities across snapshots. They are regression cohorts, +not an index of the current teaching sequence. Historical allocator tables +also retain their original labels and dates. + +Native library construction and optimizer tuning are now [appendices](../docs/appendices/README.md). +The [roadmap](../docs/roadmap.md) owns future work. + +The teaching pattern is to introduce a difficult idea, then show how to make +it useful in a larger setting. Each chapter must remain a complete, runnable +compiler snapshot that preserves the language supported by its predecessor. + +## Teaching sequence + +| Topic | Main idea | Endpoint | +|---|---|---| +| SCCP within a function | Optimistic types, reachable control, Phi inputs, worklists, fixed points, and loop widening. Keep calls conservative at their interfaces. | Fold loop-carried values and eliminate branches that pessimistic iteration cannot resolve. | +| SCCP across functions | Discover call edges during analysis; merge arguments and returns; account for unknown callers and function pointers. | Specialize parameters, resolve call targets, handle recursion, and remove unreachable functions. | +| SSA before types are known | Syntax and lexical binding choose graph shape. Keep scalar types, deferred numeric modes, conversions, guards, incomplete objects, symbolic fields, and removal of parser-supplied Load/Phi bounds together. | Compile forward definitions without provisional types determining graph topology; settle types and layouts afterward. | +| Construction and private memory | Allocation helpers, hidden initializers, user constructors, private self-memory, field-completion checks, and explicit publication. | Preserve initialization and memory facts through constructor calls and publish a completed object. | +| Escape analysis and unknown callers | Discover which aliases and functions can reach code outside the known graph. | Preserve exposed memory and callable functions while specializing private code. | +| Modules and lazy class initialization | Multiple source files, namespaces, privacy, ownership, singleton storage, first-use initialization, parent order, cycle diagnostics, and lowering. | Compile sources together with defined class-initialization behavior. | +| Separate compilation | Preserve types, ideal IR, and identities across independent compilations. | Build a library and client separately, optimize imported bodies, link remaining calls, and check initialization dependencies across imports. | + +The first incomplete-type chapter stays together. Its unresolved memory accesses +must work before construction introduces private memory; defer the private +construction protocol, not the machinery required to represent an unknown access. + +## SCCP boundary + +The first SCCP snapshot keeps unknown-caller Start inputs and disconnects call +edges during the optimistic solve. Parameters retain their declared types; +call results retain facts from the preceding pessimistic pass, which can be +sharper than the written signature. Calls still have whole-memory effects. +After SCCP, peepholes relink calls and FinalizeIR removes uncalled bodies while +preserving named exports. This is an intermediate compiler design, not a +textual division of Opto.java. + +Ordinary loop widening belongs in the first chapter. The second adds argument +and result widening for recursion. Use `f(x-1)+1` to explain why these are +separate data cycles: the recursive return value never crosses Parm(x). + +Chained comparisons do not need their own chapter. Their home is +beside Chapter 23 short-circuit expressions: evaluate each operand +once and evaluate later operands only when earlier comparisons succeed. +The current Chapter 24 cold-loop-split allocator improvement belongs with the +tuning appendix. + +## Construction and escape boundary to establish + +Separate an explicit memory-publication operation from the analysis that +discovers exposure to unknown code. + +| Construction chapter | Escape-analysis chapter | +|---|---| +| Tracks private memory while an object is built. | Tracks aliases and functions exposed through memory, calls, and returns. | +| Publishes initialized memory at an explicit boundary. | Propagates exposure through the reachable graph. | +| Conservatively preserves unknown callers for surviving function addresses, following the current Chapter 24 policy. | Uses escape facts to determine which functions need unknown callers. | +| Explains EscapeNode as a publication operation. | Explains the Start/Stop feedback cycle between known code and the external world. | + +Verify this split against constructor calls and memory summaries. The earlier +snapshot can publish conservatively; the later one refines that behavior. +The name EscapeNode does not require teaching the full escape analysis at +the point where publication is introduced. + +`chapter26` now implements that explicit boundary. Hidden initializer +and allocator functions carry whole private object memory; completion publishes +fields into public aliases. The new constructor functions reject early passing +or storing of `self`, including self-links, and access through a merged alias. +These are deliberate limits for review with the next escape snapshot. Normal +function addresses retain conservative unknown callers. Initializers capture +enclosing values as hidden arguments; writing back to captures is rejected. +Arrays follow the same construction/publication protocol. There are no modules +or class singletons in this snapshot. + +Review on 2026-10-22 removed an unnecessary representation detour. Constructor +fields now stay in local SSA values and ordinary Phis; helpers emit ordinary +Stores at exit, with precise aliases gathered in MemMerge. Private memory is +only the handoff between helpers and publication. No global escape discovery +is needed to build that representation. + +Construction and escape discovery need not be taught in this order. Keep the +current order provisionally, without inventing a constructor example to claim +that dependency. The callback example now lives in the EA chapter. + +## Modules before separate compilation + +Teach module semantics with all source files compiled in one invocation. +Include lazy initialization, parent requirements, and cycle checking there. +Separate compilation then adds serialized ideal IR and types, GlobalBits +identity remapping, imported type reconstruction and analysis, and Unlink +before machine compilation. Extend initialization checks across imported +bodies, preserving opaque CLInit nodes until lowering. + +Keep code ownership distinct from unknown-caller edges in both chapters. +Keeping a function for emission must not imply arbitrary callers can reach it. + +## Appendices and future work + +- Native library construction: numeric C data symbols, address handling, + modular sys, runtime and driver support, and larger examples. Preserve the + native functionality already taught in earlier chapters. +- Optimizer and backend tuning: existing cloning/inlining choices, spill + ranking by live-range area and copy cost, rematerialization, and splitting + policies. Compare equivalent programs when explaining costs. + +This split organizes implemented behavior. Deterministic inlining, argument +arrays, static pre-initialization of classes, CFG frequency estimates, and +constant instance-field storage remain future work. See the +[roadmap](../docs/roadmap.md). + +## Renumbering and validation + +The first runnable extractions are +[SCCP Within a Function](../docs/chapter24a/README.md) and +[SCCP Across Functions](../docs/chapter24b/README.md), followed by +[SSA Before Types Are Known](../docs/chapter25/README.md), +[Construction and Private Memory](../docs/chapter26/README.md) and +[Escape Analysis and Unknown Callers](../docs/chapter27/README.md), and +[Modules and Lazy Class Initialization](../docs/chapter28a/README.md), and +[Separate Compilation](../docs/chapter28b/README.md). +All seven main snapshots are extracted and numbered. Chained comparisons now +start in Chapter 23 beside short-circuit expressions. The linear-history +publisher follows the numbered directories, including lettered pairs. + +Validate each intermediate compiler with its own Make suite, including execution +checks. Keep declarations, conversions, diagnostics, and type bounds consistent; +do not weaken checks to make an intermediate snapshot pass. Rebuild serialized +libraries when their compiler or object format changes. + +## Escape snapshot ready for boundary review + +`chapter27` adds function/alias exposure sets to TypeMem and closes the +Start/Stop feedback cycle during SCCP. Known calls propagate memory summaries; +exposed returns, fields, external arguments and native calls discover further +addresses. Native calls contribute even on paths with no live Return. Function +addresses kept entirely in known code no longer imply unknown callers. + +The [recursive callback example](../docs/chapter27/README.md) now belongs in +that chapter. Its base parameter becomes 2 while recursive calls remain. +Returning the callback or a containing object, or handing it to native code, +restores conservative arguments. FinalizeIR separates those analysis facts +from the roots needed to emit code. + +Construction still keeps local field values and rejects early exposure of self. +Both runnable snapshots are available for Cliff's promised review/play session. +This extraction does not choose broader rules for partially initialized objects. +No modules, class singletons, serialized IR or identity remapping are included. + +## Modules snapshot ready for review + +`chapter28a` compiles all discovered source files together. It adds class +singletons, namespaces and export filtering, parent initialization, CLInit memory effects, +cycle diagnostics, runtime lowering, and shared native class storage. Source +discovery reads parents as well as referenced children. The bundled modular sys +is supplied as source; building reusable libraries remains appendix material. + +There is no Serialize phase, object reader, GlobalBits remapping, or imported +IR. LocalIds assigns dense IDs within the invocation. FinishCalls detaches +analysis call edges after loop analysis, before scheduling or instruction +selection. ELF emits all discovered live bodies in the selected entry object. + +The preceding construction representation remains: field values and loop Phis +stay local, with Stores at helper exits. Constructor reads may see an unfinished +object, but may never read an uninitialized field. The module snapshot checks +this separately from completion and publication. Escape analysis still restores +unknown callers when a private callback reaches native code, including on paths +with no Return. + +The README's lazy lookup-table example is executable. Existing tests also cover +source/type cycles, rejected initialization cycles, parent order, native cold/hot +calls, shared storage, and ARM/RISC-V encoding. The construction/escape border +remains open to Cliff's review; this extraction does not settle broader rules for +publishing unfinished objects. + +## Separate compilation snapshot ready for review + +`chapter28b` adds saved types and ideal IR, `GlobalBits` +identity remapping, object discovery, imported bodies, and `Unlink`. It carries +Modules' deferred constructor parsing, local final-field values and loop Phis, +converted private memory summaries, native callback exposure, and source-parent +initialization order. Source privacy enforcement remains a separate review. + +The runnable example compiles a library and a client from separate source roots. +The client allocates an imported struct whose constructor updates a final field +in a loop, then calls a retained library function that updates shared state. +The regression checks evaluation under three seeds and native cold/repeated +calls. A second import test preserves unknown callers for a callback passed to +native code. Existing import tests retain initialization-cycle diagnostics, +shared class data, public type reconstruction, and native function thunks. + +Serialization now saves the class declarations of owned nested structs even +when they are absent from the optimized graph's type closure. Otherwise a +client can load the instance type but cannot find its allocation signature. +An explicit source input is always recompiled; dependency objects may still +be reused. The old incremental-rebuild experiment remains ignored. + +All seven main teaching boundaries now have numbered snapshots. Broader +language choices and compiler improvements remain in the review queue. diff --git a/skills/chapter25-codex-notes.md b/skills/chapter25-codex-notes.md index 22c832f6..4632698a 100644 --- a/skills/chapter25-codex-notes.md +++ b/skills/chapter25-codex-notes.md @@ -4,20 +4,35 @@ This file is intentionally AI-facing. It records durable invariants, debugging habits, and project-owner preferences that are easy to miss when reading only the implementation. Check current code and Git state rather than treating old session handoffs as live status. -For cross-chapter work, read [the backport queue](../docs/chapter-backports.md). -It owns pending work; these notes capture reusable lessons. Completed work and -superseded validation history remain in Git. -The [completed split](../docs/chapter-split.md) maps the former combined 24/25 +The [roadmap](../docs/roadmap.md) owns unfinished compiler work and language +proposals. These notes capture reusable lessons. Completed work and superseded +validation history remain in Git. +The [completed split](chapter-split.md) maps the former combined 24/25 to 24a/24b (SCCP), 25 (incomplete types), 26 (construction), 27 (escape), and 28a/28b (modules/separate compilation). The old combined snapshots are retired. Earlier entries below retain historical chapter numbers and test names; old -Chapter 25 implementation references describe the lineage now in 25–28b, with +Chapter 25 implementation references describe the lineage now in 25–28b, with the complete implementation in 28b. Use current paths and teaching boundaries. The construction/escape border remains open to semantic review. -Future work belongs in the [roadmap](../docs/roadmap.md), including the separate -[incremental build design](../docs/incremental-builds.md). +The [incremental build design](incremental-builds.md) expands the roadmap's +dependency and output-ownership requirements. For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.md). +## Reset checkpoint (2026-10-29) + +Chapter 28b's Encoding/Export cleanup is committed as `f47c7d1f`; the tree +was clean before this notes update. All 570 chapter tests passed on Windows +and Linux. Encoding prepares per-unit objects; Export only writes/links them. +Select combined in-memory output before Encoding, and request Serialize before +Lower for file output. There is no Export fallback that re-encodes the graph. + +Cliff wants to pivot to new language work and a new chapter. The next project +has not been chosen: discuss it before implementing. His goals are readable, +simple, usable; type variables/generics for real collections and low-annotation +automatic memory reclamation are major priorities. Privacy and subclassing +are smaller candidates. The roadmap owns remaining cleanup work; do not resume +it automatically or create a chapter merely because a reset happened. + ## Collaboration preferences - For densely keyed integer maps, use indexed `Ary<T>` storage rather than @@ -36,10 +51,18 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m Static `_spillScaled` also counts each inlined loop copy separately, without credit for eliminated calls; inspect the optimized call graph before calling a higher cross-chapter score an allocator regression. -- Prefer concrete ArrayList to List, common base classes to unnecessary - interfaces, and short names (`pred`, `proj`, `def`, `use`). Avoid redundant +- Use concrete JDK collection types in fields, parameters, and return types: + ArrayList, HashMap, HashSet, etc., rather than List, Map, or Set interfaces. + Prefer common base classes to unnecessary interfaces, and short names + (`pred`, `proj`, `def`, `use`). Avoid redundant Objects.requireNonNull. Prefer int[]/AryInt to List<Integer>; avoid large boxed ID collections (use arrays/BitSet or NBHML as appropriate). Node ID 0 means null. +- When both branches assign the same field or variable, prefer one assignment + with a ternary expression over an if/else. +- Do not wrap normal compiler algorithms in try/finally when no exceptions are + expected. Keep the lifetime and phase transitions explicit. +- Space declaration initializers as `Type name = expression;`. Preserve + deliberate column alignment when it makes related declarations easier to read. - Do not introduce Java streams. Use loops and explicit sorts; ask Cliff before making any exception. - Cliff often debugs the same failure in parallel. If asked to work in a side @@ -51,7 +74,7 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m printer/accessor side effect is immediately high priority. - Keep edited text files LF-only, including on Windows. - When mixing `||` and `&&` in one expression, always parenthesize the groups - explicitly rather than relying on operator precedence. + explicitly rather than relying on operator precedence: `x || (y && z)`. - For any node consuming or producing both control and memory, control belongs in slot 0 and memory in slot 1; other inputs/results follow. This applies to Start, Return, and future forward ports of Call, CallEnd, and other such nodes. @@ -65,6 +88,40 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m ## Mutability and chapter boundaries +- Chapter 28b emits one object per source CompUnit. Imported functions have + Fun/Summary/Return graphs until an object is materialized. SummaryNodes are + never inlineable bodies. Only local calls request speculative IR; an unused + imported function must not recursively fetch its callees. Function identity + survives materialization through GlobalBits and rewired call/return edges. +- Initialization summaries retain direct call/touch edges and parent edges. + Check cycles using available IR and summaries; do not load additional bodies + just to disprove a possible cycle. Rejecting such a program is intentional. + Cliff accepts optimization-dependent errors as a language experiment and + prefers this simpler compiler to a more accepting late-loading cycle check. + Native linking uses the compiler's resolved CompUnit object paths when + available. Object-only tests without a CodeGen list all link inputs explicitly; + TestC does no dependency search. Eval2 trusts TypeCheck and a coherent set of + library objects; it does not revalidate cross-object compatibility. + SummaryNode.load uses CodeGen.getElf directly; + Eval2 needs no extra byte snapshot or protection from rewritten object files. + Tests must isolate output paths to avoid writer/reader races. + Eval2 materializes one object's checked IR on demand and schedules it + independently; never reoptimize graphs with active evaluation frames. Eval2 + executes CLInit directly using shared run-once flags. The child's clinit + body checks its parent before setting its own flag, so Eval2 and Lower need + only the requested class's flag-test/call at each active use. +- Summary escape information must include retention on nonreturning paths and + references passed by new callers. Result types alone do not capture this. + Keep imported boundaries conservative; do not restore the old whole-graph + analyzeImports Iter/Opto repair pass. Deserialization must establish its + construction invariants directly, without temporarily setting phase Parse. + +- Prefix updates must share ordinary name resolution: a method's bare field + name denotes memory, while constructor fields may still be local SSA values. + Do not update the declaration's scope slot before resolving that distinction. + Keep the returned value alive across Store folding; a narrow Store can remove + a conversion it does not need, while the prefix expression still needs it. + - Separate compilation retains the Modules parser's constructor local values and escape fixes while adding saved IR and import ownership. Save owned nested struct class declarations even when graph types omit them: callers need the @@ -73,9 +130,7 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m library/client example checks three seeds and repeated native calls; native callback exposure is also checked after saving and loading the library. -- The four README intention questions are kept in - [the review agenda](../docs/chapter-backports.md#four-readme-intentions-to-review). - Final data fields are writable constructor locals, including in loops; final +- Final data fields are writable constructor locals, including in loops; final local bindings and shared function declarations stay fixed. Do not classify a constant instance default as a shared field or skip its loop Phi merely because the published field is final. Class constants keep their static image. @@ -149,8 +204,8 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m - CLInit lowering (2026-10-17): TypeCheck queues surviving markers; optional Serialize saves opaque ideal IR, then Lower runs Iter before LoopTree. Serialization includes disconnected Returns and filters cross-object links - without mutating the live graph. Parent initialization precedes the child's - flag test/call; the true flag store remains inside the initializer. Runtime + without mutating the live graph. In Chapter 28b the child's initializer owns + both its parent-init check and its subsequent true flag store. Runtime flag reads follow ordinary Load idealization, including removing control. Handle x86 condition-flag lifetime constraints in the backend, not by pinning ideal Loads. Generated calls remain unlinked and @@ -247,6 +302,10 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m ## Tutorial backports +- Chapter 18 is the evaluator cutover: use `Eval2` and the compiler's schedule + from that chapter onward. The old `evaluator/Evaluator` and its private + `Scheduler` remain only in Chapters 10-17b; do not forward-port or revive them. + - Shared ISA encoding lives in `isa/` for Chapters 21-25: x86 bytes behind `CodeSink`, ARM/RISC-V word packing, and test-only evaluators. Chapter code still owns machine nodes, register masks, operand/type interpretation, layouts and @@ -432,9 +491,9 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m top-level runner, e.g. `make -k tests CHAPTERS="chapter10 chapter11 chapter12"`, with all affected directories explicitly listed. Establish the destination baseline before changing it; older Makefiles may need a forced rebuild after API changes. -- Keep unrelated discoveries separate in `docs/chapter-backports.md`. Completed - fixes leave the pending queue; record validation in the change description. - Keep unresolved discoveries in the actual queue. A green suite +- Record confirmed unresolved bugs and future work in `docs/roadmap.md`. + Remove completed items and record validation in the change description. + Do not turn an unconfirmed textual difference into a backport task. A green suite after changing graph/test order does not prove an intermittent failure fixed. - Store disposable probes/logs under an appropriate ignored build directory. Preserve durable reproducers in tests or the review record, not only scratch @@ -823,6 +882,13 @@ Consequences: ## Types and monotonicity +- Chapters 24a/24b retain typed fallback results for erroneous or unresolved + Loads and arithmetic. Chapter 25 introduces the surrounding Convert nodes + and removes parser-supplied bounds. Do not backport BOTTOM-on-error in + isolation: parser, conversions, loop bounds, and diagnostics must agree. + Preserve `testBrainFuck`, `testWhileWithNullInside`, `testNullGuardErrors`, + `testLinkedList0`, and `testCoRecur`, including their expected errors. + - Integer widening starts in Chapter 24. `nonZero()` preserves `_widen` when narrowing a range. B10's correction is in 24, with `TypeTest.testNonZeroWidening` in 24-25. Both now use widening 0-3 and dual widening `3-widen`, preserve @@ -938,6 +1004,44 @@ Consequences: to FULL/public-only, it must not forget precise private bits discovered from the same Return earlier. +## Shared escape analysis and the rejected phase-hook experiment + +Chapter 28b shares one `GlobalEscapeNode` across compilation units. It scans +calls and unloaded summaries, registers reads with `addDepForwards`, and wakes +StopCUs when its type changes. New calls/functions/summaries must also queue it: +they were not present to register dependencies during an earlier compute. +StopCU combines its local facts with this shared result. The shared scan still +walks all contributors; a future incremental version must handle contributions +shrinking or disappearing, not just accumulate unions. + +The helper has no ordinary inputs. Root Start slot 2 keeps it alive but does +not read its type; Unlink removes that edge. Serialize omits the helper, saving +the combined StopCU facts instead. Do not replace dependency edges with ordinary +inputs merely to make a graph drawing convenient. + +The experiment after `f7fcac0a` did NOT remove `priorEscapes`/`publicBottom`: +FULL conflates unknown external values and unresolved local values. Public +function `#self` can have generic pointer type; passing it to native code can +yield FULL beside a precise private callback. Mapping FULL to public functions +then loses that callback, so the retained private bits currently matter. +Remembering all old facts instead would trap pessimistic Iter at its initial +FULL. Bounding only CallEnd/CLInit memory effects did not fix generic arguments +and returns. Treating FULL as every local function broadened escapes and caused +precision, worklist, and lifetime failures. All trial code was restored. + +A cleaner solution likely needs external escape bounds carried through value +flow, or a deliberate acceptance of broader escapes with the resulting issues +fixed. This is larger than moving a phase test to another node. Keep the hook +until that design is chosen. Disposable patches/logs are under +`chapter28b/build/escape-boundary-*`; do not rely on their survival or reapply +them as fixes. Useful regressions include BoundaryTest's recursive callback +and native exposure, plus private initializer and separate-compilation tests. + +Scheduling scratch arrays retain capacity and clear touched entries/references; +remaining node-ID bitset costs need measurement. Use isolated fixture/output +directories for concurrent Windows/Linux tests: Eval2 trusts coherent objects, +and should not defend against tests racing to overwrite them. + ## Serialization and code generation - Narrow direct C integer returns are normalized during instruction selection @@ -1065,8 +1169,8 @@ serialize/deserialization bijection after persisted graph changes Default fuzzer wrappers use explicit regression seed lists; exploratory fuzzing is opt-in. A passing wrapper with an empty list is not exploratory coverage, and `OPEN_FAILING_SEEDS` remain unresolved until explicitly verified and promoted. -For dated suite counts and known failures, consult the backport validation record -rather than treating an old count as the current baseline. +For dated suite counts, consult the validation record in Git history; current +known work is in the roadmap. Do not treat an old count as the current baseline. Peepholes are always enabled in every chapter, by Cliff's request on 2026-09-26. Do not reintroduce an unoptimized compiler mode. Incomplete Loop/Phi nodes still @@ -1087,7 +1191,7 @@ comparisons as equal runtime results. the SUB, not copy its inputs. ARM register 31 is XZR for load destinations, so narrow loads use WMASK, never RMASK. Logical-not uses CMP to XZR followed by CSET to its result register. Shared emulator fixes and Chapter 21-25 - backports are recorded in `docs/chapter-backports.md`. + backports are recorded in Git history. - Cohort 23's dynamic increase from compiler 24 to 25 is primarily class-field initialization: Ops 630->1106. RA 160->442 includes 224 extra pointer-address and 56 integer rematerialization instructions, but only two extra splits. diff --git a/skills/current-spill-matrix.md b/skills/current-spill-matrix.md new file mode 100644 index 00000000..ee77a77a --- /dev/null +++ b/skills/current-spill-matrix.md @@ -0,0 +1,14 @@ +# Current allocator comparison + +`make spill-stats` measures Chapters 20, 21, 22, 23, 24b, and 28b against the +historical test cohorts. Cohort IDs and test names retain their old numbers; +column headings identify the current compiler directories. + +These are separate from the dated [pre-split measurements](../docs/regalloc-spills.md). +After a successful run, `python build-support/spill-matrix.py --reuse --update-docs` +updates only this table. Fresh logs use 24b/28b names so old 24/25 results cannot +silently become measurements of new compiler snapshots. + +<!-- spill-matrix:start --> +No post-split measurement has been published yet. +<!-- spill-matrix:end --> diff --git a/skills/graph-viewer-codex-notes.md b/skills/graph-viewer-codex-notes.md index d76a949d..0d67ea35 100644 --- a/skills/graph-viewer-codex-notes.md +++ b/skills/graph-viewer-codex-notes.md @@ -1,28 +1,20 @@ # Shared graph viewer: restart notes -Updated 2026-09-26. Read graph/README.md for the current design and protocol. +Updated 2026-10-29. Read graph/README.md for the current design and protocol. These are AI-facing continuity notes, not a new task list or authority to change compiler semantics. Cliff declared the viewer task complete on 2026-09-26. Peephole animations, inlining display, TypeCheck diagnostics, and arrow-key -playback are committed. Resume the independent bug-fix backport review from -docs/chapter-backports.md when requested. +playback are committed. Unfinished compiler work is tracked in +docs/roadmap.md. -## Working tree and latest change +## Playback and diagnostics -The working tree was clean when checked on 2026-09-26; the viewer changes, -including peep.js and arrow-key playback, are in a32e83c9. Preserve any later -unrelated dirty files. +Left/Right arrow keys perform Previous/Next actions, including animation stages, +outside editable content and without modifiers. Preserve slider navigation and +focus during source cursor updates so repeated playback keys keep working. -Latest request: Left/Right arrow keys perform the same actions as the Previous/Next -buttons, including animation stages. Added outside inputs/editable content with -no modifiers; slider and source-editing keys keep their existing behavior. -Source cursor updates preserve focus so repeated playback keys keep working. -Disposable headless check build/check-viewer-keys.py passed forward/back mouse -equivalence, input/modifier exclusions, slider navigation, and the first-frame -boundary. These changes need only a browser refresh. - -Previous request: run TypeCheck after optimization and end playback on a visible -diagnostic. Implemented in chapters 10-25 (the chapters with a TypeCheck pass): +Run TypeCheck after optimization and end playback on a visible diagnostic. +Historical chapter numbers below predate the split recorded in chapter-split.md. - Chapters 10-17: SimpleGraphObserver calls STOP.typeCheck() after Iter; the previously package-private method is now public. Chapters 18-25 call @@ -43,12 +35,19 @@ diagnostic. Implemented in chapters 10-25 (the chapters with a TypeCheck pass): clears the diagnostic. Node-free errors show the message without centering. - Restart make view for Java changes; refreshing is enough for JS-only changes. -Validation on 2026-09-25: all 25 chapters built with Make. Headless browser checks -passed for real TypeCheck errors in chapters 10, 17, 18, 25, including final Next, -folded error nodes, backward playback, direct jump, and a subsequent valid compile. -Dead erroneous code was removed before TypeCheck. Existing JUnit checks passed: -chapter10 Chapter10Test (27), chapter18 Chapter13Test + Chapter18Test (39), -chapter28b Chapter13Test (16). This was NOT an all-chapter full test-suite run. +## Analysis-flow diagrams + +Normal viewer edges do not show `addDepForwards` analysis dependencies. For a +type-flow explanation, distinguish those reads from ordinary node inputs and +from lifetime-only edges. Label a hand-built diagram as a schematic, not a live +IR capture. Type flow runs def-to-use, opposite the usual input-pointer arrows. + +For example, chapter28b's GlobalEscapeNode has no ordinary inputs: it reads +calls/summaries through dependencies, and StopCU reads its result. Root Start +slot 2 keeps the helper alive without consuming its type. The disposable +`chapter28b/build/global-escape-flow.svg` illustrates this distinction; its DOT +source is a standalone schematic, not restored compiler DOT support. +`graph/render_docs.py` renders saved viewer JSON when a real snapshot is needed. ## Durable design and owner preferences diff --git a/skills/incremental-builds.md b/skills/incremental-builds.md new file mode 100644 index 00000000..47c06fd9 --- /dev/null +++ b/skills/incremental-builds.md @@ -0,0 +1,76 @@ +# Incremental builds + +Implementation notes and remaining build-driver work, reviewed 2026-10-25. + +## Current reuse check + +Chapter 28b's `CompUnit` decides whether to reuse a local object by walking its +recorded dependencies before loading saved types or IR. It checks all sources +owned by each local object, missing inputs, and dependency object timestamps. +Dependencies use the same local/library resolution for checking and loading. + +For example: + +1. A depends on B and has inlined some of B. +2. B's source changes, leaving B's object unchanged. +3. A client discovers A. The check follows A's recorded dependency to B and + finds B's newer source, even though B's object has not changed. +4. A is reparsed, so its saved inlined copy of B is never accepted. + +Use a fresh visited set for each candidate object. Cycles terminate safely, +but a provisional success inside a cycle must not be cached for another root. +Each source compilation unit now owns its own object; units compiled together +share a build timestamp so cycles do not invalidate themselves due to write order. +An unchanged public signature is insufficient: Simple imports implementation +details for optimization as well as declarations for type checking. + +Object-only libraries are still supplied as a coherent build. Do not reject +them merely because one dependency has a newer installation timestamp: existing +tests compile cyclic libraries against stubs and then combine their real +objects. Validating those combinations requires saved input versions, not a +guess from file times. A newer library object does invalidate a source-backed +local dependent that can be rebuilt. + +`IncrementalBuildTest` covers unchanged reuse, transitive source edits, +cycles, missing inputs, library dependencies, and clean/incremental native +execution parity. It sets file times directly, without sleeps. + +## Required pieces + +- Plan object targets and rebuilds across invocations. Handle cycles as groups + and newly discovered dependencies. The compiler's per-object reuse check is + not a complete scheduling or linking policy. +- Record actual input identities: source units and resolved library paths and + versions. A search + path change can select a different library even with an older timestamp. +- Record compiler and serialization versions, target, ABI, and relevant + options. Content hashes would also detect changes with preserved timestamps. + Specify which changes require rebuilding source and which reject an + incompatible source-free library. +- Make build targets account for every output of a compilation request, which + can discover several source units and produce a separate object for each. +- Provide an "ensure current" build operation. Explicit source compilation + currently always recompiles. An unchanged build should do no compilation or + linking; a changed object or link input should trigger relinking. +- Publish successful outputs without leaving a partial object accepted after + failure, and keep the recorded inputs consistent with those outputs. + +Initially, rebuilding every dependent of a changed object is a reasonable +conservative rule. Fine-grained interface/body fingerprints can come later. + +## Suggested first implementation + +Put a small build driver around the compiler with explicit object targets, +recorded inputs, and dependency propagation. Keep individual compilation +requests straightforward. A clean build and an incremental build must produce +the same observable program behavior. + +Cover unchanged builds, direct and transitive source edits, inlined constants +and bodies, cyclic dependencies, multiple outputs, removed inputs, library-path +changes, compiler/target changes, and failed builds. Use controlled timestamps +or hashes instead of sleeps. Replace the ignored historical experiment with +tests matching the chosen object ownership model. + +Library IR loading is independent of build and native link dependencies. +Summary-only imports still require coherent objects at link time, and inlining +does not erase the build dependency on the object supplying the copied body. diff --git a/skills/japanese-translation.md b/skills/japanese-translation.md new file mode 100644 index 00000000..39383345 --- /dev/null +++ b/skills/japanese-translation.md @@ -0,0 +1,22 @@ +# Japanese translation / 翻訳の保守方針 + +English is authoritative. The contributed Japanese translation is maintained on +a best-effort basis; English edits and chapter reorganizations never wait for +translation updates. Move Japanese files and update navigation when renumbering +chapters; prose updates can follow separately. Contributions are welcome, without +an ongoing maintenance obligation for the original translator. + +[英語版](../README.md)が正式な文書です。この日本語訳は寄贈されたもので、可能な範囲で保守されますが、 +英語版より更新が遅れることがあります。英語版の編集や章の再構成を、翻訳の更新待ちにすることはありません。 +章を移動・改番する際は、日本語ファイルも一緒に移動して章の見出しやリンクを更新し、本文の翻訳は後から更新できます。 +翻訳の改善は歓迎しますが、元の寄贈者に継続的な保守を義務付けるものではありません。 + +初回の翻訳は英語版のコミット +[`81b654d7`](https://github.com/SeaOfNodes/Simple/tree/81b654d76444d747fdf1d053760e507b50f26e18) +を基にしています。2026年10月の目次・文法見出し・コード例・命令エンコーディング・ +`C0DE` ヘッダに関する文書修正も反映しています。 +各文書の English リンクは現在の英語版を指します。更新されていない箇所の原文を確認するには、 +上記コミットの対応するファイルを参照してください。新しい章が未翻訳の場合は、その旨を示して英語版にリンクできます。 + + +[日本語の紹介に戻る](../README.ja.md) diff --git a/skills/stage-audit.md b/skills/stage-audit.md new file mode 100644 index 00000000..194a5178 --- /dev/null +++ b/skills/stage-audit.md @@ -0,0 +1,113 @@ +# Chapter split cleanup audit + +This record distinguishes checked behavior, existing gaps, and future designs +while retiring the old combined Chapters 24/25. The final mapping is in the +[split record](chapter-split.md). Historical Java test names are retained as +regression identities; they do not identify current chapter boundaries. + +## Corrections in this cleanup + +- Chained comparisons now begin in Chapter 23. The early SCCP implementations + incorrectly compared a boolean intermediate with the third operand: + `10 < 30 < 20` returned true. Later implementations preserved the middle + operand but reused the first operator, making `0 < 2 <= 2` false. All current + snapshots from 23 onward now compare adjacent values using each written + operator. Regression cases cover both directions, strict/non-strict mixtures, + single evaluation, side effects, short-circuiting, and mixed-direction errors. +- Chapter 23's inherited, explicitly described fuzzer scratch contains many + comparisons using the former left-associative grammar. It is retained as an + ignored `testJigFuzzer`, consistent with the scratch-test policy; its mixed + directions are no longer valid source. Later snapshots already omitted it. +- Native-library and tuning material is preserved in [appendices](../docs/appendices/README.md), + and unfinished work is collected in the [roadmap](../docs/roadmap.md). +- The recursive-return diagnostic already passed. It remains enabled in the + earlier snapshots and is enabled again in 28a/28b. The mutual-recursion test had an obsolete graph-string + expectation; it now checks its existing four execution results in 24b–28b. + In 24a, local SCCP cannot infer the recursive return type; the same example + checks that conservative diagnostic instead. This makes the boundary explicit. +- Allocation/runtime measurement runs passed after the comparison correction; + the changed spill goldens record the resulting smaller graphs. Historical + published tables retain their original compiler labels and dates. + +## Prefix-update follow-up + +Prefix `++` and `--` now use the same name lookup as ordinary reads and postfix +updates in Chapters 23 through 28b. A method's field goes through memory; a +constructor-local field stays in SSA. Previously, the prefix path updated the +field declaration's parser slot and silently lost the write. The two-call +reproducer now returns 21 instead of 11. Narrow update results stay alive while +Store folding drops a redundant conversion from its own input. + +`PrefixUpdateTest` covers increment/decrement, integer/float/byte fields, +multiple worklist seeds, local shadowing, final-field errors, native byte +wraparound, and module fields where supported. Existing final-constructor +coverage includes prefix updates in loops. + +The focused Linux checks pass in all eight affected chapters, including native +SystemV byte-wraparound execution and the existing final-constructor tests. +The early final-field rejection is retained so an illegal prefix update never +enters the optimizer as a Store; native data bindings still update their writable +contents despite having a fixed address. + +The Windows full suites cover 4,169 tests plus eight fuzzer regression runs; +shared printer and ISA checks pass. Focused reruns cover the final diagnostic +and prefix-result lifetime adjustments. The full Chapter 27 suite was rerun +after restricting the temporary keep to prefix results; its existing spill +expectations remain unchanged. No allocator goldens were updated. + +## Implemented, with narrower limits than earlier prose suggested + +The constructor-loop assertion is fixed in Chapters 25–27. Load forwarding now +waits for a stored loop Phi to acquire a type at least as precise as the Load, +and records a dependency to retry. The reduced case needs only a conditional +assignment in a loop; break/continue and final fields are not required. +Regressions cover the reduced case and the original constructor loop across +worklist seeds and execution inputs. Chapters 28a/28b already had the guard; +the corresponding earlier-chapter probes pass in 23/24a/24b. + +- Separate compilation imports types and optimized IR without source parsing; + it reruns analysis and type checking with the new callers. +- Loading is by object and recorded dependencies, not individual used members. + A call to `sys.io.p` loads all ten bundled sys units. Independent, unrelated + objects can remain unloaded. Loading does not imply class initialization. +- Timestamp checks do not yet provide a reliable incremental build system. + The [design note](incremental-builds.md) records the missing pieces. +- Underscores select exports in modules; source privacy checks are still a + proposal awaiting review. Do not treat hidden exports as inaccessible source. +- Constructor-local final fields use SSA values and exit Stores. Reads of an + unfinished object are allowed only for initialized fields. Passing unfinished + self to arbitrary helpers and accepting completion only after SCCP are not + promised by the current chapter boundary. + +## Existing failures and deferred validation + +- The ideal evaluator does not truncate narrow stores in Chapter 23. A byte + update can therefore disagree with generated code; the postfix-only + reproducer is retained in the [roadmap](../docs/roadmap.md#narrow-stores-in-the-ideal-evaluator). +- `testModule0` is an ignored historical incremental-build experiment. It + assumes per-source output objects and unchanged explicit-input builds, neither + of which matches the present compiler's contract. +- Serialization's expensive byte-for-byte round-trip assertion remains disabled. + Ordinary import tests do not establish that stronger property. +- Direct probes of the remaining ignored tests found a stale forward-method + graph expectation and a missing `progs/sieve.smp` fixture in 28b. These and + the interactive stdin test remain ignored. Ignored Jig and + random-fuzzer entry points are intentional scratch/long-running tools. + Audit these separately before advertising their behavior as tested. + +Selective loading, full incremental builds, source privacy, compact pointers, +argument arrays, deterministic inlining, and broader constructor rules remain +future work. They are not prerequisites for completing the chapter numbering. + +## Validation + +The affected snapshots (23, 24a, 24b, 25, 26, 27, 28a, 28b) were checked with +their full Make suites on Windows and Linux, plus focused reruns after updating +test expectations. The final Windows coverage is 4,143 tests plus eight fuzzer +regression runs, with the shared printer and ISA tests also passing. The +native library/client cases and final Chapter 28b release build pass on both OSes. + +A local linear-history dry run produced all 31 tags in teaching order. The +final source snapshot, chapter navigation, Maven module IDs, and reviewed local +documentation links were checked. Measurement capture in 24b and dynamic +measurements in 24b/28b pass; the historical published tables were not rewritten. |