path: root/docs/roadmap.md

# Compiler roadmap

English | [日本語](roadmap.ja.md)

Open compiler work and language proposals, reviewed against the current
snapshots on 2026-11-08. These are future changes, not promises about the
language implemented by the tutorial. Language changes still need their own
design decisions and chapter boundaries.

## Contents

- [Compilation and optimization](#compilation-and-optimization) — diagnostics, analysis costs, builds, loading, and code quality.
  - [General cleanup](#general-cleanup) - Some general quality-of-life suggestions
  - [Phase ordering and incremental analysis](#phase-ordering-and-incremental-analysis) — scheduling costs, shared escape scans, and late identity precision.
  - [Reliable incremental builds](#reliable-incremental-builds) — input identities, compatibility, and relinking.
  - [Function-level loading](#function-level-loading-of-library-ir) — load individual library bodies.
  - [Deterministic inlining](#deterministic-inlining) — stable choices using optimized body sizes.
  - [CFG frequency estimates](#cfg-frequency-estimates) — guide scheduling and layout.
  - [Loop rotation](#loop-rotation) — bottom-tested loops with correct entry and exit behavior.
  - [Integer instruction widths](#integer-instruction-widths) — local narrowing patterns, 32-bit instructions, and later demanded-bit analysis.
  - [Per-function calling conventions](#per-function-calling-conventions) — precise clobbers, custom interfaces, and later joint allocation.
  - [Constant instance-field storage](#constant-instance-field-storage) — omit slots whose values are constant across instances.
- [Language boundaries](#language-boundaries) — native argument representation, source access rules, and symbol visibility.
  - [C byte pointers](#c-byte-pointers) — distinct native pointers with byte loads/stores and C-owned memory.
  - [Source privacy](#source-privacy) — enforce private access through imports and optimization.
  - [ELF visibility](#elf-visibility-and-unknown-callers) — separate linker exports from unknown-caller reachability.
- [Initialization and runtime](#initialization-and-runtime) — entry arguments, static initialization, and pointer storage.
  - [Entry arguments](#entry-arguments) — pass argument arrays to class initializers.
  - [Pre-initialize classes](#pre-initialize-classes-from-elf-data) — replace provably static initialization with ELF data.
  - [Compact arenas](#compact-code-and-data-arenas) — four-byte references and explicit native pointer boundaries.
- [Longer-term proposals](#longer-term-language-and-library-proposals) — deferred language and library designs.
  - [Partial-initialization contracts](#inferred-partial-initialization-contracts--deferred) — inferred requirements, nested guarantees, and imported contracts.
  - [Automatic memory management](#automatic-memory-management) — ownership, sharing, cycles, and native retention.
  - [A graphical game](#a-small-graphical-game) — exercise graphics, keyboard, and audio APIs.
  - [Time and animation](#time-timers-and-animation) — scoped follow-on chapter with clocks, calendar values, cooperative timers, and responsive waits.
  - [Trailing arrays](#structs-with-trailing-arrays) — inline elements after named fields.
  - [Array initialization](#array-initialization) — generate each element during allocation.
  - [String libraries](#string-libraries) — ownership, Unicode, and additional representations beyond byte buffers.
  - [Struct values](#struct-values) — inline storage, copying, and by-value calls.

## Compilation and optimization

### General cleanup

Keep the recursive-descent path aligned with the grammar. Review moving `TYPES`
to `CodeGen`.

Consider whether an allocation-free use of try-with-resources makes keep/unkeep
pairing clearer; this remains an experiment, not a new resource-management
convention.



### Phase ordering and incremental analysis

Remaining work:

- Review the remaining node-ID bitset costs in Eval2 scheduling. Scratch
  arrays now retain capacity and reset only touched entries, but bitsets
  still span previously allocated node IDs. Scheduling a newly loaded
  object should cost only for that graph and any affected existing nodes.
- Make `GlobalEscapeNode`'s shared call/summary scan incremental. Compilation
  unit Stops now reuse its result, but updating that result still scans all
  calls and unloaded summaries. Handle contributors changing or disappearing,
  rather than only accumulating unions. The
  [outside-world escape flow](../chapter28a/README.md#escaping-or-the-startstop-cycle) already shares
  the result; the remaining cost is recomputing it.
- Improve precision for identities discovered after Opto's foreign seed is
  established. They currently remain in the cofinite foreign tail. Preserve
  monotone propagation and imported callback behavior when refining this bound.

Whole-program TypeCheck is acceptable for now; incremental checking is an
optional later improvement. Expensive assertion walks are outside this
performance work.

### Reliable incremental builds

Chapter 28b now checks the reachable source-backed inputs before reusing saved
IR, including transitive dependencies and missing inputs.
The remaining work is a build driver with recorded input identities, beyond
those conservative timestamp checks. An unchanged public signature does not
establish compatibility: optimization imports bodies and constants too.

Record the actual resolved input paths and versions, compiler/serialization
versions, target, ABI, and relevant options. Content hashes can cover preserved
timestamps and changed library search paths. Distinguish rebuildable source
from incompatible source-free libraries; their installation timestamps alone
cannot validate a set of objects compiled against one another.

Native separate compilation should also detect a library changing between
reading its types and linking the executable. One possible mechanism is a
SHA-256-named symbol that callers require and the matching library defines.
This remains future link-time validation; Eval2 assumes a coherent build.

An unchanged build should do no compilation or linking. Publish outputs only
after successful compilation, and relink when an object or other link input
changes. Start with a small build driver and conservative rebuilding of every
dependent; finer interface/body fingerprints can follow. The
[incremental-build design](../skills/incremental-builds.md) records the input
ownership rules and validation cases.

### Function-level loading of library IR

Chapter 28b emits one object per compilation unit and reads declarations and
`SummaryNode` information without loading bodies. Optimization can request an
object's IR for inlining; Eval2 materializes bodies on executed calls and
assumes a coherent build. Initialization checks reject possible cycles using
the available graph and summaries, without loading more bodies to disprove
them. A requested object still reconstructs all of its functions.

Further selectivity would require a function-body index within each object
and narrower metadata discovery. Preserve type identity, initialization rules,
and imported escape summaries. Loading, initializing, and emitting a class
remain separate decisions. This optimization is independent of making
incremental rebuilding correct.

### Deterministic inlining

The current inliner has a bounded exception for small initializer boilerplate,
but still uses parse-size estimates and worklist order. After shrinking
rewrites converge, choose one growing inline using current body sizes and a
deterministic ranking. Cache sizes per convergence round and use stable
call-site provenance to break ties, rather than newly allocated node IDs.
Avoid repeated whole-body scans and per-node dependencies while a large
function gradually shrinks.

### CFG frequency estimates

After lowering and loop-tree construction, estimate block frequencies for GCM
and code layout. Treat initialization and known-failure paths as rare; use
bounded loop-trip estimates and branch probabilities when no profile is
available. Keep estimates consistent with control flow and shared between
scheduling and layout. These are backend estimates, not ideal-IR types.

### Loop rotation

Consider lowering `for` loops with a zero-trip entry test and a bottom-tested
loop, or adding partial peeling. Preserve condition side effects, loop-local
scope, and the destinations of `break` and `continue`. The current parser
constructs a loop with its test at the head.

### Integer instruction widths

Keep the Chapter 30 language rule: `int` aliases `i32`, but inferred values
and intermediate arithmetic retain their existing rules. Two `int` operands
can produce a 33-bit sum; narrowing on assignment is a separate operation.

Start with local instruction-selection improvements, without a new analysis:

- Recognize `(x<<32)>>32` as sign-extension instead of two shifts.
- Add an operand-width choice to integer machine operations and the shared
  encoder. On x86, dropping REX.W can remove a prefix byte when no extended
  register needs REX. The encoder already has `rexF(..., wide, ...)`, but
  register arithmetic currently forces wide operations.
- Use range proofs for suitable 32-bit arithmetic with a result in `u32`,
  and 32-bit signed comparisons between operands proven to fit `i32`.
- Combine arithmetic with explicit narrowing: a narrowed add can use a
  32-bit add followed by sign-extension. Bypass input extensions where only
  low bits are consumed, preserving shared producers' other wide uses.

An x86 32-bit result zeroes the upper half; negative signed results still
need sign-extension before a full-width consumer. Keep shifts and division
under separate rules: operand width changes shift-count masking and which
high bits can flow down, and narrow signed division introduces the
`INT_MIN / -1` overflow case. Do not infer signed representability solely
from `TypeInteger.log_size()`; use actual range bounds.

Longer term, distinguish computed bits from demanded bits. `TypeInteger`
already provides ranges and a derived possible-bits mask. Tracking which
bits consumers observe could keep arithmetic chains and loops narrow and
extend only at wide consumers, but requires backwards propagation through
shared uses and Phis. This is a separate analysis proposal, not part of the
local selection work. Narrow operations still occupy whole registers.

### Per-function calling conventions

Chapter 30 now allocates functions independently. Give known direct-call targets
their own machine contracts: argument/return locations, clobbered registers,
stack argument layout/alignment, and integer-extension requirements. Start
with precise clobber summaries under the existing convention, then consider
custom argument/return locations. Allocate callees before callers and freeze
each contract; caller preferences can guide choices without forcing every
call site to agree on its preferred registers.

Preserve machine-call relationships before Unlink discards the ideal call
graph. Replace compilation-wide parameter, return, preservation, and call-kill
masks with contract lookups. Keep C entry points and callbacks on the C ABI,
using wrappers if an internal custom entry is worthwhile. Source privacy
alone does not establish that all calls are known.

Indirect calls need a convention shared by all possible targets, entry
thunks, or a richer function-pointer representation. Public Simple functions
could export their selected contract in target-specific metadata after
allocation; ideal IR is currently serialized before allocation. A changed
contract requires rebuilding callers or rejecting stale objects even when
the source type is unchanged.

Joint caller/callee coloring is a larger, separate proposal. Model argument
and result transfers as preferences, retaining moves when needed; current
LRGs and spill slots belong to individual functions. Values live across a
call must survive the callee's transitive clobbers. Recursive activations
require saves/spills and cannot be represented just by merging static live
ranges. Initially use fixed conventions for recursive groups. If interfaces
are later renegotiated, budget retries by function or recursive group, not
one global round counter shared by unrelated functions.

### Constant instance-field storage

When a final instance field has the same constant value in every instance,
fold accesses or store the value on the class object and omit the instance
slot. Preserve field-access semantics and public function reachability:
externally accessible function fields remain callable regardless of storage.
Existing static images for class constants do not implement this general
instance-layout optimization.

## Language boundaries

### C byte pointers

Introduce an eight-byte C pointer type, initially pointing only to bytes, with
indexed byte Loads and Stores, null checks, and defined offset operations.
Track whether the storage is C-managed or Simple-managed separately from
pointer width and address range. The prospective low-4-GiB Simple heap does not
make low-address C allocations Simple-owned; high-address buffers require the
full-width C pointer. A low-address C buffer might admit a compact reference,
but only with compatible layout and retained C ownership. Revisit whether that
is useful when designing the heap and automatic memory management.

Allow Simple buffers to be borrowed by C under a documented promise that C
does not retain the pointer. Initially this lifetime promise is not enforced.
Returning a pointer must not silently transfer ownership, and Simple memory
management must not free or traverse C-owned storage.

Use normal memory SSA. C views of Simple buffers can alias Simple Loads and
Stores, so representation alone cannot select a disjoint native-memory alias
class. Native calls must expose effects on borrowed Simple memory as well as
C memory. Lengths and writable access need API contracts; a raw C pointer
carries no Simple array header or length.

This would let the runtime scan and copy argument bytes in Simple. Access to
`argv[i]` would still need the small C helper: `char **` requires pointer loads,
which are outside the initial byte-only scope.

Replace existing I/O's `i64` address convention with explicit native views of
Simple buffers, preserving their ownership and the non-retention contract.
A later Simple `C` struct facility could supply C-compatible field layout for
libc interfaces. Such structs have no Simple constructor/destructor and do not
intermingle with the managed Simple object graph; specify native layout and
ownership independently of ordinary Simple structs.

### Source privacy

Underscores already select the public interface, but source access to another
class's private fields is not rejected. Revisit source enforcement and its
preservation across imports. This language change still needs review; it is not
a correction to impose on earlier chapters.

- A field or method beginning with `_` is accessible only from its declaring
  class, including constructors and methods. Enclosing, nested, and sibling
  classes gain no special access. Same-class code may access another instance.
- A private class name is available in its declaring scope and inside the
  class itself. Naming `App._Cache` must not grant access to `_Cache._data`.
  Dotted lookup cannot bypass access rules, and private ancestors keep their
  descendants out of the public interface. Distinguish naming a private class
  from using a public value returned by a factory.
- Reads, writes, updates, method references, type names, and class-value
  references must follow the same rule. Preserve source owner and location
  through unresolved receiver types, inlining, and imports. Folding away a
  Load or an unused result must not erase an illegal source access.
- Native declarations follow the same rules. Public wrappers may deliberately
  expose private bindings; `sys` has no special exemption.

The resizable-array helpers call `libc._exit` directly, and `sys.io.read`
calls `aryu8._grow`. Keep the native exit binding private, add a public `exit`
wrapper in `sys/libc.smp`, and define a public capacity-reservation method for
the buffer. Bitset and array helpers accessing their own private fields remain
legal. String spill fixtures with helpers outside `_String` accessing
`_hashCode` need explicit adaptation; do not grant file-wide access merely to
keep those fixtures unchanged. Validate native execution and imported source
ownership alongside the access diagnostics.

### ELF visibility and unknown callers

Escape analysis tracks functions reachable by unknown callers, while the ELF
writer exports retained named functions owned by the compilation unit. Decide
which symbols should be public, local, or hidden across separate compilations.
Keep source privacy, ownership for emission, and unknown-caller edges distinct;
keeping a body for linking must not imply arbitrary callers can reach it.

## Initialization and runtime

### Entry arguments

Replace the integer class-initializer argument with an argument array. The
selected entry class receives a non-null array, empty when there are no
arguments; other class initializers receive null. Coordinate the parser,
entry signature, runtime driver, and separately compiled interfaces.

The native runtime already adapts process arguments for an explicit
`simple_main` function. That does not implement argument arrays for class
initializers.

### Pre-initialize classes from ELF data

Prove when a class's entire initialization can be represented by static data
and relocations, without remaining runtime effects. Emit those contents and
start its `$init` flag as true. Carry that fact through separate compilation
so importing compilers can remove checks too.

Constant final fields already have static images; that alone does not prove
the whole class can be pre-initialized. I/O and argument-dependent work still
require execution. Preserve dynamic parent requirements, and do not let cyclic
dependencies justify their own pre-initialization. Static reference cycles
can use relocations without requiring a dynamic initialization cycle.

Mutable fields may have static initial contents without becoming constants.
For a wholly immutable, pre-initialized class, eliminate the mutable `$init`
state as well as the checks so the class image can reside in ROM. Native
function thunks already preserve static initialization of function fields.

### Compact code and data arenas

Use four-byte Simple references into bounded code and data arenas. The arenas
need not overlap or live near zero; choose absolute addresses or offsets from
an arena base when implementing placement and indirect calls. Native function
thunks can occupy the code arena while native libraries live elsewhere.
Currently function pointers occupy four bytes and use low-address thunks;
ordinary data pointers still occupy eight bytes.  See [Automatic memory
management](#automatic-memory-management).

Give full-width native pointers separate types and foreign lifetime contracts.
Opaque device addresses may need to pass through without CPU dereference.
Automatic Simple object lifetimes without a GC still need a design for
ownership, borrowing, cycles, and native retention. Compact storage alone
does not establish those rules. Preserve static class images and explicit
native boundary conversions when choosing the representation.

Defer the four-byte data-pointer switch to a later chapter alongside memory
management, which will also establish the allocator and native ownership rules.

## Longer-term language and library proposals

These are language-design alternatives. The sketches below are not executable
examples of the current Simple grammar.

### Inferred partial-initialization contracts — deferred

Chapter 29 already permits local reads of proven initialized fields and limited
retention of partial references through normal forward type flow. The deferred
extension is inferred field requirements, nested guarantees, and contracts for
separately compiled callees, while reserving final-field writes to the original
constructor. Tracing requirements through aliases, nested fields, forwarding,
recursive calls, and returned/stored references adds more infrastructure than
is justified for this feature alone.

Revisit alongside a general backwards/liveness analysis, which may also serve
future memory management. The normal-type contract proposal and its unresolved
details are recorded in the
[construction notes](../chapter29/docs/construction-groups.md#deferred-field-aware-contracts-2026-11-07).
Keep the separate-compilation restriction on `^` functions meanwhile.

### Automatic memory management

Explore automatic memory reclamation with few explicit lifetime annotations,
using Rust and [Goose](https://github.com/aardappel/goose/blob/master/docs/tutorial.md)
as design references. Decide ownership, borrowing, sharing, cycles, and native
retention together with the existing object and array model. This is a language
design candidate, independent of compact pointer storage.

### A small graphical game

Chapter 31 follows ASCII tic-tac-toe with a graphical version, using
[software drawing in Simple](../chapter31/docs/drawing.md) and an optional
MiniFB window/input backend. The initial library supplies colors, rectangles,
circles, thick lines, and bitmap labels. Mouse/keyboard turns and replay
exercise that boundary without requiring an animated game engine.

Optional [background audio](../chapter31/docs/audio.md) uses miniaudio for
soft move sounds and ascending/descending win/loss cues. Further games remain
future work; MiniFB supplies only the window and input boundary.

### Time timers and animation

Scoped on 2026-11-12 as a proposed Chapter 32 following Chapter 31's drawing,
graphics, and audio work. Save this scope while closing Chapter 31; the time
interface and animation work are not implemented yet. This is primarily a
standard-library/runtime chapter, with no anticipated compiler changes or new
external dependencies. Keep the game loop and timer dispatch single-threaded.

Default instants are bare `i64` milliseconds since Unix epoch, UTC. Durations
are also bare `i64` milliseconds. Distinguish wall-clock readings from monotonic
readings by operation names and documentation: animation and timer deadlines
must not depend on system-clock corrections.

Proposed `sys.time` core:

| Operation | Meaning |
| --- | --- |
| `now()` | UTC Unix-epoch milliseconds. |
| `monotonic()` | Milliseconds from an unspecified monotonic origin, for deadlines and elapsed time. |
| `nanoTime()` | Monotonic nanoseconds; units do not promise nanosecond clock resolution. |
| `sleep(ms)` | Suspend the calling thread for a duration. |
| `sleepUntil(deadline)` | Wait until a monotonic millisecond deadline. |
| `seconds(n)`, `minutes(n)`, `hours(n)`, `days(n)` | Convert durations to milliseconds; a day here is exactly 24 hours. |
| `toSeconds(ms)`, `toMinutes(ms)`, etc. | Convert back with explicitly documented rounding. |

Cygwin's existing runtime headers expose `clock_gettime`, `clock_nanosleep`,
and `localtime_r`; investigate sharing the native clock/calendar bridge with
Linux. Native Windows interval measurement can use QueryPerformanceCounter.
Epoch nanoseconds are optional: signed `i64` would cover only about 292 years
on either side of 1970, so retain milliseconds as the default representation.
Sleep deadlines are approximate; handle interruptions and avoid busy waiting.

Initial calendar/text support:

- Format UTC as `2026-11-12T15:04:05.123Z`; parse that form and explicit numeric
  UTC offsets such as `-08:00`.
- Break instants into UTC or system-local calendar fields. An auxiliary
  `DateTime` structure can contain year, month, day, hour, minute, second,
  millisecond, weekday, UTC offset, daylight-saving status, and zone abbreviation.
  The underlying instant remains a bare `i64`.
- Construct instants from UTC fields or calendar fields with explicit offsets.
- Query the local UTC offset, DST status, and zone abbreviation at a supplied
  instant, using the operating system's existing time-zone machinery.
- Defer arbitrary named-zone conversion, bundled time-zone databases,
  natural-language parsing, and locale-specific formatting. Parsing local
  calendar times without an offset needs a policy for ambiguous/nonexistent
  times around daylight-saving transitions.

Keep arithmetic, parsing, formatting, and timer scheduling in Simple where
practical. Limit native code to clock readings, OS waits, and local-zone
conversion. Validate overflow, negative epoch division/remainders, leap days,
invalid dates, and supported calendar ranges. A parse error cannot use `-1`
as a sentinel because that is a valid instant. Fixed-duration days and calendar
day arithmetic are separate operations.

Animation needs an input-aware wait in addition to ordinary sleep. Replace
the graphics adapter's fixed 17ms delay with a proposed `gfx.wait(timeoutMs)`
that processes events and wakes for either input or timeout. The game loop
waits until the earlier of its next frame and next timer, then handles input,
due timers, simulation, and drawing on the same thread. Windows has a
message-aware timed wait; X11 needs corresponding backend integration.

A small cooperative timer queue in Simple can expose `after(delay)` and
`every(period)` returning timer IDs, plus `cancel(id)`, `nextDeadline()`, and
`poll(now)` returning due IDs. Returning IDs avoids requiring closures or
background callbacks. Advance repeating deadlines from their previous deadline
to avoid drift. Define a missed-tick policy and bound animation catch-up;
reset or clamp frame deltas after laptop sleep rather than replaying hours of
frames. Explicitly exclude waking the computer from system suspend.

Use an animated game example to drive the chapter. Planning estimate:
roughly 1,000-2,000 implementation lines, plus tests and documentation. Clock
readings, sleep, and unit conversion are small; calendar/text/zone support,
cooperative scheduling, responsive graphics waiting, and an animated example
make the complete scope medium-sized. Test timer ordering, cancellation,
repetition, and clock jumps with injected/fake readings; keep actual sleep tests
broadly bounded and hardware/display checks optional.

Platform references: [clock_gettime](https://man7.org/linux/man-pages/man2/clock_gettime.2.html),
[clock_nanosleep](https://man7.org/linux/man-pages/man2/clock_nanosleep.2.html),
[calendar conversion](https://man7.org/linux/man-pages/man3/localtime.3.html),
[Windows clock guidance](https://learn.microsoft.com/en-us/windows/win32/sysinfo/acquiring-high-resolution-time-stamps),
and [message-aware waiting](https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-msgwaitformultipleobjectsex).

### Structs with trailing arrays

Allow named fields followed by an array length and inline element storage:

```simple
struct vecInt {
    u32 size; // Used elements
    u32 #;    // Capacity
    int [];  // Trailing elements in the same allocation
};
```

The final field would be named `[]`, with an unsigned length field `#`
immediately before it. Element types could include references to the same
struct. Current arrays have a fixed `u32` length and a special two-field
layout; supporting arbitrary leading fields requires parser and layout work.

Consider `u8`, `u16`, `u32`, or `u64` length fields to reduce small-array
overhead. Arrays with different length representations would be distinct
types, with explicit conversions and capacity checks. The spelling and
alignment rules remain to be decided.

Growing such an object requires a replacement allocation. An extra level of
indirection, as in a growable-buffer wrapper, instead preserves the wrapper's
identity. Specify ownership before treating a growing call as consuming or
freeing its receiver.

### Array initialization

Allow an element-producing function at allocation, for example
`new u8[buffer#, { u32 idx -> buffer[idx] }]`. This would support final arrays
and fully initialized non-null reference arrays. Define when each element is
evaluated and when the array can be published. Existing zero-filled array
allocation does not provide this constructor operation.

### String libraries

`sys.aryu8` already provides a growable buffer wrapper; `sys.text` formats into
it and copies the result into an immutable byte array. Extend these foundations
with explicit ownership contracts and UTF-8 decoding. Consider:

- A growable string with inline bytes and cached metadata, returning a
  potentially new receiver when capacity increases.
- Immutable-string ownership and sharing beyond the existing formatting result.
  Read-only access alone does not establish exclusive ownership.
- Unicode decoding and iteration over the byte representation, including
  behavior at invalid character boundaries.
- A C-compatible string with a trailing zero and directly available length.
- A short string packed with its length into a machine word, passed by value.

Keep mutable storage private by default and specify sharing, concatenation,
capacity growth, and native I/O contracts. Existing `sys` byte-array and
buffer helpers provide the basic builder and formatting operations.

### Struct values

Allow a struct to be stored inline in another object, array, or argument list
rather than always through a reference. A proposed `*Complex` or `inline`
spelling would place both coordinates of `struct Complex { f64 x,y; };`
directly in the containing value.

Define whole-value copying, reference-to-value conversion, and assignment
through an existing reference. A value must not silently become a reference
or expose an interior address with an invalid lifetime. Decide how consuming
an enclosing object's lifetime could permit such an address. Keep method
syntax uniform and define the calling convention for by-value arguments.