summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorCliff Click <cliffc@acm.org>2026-10-25 14:24:09 -0700
committerCliff Click <cliffc@acm.org>2026-10-25 14:24:09 -0700
commit0e7c50b22d7f900b9c3c8db3765f3b7af41e5dd1 (patch)
tree6bfc8e100d1e1a727fdeabbc8dcf0710d04ff29b
parent2d80a348bad4b1e3165bd955ceaff3e18a96e516 (diff)
downloadSimple-0e7c50b22d7f900b9c3c8db3765f3b7af41e5dd1.tar.gz
Simple-0e7c50b22d7f900b9c3c8db3765f3b7af41e5dd1.zip
Update incremental build checkmain
-rw-r--r--chapter28b/README.ja.md14
-rw-r--r--chapter28b/README.md19
-rw-r--r--chapter28b/src/main/java/com/seaofnodes/simple/codegen/CompUnit.java42
-rw-r--r--chapter28b/src/main/java/com/seaofnodes/simple/codegen/ParseAll.java10
-rw-r--r--chapter28b/src/test/java/com/seaofnodes/simple/IncrementalBuildTest.java172
-rw-r--r--docs/roadmap.md22
-rw-r--r--skills/incremental-builds.md48
7 files changed, 280 insertions, 47 deletions
diff --git a/chapter28b/README.ja.md b/chapter28b/README.ja.md
index c4f60712..a3221da5 100644
--- a/chapter28b/README.ja.md
+++ b/chapter28b/README.ja.md
@@ -89,6 +89,16 @@ java -ea -cp "build/classes/main:build/classes/test:lib/*" \
## 検証と今後
+明示的に指定したソースは常にコンパイルします。他のローカルオブジェクトを再利用する前に、
+記録された依存関係をたどり、各オブジェクトにまとめられた全ソースを検査します。
+入力の欠落や、より新しいソース・依存オブジェクトがあれば、保存した型や IR を使わず
+再解析します。A が B をインライン化していれば、B のオブジェクトが未更新でも、
+B のソース変更で A は無効になります。循環は訪問済み集合で処理します。
+
+ライブラリは検査と読み込みで同じ検索パスを使います。ソースのないライブラリは整合した
+組み合わせで提供する必要があります。時刻だけでは互換性は判断できません。入力ハッシュ、
+コンパイラ・ABI のバージョン、自動再リンクを含むビルドドライバは今後の課題です。
+
リポジトリのルートから次を実行します。
```sh
@@ -100,6 +110,10 @@ make tests CHAPTERS=chapter28b
オブジェクトにします。`make release` は jar と標準ライブラリを作り、`make view` は
共通のグラフビューアーを起動します。
+`IncrementalBuildTest` は時刻を明示的に設定し、再利用、間接的な変更、まとめられた
+循環、入力の欠落、ライブラリ更新を検査します。変更後のネイティブ実行結果も
+クリーンビルドと比較します。
+
保存形式は `C0DE` 以外のバージョン番号をまだ持ちません。コンパイラや形式を変えたら
オブジェクトを再構築してください。高コストの往復一致検査は無効のままで、古い増分
ビルドの実験 `testModule0` も実行対象外です。
diff --git a/chapter28b/README.md b/chapter28b/README.md
index f464fed4..0a7baa68 100644
--- a/chapter28b/README.md
+++ b/chapter28b/README.md
@@ -116,10 +116,18 @@ normal inlining heuristics. The combined graph goes through the normal
compilation phases, including peepholes, Opto and TypeCheck.
Dependencies between `CompUnit`s are recorded. An explicitly requested source
-file is always compiled. Other source files are recompiled if their object is
-missing, their source is newer, or a recorded dependency's object in the build
-directory is newer. These timestamp checks are not yet a complete incremental
-build system, there is some modest future work needed to finish this.
+file is always compiled. Before reusing another local object, `CompUnit` walks
+its recorded dependencies and checks every source bundled into each local
+object. A missing input or a newer source or dependency object forces parsing
+again, before saved types or IR are imported. This includes transitive
+dependencies: if A has inlined B, editing B's source invalidates A even while
+B's object is unchanged. A visited set lets this check follow cycles.
+
+Library dependencies use the same search paths for checking and loading.
+Object-only libraries must be supplied as a coherent build; timestamps alone
+cannot establish their compatibility. These checks are not a complete build
+driver: explicit requests still compile, and input hashes, compiler/ABI
+versions, and automatic relinking remain future work.
Loading currently works at the object granularity. Unrelated objects can stay
unloaded, but selecting an object loads its entire saved graph and follows its
@@ -163,6 +171,9 @@ constructors, initialization cycles across objects, shared static data, native
function thunks, and native library/client execution. Make also builds the
modular `sys` object used by the library tests. Clients can import that object
with `-L`; they do not need to parse the standard library's source again.
+`IncrementalBuildTest` checks reuse, transitive edits, bundled cycles, missing
+inputs, and changed library objects using controlled timestamps. It also
+compares native execution after an edit with a clean build.
`make release` builds the launcher jar and standard library; `make view`
starts the shared graph viewer.
diff --git a/chapter28b/src/main/java/com/seaofnodes/simple/codegen/CompUnit.java b/chapter28b/src/main/java/com/seaofnodes/simple/codegen/CompUnit.java
index 71126795..8b303008 100644
--- a/chapter28b/src/main/java/com/seaofnodes/simple/codegen/CompUnit.java
+++ b/chapter28b/src/main/java/com/seaofnodes/simple/codegen/CompUnit.java
@@ -60,7 +60,7 @@ public class CompUnit {
( fname.equals(CODE._srcName) || // Explicit input is a compilation request
!_obj.exists() || // No object file
_smp.lastModified() > _obj.lastModified() || // Out-of-date with source?
- checkDependentObjs() ); // Out-of-date with other objs?
+ !current(_obj,new HashSet<>()) ); // Out-of-date anywhere in its inputs?
_src = mustCompile ? new String(Files.readAllBytes(_smp.toPath())) : null;
}
@@ -101,16 +101,42 @@ public class CompUnit {
_src = null;
}
- // Return true if _obj is out-of-date relative to dependents.
- private boolean checkDependentObjs() {
- ElfReader elf = CODE.getElf(_obj);
+ // Check source-backed inputs before importing any saved types or IR.
+ // A dependency's source can be newer while its object is unchanged.
+ // Use a fresh visited set for each root: encountering a cycle is safe, but
+ // caching a provisional "current" answer from inside it would not be.
+ private static boolean current(File obj, HashSet<File> visit) {
+ if( obj==null || !obj.isFile() ) return false;
+ if( !visit.add(obj.getAbsoluteFile()) ) return true;
+ ElfReader elf = CODE.getElf(obj);
+ if( elf==null ) return false;
elf.loadPublicSymbols();
+ if( elf._units==null ) return false;
+ long time = obj.lastModified();
+ for( String unit : elf._units ) {
+ // Never import another copy of a unit explicitly being compiled.
+ CompUnit cu = CODE._compunits.get(unit);
+ if( unit.equals(CODE._srcName) || (cu!=null && cu._src!=null) ) return false;
+ File src = new File(CODE._modDir+"/"+unit+".smp");
+ if( !src.isFile() || src.lastModified()>time ) return false;
+ }
for( String dep : elf._deps ) {
- File obj = new File(CODE._buildDir+"/"+dep+".o");
- if( obj.lastModified() > _obj.lastModified() ) // Out-of-date with deps?
- return true;
+ if( Arrays.asList(elf._units).contains(dep) ) continue; // Owned by this object.
+ File file = dependencyObj(dep,true);
+ if( file==null || !file.isFile() || file.lastModified()>time ) return false;
+ // Object-only libraries are supplied as a coherent build. Their
+ // input versions cannot be inferred from installation timestamps.
+ if( file.equals(new File(CODE._buildDir+"/"+dep+".o")) && !current(file,visit) ) return false;
}
- return false;
+ return true;
+ }
+
+ // Resolve dependencies the same way for freshness checks and actual loads.
+ static File dependencyObj(String fname, boolean local) {
+ File obj = new File(CODE._buildDir+"/"+fname+".o");
+ if( local && (obj.isFile() || new File(CODE._modDir+"/"+fname+".smp").isFile()) ) return obj;
+ ElfReader elf = CODE.findExternalSimple(Parser.addClzPrefix(fname.replace('/','.')));
+ return elf==null ? null : elf._file;
}
// Add a discovered dependency, for later writing out in an ELF file, ".simple" section
diff --git a/chapter28b/src/main/java/com/seaofnodes/simple/codegen/ParseAll.java b/chapter28b/src/main/java/com/seaofnodes/simple/codegen/ParseAll.java
index c55f9a90..d7c0f7e2 100644
--- a/chapter28b/src/main/java/com/seaofnodes/simple/codegen/ParseAll.java
+++ b/chapter28b/src/main/java/com/seaofnodes/simple/codegen/ParseAll.java
@@ -258,12 +258,10 @@ public abstract class ParseAll {
}
private static File depObj(CodeGen code, CompUnit cunit, String fname) {
- if( cunit._smp != null )
- return null;
- ElfReader elf = code.findExternalSimple(Parser.addClzPrefix(fname.replace('/','.')));
- if( elf == null )
- throw new RuntimeException("Cannot find external Simple dependency '"+fname+"'");
- return elf._file;
+ File obj = CompUnit.dependencyObj(fname,cunit._smp!=null);
+ if( obj == null )
+ throw new RuntimeException("Cannot find Simple dependency '"+fname+"'");
+ return cunit._smp!=null && obj.equals(new File(code._buildDir+"/"+fname+".o")) ? null : obj;
}
private static File externalObj(CodeGen code, CompUnit par, String name) {
diff --git a/chapter28b/src/test/java/com/seaofnodes/simple/IncrementalBuildTest.java b/chapter28b/src/test/java/com/seaofnodes/simple/IncrementalBuildTest.java
new file mode 100644
index 00000000..501392f3
--- /dev/null
+++ b/chapter28b/src/test/java/com/seaofnodes/simple/IncrementalBuildTest.java
@@ -0,0 +1,172 @@
+package com.seaofnodes.simple;
+
+import com.seaofnodes.simple.codegen.*;
+import com.seaofnodes.simple.type.TypeInteger;
+import com.seaofnodes.simple.util.Ary;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.attribute.FileTime;
+import org.junit.Test;
+import static org.junit.Assert.*;
+
+public class IncrementalBuildTest {
+ @Test public void testTransitiveSourceChange() throws Exception {
+ Path base=Path.of("build/objs/incremental");
+ Files.createDirectories(base);
+ Path root=Files.createTempDirectory(base,"chain-");
+ Path src=Files.createDirectory(root.resolve("src"));
+ Path obj=Files.createDirectory(root.resolve("obj"));
+ write(src,"App","val marker=0;",1000);
+ write(src,"App/B","val value={ -> 5; };",1000);
+ write(src,"App/A","val value={ -> B.value()+1; };",1000);
+ write(src,"App/C","val value={ -> A.value()+1; };",1000);
+ write(src,"App/Main","val run={ -> C.value(); }; return run();",1000);
+ compile(src,obj,"App");
+ stamp(obj.resolve("App.o"),1500);
+ compile(src,obj,"App/B");
+ stamp(obj.resolve("App/B.o"),2000);
+ compile(src,obj,"App/A");
+ stamp(obj.resolve("App/A.o"),3000);
+ compile(src,obj,"App/C");
+ stamp(obj.resolve("App/C.o"),3500);
+ CodeGen code=client(src,obj);
+ assertNull("Reuse unchanged A",code._compunits.get("App/A")._src);
+ assertNull("Reuse unchanged B",code._compunits.get("App/B")._src);
+ assertNull("Reuse unchanged C",code._compunits.get("App/C")._src);
+ assertEquals("7",Eval2.eval(code,0));
+
+ write(src,"App/B","val value={ -> 7; };",4000);
+ code=client(src,obj);
+ assertEquals("9",Eval2.eval(code,0));
+ assertNotNull(code._compunits.get("App/C")._src);
+ assertNotNull("Reparse A before loading its stale inlined body",code._compunits.get("App/A")._src);
+ assertNotNull(code._compunits.get("App/B")._src);
+ compile(src,obj,"App/Main");
+ checkNative(obj,new Ary<>(new String[]{obj.resolve("App.o").toString()}));
+
+ Path clean=Files.createDirectory(root.resolve("clean"));
+ assertEquals("9",Eval2.eval(client(src,clean),0));
+ compile(src,clean,"App/Main");
+ checkNative(clean,null);
+ }
+
+ @Test public void testBundledSourceChangeAndCycle() throws Exception {
+ Path root=fixture("bundle-");
+ Path src=root.resolve("src"), obj=root.resolve("obj");
+ write(src,"App/A","val value={int n -> n ? B.value(n-1) : 1; };",1000);
+ write(src,"App/B","val value={int n -> n ? A.value(n-1) : 5; };",1000);
+ write(src,"App/Main","return A.value(1);",1000);
+ compile(src,obj,"App/A");
+ stamp(obj.resolve("App/A.o"),3000);
+ assertFalse("A owns both sides of the cycle",Files.exists(obj.resolve("App/B.o")));
+ CodeGen code=client(src,obj);
+ assertEquals("5",Eval2.eval(code,0));
+ assertNull(code._compunits.get("App/A")._src);
+ assertNull(code._compunits.get("App/B")._src);
+
+ write(src,"App/B","val value={int n -> n ? A.value(n-1) : 7; };",4000);
+ code=client(src,obj);
+ assertEquals("7",Eval2.eval(code,0));
+ assertNotNull(code._compunits.get("App/A")._src);
+ assertNotNull(code._compunits.get("App/B")._src);
+ }
+
+ @Test public void testMissingDependency() throws Exception {
+ Path root=fixture("missing-");
+ Path src=root.resolve("src"), obj=root.resolve("obj");
+ write(src,"App/A","val value={ -> B.value()+1; };",1000);
+ write(src,"App/B","val value={ -> 5; };",1000);
+ write(src,"App/Main","return A.value();",1000);
+ compile(src,obj,"App/B");
+ stamp(obj.resolve("App/B.o"),2000);
+ compile(src,obj,"App/A");
+ stamp(obj.resolve("App/A.o"),3000);
+ Files.delete(obj.resolve("App/B.o"));
+ CodeGen code=client(src,obj);
+ assertEquals("6",Eval2.eval(code,0));
+ assertNotNull(code._compunits.get("App/A")._src);
+ assertNotNull(code._compunits.get("App/B")._src);
+ Files.delete(src.resolve("App/B.smp"));
+ try {
+ client(src,obj);
+ fail("Missing source and object must not reuse A's inlined body");
+ } catch( RuntimeException e ) {
+ assertTrue(e.getMessage(),e.getMessage().contains("Undefined name 'B'"));
+ }
+ }
+
+ @Test public void testLibraryDependency() throws Exception {
+ Path root=fixture("library-");
+ Path src=root.resolve("src"), obj=root.resolve("obj");
+ Path lib=Files.createDirectory(root.resolve("lib"));
+ var paths=new Ary<>(new String[]{lib.toString(),obj.toString()});
+ library(lib,5);
+ stamp(lib.resolve("LibB.o"),2000);
+ write(src,"App/A","val value={ -> LibB.value()+1; };",1000);
+ write(src,"App/Main","return A.value();",1000);
+ new CodeGen(src.toString(),obj.toString(),paths,"App/A",null,123,true,TypeInteger.BOT)
+ .driver(TestC.CPU_PORT,TestC.CALL_CONVENTION,false,false);
+ stamp(obj.resolve("App/A.o"),3000);
+ CodeGen code=new CodeGen(src.toString(),obj.toString(),paths,"App/Main",null,123,true,TypeInteger.BOT)
+ .driver(CodeGen.Phase.TypeCheck);
+ assertEquals("6",Eval2.eval(code,0));
+ assertNull("Unchanged library inputs permit reuse",code._compunits.get("App/A")._src);
+
+ library(lib,7);
+ stamp(lib.resolve("LibB.o"),4000);
+ code=new CodeGen(src.toString(),obj.toString(),paths,"App/Main",null,123,true,TypeInteger.BOT)
+ .driver(CodeGen.Phase.TypeCheck);
+ assertEquals("8",Eval2.eval(code,0));
+ assertNotNull(code._compunits.get("App/A")._src);
+ }
+
+ private static void library(Path lib, int value) {
+ new CodeGen(null,lib.toString(),null,"LibB","val value={ -> "+value+"; };",123,true,TypeInteger.BOT)
+ .driver(TestC.CPU_PORT,TestC.CALL_CONVENTION,false,false);
+ }
+
+ private static void checkNative(Path obj, Ary<String> libs) throws Exception {
+ Path driver=obj.resolve("driver.c");
+ Files.writeString(driver,"""
+ __attribute__((CALL_CONV)) extern long long run(void) __asm__("App.Main.run");
+ int main(void) { return run()==9 ? 0 : 1; }
+ """);
+ String exe=obj.resolve(TestC.OS.startsWith("Windows")?"test.exe":"test").toString();
+ TestC.linkExe(obj.resolve("App/Main.o").toString(),
+ TestC.CALL_CONVENTION.equals("win64")?"ms_abi":"sysv_abi",driver.toString(),libs,exe);
+ assertEquals("",TestC.exec(TestC.TEST_TIMEOUT_SECONDS,exe));
+ }
+
+ private static Path fixture(String prefix) throws Exception {
+ Path base=Path.of("build/objs/incremental");
+ Files.createDirectories(base);
+ Path root=Files.createTempDirectory(base,prefix);
+ Path src=Files.createDirectory(root.resolve("src"));
+ Path obj=Files.createDirectory(root.resolve("obj"));
+ write(src,"App","val marker=0;",1000);
+ compile(src,obj,"App");
+ stamp(obj.resolve("App.o"),1500);
+ return root;
+ }
+
+ private static void write(Path src, String name, String text, long time) throws Exception {
+ Path path=src.resolve(name+".smp");
+ Files.createDirectories(path.getParent());
+ Files.writeString(path,text);
+ stamp(path,time);
+ }
+
+ private static void stamp(Path path, long time) throws Exception {
+ Files.setLastModifiedTime(path,FileTime.fromMillis(time));
+ }
+
+ private static void compile(Path src, Path obj, String name) {
+ new CodeGen(src.toString(),obj.toString(),null,name,null,123,true,TypeInteger.BOT)
+ .driver(TestC.CPU_PORT,TestC.CALL_CONVENTION,false,false);
+ }
+
+ private static CodeGen client(Path src, Path obj) {
+ return new CodeGen(src.toString(),obj.toString(),null,"App/Main",null,123,true,TypeInteger.BOT)
+ .driver(CodeGen.Phase.TypeCheck);
+ }
+}
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 62c2e509..75ee8d54 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -34,17 +34,17 @@ signed and unsigned fields and compare ideal evaluation with generated code.
### Reliable incremental builds
-`CompUnit` checks source and dependency-object timestamps when a unit is
-discovered. It can accept A's saved IR before discovering that B, whose body
-was inlined into A, needs rebuilding. An unchanged public signature does not
-make A current: optimization imports bodies and constants too.
-
-Discover the dependency graph and propagate rebuild decisions to a fixed
-point before accepting saved IR. Handle cycles together, bundled source units,
-missing inputs, and objects resolved through library search paths. Record
-compiler/serialization versions, target, ABI, and relevant options; content
-hashes can cover preserved timestamps. Distinguish rebuildable source from
-incompatible source-free libraries.
+Chapter 28b now checks the reachable source-backed inputs before reusing saved
+IR, including transitive dependencies, bundled sources, and missing inputs.
+The remaining work is a build driver with recorded input identities, beyond
+those conservative timestamp checks. An unchanged public signature does not
+establish compatibility: optimization imports bodies and constants too.
+
+Record the actual resolved input paths and versions, compiler/serialization
+versions, target, ABI, and relevant options. Content hashes can cover preserved
+timestamps and changed library search paths. Distinguish rebuildable source
+from incompatible source-free libraries; their installation timestamps alone
+cannot validate a set of objects compiled against one another.
An unchanged build should do no compilation or linking. Publish outputs only
after successful compilation, and relink when an object or other link input
diff --git a/skills/incremental-builds.md b/skills/incremental-builds.md
index 201922a0..050bd1ba 100644
--- a/skills/incremental-builds.md
+++ b/skills/incremental-builds.md
@@ -1,35 +1,47 @@
-# Incremental builds: future work
+# Incremental builds
-Recorded after the Separate Compilation README review, 2026-10-24. This is a
-design outline, not a description of implemented incremental builds.
+Implementation notes and remaining build-driver work, reviewed 2026-10-25.
-## Current gap
+## Current reuse check
-`CompUnit` decides whether to reuse an object when the unit is discovered. It
-checks its source timestamp and its dependencies' existing object timestamps.
-It does not settle the whole dependency graph before accepting saved IR.
+Chapter 28b's `CompUnit` decides whether to reuse a local object by walking its
+recorded dependencies before loading saved types or IR. It checks all sources
+owned by each local object, missing inputs, and dependency object timestamps.
+Dependencies use the same local/library resolution for checking and loading.
For example:
1. A depends on B and has inlined some of B.
2. B's source changes, leaving B's object unchanged.
-3. A client discovers A. A's check sees the old B object and accepts A's object.
-4. Later discovery notices that B needs rebuilding, without reconsidering A.
+3. A client discovers A. The check follows A's recorded dependency to B and
+ finds B's newer source, even though B's object has not changed.
+4. A is reparsed, so its saved inlined copy of B is never accepted.
-The compiler can therefore reuse code based on stale bodies or constants.
+Use a fresh visited set for each candidate object. Cycles terminate safely,
+but a provisional success inside a cycle must not be cached for another root.
+Dependencies owned by the same object are checked through its bundled sources.
An unchanged public signature is insufficient: Simple imports implementation
details for optimization as well as declarations for type checking.
+Object-only libraries are still supplied as a coherent build. Do not reject
+them merely because one dependency has a newer installation timestamp: existing
+tests compile cyclic libraries against stubs and then combine their real
+objects. Validating those combinations requires saved input versions, not a
+guess from file times. A newer library object does invalidate a source-backed
+local dependent that can be rebuilt.
+
+`IncrementalBuildTest` covers unchanged reuse, transitive source edits, bundled
+cycles, missing inputs, library dependencies, and clean/incremental native
+execution parity. It sets file times directly, without sleeps.
+
## Required pieces
-- Discover dependencies, mark changed units, and propagate that decision to
- dependents until it settles. Handle cycles as groups. Do this before loading
- IR that may be stale; newly discovered dependencies must also join the plan.
-- Record actual inputs. An object can own several source units, so track all
- of their sources and the objects resolved through library search paths.
- The current check assumes `buildDir/name.o`; a missing file has timestamp
- zero and does not trigger rebuilding. Missing inputs must invalidate or
- produce a useful error rather than silently accepting an old result.
+- Plan object targets and rebuilds across invocations. Handle cycles as groups
+ and newly discovered dependencies. The compiler's per-object reuse check is
+ not a complete scheduling or linking policy.
+- Record actual input identities. An object can own several source units, so
+ track all of their sources and resolved library paths and versions. A search
+ path change can select a different library even with an older timestamp.
- Record compiler and serialization versions, target, ABI, and relevant
options. Content hashes would also detect changes with preserved timestamps.
Specify which changes require rebuilding source and which reject an