diff options
| author | Cliff Click <cliffc@acm.org> | 2026-09-30 19:25:04 -0700 |
|---|---|---|
| committer | Cliff Click <cliffc@acm.org> | 2026-09-30 19:25:04 -0700 |
| commit | c4dcf6f751b35c7a2a3b46fa2d69b95d37d082a6 (patch) | |
| tree | e5ad31706399578310cfbb3b0387a77cf83372ba | |
| parent | a996996a1ad776089a4cb9cbf7744f5f4e8dde6d (diff) | |
| download | Simple-c4dcf6f751b35c7a2a3b46fa2d69b95d37d082a6.tar.gz Simple-c4dcf6f751b35c7a2a3b46fa2d69b95d37d082a6.zip | |
Finish forward-port mem ops thru ch25
119 files changed, 4410 insertions, 1839 deletions
@@ -83,10 +83,11 @@ The top-level Makefile runs each chapter's `tests`, `tags` (also `tag`), The [chapter backport queue](docs/chapter-backports.md) records proposed small corrections and the per-chapter test/review workflow. Larger architectural moves are tracked separately there. The 10a/10b split is complete; forwarding 10b's -memory representation now runs through Chapter 21, including constructors, -mutability, whole-memory function calls, instruction selection, and register -allocation and machine-code execution. Chapter 21 is ready for review; Chapters 22-24 still use their earlier -parser-managed alias chains. +memory representation now runs through Chapter 24, including constructors, +mutability, whole-memory function calls, machine-code execution, cyclic field +types, and SCCP. Chapters 22-25 are ready for review. Chapter 25 includes the +related Load-search, Phi-factoring, and scheduling corrections while retaining +its separate constructor-memory and incomplete-type SSA design. To build and test just the memory chapters: diff --git a/chapter22/README.md b/chapter22/README.md index 96c02ef1..426db34f 100644 --- a/chapter22/README.md +++ b/chapter22/README.md @@ -29,6 +29,12 @@ Let's break it down. pool*, and from there in an ELF file's RODATA section. +Chapter 10b's lazy memory partitioning continues here: the parser carries one +`$mem`, and BulkMemPhi/MemPhi discover field aliases during optimization. +Allocations consume a partial MemMerge and produce `{ptr, $mem}`; calls carry +whole memory. The loop Load search follows every arm of a memory merge, so +BrainFuck's invariant program length folds out of its main loop. + You can also read [this chapter](https://github.com/SeaOfNodes/Simple/tree/linear-chapter22) in a linear Git revision history on the [linear](https://github.com/SeaOfNodes/Simple/tree/linear) branch and [compare](https://github.com/SeaOfNodes/Simple/compare/linear-chapter21...linear-chapter22) it to the previous chapter. @@ -192,7 +198,7 @@ class is left for Chapter 23, cold loop splits for 24, and area/cost ranking for Chapter 21's adjacent-copy forwarding also carries forward: after coloring, a sole use in the next instruction can read the source register when its operand mask permits it and no two-address tie is broken. The table below -includes this cleanup. Cohort 22 now also includes the two zero-move C return +includes this cleanup and the lazy memory forward port. Cohort 22 now also includes the two zero-move C return ABI checks added since the earlier audit (26 entries instead of 24). Run `make spill-stats` in this directory. Each row below uses **this chapter's @@ -202,10 +208,10 @@ SystemV. Diagnostic graphs and mask regressions do not contribute to the totals. | Program cohort | Compilations | Retained moves | Loop-weighted moves | |---|---:|---:|---:| -| Chapter 20 | 39 | 321 | 433 | -| Chapter 21 | 52 | 438 | 970 | +| Chapter 20 | 39 | 324 | 443 | +| Chapter 21 | 52 | 432 | 964 | | Chapter 22 | 26 | 67 | 67 | -| **Total** | **117** | **826** | **1,470** | +| **Total** | **117** | **823** | **1,474** | `_spills` counts retained SplitNodes, including register moves; `_spillScaled` weights each by `8^loopDepth`. These are compiler estimates, not measured runtime @@ -225,7 +231,9 @@ The original Chapter 22 snapshot, which mixed several later heuristics, produced staged version saves 240 weighted moves (13.8%). Neither comparison should be confused with comparing entire compiler chapters: lowering also changes. For example, the frozen Chapter 20 String input has no explicit return, and Chapter -22's default-return handling eliminates that workload at the default seed. +22's default-return handling eliminated that workload in the earlier audit. +With lazy memory, some target compilations retain callable bodies; the current +table includes their moves. The full suite and statistics runner pass. Routine backend comparisons use a fixed optimizer seed: diff --git a/chapter22/src/main/java/com/seaofnodes/simple/Parser.java b/chapter22/src/main/java/com/seaofnodes/simple/Parser.java index 8132f782..7a60a2dc 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/Parser.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/Parser.java @@ -137,7 +137,7 @@ public class Parser { _scope.define(ScopeNode.ARG0, TypeInteger.BOT, false, null, _lexer); ctrl(XCTRL); - _scope.mem(new MemMergeNode(false)); + _scope.mem(con(TypeMem.BOT)); // Parse the sys import _lexer = new Lexer(com.seaofnodes.simple.sys.SYS); @@ -207,7 +207,7 @@ public class Parser { // Build a multi-exit return point for all function returns RegionNode r = new RegionNode((Lexer)null,null,null).init(); assert r.inProgress(); - PhiNode rmem = new PhiNode(ScopeNode.MEM0,TypeMem.BOT,r,null).init(); + PhiNode rmem = new BulkMemPhiNode(ScopeNode.MEM0,r,null).init(); PhiNode rrez = new PhiNode(ScopeNode.ARG0,Type.BOTTOM,r,null).init(); ReturnNode ret = new ReturnNode(r, rmem, rrez, rpc, fun).init(); fun.setRet(ret); @@ -219,11 +219,8 @@ public class Parser { // external calls. _scope.push(ScopeNode.Kind.Function); ctrl(fun); // Scope control from function - // Private mem alias tracking per function - MemMergeNode mem = new MemMergeNode(true); - mem.addDef(null); // Alias#0 - mem.addDef(new ParmNode(ScopeNode.MEM0,1,TypeMem.BOT,fun,con(TypeMem.BOT)).peephole()); // All aliases - _scope.mem(mem); + // Unknown callers supply whole memory; field aliases are discovered lazily. + _scope.mem(new ParmNode(ScopeNode.MEM0,1,TypeMem.BOT,fun,con(TypeMem.BOT)).peephole()); // All args, "as-if" called externally for( int i=0; i<ids.length; i++ ) { Type t = sig.arg(i); @@ -238,7 +235,7 @@ public class Parser { // Last expression is the return except for the top-level main if( ctrl()._type==Type.CONTROL ) if( fun.sig() != _code._main ) - fun.addReturn(ctrl(), _scope.mem().merge(), last); + fun.addReturn(ctrl(), _scope.mem(), last); else { fun.setDef(1,XCTRL); // Kill default main _code.addAll(fun._outputs); @@ -604,9 +601,8 @@ public class Parser { */ private Node parseReturn() { var expr = require(parseAsgn(), ";"); - // Need default memory, since it can be lazy, need to force - // a non-lazy Phi - _fun.addReturn(ctrl(), _scope.mem().merge(), expr); + // Force the lazy whole-memory Phi before closing this exit. + _fun.addReturn(ctrl(), _scope.mem(), expr); ctrl(XCTRL); // Kill control return expr; } @@ -1272,29 +1268,31 @@ public class Parser { Field[] fs = obj._fields; if( fs==null ) throw error("Unknown struct type '" + obj._name + "'"); - int len = fs.length; - Node[] ns = new Node[2+len]; - ns[0] = ctrl(); // Control in slot 0 - // Total allocated length in bytes - ns[1] = size; - // Memory aliases for every field - for( int i = 0; i < len; i++ ) - ns[2+i] = memAlias(fs[i]._alias); - Node nnn = new NewNode(TypeMemPtr.make(obj), ns).peephole().keep(); - for( int i = 0; i < len; i++ ) - memAlias(fs[i]._alias, new ProjNode(nnn,i+2,memName(fs[i]._alias)).peephole()); - Node ptr = new ProjNode(nnn.unkeep(),1,obj._name).peephole().keep(); - - // Initial nonzero values for every field - for( int i = 0; i < len; i++ ) { - Node val = init.get( i + idx ); + Node prior = mem().keep(); + MemMergeNode input = new MemMergeNode((Node)null); + for( Field f : fs ) input.alias(f._alias,prior); + // Allocation zeroes its fields; nonzero initializers become explicit Stores. + Node[] ns = new Node[3]; + ns[0] = ctrl(); + ns[1] = input.peephole(); + ns[2] = size; + Node nnn = new NewNode(TypeMemPtr.make(obj),ns).peephole().keep(); + Node ptr = new ProjNode(nnn,0,obj._name).peephole().keep(); + Node out = new ProjNode(nnn,1,"$mem").peephole().keep(); + MemMergeNode after = new MemMergeNode(prior); + for( Field f : fs ) after.alias(f._alias,out); + mem(after.peephole()); + out.unkeep(); + nnn.unkeep(); + prior.unkeep(); + for( int i=0; i<fs.length; i++ ) { + Node val = init.get(idx+i); if( val._type != val._type.makeZero() ) { - Node mem = memAlias(fs[i]._alias); - Node st = new StoreNode(loc(),fs[i]._fname,fs[i]._alias,fs[i]._type,mem,ptr,con(obj.offset(i)),val,true).peephole(); - memAlias(fs[i]._alias,st); + Field f = fs[i]; + Node st = new StoreNode(loc(),f._fname,f._alias,f._type,mem(),ptr,con(obj.offset(i)),val,true).peephole(); + mem(new MemMergeNode(mem(),f._alias,st).peephole()); } } - return ptr.unkeep(); } @@ -1319,13 +1317,18 @@ public class Parser { return con(str); } - // We set up memory aliases by inserting special vars in the scope these - // variables are prefixed by $ so they cannot be referenced in Simple code. - // Using vars has the benefit that all the existing machinery of scoping - // and phis work as expected - private Node memAlias(int alias ) { return _scope.mem(alias ); } - private void memAlias(int alias, Node st) { _scope.mem(alias, st); } - public static String memName(int alias) { return ("$"+alias).intern(); } + // Memory is one hidden SSA variable, including across branches and loops. + private Node mem() { return _scope.mem(); } + private void mem(Node n) { _scope.mem(n); } + + private void store(String name, int alias, Type glb, Node ptr, Node off, Node val, boolean init, Node ctrl) { + Node prior = mem().keep(); + Node st = new StoreNode(loc(),name,alias,glb,prior,ptr,off,val,init); + st.setDef(0,ctrl); + st = st.peephole(); + mem(new MemMergeNode(prior,alias,st).peephole()); + prior.unkeep(); + } /** * Parse postfix expression; this can be a field expression, an array @@ -1366,6 +1369,7 @@ public class Parser { if( tf instanceof TypeMemPtr ftmp && ftmp.isFRef() ) tf = ftmp.makeFrom(((TypeMemPtr)(TYPES.get(ftmp._obj._name)))._obj); + expr.keep(); // Field offset; fixed for structs, computed for arrays Node off = (name.equals("[]") // If field is an array body // Array index math @@ -1378,16 +1382,11 @@ public class Parser { Node val = parseAsgn().keep(); Node lift = liftExpr( val, tf, f._final, false ); - Node st = new StoreNode(loc(), name, f._alias, tf, memAlias(f._alias), expr, off.unkeep(), lift, false); - // Arrays include control, as a proxy for a safety range check. - // Structs don't need this; they only need a NPE check which is - // done via the type system. - if( base.isAry() ) st.setDef(0,ctrl()); - memAlias(f._alias, st.peephole()); + store(name,f._alias,tf,expr.unkeep(),off.unkeep(),lift,false,base.isAry() ? ctrl() : null); return val.unkeep(); // "obj.a = expr" returns the expression while updating memory } - Node load = new LoadNode(loc(),name, f._alias, tf, memAlias(f._alias), expr.keep(), off); + Node load = new LoadNode(loc(),name, f._alias, tf, mem(), expr, off); // Arrays include control, as a proxy for a safety range check // Structs don't need this; they only need a NPE check which is // done via the type system. @@ -1398,12 +1397,7 @@ public class Parser { char ch = _lexer.matchOperAssign(); if( ch!=0 ) { Node op = opAssign(ch,load,f._final,tf,name); - Node st = new StoreNode(loc(), name, f._alias, tf, memAlias(f._alias), expr.unkeep(), off, op, false); - // Arrays include control, as a proxy for a safety range check. - // Structs don't need this; they only need a NPE check which is - // done via the type system. - if( base.isAry() ) st.setDef(0,ctrl()); - memAlias(f._alias, peep(st)); + store(name,f._alias,tf,expr.unkeep(),off,op,false,base.isAry() ? ctrl() : null); load = postfix(ch) ? load.unkeep() : op; // And use the original loaded value as the result } else expr.unkill(); @@ -1484,7 +1478,7 @@ public class Parser { } // Control & memory after parsing args args.set(0,ctrl().keep()); - args.set(1,_scope.mem().merge().keep()); + args.set(1,_scope.mem().keep()); args.push(expr); // Function pointer // Unkeep them all for( Node arg : args ) @@ -1505,11 +1499,8 @@ public class Parser { call.peephole(); // Rerun peeps after CallEnd, allows early inlining // Control from CallEnd ctrl(new CProjNode(cend,0,ScopeNode.CTRL).peephole()); - // Memory from CallEnd - MemMergeNode mem = new MemMergeNode(true); - mem.addDef(null); // Alias#0 - mem.addDef(new ProjNode(cend,1,ScopeNode.MEM0).peephole()); - _scope.mem(mem); + // A call may update any alias, so its memory result is the new whole state. + _scope.mem(new ProjNode(cend,1,ScopeNode.MEM0).peephole()); // Call result return new ProjNode(cend,2,"#2").peephole(); } diff --git a/chapter22/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java b/chapter22/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java index 68b310bf..21a8ad76 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java @@ -50,7 +50,7 @@ public abstract class GlobalCodeMotion { var cons = new ArrayList<Node>(); for( Node con : start._outputs ) if( con!=null && !(con instanceof CFGNode) && - (con.isConst() || con instanceof MachNode mach && mach.outregmap()!=null) ) { + (con.isConst() || (con instanceof MachNode mach && mach.outregmap()!=null)) ) { globals.put(con,true); cons.add(con.keep()); // Preserve original inputs while rewiring users. } @@ -172,16 +172,8 @@ public abstract class GlobalCodeMotion { // Loads need their memory inputs' uses also done if( n instanceof MemOpNode ld && ld._isLoad ) - for( Node memuse : ld.mem()._outputs ) - if( late[memuse._nid]==null && - // New makes new memory, never crushes load memory - !(memuse instanceof NewNode) && - // Load-use directly defines memory - (memuse._type instanceof TypeMem || - // Load-use directly defines memory - memuse instanceof CallNode || - // Load-use indirectly defines memory - (memuse._type instanceof TypeTuple tt && tt._types[ld._alias] instanceof TypeMem)) ) + for( Node memuse : ld.antiDeps() ) + if( late[memuse._nid]==null ) continue outer; // All uses done, schedule @@ -193,9 +185,7 @@ public abstract class GlobalCodeMotion { for( Node def : n._inputs ) { if( def==null ) continue; if( late[def._nid]==null ) work.push(def); - for( Node out : def._outputs ) - if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) - work.push(ld); + wakeLoads(def,late,work,new BitSet()); } if( n instanceof LoopNode loop ) for( Node phi : loop._outputs ) @@ -255,23 +245,32 @@ public abstract class GlobalCodeMotion { best instanceof IfNode; } + // An allocation may consume a load's memory through a partial aggregate. + private static void wakeLoads(Node def, CFGNode[] late, WorkList<Node> work, BitSet visit) { + if( visit.get(def._nid) ) return; + visit.set(def._nid); + for( Node out : def._outputs ) + if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) work.push(ld); + if( def instanceof MemMergeNode ) + for( int i=1; i<def.nIns(); i++ ) + if( def.in(i)!=null ) wakeLoads(def.in(i),late,work,visit); + } + private static CFGNode find_anti_dep(CFGNode lca, MemOpNode load, CFGNode early, CFGNode[] late, int[] anti) { // We could skip final-field loads here. // Walk LCA->early, flagging Load's block location choices for( CFGNode cfg=lca; early!=null && cfg!=early.idom(); cfg = cfg.idom() ) anti[cfg._nid] = load._nid; // Walk load->mem uses, looking for Stores causing an anti-dep - for( Node mem : load.mem()._outputs ) { + for( Node mem : load.antiDeps() ) { switch( mem ) { case MemOpNode st: - if( !st._isLoad && load._alias == st._alias ) { - assert late[mem._nid] != null; - lca = anti_dep(load,late[mem._nid],lca,st,anti); - } - break; // Loads do not cause anti-deps on other loads - case CallNode call: - assert late[call._nid]!=null; - lca = anti_dep(load,late[call._nid],lca,call,anti); + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); + break; + case CallNode st: + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); break; case PhiNode phi: // Repeat anti-dep for matching Phi inputs. @@ -280,10 +279,10 @@ public abstract class GlobalCodeMotion { if( phi.in(i)==load.mem() ) lca = anti_dep(load,phi.region().cfg(i),lca,null,anti); break; - case NewNode st: break; - case ReturnNode ret: break; // Load must already be ahead of Return - case MemMergeNode ret: break; // Mem uses now on ScopeMin - case NeverNode never: break; + case NewNode st: + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); + break; default: throw Utils.TODO(); } } @@ -295,7 +294,7 @@ public abstract class GlobalCodeMotion { // Stores are already placed. Constrain the load at that block only. if( anti[stblk._nid]==load._nid ) { lca = stblk.domLCA(lca,null); - if( lca==stblk && st!=null && st._inputs.find(load) == -1 ) + if( lca==stblk && st!=null && !(st instanceof CallNode) && st._inputs.find(load) == -1 ) st.addDef(load); // Same-block load must precede the store. } return lca; diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java new file mode 100644 index 00000000..7d4f1fb8 --- /dev/null +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java @@ -0,0 +1,240 @@ +package com.seaofnodes.simple.node; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.type.Type; +import com.seaofnodes.simple.type.TypeMem; +import java.util.BitSet; + +/** + * A Phi for bulk memory. It covers all aliases except {@link #_aliases}; + * those aliases must be represented by parallel precise MemPhis at the same + * program slice. + */ +public class BulkMemPhiNode extends PhiNode { + + private static final BitSet EMPTY = new BitSet(); + + // Excluded Aliases; empty means NO exclusions. + // Each split clones the exclusions; published sets are never mutated. + public final BitSet _aliases; + + public BulkMemPhiNode(String label, Node... inputs) { + this(label,EMPTY,inputs); + } + + private BulkMemPhiNode(String label, BitSet aliases, Node... inputs) { + super(label,TypeMem.BOT,inputs); + _aliases = aliases; + } + + public boolean isSplit(int alias) { return _aliases.get(alias); } + + public BulkMemPhiNode(BulkMemPhiNode phi) { super(phi); _aliases = phi._aliases; } + + @Override public String label() { return "BulkPhi"+_aliases; } + + @Override + public Type compute() { + Type t = super.compute(); + return t instanceof TypeMem mem ? mem : + t==Type.BOTTOM ? TypeMem.BOT : + TypeMem.TOP; + } + + @Override + public Node idealize() { + Node progress = super.idealize(); + if( progress != null ) return progress; + if( ((RegionNode)region()).inProgress() || region().nIns()<=1 ) + return null; // Input is in-progress + + // "Peek through" a MemMerge that covers this alias set on its default + for( int i=1; i<nIns(); i++ ) + if( in(i) instanceof MemMergeNode mmm && canPeek(mmm) ) { + setDef(i,CodeGen.CODE.add(mmm.in(1))); + return this; + } + + // If any input or output uses or defines a specific alias we cover, + // make a private MemPhi for it, and add it to the excluded list. + for( int i=1; i<nIns(); i++ ) { + int alias = inputAlias(in(i)); + if( alias!=0 ) + return slice(alias); + } + + for( int i=0; i<_outputs.size(); i++ ) { + int alias = outputAlias(_outputs.get(i)); + if( alias!=0 ) + return slice(alias); + } + + return null; + } + + static public boolean checkMem(Node n, int alias) { + for( Node m : n._inputs ) + // Bulk memory supplies this alias + if( m instanceof BulkMemPhiNode bulk ) { + if( bulk._aliases.get(alias) ) + return false; + } else if( m instanceof MemMergeNode mmm && + mmm.in(1) instanceof BulkMemPhiNode bulk && + bulk._aliases.get(alias) && + (alias >= mmm.nIns() || mmm.in(alias)==null) ) + return false; + return true; + } + + + // True if the MemMerge has no precise slice still covered by this Phi. + private boolean canPeek(MemMergeNode mmm) { + return mmm.in(1)!=null && missingAlias(mmm)==0; + } + + // Return an alias required by an input but still covered by this Phi, or 0. + private int inputAlias(Node n) { + return switch(n) { + case ProjNode proj -> 0; + case ParmNode parm -> 0; + case ConstantNode con -> 0; + case MemMergeNode mmm -> missingAlias(mmm); + case MemOpNode mem -> unsplit(mem._alias); + case BulkMemPhiNode bulk -> missingAlias(bulk); + default -> throw new AssertionError("Unexpected memory input "+n); + }; + } + + // Return an alias required by a user but still covered by this Phi, or 0. + private int outputAlias(Node use) { + // This decision depends backwards on an immediate user's alias/set. + // Normal graph-neighbor enqueueing flows from defs to uses, so retain + // an explicit forward dependency to revisit this Phi when the user + // sharpens. + addDepForwards(use); + return switch(use) { + case ScopeNode scope -> 0; + case MemMergeNode mmm -> missingAlias(mmm); + case BulkMemPhiNode bulk -> missingAlias(bulk); + case MemOpNode mem -> unsplit(mem._alias); + case MemPhiNode phi -> unsplit(phi._alias); + default -> { + assert use instanceof ReturnNode || use instanceof CallNode || use instanceof ParmNode + : "Unexpected bulk-memory user "+use; + yield 0; + } + }; + } + + // Alias zero/one is bulk, and an already excluded precise alias cannot + // trigger another split of this Phi. + private int unsplit(int alias) { + if( alias <= 1 ) return 0; + assert !_aliases.get(alias); + return alias; + } + + // First alias excluded by the neighboring bulk Phi but not by this one. + private int missingAlias(BulkMemPhiNode bulk) { + for( int alias = bulk._aliases.nextSetBit(0); alias >= 0; alias = bulk._aliases.nextSetBit(alias+1) ) + if( !_aliases.get(alias) ) + return alias; + return 0; + } + + // Every explicit slot requests that alias, including a partial allocation + // input whose slot still points to this bulk Phi. + private int missingAlias(MemMergeNode mmm) { + for( int alias=2; alias<mmm.nIns(); alias++ ) + if( mmm.in(alias)!=null && + mmm.alias(alias)!=mmm.in(1) && + !_aliases.get(alias) ) + return alias; + return 0; + } + + // Slice out given alias + private MemMergeNode slice( int alias ) { + assert !_aliases.get(alias); + // Inputs are the same; MemPhi will sharpen his own inputs. If a + // parallel precise Phi already exists, reuse it instead of building a + // duplicate for the same Region/alias pair. + Node mem = precisePhi(alias); + + BitSet aliases = ((BitSet)_aliases.clone()); + aliases.set(alias); + BulkMemPhiNode bphi = new BulkMemPhiNode(_label,aliases); + for( int i=0; i<nIns(); i++ ) + bphi.addDef(in(i)); + // Installing bphi creates a new bulk user with a different exclusion + // set. Queue the original memory inputs before bphi itself can peep + // away and hide those neighbor relationships. + for( int i=1; i<bphi.nIns(); i++ ) + CodeGen.CODE.add(bphi.in(i)); + // Do not peephole bphi here, as BulkMemPhi can recursively start a + // second bulk rewrite before this one has finished. Let the worklist + // discover any further splits + bphi.setType(TypeMem.BOT); + Node bulk = bphi; + CodeGen.CODE.add(bulk); + CodeGen.CODE.add(mem); + return aggregate(bulk,alias,mem); + } + + // Build aggregate memory over a bulk Phi, preserving every precise slice + // which the bulk has already excluded. The named alias is newly supplied + // (or replaces its old slice). + MemMergeNode aggregate(Node bulk, int alias, Node precise) { + MemMergeNode mmm = new MemMergeNode(bulk); + for( int old = _aliases.nextSetBit(0); old >= 0; old = _aliases.nextSetBit(old+1) ) + { assert old != alias; mmm.alias(old,precisePhi(old));} + mmm.alias(alias,precise); + assert checkMerge(mmm); + return mmm; + } + + // Find or make the precise Phi parallel to this bulk Phi at the same merge. + private Node precisePhi(int alias) { + MemPhiNode mphi = _findPhi(alias); + if( mphi!=null ) return mphi; + + mphi = new MemPhiNode("$"+alias,alias); + mphi.addDef(region()); + // Due to cycles, must set before calling peephole + mphi.setType(TypeMem.make(alias,Type.BOTTOM)); + for( int i=1; i<nIns(); i++ ) + mphi.addDef(CodeGen.CODE.add(preciseInput(in(i),alias))); + return CodeGen.CODE.add(mphi); + } + + MemPhiNode _findPhi(int alias) { + for( Node use : region()._outputs ) + if( use instanceof MemPhiNode mphi && mphi._alias==alias ) + return mphi; + return null; + } + + // Select an alias already split out of a predecessor's bulk memory. + private Node preciseInput(Node mem, int alias) { + if( mem instanceof MemMergeNode mmm ) + return mmm.alias(alias); + if( mem instanceof BulkMemPhiNode bulk && bulk.isSplit(alias) ) + return bulk.precisePhi(alias); + return mem; + } + + // Every alias excluded by a bulk default must be supplied explicitly. + static boolean checkMerge(MemMergeNode mmm) { + if( !(mmm.in(1) instanceof BulkMemPhiNode bulk) ) return true; + for( int alias = bulk._aliases.nextSetBit(0); alias >= 0; alias = bulk._aliases.nextSetBit(alias+1) ) + if( alias >= mmm.nIns() || mmm.in(alias)==null ) + return false; + return true; + } + + @Override public boolean eq(Node n) { + return _aliases.equals(((BulkMemPhiNode)n)._aliases) && super.eq(n); + } + + @Override int hash() { return _aliases.hashCode(); } +} diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/LoadNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/LoadNode.java index e0a6d09a..0432bcbc 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/LoadNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/LoadNode.java @@ -1,7 +1,7 @@ package com.seaofnodes.simple.node; +import com.seaofnodes.simple.codegen.CodeGen; import com.seaofnodes.simple.Parser; -import com.seaofnodes.simple.Utils; import com.seaofnodes.simple.type.*; import java.util.BitSet; @@ -23,6 +23,25 @@ public class LoadNode extends MemOpNode { super(loc, name, alias, true, glb, mem, ptr, off); } + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && !clobbered(dep); + } + + // Check only immediate memory users. Writers clobber memory; Phis and + // aggregates might lead to a clobber, so stop rather than search further. + private boolean clobbered(Node dep) { + Node mem = mem(); + dep.addDepForwards(mem); + for( Node use : mem._outputs ) { + if( use==null ) continue; + dep.addDepForwards(use); + if( use instanceof StoreNode || use instanceof PhiNode || + use instanceof MemMergeNode || + use instanceof NewNode || use instanceof CallNode ) return true; + } + return false; + } + // Debugger label @Override public String label() { return "ld_"+mlabel(); } @Override @@ -30,39 +49,24 @@ public class LoadNode extends MemOpNode { @Override public Type compute() { - if( mem()._type instanceof TypeMem mem ) { - // Update declared forward ref to the actual - if( _declaredType.isFRef() && mem._t instanceof TypeMemPtr tmp && !tmp.isFRef() ) - _declaredType = tmp; - // No lifting if ptr might null-check - if( err()==null ) - return _declaredType.join(mem._t); - } - return _declaredType; + Type t = MemMergeNode.contents(mem(),_alias,this); + // Update declared forward ref to the actual. + if( _declaredType.isFRef() && t instanceof TypeMemPtr tmp && !tmp.isFRef() ) + _declaredType = tmp; + return err()==null ? _declaredType.join(t) : _declaredType; } @Override public Node idealize() { - Node ptr = ptr(); - Node mem = mem(); - - // Simple Load-after-Store on same address. - if( mem instanceof StoreNode st && - ptr == st.ptr() && off() == st.off() ) { // Must check same object - assert _name.equals(st._name); // Equiv class aliasing is perfect - return extend(st.val()); - } - - // Simple load-after-MemMerge to a known alias can bypass. Happens when inlining. - if( mem instanceof MemMergeNode mem2 ) { - setDef(1,mem2.alias(_alias)); + if( mem() instanceof MemMergeNode merge ) { + setDef(1,CodeGen.CODE.add(merge.alias(_alias))); return this; } + Node ptr = ptr(); + Node mem = mem(); - // Simple Load-after-New on same address. - if( mem instanceof ProjNode p && p.in(0) instanceof NewNode nnn && - ptr == nnn.proj(1) ) // Must check same object - return zero(nnn); // Load zero from new + Node win = find(mem,ptr,null,null); + if( win!=null ) return folded(win); // Uplift control to a prior dominating load. for( Node memuse : mem._outputs ) @@ -71,49 +75,7 @@ public class LoadNode extends MemOpNode { cfg0()!=null && cfg0().domLCA(ld.cfg0(),this) == ld.cfg0() ) // Higher control means load is legal earlier return ld; - // Load-after-Store on same address, but bypassing provably unrelated - // stores. This is a more complex superset of the above two peeps. - // "Provably unrelated" is really weak. - if( ptr instanceof ReadOnlyNode ro ) - ptr = ro.in(1); - outer: - while( true ) { - switch( mem ) { - case StoreNode st: - if( ptr == addDep(st.ptr()) && off() == st.off() ) - return extend(castRO(st.val())); // Proved equal - // Can we prove unequal? Offsets do not overlap? - if( !off()._type.join(st.off()._type).isHigh() && // Offsets overlap - !neverAlias(ptr,st.ptr()) ) // And might alias - break outer; // Cannot tell, stop trying - // Pointers cannot overlap - mem = st.mem(); // Proved never equal - break; - case PhiNode phi: - // Assume related - addDep(phi); - break outer; - case ConstantNode top: break outer; // Assume shortly dead - case ProjNode mproj: // Memory projection - switch( mproj.in(0) ) { - case NewNode nnn1: - if( ptr instanceof ProjNode pproj && pproj.in(0) == mproj.in(0) ) - return zero(nnn1); - if( !(ptr instanceof ProjNode pproj && pproj.in(0) instanceof NewNode) ) - break outer; // Cannot tell, ptr not related to New - mem = nnn1.in(nnn1.findAlias(_alias));// Bypass unrelated New - break; - case StartNode start: break outer; - case CallEndNode cend: break outer; // TODO: Bypass no-alias call - default: throw Utils.TODO(); - } - break; - case MemMergeNode merge: mem = merge.alias(_alias); break; - - default: - throw Utils.TODO(); - } - } + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); // Push a Load up through a Phi, as long as it collapses on at least // one arm. If at a Loop, the backedge MUST collapse - else we risk @@ -122,19 +84,27 @@ public class LoadNode extends MemOpNode { // if( pred ) ptr.x = e0; val = pred ? e0 // else ptr.x = e1; : e1; // val = ptr.x; ptr.x = val; - if( mem() instanceof PhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + if( mem() instanceof MemPhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + // Array access control must not be moved across the merge. + in(0)==null && // Offset can be hoisted off() instanceof ConstantNode && // Pointer can be hoisted hoistPtr(ptr,memphi) ) { - // Profit on RHS/Loop backedge - if( profit(memphi,2) || - // Else must not be a loop to count profit on LHS. - (!(memphi.region() instanceof LoopNode) && profit(memphi,1)) ) { - Node ld1 = ld(1); - Node ld2 = ld(2); - return new PhiNode(_name,_declaredType,memphi.region(),ld1,ld2); + // Returning to this memory means no change only if the pointer + // is also loop invariant. A pointer Phi may select another object. + Node stop = memphi.region() instanceof LoopNode && + !(ptr instanceof PhiNode p && p.region()==memphi.region()) ? memphi : null; + if( find(memphi.in(2),ptr,stop,new BitSet())!=null || + (!(memphi.region() instanceof LoopNode) && find(memphi.in(1),ptr,null,new BitSet())!=null) ) { + PhiNode phi = new PhiNode(_name,_declaredType,memphi.region()).keep(); + phi.setType(_type); // Backedge searches may return this value Phi. + for( int i=1; i<memphi.nIns(); i++ ) { + Node p = ptr instanceof PhiNode pp && pp.region()==memphi.region() ? pp.in(i) : ptr; + phi.addDef(load(memphi.in(i),p,stop,phi)); + } + return CodeGen.CODE.add(phi.unkeep()); } } @@ -149,9 +119,66 @@ public class LoadNode extends MemOpNode { return castRO(new ConstantNode(zero).peephole()); } - private Node ld( int idx ) { - Node mem = mem(), ptr = ptr(); - return new LoadNode(_loc,_name,_alias,_declaredType,mem.in(idx),ptr instanceof PhiNode && ptr.in(0)==mem.in(0) ? ptr.in(idx) : ptr, off()).peephole(); + // Search only; no new nodes or rewiring. Return a Store/New that folds, + // the unchanged loop memory, or a merge whose every arm folds. Null fails. + // Register dependencies so a later pointer/offset rewrite retries the query. + private Node find(Node mem, Node ptr, Node stop, BitSet visit) { + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); + while( mem!=null ) { + addDep(mem); + if( mem==stop ) return mem; + switch( mem ) { + case MemMergeNode merge: mem=merge.alias(_alias); break; + case StoreNode st: + addDep(st.ptr()); + if( _alias==st._alias && ptr==st.ptr() && off()==st.off() ) return st; + if( _alias==st._alias && !neverAlias(ptr,st.ptr()) && + !addDep(off())._type.join(addDep(st.off())._type).isHigh() ) return null; + mem=st.mem(); + break; + case ProjNode proj: + if( !(proj.in(0) instanceof NewNode nnn) ) return null; + if( ptr instanceof ProjNode p && p.in(0)==nnn ) return nnn; + if( !(ptr instanceof ProjNode p && p.in(0) instanceof NewNode) ) return null; + mem=nnn.mem(); // Distinct allocations cannot overlap. + break; + case MemPhiNode phi: + addDep(phi.region()); + // Only the original loop closes a successful cycle. Another + // loop or an unfinished merge needs a separate proof. + if( visit==null || !(phi.region() instanceof RegionNode r) || r instanceof LoopNode || + r.inProgress() || phi.inProgress() || r._type!=Type.CONTROL ) return null; + if( visit.get(phi._nid) ) return phi; // Already proved every arm. + for( int i=1; i<phi.nIns(); i++ ) + if( find(phi.in(i),ptr,stop,visit)==null ) return null; + visit.set(phi._nid); + return phi; + default: return null; + } + } + return null; + } + + private Node folded(Node win) { + return win instanceof StoreNode st ? extend(castRO(st.val())) : zero((NewNode)win); + } + + // Materialize only after the profitability search succeeds. The value Phi + // stands for a load which comes back around the loop with unchanged memory. + private Node load(Node mem, Node ptr, Node stop, PhiNode value) { + Node win=find(mem,ptr,stop,new BitSet()); + if( win!=null ) { + if( win==stop ) return value; + if( win instanceof MemPhiNode mp ) { + PhiNode phi=new PhiNode(_name,_declaredType,mp.region()).keep(); + for( int i=1; i<mp.nIns(); i++ ) phi.addDef(load(mp.in(i),ptr,stop,value)); + return phi.unkeep().peephole(); + } + return folded(win).peephole(); + } + LoadNode ld=new LoadNode(_loc,_name,_alias,_declaredType,mem,ptr,off()); + ld.setDef(0,in(0)); + return ld.peephole(); } private static boolean neverAlias( Node ptr1, Node ptr2 ) { @@ -180,16 +207,6 @@ public class LoadNode extends MemOpNode { return false; } - // Profitable if we find a matching Store on this Phi arm. - private boolean profit(PhiNode phi, int idx) { - Node px = phi.in(idx); - if( px==null ) return false; - if( px._type instanceof TypeMem mem && mem._t.isHighOrConst() ) return true; - if( px instanceof StoreNode st1 && ptr()==st1.ptr() && off()==st1.off() ) return true; - addDep(px); - return false; - } - // Read-Only is a deep property, and cannot be cast-away private Node castRO(Node rez) { if( ptr()._type.isFinal() && !rez._type.isFinal() ) @@ -211,4 +228,5 @@ public class LoadNode extends MemOpNode { Node shl = new ShlNode(null,val,shf.keep()).peephole(); return new SarNode(null,shl,shf.unkeep()); } + } diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/LoopNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/LoopNode.java index 334ac686..c645968e 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/LoopNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/LoopNode.java @@ -57,11 +57,12 @@ public class LoopNode extends RegionNode { // Now fold control into the exit. Might have 1 valid exit, or an // XCtrl or a bunch of prior NeverNode exits. Node top = new ConstantNode(Type.TOP).peephole(); - Node memout = new MemMergeNode(false); - memout.addDef(f); // placeholder for control + MemMergeNode memout = new MemMergeNode(new ConstantNode(TypeMem.BOT).peephole()); + memout.setDef(0,f); for( Node u : _outputs ) - if( u instanceof PhiNode phi && phi._type.isa(TypeMem.BOT) ) - memout.addDef(phi); + if( u instanceof BulkMemPhiNode ) memout.setDef(1,u); + else if( u instanceof MemPhiNode phi ) memout.alias(phi._alias,phi); + memout.init(); Node ctrl = ret.ctrl(), mem = ret.mem(), expr = ret.expr(); if( ctrl!=null && ctrl._type != Type.XCONTROL ) { @@ -72,7 +73,7 @@ public class LoopNode extends RegionNode { // Nope, insert an aligned exit layer RegionNode r = new RegionNode(_loc,null,ctrl).init(); ctrl = r; r._ltree = stop._ltree; - mem = new PhiNode(r,mem ).init(); + mem = new BulkMemPhiNode(ScopeNode.MEM0,r,mem).init(); expr = new PhiNode(r,expr).init(); } // Append new Never exit diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java index 087cc18e..c1d11f88 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java @@ -5,173 +5,84 @@ import com.seaofnodes.simple.codegen.CodeGen; import com.seaofnodes.simple.type.*; import java.util.*; -/** - * Memory Merge - a merge of many aliases into a "fat memory". All aliases - * are here, but most will be lazy - take the default fat memory. +/** A default slice and explicit alias overrides. A null default means partial + * memory: absent aliases are not covered, rather than having unknown contents. */ public class MemMergeNode extends Node { + public MemMergeNode(MemMergeNode mem) { super(mem); } - /* - * In-Progress means this is being used by the Parser to track memory - * aliases. No optimizations are allowed. When no longer "in progress" - * normal peeps work. - */ - public final boolean _inProgress; - - public MemMergeNode( boolean inProgress) { _type = TypeMem.BOT; _inProgress = inProgress; } - public MemMergeNode(MemMergeNode mem) { super(mem); _inProgress = false; } - - - // If being used by a Scope, this is "in progress" from the Parser. - // Otherwise, it's a memory merge - boolean inProgress() { return _inProgress; } + public MemMergeNode(Node bulk) { super(null,bulk); _type = TypeMem.BOT; } + public MemMergeNode(Node bulk, int alias, Node precise) { + this(bulk); + alias(alias,precise); + } + public Node alias(int alias) { + Node n = alias < nIns() ? in(alias) : null; + assert n!=null || !(in(1) instanceof BulkMemPhiNode bulk) || !bulk.isSplit(alias); + n = n==null ? in(1) : n; + assert n!=null : "Alias not covered: "+alias; + return n; + } - @Override public String label() { return "ALLMEM"; } - @Override public boolean isMem() { return true; } + public void alias(int alias, Node mem) { + assert alias > 1; + while( nIns()<=alias ) addDef(null); + setDef(alias,mem); + } - @Override - public StringBuilder _print1(StringBuilder sb, BitSet visited) { - sb.append(_inProgress ? "Merge[" : "MEM[ "); - for( int j=2; j<nIns(); j++ ) { - sb.append(j); - sb.append(":"); - Node n = in(j); - while( n instanceof ScopeNode loop ) { - sb.append("Lazy_"); - n = loop.mem().alias(j); - } - if( n==null ) sb.append("___ "); - else n._print0(sb, visited).append(" "); + // Scalar contents of one alias. New zeroes the allocated fields; + // join it with the incoming contents for all previously allocated objects. + // Phis use their cached types, so this query does not recurse around loops. + static Type contents(Node mem, int alias, Node dep) { + dep.addDep(mem); + if( mem instanceof MemMergeNode merge ) + return contents(merge.alias(alias),alias,dep); + if( mem instanceof ProjNode proj && proj.in(0) instanceof NewNode nnn ) { + assert proj._idx==1 && nnn.field(alias)!=null; + dep.addDep(nnn); + return contents(nnn.mem(),alias,dep).meet(nnn.field(alias)._type.makeZero()); } - sb.setLength(sb.length()-1); - return sb.append("]"); + // Function parameters and call results may contain arbitrary heap values. + // Their bulk memory types do not imply empty or zero-filled storage. + if( mem._type==Type.TOP || mem._type==TypeMem.TOP ) return Type.TOP; + if( mem._type instanceof TypeMem mt && mt._alias==alias ) return mt._t; + return Type.BOTTOM; } - // Make a memory merge: no longer a Scope really, tracking memory state but - // not related to the parser in any way. - public Node merge() { - // Force default memory to not be lazy - MemMergeNode merge = new MemMergeNode(false); - for( Node n : _inputs ) - merge.addDef(n); + @Override public String label() { return "MemMerge"; } + @Override public boolean isMem() { return true; } + @Override public Type compute() { return TypeMem.BOT; } + @Override public StringBuilder _print1(StringBuilder sb, BitSet visited) { + sb.append("MEM["); for( int i=1; i<nIns(); i++ ) - merge._mem(i,null); - return merge.peephole(); + if( in(i)!=null ) { + sb.append(i==1 ? "default:" : i+":"); + in(i)._print0(sb,visited).append(" "); + } + return sb.append("]"); } - - @Override public Type compute() { return TypeMem.BOT; } - @Override public Node idealize() { - if( inProgress() ) return null; - - // Fold defaults into the default boolean progress=false, allDefault=true; - for( int i=2; i<nIns(); i++ ) - if( in(1) == in(i) ) { setDef(i,null); progress=true; } - else { allDefault=false; } - - // If not merging any memory (all memory is just the default) - if( allDefault ) - return in(1); // Become default memory + for( int i=2; i<nIns(); i++ ) { + if( in(i)!=null && in(i)==in(1) ) { setDef(i,null); progress=true; } + if( in(i) instanceof MemMergeNode mem ) { + setDef(i,CodeGen.CODE.add(mem.alias(i))); + progress=true; + } + if( in(i)!=null ) allDefault=false; + } + if( allDefault && in(1)!=null ) return in(1); - // Collapse stacked all-mem if( in(1) instanceof MemMergeNode mem ) { - // Goal is to swap my default mem with mem's default mem - for( int i=2; i<mem.nIns(); i++ ) { - if( mem.in(i) != null ) { - // deeper default mem has a non-default - if( i>=nIns() || in(i)==null ) - setDefX(i,mem.in(i)); - } - } + for( int i=2; i<mem.nIns(); i++ ) + if( mem.in(i)!=null && (i>=nIns() || in(i)==null) ) + alias(i,mem.in(i)); + CodeGen.CODE.add(mem.in(1)); setDef(1,mem.in(1)); return this; } - + assert BulkMemPhiNode.checkMerge(this); return progress ? this : null; } - - - public Node in( Var v ) { return in(v._idx); } - - public Node alias( int alias ) { - return alias < nIns() && in(alias)!=null ? in(alias) : in(1); - } - - Node alias( int alias, Node st ) { - while( alias >= nIns() ) addDef(null); - return setDef(alias,st); - } - - // Read or update from memory. - // A shared implementation allows us to create lazy phis both during - // lookups and updates; the lazy phi creation is part of chapter 8. - Node _mem( int alias, Node st ) { - // Memory projections are made lazily; if one does not exist - // then it must be START.proj(1) - Node old = alias(alias); - if( old instanceof ScopeNode loop ) { - MemMergeNode loopmem = loop.mem(); - Node memdef = loopmem.alias(alias); - // Lazy phi! - old = memdef instanceof PhiNode phi && loop.ctrl()==phi.region() - // Loop already has a real Phi, use it - ? memdef - // Set real Phi in the loop head - // The phi takes its one input (no backedge yet) from a recursive - // lookup, which might have insert a Phi in every loop nest. - : loopmem.alias(alias, new PhiNode(Parser.memName(alias), TypeMem.BOT,loop.ctrl(),loopmem._mem(alias,null),null).peephole() ); - alias(alias,old); - } - // Memory projections are made lazily; expand as needed - return st==null ? old : alias(alias,st); // Not lazy, so this is the answer - } - - - void _merge( MemMergeNode that, RegionNode r) { - int len = Math.max(nIns(),that.nIns()); - for( int i = 1; i < len; i++) - if( alias(i) != that.alias(i) ) { // No need for redundant Phis - // If we are in lazy phi mode we need to a lookup - // by alias as it will trigger a phi creation - Node lhs = this._mem(i,null); - Node rhs = that._mem(i,null); - alias(i, new PhiNode(Parser.memName(i), lhs._type.glb(true).meet(rhs._type.glb(true)), r, lhs, rhs).peephole()); - } - } - - // Fill in the backedge of any inserted Phis - void _endLoopMem( ScopeNode scope, MemMergeNode back, MemMergeNode exit ) { - Node exit_def = exit.alias(1); - for( int i=1; i<nIns(); i++ ) { - if( in(i) instanceof PhiNode phi && phi.region()==scope.ctrl() ) { - assert phi.in(2)==null; - phi.setDef(2,back.alias(i)==scope ? phi : back.alias(i)); // Fill backedge - } - if( exit_def == scope ) // Replace a lazy-phi on the exit path also - exit.alias(i,in(i)); - } - } - - // Now one-time do a useless-phi removal - void _useless( ) { - for( int i=2; i<nIns(); i++ ) { - if( in(i) instanceof PhiNode phi ) { - // Do an eager useless-phi removal - Node in = phi.peephole(); - CodeGen.CODE.addAll(phi._outputs); - phi.moveDepsToWorklist(); - if( in != phi ) { - if( !phi.iskeep() ) // Keeping phi around for parser elsewhere - phi.subsume(in); - setDef(i,in); // Set the update back into Scope - } - } - } - } - - @Override public boolean eq( Node n ) { - return this==n || !_inProgress; - } } diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/MemOpNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/MemOpNode.java index bb84019e..3428b5a5 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/MemOpNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/MemOpNode.java @@ -6,6 +6,7 @@ import com.seaofnodes.simple.type.Type; import com.seaofnodes.simple.type.TypeMemPtr; import java.lang.StringBuilder; import java.util.BitSet; +import java.util.ArrayList; /** * Convenience common base for Load and Store. @@ -78,6 +79,8 @@ public abstract class MemOpNode extends Node { static String mlabel(String name) { return "[]".equals(name) ? "ary" : ("#".equals(name) ? "len" : name); } String mlabel() { return mlabel(_name); } + @Override public boolean isMem() { return !_isLoad; } + public Node mem() { return in(1); } public Node ptr() { return in(2); } public Node off() { return in(3); } @@ -86,6 +89,13 @@ public abstract class MemOpNode extends Node { public StringBuilder _printMach( StringBuilder sb, BitSet visited ) { throw Utils.TODO(); } public int log_size() { return _declaredType.log_size(); } + // Extra conditions for factoring matching memory operations through a Phi. + // The caller has already checked the opcode, input shape and control. + boolean canDrop(MemOpNode other, Node dep) { + dep.addDep(ptr()); + return _alias==other._alias && _declaredType==other._declaredType && err()==null; + } + @Override public boolean eq(Node n) { MemOpNode mem = (MemOpNode)n; // Invariant @@ -106,4 +116,29 @@ public abstract class MemOpNode extends Node { return null; return Parser.error( "Might be null accessing '" + _name + "'",_loc); } + // Memory writers which must follow this read. A packaging node is not a + // write, but can hide New's input, so follow this alias through aggregates. + public ArrayList<Node> antiDeps() { + ArrayList<Node> deps = new ArrayList<>(); + antiDeps(mem(),deps,new BitSet()); + return deps; + } + + private void antiDeps(Node mem, ArrayList<Node> deps, BitSet visit) { + if( visit.get(mem._nid) ) return; + visit.set(mem._nid); + for( Node use : mem._outputs ) { + if( use instanceof MemMergeNode merge ) { + Node slice = _alias<merge.nIns() ? merge.in(_alias) : null; + if( (slice==mem) || (slice==null && merge.in(1)==mem) ) + antiDeps(merge,deps,visit); + } else if( use instanceof CallNode || + (use instanceof MemOpNode st && !st._isLoad && st._alias==_alias) || + (use instanceof NewNode nnn && nnn.field(_alias)!=null) || + (use instanceof MemPhiNode phi && phi._alias==_alias) || + (use instanceof BulkMemPhiNode phi && !phi.isSplit(_alias)) ) { + if( !deps.contains(use) ) deps.add(use); + } + } + } } diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java new file mode 100644 index 00000000..e5a29329 --- /dev/null +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java @@ -0,0 +1,44 @@ +package com.seaofnodes.simple.node; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.type.Type; +import com.seaofnodes.simple.type.TypeMem; + +/** A Phi for one field alias, parallel to a bulk Phi at the same Region. */ +public class MemPhiNode extends PhiNode { + public final int _alias; + + public MemPhiNode(String label, int alias, Node... inputs) { + super(label,TypeMem.make(alias,Type.BOTTOM),inputs); + assert alias > 1; + _alias = alias; + } + + public MemPhiNode(MemPhiNode phi) { super(phi); _alias = phi._alias; } + + @Override public String label() { return "MemPhi_"+_alias; } + + @Override public Type compute() { + assert BulkMemPhiNode.checkMem(this,_alias); + if( !(region() instanceof RegionNode r) ) + return region()._type==Type.XCONTROL ? TypeMem.TOP : _type; + if( r.inProgress() ) return _declaredType; + Type t = Type.TOP; + for( int i=1; i<nIns(); i++ ) + if( addDep(r.in(i))._type!=Type.XCONTROL ) + t = t.meet(MemMergeNode.contents(in(i),_alias,this)); + return TypeMem.make(_alias,t); + } + + @Override public Node idealize() { + for( int i=1; i<nIns(); i++ ) + if( in(i) instanceof MemMergeNode mem ) { + setDef(i,CodeGen.CODE.add(mem.alias(_alias))); + return this; + } + return super.idealize(); + } + + @Override public boolean eq(Node n) { return _alias==((MemPhiNode)n)._alias && super.eq(n); } + @Override int hash() { return _alias; } +} diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/NewNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/NewNode.java index 27f7d634..57d351ef 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/NewNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/NewNode.java @@ -6,34 +6,35 @@ import com.seaofnodes.simple.codegen.*; import com.seaofnodes.simple.type.*; import java.util.BitSet; -/** - * Allocation! Allocate a chunk of memory, and pre-zero it. - * The inputs include control and size, and ALL aliases being set. - * The output is large tuple, one for every alias plus the created pointer. - * New is expected to be followed by projections for every alias. +/** Allocate a zeroed object. Inputs {ctrl, $mem, size}; + * results {ptr, $mem}. + * The memory input and result cover only the aliases in the allocated struct. */ public class NewNode extends Node implements MultiNode { public final TypeMemPtr _ptr; - public final int _len; public NewNode(TypeMemPtr ptr, Node... nodes) { super(nodes); assert !ptr.nullable(); _ptr = ptr; - _len = ptr._obj._fields.length; - // Control in slot 0 - assert nodes[0]._type==Type.CONTROL || nodes[0]._type == Type.XCONTROL; - // Malloc-length in slot 1 - assert nodes[1]._type instanceof TypeInteger || nodes[1]._type==Type.NIL; - for( int i=0; i<_len; i++ ) - // Memory slices for all fields. - assert nodes[2+i]._type.isa(TypeMem.BOT); + assert nodes.length==3; + assert nodes[0]._type==Type.CONTROL || nodes[0]._type==Type.XCONTROL; + assert nodes[1]._type instanceof TypeMem; + assert nodes[2]._type instanceof TypeInteger || nodes[2]._type==Type.NIL; } - public NewNode(NewNode nnn) { super(nnn); _ptr = nnn._ptr; _len = nnn._len; } + public Node mem() { return in(1); } + public Node size() { return in(2); } + + public Field field(int alias) { + for( Field f : _ptr._obj._fields ) + if( f._alias==alias ) return f; + return null; + } + + public NewNode(NewNode nnn) { super(nnn); _ptr = nnn._ptr; } - public Node mem (int idx) { return in(idx+2); } @Override public String label() { return "new_"+(_ptr._obj.isAry() ? "ary_"+_ptr._obj._fields[1]._type.str() : _ptr._obj.str()); } @Override @@ -42,32 +43,10 @@ public class NewNode extends Node implements MultiNode { return sb.append(_ptr._obj.str()); } - int findAlias(int alias) { - int idx = _ptr._obj.findAlias(alias); - assert idx!= -1; // Error, caller should be calling - return idx+2; // Skip ctrl, size - } - - // 0 - ctrl - // 1 - byte size - // 2-len+2 - aliases, one per field - // len+2 - 2*len+2 - initial values, one per field - public Node size() { return in(1); } @Override public TypeTuple compute() { - Field[] fs = _ptr._obj._fields; - Type[] ts = new Type[fs.length+2]; - ts[0] = Type.CONTROL; - ts[1] = _ptr; - for( int i=0; i<fs.length; i++ ) { - Type mt = in(i+2)._type; - TypeMem mem = mt==Type.TOP ? TypeMem.TOP : (TypeMem)mt; - Type tfld = mem._t.meet(mem._t.makeZero()); - Type tfld2 = tfld.join(fs[i]._type); - ts[i+2] = TypeMem.make(fs[i]._alias,tfld2); - } - return TypeTuple.make(ts); + return TypeTuple.make(_ptr,TypeMem.BOT); } @Override @@ -105,9 +84,9 @@ public class NewNode extends Node implements MultiNode { _kills = kills; } public String op() { return "alloc"; } - public RegMask regmap(int i) { return i==1 ? _arg3Mask : null; } + public RegMask regmap(int i) { return i==2 ? _arg3Mask : null; } public RegMask outregmap() { return null; } - public RegMask outregmap(int idx) { return idx==1 ? _retMask : null; } + public RegMask outregmap(int idx) { return idx==0 ? _retMask : null; } public RegMask killmap() { return _kills; } public void asm(CodeGen code, SB sb) { sb.p("#calloc, ").p(code.reg(size())); diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/Node.java b/chapter22/src/main/java/com/seaofnodes/simple/node/Node.java index be26fc2f..339ef9af 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/Node.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/Node.java @@ -230,6 +230,7 @@ public abstract class Node implements Cloneable { // Error is 'use' does not exist; ok for 'use' to be null. protected boolean delUse( Node use ) { _outputs.del(_outputs.find(use)); + moveDepsToWorklist(); // User-count and anti-dependence queries can now change. return _outputs.isEmpty(); } @@ -303,6 +304,7 @@ public abstract class Node implements Cloneable { n.unlock(); int idx = n._inputs.find(this); n._inputs.set(idx,nnn); + n.moveDepsToWorklist(); // Rewiring can change a dependent query without changing type. nnn.addUse(n); CODE.addAll(n._outputs); } @@ -549,7 +551,12 @@ public abstract class Node implements Cloneable { * or output of this node, that is, it is at least one step away. The node * being added must benefit from this node being peepholed. */ - <N extends Node> N addDep( N dep ) { + <N extends Node> N addDep( N dep ) { return addDep(dep,false); } + + // Keep an explicit dependency when inspecting an immediate user's shape. + <N extends Node> N addDepForwards(N dep) { return addDep(dep,true); } + + private <N extends Node> N addDep(N dep, boolean forwards) { // Running peepholes during the big assert cannot have side effects // like adding dependencies. if( CODE._midAssert ) return dep; @@ -557,8 +564,8 @@ public abstract class Node implements Cloneable { if( obs != null ) obs.dep(this, dep); if( dep._deps==null ) dep._deps = new Ary<>(Node.class); if( dep._deps .find(this) != -1 ) return dep; // Already on list - if( dep._inputs .find(this) != -1 ) return dep; // No need for deps on immediate neighbors - if( dep._outputs.find(this) != -1 ) return dep; + if( !forwards && dep._inputs .find(this) != -1 ) return dep; // No need for deps on immediate neighbors + if( !forwards && dep._outputs.find(this) != -1 ) return dep; dep._deps.add(this); return dep; } diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/PhiNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/PhiNode.java index f95a452d..5d0c842f 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/PhiNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/PhiNode.java @@ -24,6 +24,10 @@ public class PhiNode extends Node { addDef(sample); } + public static PhiNode make(String label, Type type, Node... inputs) { + return type instanceof TypeMem ? new BulkMemPhiNode(label,inputs) : new PhiNode(label,type,inputs); + } + @Override public String label() { return "Phi_"+MemOpNode.mlabel(_label); } @Override @@ -77,10 +81,9 @@ public class PhiNode extends Node { if( r.in(i)._type == Type.XCONTROL ) return null; - // Generic "pull down op" - Node progress; - if( same_op() && (progress = drop_same_op()) != null ) - return progress; + // Phi(op(a,b,...),op(c,d,...)) -> op(Phi(a,c),Phi(b,d),...). + Node down; + if( same_op() && (down=drop_same_op())!=null ) return down; // If merging Phi(N, cast(N)) - we are losing the cast JOIN effects, so just remove. if( nIns()==3 ) { @@ -114,50 +117,71 @@ public class PhiNode extends Node { // Same op on all Phi paths; all ops have only the Phi as a use. // None have a control input. private boolean same_op() { + Node op = in(1); + if( op instanceof CFGNode || op instanceof ConstantNode || op instanceof PhiNode || + op instanceof ProjNode || op instanceof NewNode || op instanceof ScopeNode || op instanceof MemMergeNode ) return false; + // A bulk Phi must split its aliases before factoring precise Stores. + if( this instanceof BulkMemPhiNode ) return false; + // Bulk splitting currently identifies parallel slices by Region/alias. + // Factoring can introduce another slice at a different memory point; + // wait until that Region's bulk partitioning has finished. + if( op instanceof MemOpNode ) + for( Node use : region()._outputs ) + if( use instanceof BulkMemPhiNode ) { + addDepForwards(use); + return false; + } + Node busy = null; for( int i=1; i<nIns(); i++ ) { - Node op = in(i); - if( in(1).getClass() != op.getClass() || op.in(0)!=null || in(1).nIns() != op.nIns() ) - return false; // Wrong class or CFG bound or mismatched inputs - if( in(1) instanceof MemOpNode mem && mem._alias != ((MemOpNode)op)._alias ) - return false; - if( op.nOuts() > 1 ) { // Too many users, but addDep in case lose users - for( Node out : op._outputs ) - if( out!=null && out!=this ) - addDep(out); - return false; + Node n = in(i); + if( addDep(region().in(i))._type==Type.XCONTROL || + op.getClass()!=n.getClass() || n.nIns()!=op.nIns() || n.in(0)!=null || !op.eq(n) ) return false; + if( n instanceof MemOpNode mem && !mem.canDrop((MemOpNode)op,this) ) return false; + // Moving a Store must remove the old effect, not duplicate it. + // Allow one shared arm: the other arms disappear into the single + // factored operation, usually reducing the total operation count. + addDepForwards(n); + if( n.nOuts()>1 ) { + for( Node use : n._outputs ) + if( use!=null && use!=this ) addDepForwards(use); + if( n instanceof StoreNode || busy!=null ) return false; + busy=n; } - for( int j=1; j<in(1).nIns(); j++ ) - if( op.in(j) instanceof ScopeNode || (op.in(j)==null ^ in(1).in(j)==null) ) - return false; // Lazy Phi input + for( int j=1; j<n.nIns(); j++ ) + if( n.in(j) instanceof ScopeNode || ((n.in(j)==null) != (op.in(j)==null)) ) return false; } return true; } private Node drop_same_op() { - assert !(in(1) instanceof CFGNode); - Node op = in(1); - Node cp = op.copy(); - cp._type = null; // Fresh type - cp.addDef(null); // No control - + Node op = in(1), cp = op.copyEmpty(); + cp._type = null; + cp.addDef(null); for( int j=1; j<op.nIns(); j++ ) { - boolean needsPhi = false; - Node x = op.in(j); // Jth input from sample #1 + Node x = op.in(j); + boolean different = false; for( int i=2; i<nIns(); i++ ) - if( in(i).in(j) != x ) - { needsPhi=true; break; } - if( needsPhi ) { - x = new PhiNode(_label,op.in(j)._type.glb(false)); - x.addDef(region()); - for( int i=1; i<nIns(); i++ ) - x.addDef(in(i).in(j)); - x = x.peephole(); + if( in(i).in(j)!=x ) different=true; + if( different ) { + Node[] ins = new Node[nIns()]; + ins[0] = region(); + Type t = Type.TOP; + for( int i=1; i<nIns(); i++ ) { + ins[i] = in(i).in(j); + t = t.meet(ins[i]._type); + } + PhiNode phi = j==1 && op instanceof MemOpNode mem + ? new MemPhiNode(_label,mem._alias,ins) + : PhiNode.make(_label,t.glb(false),ins); + x = phi.peephole(); } cp.addDef(x); } - // Test not running backwards, which can happen for e.g. And's - if( cp.compute().isa(compute()) ) - return cp; + // Factoring must not widen the result (e.g. correlated And operands). + if( cp.compute().isa(compute()) ) return cp; + for( int i=1; i<nIns(); i++ ) + for( int j=1; j<in(i).nIns(); j++ ) + if( in(i).in(j)!=null ) addDep(in(i).in(j)); cp.kill(); return null; } diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/RegionNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/RegionNode.java index 64747c63..84383845 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/RegionNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/RegionNode.java @@ -48,7 +48,7 @@ public class RegionNode extends CFGNode { if( !hasPhi() && // No Phi users, just a control user in(1) instanceof CProjNode p1 && in(2) instanceof CProjNode p2 && - addDep(p1.in(0))==addDep(p2.in(0)) && + addDepForwards(p1).in(0)==addDepForwards(p2).in(0) && p1.in(0) instanceof IfNode iff ) { // Replace with the iff.ctrl directly if( nIns()==3 ) return iff.ctrl(); diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/ReturnNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/ReturnNode.java index 618c8afa..05b19682 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/ReturnNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/ReturnNode.java @@ -9,15 +9,7 @@ import com.seaofnodes.simple.codegen.Encoding; import com.seaofnodes.simple.type.*; import java.util.BitSet; -/** - * The Return node has two inputs. The first input is a control node and the - * second is the data node that supplies the return value. - * <p> - * In this presentation, Return functions as a Stop node, since multiple <code>return</code> statements are not possible. - * The Stop node will be introduced in Chapter 6 when we implement <code>if</code> statements. - * <p> - * The Return's output is the value from the data node. - */ +/** Keeps all preceding memory effects and the return value alive: {ctrl, $mem, value, rpc}. */ public class ReturnNode extends CFGNode { public FunNode _fun; @@ -51,7 +43,8 @@ public class ReturnNode extends CFGNode { @Override public Type compute() { if( inProgress () ) return TypeTuple.RET; // In progress - return TypeTuple.make(ctrl()._type,mem()._type,expr()._type); + // Inlining can delete the entry while callers still use this return. + return TypeTuple.make(ctrl()._type,TypeMem.BOT,expr()._type); } @Override public Node idealize() { diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/ScopeNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/ScopeNode.java index 804d9cdd..0c9bb881 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/ScopeNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/ScopeNode.java @@ -7,9 +7,9 @@ import java.util.*; /** * The Scope node is purely a parser helper - it tracks names to nodes with a - * stack of variables. + * stack of variable records. */ -public class ScopeNode extends MemMergeNode { +public class ScopeNode extends Node { /** * The control is a name that binds to the currently active control @@ -19,6 +19,11 @@ public class ScopeNode extends MemMergeNode { public static final String ARG0 = "arg"; public static final String MEM0 = "$mem"; + public Node in(Var v) { return in(v._idx); } + + @Override public Type compute() { return Type.BOTTOM; } + @Override public Node idealize() { return null; } + // All active/live variables in all nested scopes, all run together public final Ary<Var> _vars; @@ -37,7 +42,7 @@ public class ScopeNode extends MemMergeNode { // A new ScopeNode public ScopeNode() { - super(true); + _type = Type.BOTTOM; _vars = new Ary<>(Var .class); _lexSize= new Ary<>(Integer.class); _kinds = new Ary<>(Kind .class); @@ -73,7 +78,7 @@ public class ScopeNode extends MemMergeNode { public Node ctrl() { return in(0); } - public MemMergeNode mem() { return (MemMergeNode)in(1); } + public Node mem() { return in(lookup(MEM0)); } public Var var(int i) { return _vars.at(i); } /** @@ -87,7 +92,7 @@ public class ScopeNode extends MemMergeNode { * @return Node that was bound */ public <N extends Node> N ctrl(N n) { return setDef(0,n); } - public Node mem(Node n) { return setDef(1,n); } + public Node mem(Node n) { update(MEM0,n); return n; } public void push( Kind kind ) { assert _lexSize._len==_kinds._len; @@ -170,12 +175,6 @@ public class ScopeNode extends MemMergeNode { return true; } - // Read from memory - public Node mem( int alias ) { return mem()._mem(alias,null); } - // Write to memory - public void mem( int alias, Node st ) { mem()._mem(alias,st); } - - /** * Lookup a name in all scopes starting from most deeply nested. * @@ -211,7 +210,7 @@ public class ScopeNode extends MemMergeNode { // Set real Phi in the loop head // The phi takes its one input (no backedge yet) from a recursive // lookup, which might have insert a Phi in every loop nest. - : loop.setDef(v._idx,new PhiNode(v._name, v.type(), loop.ctrl(), loop.in(loop.update(v,null)._idx),null).peephole()); + : loop.setDef(v._idx,PhiNode.make(v._name, v.type(), loop.ctrl(), loop.in(loop.update(v,null)._idx),null).peephole()); setDef(v._idx,old); } assert !v._final || st==null; @@ -250,21 +249,9 @@ public class ScopeNode extends MemMergeNode { n.keep(); dup.addDef(ctrl()); // Control input is just copied - // Memory input is a shallow copy - MemMergeNode memdup = new MemMergeNode(true), mem = mem(); - memdup.addDef(null); - memdup.addDef(loop ? this : mem.in(1)); - for( int i=2; i<mem.nIns(); i++ ) - // For lazy phis on loops we use a sentinel - // that will trigger phi creation on update - memdup.addDef(loop ? this : mem.in(i)); - dup.addDef(memdup); - - // Copy of other inputs - for( int i=2; i<nIns(); i++ ) - // For lazy phis on loops we use a sentinel - // that will trigger phi creation on update - dup.addDef(loop && !var(i)._final ? this : in(i)); + // Memory is one binding, with the same lazy loop Phi as scalar names. + for( int i=1; i<nIns(); i++ ) + dup.addDef(loop && !_vars.at(i)._final ? this : in(i)); return dup; } @@ -278,22 +265,21 @@ public class ScopeNode extends MemMergeNode { */ public RegionNode mergeScopes(ScopeNode that, Parser.Lexer loc) { RegionNode r = ctrl(new RegionNode(loc,null,ctrl(), that.ctrl()).keep()); - mem()._merge(that.mem(),r); - this ._merge(that ,r); + _merge(that,r); that.kill(); // Kill merged scope CodeGen.CODE.add(r); return r.unkeep(); } private void _merge(ScopeNode that, RegionNode r) { - for( int i = 2; i < nIns(); i++) + for( int i = 1; i < nIns(); i++) if( in(i) != that.in(i) ) { // No need for redundant Phis // If we are in lazy phi mode we need to a lookup // by name as it will trigger a phi creation Var v = var(i); Node lhs = this.in(this.update(v,null)); Node rhs = that.in(that.update(v,null)); - setDef(i, new PhiNode(v._name, v.type(), r, lhs, rhs).peephole()); + setDef(i, PhiNode.make(v._name, v.type(), r, lhs, rhs).peephole()); } } @@ -318,23 +304,17 @@ public class ScopeNode extends MemMergeNode { assert ctrl instanceof LoopNode loop && loop.inProgress(); ctrl.setDef(2,back.ctrl()); - mem()._endLoopMem( this, back.mem(), exit.mem() ); - this ._endLoop ( this, back , exit ); + _endLoop(this,back,exit); back.kill(); // Loop backedge is dead // Now one-time do a useless-phi removal - mem()._useless(); - this ._useless(); - - // The exit mem's lazy default value had been the loop top, - // now it goes back to predating the loop. - exit.mem().setDef(1,mem().in(1)); + _useless(); } // Fill in the backedge of any inserted Phis void _endLoop( ScopeNode scope, Node back, Node exit ) { - for( int i=2; i<nIns(); i++ ) { - if( var(i)._final ) continue; // Final vars did not get modified in the loop - if( var(i).type().isHighOrConst() ) continue; // Cannot lift higher than a constant, so no Phi + for( int i=1; i<nIns(); i++ ) { + if( _vars.at(i)._final ) continue; // Final vars did not get modified in the loop + if( _vars.at(i).type().isHighOrConst() ) continue; // Cannot lift higher than a constant, so no Phi if( back.in(i) != scope ) { PhiNode phi = (PhiNode)in(i); assert phi.region()==scope.ctrl() && phi.in(2)==null; @@ -346,6 +326,23 @@ public class ScopeNode extends MemMergeNode { } + // Now one-time do a useless-phi removal + void _useless( ) { + for( int i=1; i<nIns(); i++ ) { + if( in(i) instanceof PhiNode phi ) { + // Do an eager useless-phi removal + Node in = phi.peephole(); + CodeGen.CODE.addAll(phi._outputs); + phi.moveDepsToWorklist(); + if( in != phi ) { + if( !phi.iskeep() ) // Keeping phi around for parser elsewhere + phi.subsume(in); + setDef(i,in); // Set the update back into Scope + } + } + } + } + // Up-casting: using the results of an If to improve a value. // E.g. "if( ptr ) ptr.field;" is legal because ptr is known not-null. public void addGuards( Node ctrl, Node pred, boolean invert ) { diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/StartNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/StartNode.java index 4bd76f50..91bf85a1 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/StartNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/StartNode.java @@ -3,18 +3,11 @@ package com.seaofnodes.simple.node; import com.seaofnodes.simple.codegen.CodeGen; import com.seaofnodes.simple.Parser; import com.seaofnodes.simple.type.*; -import java.util.Arrays; import java.util.BitSet; import java.util.HashSet; import static com.seaofnodes.simple.Utils.TODO; -/** - * The Start node represents the start of the function. - * - * Start initially has 1 input (arg) from outside and the initial control. - * In ch10 we also add mem aliases as structs get defined; each field in struct - * adds a distinct alias to Start's tuple. - */ +/** Start supplies control, whole memory, and the argument, in that order. */ public class StartNode extends LoopNode implements MultiNode { final Type _arg; @@ -45,7 +38,7 @@ public class StartNode extends LoopNode implements MultiNode { @Override public TypeTuple compute() { - return TypeTuple.make(Type.CONTROL,TypeMem.TOP,_arg); + return TypeTuple.make(Type.CONTROL,TypeMem.BOT,_arg); } @Override public Node idealize() { return null; } diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/StoreNode.java b/chapter22/src/main/java/com/seaofnodes/simple/node/StoreNode.java index e151911c..99adfdfc 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/StoreNode.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/StoreNode.java @@ -1,5 +1,6 @@ package com.seaofnodes.simple.node; +import com.seaofnodes.simple.codegen.*; import com.seaofnodes.simple.*; import com.seaofnodes.simple.type.*; import java.util.BitSet; @@ -25,6 +26,10 @@ public class StoreNode extends MemOpNode { } // Debugger label + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && _init==((StoreNode)other)._init; + } + @Override public String label() { return "st_"+mlabel(); } @Override public boolean isMem() { return true; } @@ -38,28 +43,25 @@ public class StoreNode extends MemOpNode { @Override public Type compute() { Type val = val()._type; - TypeMem mem = (TypeMem)mem()._type; // Invariant - if( mem == TypeMem.TOP ) return TypeMem.TOP; - Type t = Type.BOTTOM; // No idea on field contents - // Same alias, lift val to the declared type and then meet into other fields - if( mem._alias == _alias ) { - // Update declared forward ref to the actual - if( _declaredType.isFRef() && val instanceof TypeMemPtr tmp && !tmp.isFRef() ) - _declaredType = tmp; - val = val.join(_declaredType); - t = val.meet(mem._t); - } + if( _declaredType.isFRef() && val instanceof TypeMemPtr tmp && !tmp.isFRef() ) + _declaredType = tmp; + Type t = val.join(_declaredType).meet(MemMergeNode.contents(mem(),_alias,this)); return TypeMem.make(_alias,t); } @Override public Node idealize() { + if( mem() instanceof MemMergeNode merge ) { + setDef(1,CodeGen.CODE.add(merge.alias(_alias))); + return this; + } + // Simple store-after-store on same address. Should pick up the // required init-store being stomped by a first user store. if( mem() instanceof StoreNode st && - ptr()==st.ptr() && // Must check same object - off()==st.off() && // And same offset (could be "same alias" but this handles arrays to same index) + ptr()==st.ptr() && _alias==st._alias && // Must check same object + off()==st.off() && // And same offset ptr()._type instanceof TypeMemPtr && // No bother if weird dead pointers // Must have exactly one use of "this" or you get weird // non-serializable memory effects in the worse case. @@ -69,12 +71,6 @@ public class StoreNode extends MemOpNode { return this; } - // Simple store-after-MemMerge to a known alias can bypass. Happens when inlining. - if( mem() instanceof MemMergeNode mem ) { - setDef(1,mem.alias(_alias)); - return this; - } - // Value is automatically truncated by narrow store if( val() instanceof AndNode and && and.in(2)._type.isConstant() ) { int log = _declaredType.log_size(); diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java index 839ab50e..7f175f52 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java @@ -673,6 +673,8 @@ public class arm extends Machine { case NotNode not -> new NotARM(not); case OrNode or -> or(or); case ParmNode parm -> new ParmARM(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read -> new ReadOnlyMach(read); diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java index 8775f228..1618334f 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java @@ -345,6 +345,8 @@ public class riscv extends Machine { case NotNode not -> new NotRISC(not); case OrNode or -> or(or); case ParmNode parm -> new ParmRISC(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read-> new ReadOnlyMach(read); diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java index 6e72a540..c712eb7d 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java @@ -27,6 +27,11 @@ public class LoadX86 extends MemOpX86 { } static void enc( Encoding enc, Type decl, short dst, short ptr, short idx, int off, int scale ) { + // Allocation may put a full-width integer in an XMM register (or a + // floating value in a GPR). Move its bits using the selected register bank. + if( dst>=x86_64_v2.XMM_OFFSET && decl.log_size()==3 ) decl=TypeFloat.F64; + else if( dst<x86_64_v2.XMM_OFFSET && decl instanceof TypeFloat ) + decl = decl==TypeFloat.F32 ? TypeInteger.U32 : TypeInteger.BOT; if( decl == TypeFloat.F32) enc.add1(0xF3); if( decl == TypeFloat.F64) enc.add1(0xF2); diff --git a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java index 8a6ae188..f9dfe8f4 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java @@ -363,6 +363,8 @@ public class x86_64_v2 extends Machine { case NotNode not -> new NotX86(not); case OrNode or -> or(or); case ParmNode parm -> new ParmX86(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read-> new ReadOnlyMach(read); @@ -396,9 +398,8 @@ public class x86_64_v2 extends Machine { if( lhs instanceof LoadNode ld && ld.nOuts() == 1 && ld._declaredType.log_size() >= 3) return new AddMemX86(add, address(ld), ld.ptr(), idx, off, scale, 0, rhs); -// if(rhs instanceof LoadNode ld && ld.nOuts() == 1 && ld._declaredType.log_size() >= 3) { -// throw Utils.TODO(); // Swap load sides -// } + if( rhs instanceof LoadNode ld && ld.nOuts()==1 && ld._declaredType.log_size()>=3 ) + return new AddMemX86(add,address(ld),ld.ptr(),idx,off,scale, 0,lhs); // Attempt a full LEA-style break down. // Returns one of AddX86, AddIX86, LeaX86, or LHS diff --git a/chapter22/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java b/chapter22/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java index d4906a1b..d3f22a6d 100644 --- a/chapter22/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java +++ b/chapter22/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java @@ -67,7 +67,7 @@ public class SimpleGraphAdapter extends GraphAdapter<Node> { return n instanceof CFGNode ? Role.CTRL : isMem(n) ? Role.MEM : Role.DATA; if( n instanceof RegionNode && i == 0 ) return Role.ASSOC; if( i == 0 || n instanceof RegionNode || n instanceof StopNode ) return Role.CTRL; - if( n instanceof MemMergeNode || i == 1 && (n instanceof MemOpNode || n instanceof ReturnNode || n instanceof CallNode) ) return Role.MEM; + if( n instanceof MemMergeNode || (i == 1 && (n instanceof MemOpNode || n instanceof ReturnNode || n instanceof CallNode || n instanceof NewNode)) ) return Role.MEM; Node def = n.in(i); if( def != null && isMem(def) ) return Role.MEM; return Role.DATA; diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter10Test.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter10Test.java index aa2cd941..adfdd1ae 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Chapter10Test.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter10Test.java @@ -2,6 +2,8 @@ package com.seaofnodes.simple; import com.seaofnodes.simple.codegen.CodeGen; import org.junit.Test; +import com.seaofnodes.simple.node.*; +import java.util.BitSet; import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; @@ -452,4 +454,121 @@ return 0; assertEquals("return 0;", code.print()); } + @Test public void testReadBeforeStores() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !a = new S; S !b = new S; + a.x = 11; b.x = 22; + S !p = a; if (arg) p = b; + int before = p.x; + a.y = 55; b.y = 66; + a.x = 33; b.x = 44; + return before*100 + p.x; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals("1133",Eval2.eval(code,0)); + assertEquals("2244",Eval2.eval(code,1)); + } + + static final String NESTED_MEMORY = """ + struct S { int x; int y; int z; }; + S !s = new S; + s.x = 5; s.y = 7; s.z = 11; + while (arg > 0) { + s.x = s.x + 1; + int j = 3; + while (j > 0) { + j = j - 1; + if (j == 1) continue; + s.y = s.y + arg; + if (arg == 2) break; + s.z = s.z + 1; + } + arg = arg - 1; + } + return s; + """; + + @Test public void testMemoryAcrossNestedLoops() { + var code = new CodeGen(NESTED_MEMORY).parse().opto(); + StopNode stop = code._stop; + for (int arg=0; arg<7; arg++) { + assertEquals("S{x="+(5L+arg)+",y="+(7L+arg*(arg+1)-(arg>=2 ? 2 : 0))+ + ",z="+(11L+2*arg-(arg>=2 ? 2 : 0))+"}",Eval2.eval(code,arg)); + } + } + + @Test public void testMemoryAtEarlyReturns() { + var code = new CodeGen(""" + struct S { int x; int y; int z; }; + S !s = new S; + s.x = 3; s.y = 5; s.z = 7; + if (arg) { s.x = 11; s.z = 13; return s; } + s.y = 17; + return s; + """).parse().opto(); + StopNode stop = code._stop; + for (int arg=0; arg<2; arg++) { + assertEquals(arg==0 ? "S{x=3,y=17,z=7}" : "S{x=11,y=5,z=13}",Eval2.eval(code,arg)); + } + } + + @Test public void testKeepLoadsAtMerge() { + var code = new CodeGen(""" + struct S { int x; }; + S !a = new S; S !b = new S; + a.x = arg; b.x = arg+1; + S !p = a; S !q = b; + if (arg<0) { p=b; q=a; } + int v; + if (arg>1) v=p.x; else v=q.x; + return v; + """).parse().opto(); + StopNode stop = code._stop; + // The one-step safety check stops at the memory aggregate. + assertEquals(2,countMemoryNodes(stop,LoadNode.class,new BitSet())); + assertEquals("-1",Eval2.eval(code,-1)); + assertEquals("1",Eval2.eval(code,0)); + assertEquals("3",Eval2.eval(code,3)); + } + + @Test public void testDropStores() { + var code = new CodeGen(""" + struct S { int x; }; + S !s = new S; + if (arg) s.x=arg+1; else s.x=arg+2; + return s; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals(1,countMemoryNodes(stop,StoreNode.class,new BitSet())); + assertEquals("S{x=2}",Eval2.eval(code,0)); + assertEquals("S{x=4}",Eval2.eval(code,3)); + } + + @Test public void testKeepReadsBeforeWrites() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !a = new S; S !b = new S; + a.x=arg; b.x=arg+1; + S !p=a; S !q=b; + if (arg<0) { p=b; q=a; } + int v; + if (arg>1) { v=p.x; p.x=41; p.y=5; } + else { v=q.x; q.x=42; } + return v; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals(2,countMemoryNodes(stop,LoadNode.class,new BitSet())); + assertEquals("-1",Eval2.eval(code,-1)); + assertEquals("1",Eval2.eval(code,0)); + assertEquals("3",Eval2.eval(code,3)); + } + + private static int countMemoryNodes(Node n, Class<?> kind, BitSet seen) { + if (n==null || seen.get(n._nid)) return 0; + seen.set(n._nid); + int cnt=kind.isInstance(n) ? 1 : 0; + for (Node def : n._inputs) cnt+=countMemoryNodes(def,kind,seen); + return cnt; + } } diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter10bTest.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter10bTest.java new file mode 100644 index 00000000..5d78ae06 --- /dev/null +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter10bTest.java @@ -0,0 +1,60 @@ +package com.seaofnodes.simple; + +import com.seaofnodes.simple.codegen.*; +import com.seaofnodes.simple.node.*; +import org.junit.Test; +import static org.junit.Assert.*; + +public class Chapter10bTest { + @Test public void testIndependentField() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !s = new S; + s.x = 42; + while (arg > 0) { s.y = s.y + arg; arg = arg - 1; } + return s.x; + """).parse().opto(); + var stop = code._stop; + assertEquals("return 42;",stop.toString()); + assertEquals("42",Eval2.eval(code,5)); + assertNull(stop.walk(n -> n instanceof LoadNode ? n : null)); + } + + @Test public void testParallelMemoryPhis() { + var code = new CodeGen(""" + struct S { int x; int y; int z; }; + S !a = new S; S !b = new S; + S !p = a; if (arg) p = b; + while (arg > 0) { + a.x = p.x + arg; + a.y = p.y + 2; + a.z = p.z + 3; + arg = arg - 1; + } + return a; + """).parse().opto(); + var stop = code._stop; + int[] phis = {0}; + stop.walk(n -> { + if (n instanceof MemPhiNode) phis[0]++; + if (n instanceof MemMergeNode m && m.in(1) instanceof BulkMemPhiNode b) + for(int a=b._aliases.nextSetBit(0); a>=0; a=b._aliases.nextSetBit(a+1)) + assertTrue(a<m.nIns() && m.in(a)!=null); + assertFalse(n instanceof PhiNode && n.isMem() && + !(n instanceof MemPhiNode) && !(n instanceof BulkMemPhiNode) && !(n instanceof ParmNode)); + return null; + }); + assertTrue(phis[0]>=3); + assertEquals("S{x=1,y=2,z=3}",Eval2.eval(code,4)); + } + + @Test public void testLateSliceWorklist() { + // Seed 97 exposed a store selecting a bulk predecessor without queuing + // it. Keep both the worklist assertion and the final heap check. + for (int seed : new int[]{0,97,123,456}) { + var code = new CodeGen(Chapter10Test.NESTED_MEMORY,com.seaofnodes.simple.type.TypeInteger.BOT,seed).parse(); + code.opto(); + assertEquals("S{x=9,y=25,z=17}",Eval2.eval(code,4)); + } + } +} diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter16Test.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter16Test.java index e7163075..84c1637a 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Chapter16Test.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter16Test.java @@ -255,4 +255,27 @@ return new Square; assertEquals("Square{side=3.0,diag=1.7320508075688772}", Eval2.eval(code, 3)); assertEquals("Square{side=4.0,diag=2.0}", Eval2.eval(code, 4)); } + static final String CONSTRUCTOR_MEMORY = """ + struct S { int x; int y; }; + struct T { int z=arg+40; }; + T !t = new T; + S !a = new S { x=11; y=7; }; + S !b = new S { x=22; y=9; }; + S !p=a; + if (arg) p=b; + int before=p.x; + S !c = new S { + x=p.x+1; + { int i=0; while (i<2) { p.y=p.y+1; i=i+1; } } + y=p.y; + }; + p.x=33; + return before*10000+c.x*100+c.y+t.z; + """; + + @Test public void testConstructorMemory() { + var code = new CodeGen(CONSTRUCTOR_MEMORY).parse().opto(); + assertEquals("111249",Eval2.eval(code,0)); + assertEquals("222352",Eval2.eval(code,1)); + } } diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter17Test.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter17Test.java index bdeb645c..f108b6c5 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Chapter17Test.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter17Test.java @@ -368,7 +368,7 @@ var b = arg ? new Bar { next = (arg==2) ? new Bar{x=2;}; x=1; }; return b ? b.next ? b.next.x : b.x; // parses "b ? (b.next ? b.next.x : b.x) : 0" """); code.parse().opto(); - assertEquals("return Phi(Region,.x,0);", code.print()); + assertEquals("return Phi(Region,.x,0,.x);", code.print()); assertEquals("0", Eval2.eval(code, 0)); assertEquals("1", Eval2.eval(code, 1)); assertEquals("2", Eval2.eval(code, 2)); diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter18Test.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter18Test.java index 7cd2831a..7d48db49 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Chapter18Test.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter18Test.java @@ -439,4 +439,70 @@ return p.coffee_count; assertEquals("1", Eval2.eval(code, 2)); } + static final String CALL_MEMORY = """ + struct S { int x; int y; }; + val bump = { S s, int d -> int old=s.x; s.x=old+d; return old; }; + S !a = new S { x=10; y=7; }; + S !b = new S { x=20; y=9; }; + S !p=a; if (arg) p=b; + int before=p.x; + int old=bump(p,3); + int after=p.x; + bump(a,5); + return before*1000000+old*10000+after*100+a.x+a.y; + """; + + @Test public void testCallMemory() { + var code = new CodeGen(CALL_MEMORY).parse().opto(); + int[] calls={0}; + code._stop.walk(n -> { if(n instanceof CallNode) calls[0]++; return null; }); + assertEquals(2,calls[0]); // Keep real calls so this checks scheduling across them. + assertEquals("10101325",Eval2.eval(code,0)); + assertEquals("20202322",Eval2.eval(code,1)); + code._stop.walk(n -> { + if(n instanceof CallNode call) { + assertEquals(5,call.nIns()); // No scheduling edges may become extra arguments. + assertTrue(call.fptr()._type instanceof TypeFunPtr); + } + return null; + }); + } + + static final String RECURSIVE_MEMORY = """ + struct S { int x; }; + val rec = { S s, int n -> + if (n==0) return s.x; + int before=s.x; + s.x=before+1; + return before+rec(s,n-1); + }; + S !s=new S { x=10; }; + int value=rec(s,arg); + return value*100+s.x; + """; + + @Test public void testRecursiveMemory() { + var code = new CodeGen(RECURSIVE_MEMORY).parse().opto(); + for(int n=0; n<7; n++) + assertEquals(Long.toString((10L*(n+1)+n*(n+1)/2)*100+10+n),Eval2.eval(code,n)); + } + + static final String INLINE_MEMORY = """ + struct S { int x; int y; }; + val make = { S s -> + S t = new S { x=s.x+1; y=3; }; + s.y=9; + return t; + }; + S !a = new S { x=arg+40; y=7; }; + S t=make(a); + return a.x*10000+a.y*100+t.x+t.y; + """; + + @Test public void testInlineAllocationMemory() { + var code = new CodeGen(INLINE_MEMORY).parse().opto(); + assertNull(code._stop.walk(n -> n instanceof CallNode ? n : null)); + assertEquals("400944",Eval2.eval(code,0)); + assertEquals("420946",Eval2.eval(code,2)); + } } diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter19Test.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter19Test.java index 7ae094ad..cd1f180c 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Chapter19Test.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter19Test.java @@ -47,7 +47,7 @@ return 0; public void testString() throws IOException { String src = Files.readString(Path.of("src/test/java/com/seaofnodes/simple/progs/stringHash.smp")); CodeGen code = new CodeGen(src).parse().opto().typeCheck().GCM().localSched(); - assertEquals("Stop[ return Phi(Region,._hashCode,Phi(Region,123456789,Phi(Loop,0,(.[]+((Phi_hash<<5)-Phi_hash))))); return Phi(Region,1,0,0,1); ]", code._stop.toString()); + assertEquals("Stop[ return Phi(Region,1,0,0,1); return Phi(Region,._hashCode,Phi(Region,123456789,Phi(Loop,0,(.[]+((Phi_hash<<5)-Phi_hash))))); ]", code._stop.toString()); //assertEquals("-4898613127354160978", Eval2.eval(code, 2)); } @@ -351,4 +351,72 @@ return sq(arg) + sq(3); code.driver(Phase.LocalSched,"x86_64_v2", "SystemV"); assertEquals("Stop[ return (mul,Parm_x(sq,int),x); return (add,#2,#2); ]", code.print()); } + + @Test public void testSelectedMemory() { + for( String cpu : new String[]{"x86_64_v2","riscv","arm"} ) + for( String src : new String[]{Chapter10Test.NESTED_MEMORY,Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY,Chapter18Test.RECURSIVE_MEMORY} ) { + var code = new CodeGen(src).parse().opto().typeCheck(); + var aliases = memoryPhis(code._stop); + code.loopTree().instSelect(cpu,"SystemV"); + assertEquals(aliases,memoryPhis(code._stop)); + code.GCM().localSched(); + code._stop.walk(n -> { + if( !(n instanceof CFGNode) && !(n instanceof ProjNode) ) + assertTrue(n.in(0) instanceof CFGNode); + if( n instanceof NewNode nn ) { + assertEquals(2,((TypeTuple)nn._type)._types.length); + assertTrue(nn.mem() instanceof MemMergeNode); + assertNull(nn.mem().in(1)); // Partial allocation input. + nn.cacheRegs(code); + MachNode mach = (MachNode)nn; + assertNull(mach.regmap(1)); + assertNotNull(mach.regmap(2)); + assertNotNull(mach.outregmap(0)); + assertNull(mach.outregmap(1)); + } + if( n instanceof CallNode call ) { + assertEquals(4,call.nIns()); // ctrl, memory, two args; direct target is embedded. + } + return null; + }); + } + } + + private static java.util.ArrayList<String> memoryPhis(Node stop) { + var phis = new java.util.ArrayList<String>(); + stop.walk(n -> { + if( n instanceof MemPhiNode phi ) phis.add("alias:"+phi._alias); + if( n instanceof BulkMemPhiNode phi ) phis.add("bulk:"+phi._aliases); + assertFalse(n instanceof PhiNode && n.isMem() && + !(n instanceof MemPhiNode) && !(n instanceof BulkMemPhiNode) && !(n instanceof ParmNode)); + return null; + }); + java.util.Collections.sort(phis); + return phis; + } + + @Test public void testFoldedReadBeforeWrites() { + var code = new CodeGen(""" + struct S { int x; }; + S !a = new S { x=11; }; S !b = new S { x=22; }; + S !p = a; if( arg ) p = b; + int before = p.x + arg; + a.x=33; b.x=44; + return before; + """).parse().opto().typeCheck().loopTree().instSelect("x86_64_v2","SystemV").GCM().localSched(); + var read = (MemOpNode)code._stop.walk(n -> + n instanceof com.seaofnodes.simple.node.cpus.x86_64_v2.AddMemX86 ? n : null); + assertNotNull(read); + int writes = 0; + for( Node use : read.antiDeps() ) + if( use instanceof MemOpNode st && st.isMem() ) { + assertSame(read.cfg0(),st.cfg0()); + assertTrue(st._inputs.find(read)>=0); + assertTrue(read.cfg0()._outputs.find(read)<st.cfg0()._outputs.find(st)); + assertNull(((MachNode)st).regmap(st._inputs.find(read))); + writes++; + } + assertEquals(1,writes); // The next Store follows this first clobber through memory. + } } diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter20Test.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter20Test.java index 828a2824..0d46afb5 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Chapter20Test.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter20Test.java @@ -162,7 +162,7 @@ for( int i=0; i<ary#-1; i++ ) ary[i+1] += ary[i]; return ary[1] * 1000 + ary[3]; // 1 * 1000 + 6 """; - testTarget(src,"x86_64_v2", "SystemV",7,"return .[];"); + testTarget(src,"x86_64_v2", "SystemV",5,"return .[];"); testTarget(src,"riscv" , "SystemV",7,"return (add,.[],(mul,.[],1000));"); testTarget(src,"arm" , "SystemV",5,"return (add,.[],(mul,.[],1000));"); } @@ -210,8 +210,8 @@ hashCode(s); for( String cpu : new String[]{"x86_64_v2","riscv","arm"} ) new CodeGen(src,com.seaofnodes.simple.type.TypeInteger.BOT,0) .driver(CodeGen.Phase.Encoding,cpu,"SystemV"); - testTarget(src,"x86_64_v2", "SystemV",0,null); - testTarget(src,"riscv" , "SystemV",0,null); + testTarget(src,"x86_64_v2", "SystemV",9,null); + testTarget(src,"riscv" , "SystemV",3,null); testTarget(src,"arm" , "SystemV",0,null); } @@ -371,8 +371,29 @@ return a; """; testTarget(src,"x86_64_v2", "SystemV",52,null); testTarget(src,"riscv" , "SystemV",44,null); - testTarget(src,"arm" , "SystemV",44,null); + testTarget(src,"arm" , "SystemV",41,null); //assertEquals("int[ 1,2,3,4,5,6,7,8,9,10,11]", Eval2.eval(code, 11)); } + + @Test public void testMemoryAllocation() { + for( String cpu : new String[]{"x86_64_v2","riscv","arm"} ) + for( String src : new String[]{Chapter10Test.NESTED_MEMORY,Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY,Chapter18Test.RECURSIVE_MEMORY, + "int[] !a = new int[3]; a[arg]=arg; return a[0]+1;"} ) { + var code = new CodeGen(src).parse().opto().typeCheck().loopTree() + .instSelect(cpu,"SystemV").GCM().localSched().regAlloc(); + com.seaofnodes.simple.codegen.RegAllocTestSupport.checkRegisters(code); + code._stop.walk(n -> { + // Whole memory, precise Phis, and memory projections need no register. + if( n._type instanceof com.seaofnodes.simple.type.TypeMem ) + assertEquals(-1,code._regAlloc.regnum(n)); + if( n instanceof com.seaofnodes.simple.node.NewNode nn ) { + assertTrue(code._regAlloc.regnum(nn.proj(0))>=0); + assertEquals(-1,code._regAlloc.regnum(nn.proj(1))); + } + return null; + }); + } + } } diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Chapter21Test.java b/chapter22/src/test/java/com/seaofnodes/simple/Chapter21Test.java index 4e1aadd9..df58f32a 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Chapter21Test.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Chapter21Test.java @@ -367,11 +367,11 @@ public class Chapter21Test { String sprimes = sb.p("]").toString(); // Compile, link against native C; expect the above string of primes to be printed out by C - TestC.run("sieve",sprimes, 178); + TestC.run("sieve",sprimes, 185); // Evaluate on RISC5 emulator; expect return of an array of primes in // the simulated heap. - EvalRisc5 R5 = TestRisc5.build("sieve", 100, 89, false); + EvalRisc5 R5 = TestRisc5.build("sieve", 100, 92, false); int trap = R5.step(10000); assertEquals(0,trap); // Return register A0 holds sieve(100) @@ -383,7 +383,7 @@ public class Chapter21Test { // Evaluate on ARM5 emulator; expect return of an array of primes in // the simulated heap. - EvalArm64 A5 = TestArm64.build("sieve", 100, 93, false); + EvalArm64 A5 = TestArm64.build("sieve", 100, 91, false); int trap_arm = A5.step(10000); assertEquals(0, trap_arm); int ary_arm = (int)A5.regs[arm.X0]; diff --git a/chapter22/src/test/java/com/seaofnodes/simple/Eval2.java b/chapter22/src/test/java/com/seaofnodes/simple/Eval2.java index 1e1612ba..cbe7bde0 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/Eval2.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/Eval2.java @@ -329,7 +329,7 @@ public abstract class Eval2 { private static Object alloc(NewNode alloc) { TypeStruct type = alloc._ptr._obj; if( type.isAry() ) { - long sz = (Long)val(alloc.in(1)); + long sz = (Long)val(alloc.size()); long x = offToIdx(sz, type); int n = (int)x; if( n!=x || n<0 ) @@ -451,9 +451,9 @@ public abstract class Eval2 { case TypeRPC rpc -> sb.p(x.toString()); case TypeMem mem -> sb.p("$mem"); case TypeTuple tt -> { - if( tt._types.length>1 && tt._types[1] instanceof TypeMemPtr ) + if( tt._types.length>1 && tt._types[0] instanceof TypeMemPtr ) // Assume a NewNode - yield _print( tt._types[1], x, sb, visit ); + yield _print( tt._types[0], x, sb, visit ); throw Utils.TODO(); } case Type tt -> { diff --git a/chapter22/src/test/java/com/seaofnodes/simple/EvalArm64.java b/chapter22/src/test/java/com/seaofnodes/simple/EvalArm64.java index cd5a8371..0a3e9196 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/EvalArm64.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/EvalArm64.java @@ -541,10 +541,13 @@ public class EvalArm64 { break; } - case 0xD2: { + case 0xD2: // MOVZ + case 0xF2: { // MOVK preserves the other halfwords // movz // get immediate - rval = (ir >> 5) & 0x7FFF; + int shift = ((ir >> 21) & 3)*16; + rval = ((ir >> 5) & 0xFFFFL) << shift; + if( opcode1==0xF2 ) rval |= regs[rdid] & ~(0xFFFFL << shift); break; } diff --git a/chapter22/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java b/chapter22/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java new file mode 100644 index 00000000..1717fbcd --- /dev/null +++ b/chapter22/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java @@ -0,0 +1,178 @@ +package com.seaofnodes.simple; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.codegen.RegAllocTestSupport; +import com.seaofnodes.simple.node.*; +import com.seaofnodes.simple.node.cpus.arm.arm; +import com.seaofnodes.simple.node.cpus.riscv.riscv; +import com.seaofnodes.simple.type.TypeMemPtr; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import org.junit.Test; +import static org.junit.Assert.*; + +// Execution regressions are separate from the fixed spill-measurement cohorts. +public class MemoryEncodingTest { + @Test public void testLoopLoadSearch() { + String[] sources = { + """ + u8[] !program=new u8[106]; u8[] !out=new u8[0]; + int sum=0; + for(int i=0; i<arg; i++) { + sum+=program#; + if(i==1) out=new u8[i+1]; + else if(i==3) out=new u8[2]; + } + return sum+out#; + """, + // One possibly aliasing arm must reject the unchanged-memory proof. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; if(arg&1) p=b; + int sum=0; + for(int i=0; i<6; i++) { + if(i==2) p.x=30; + else if(i==4) b.x=40; + sum+=a.x; + } + return sum; + """, + // Unchanged memory does not imply an unchanged loop-carried pointer. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; int sum=0; + for(int i=0; i<6; i++) { + if(i==2) a.x=30; + sum+=p.x; + p=b; if(i&1) p=a; + } + return sum; + """, + // All arms fold, but their values differ: build a value Phi. + """ + struct S { int x; }; S !a=new S; a.x=1; int sum=0; + for(int i=0; i<6; i++) { + sum+=a.x; + if(i&1) a.x=2; + else if(i==2) a.x=3; + else a.x=4; + } + return sum; + """ + }; + for( int p=0; p<sources.length; p++ ) { + CodeGen code=new CodeGen(sources[p]).parse().opto(); + if( p==0 ) assertNull(code._stop.walk(n -> + n instanceof LoadNode ld && ld.ptr() instanceof ProjNode ptr && + ptr.in(0) instanceof NewNode nn && + nn.size()._type==com.seaofnodes.simple.type.TypeInteger.constant(110) ? ld : null)); + for( int arg=0; arg<7; arg++ ) { + long expected=switch(p) { + case 0 -> 106*arg+(arg>=2 ? 2 : 0); + case 1 -> (arg&1)==0 ? 140 : 60; + case 2 -> 130; + default -> 16; + }; + assertEquals(Long.toString(expected),Eval2.eval(code,arg)); + } + } + } + + private static final String[] SOURCES = { + Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY, + Chapter18Test.RECURSIVE_MEMORY, + Chapter18Test.INLINE_MEMORY, + "int[] !a = new int[7]; a[arg]=arg+3; return a[0]+1;", + """ + int[] !a = new int[7]; + for( int i=0; i<7; i++ ) a[i]=i+arg; + for( int i=0; i<6; i++ ) a[i+1]+=a[i]; + return a[6]; + """ + }; + + private static long expected(int program, long arg) { + return switch(program) { + case 0 -> arg==0 ? 111249 : 222351+arg; + case 1 -> arg==0 ? 10101325 : 20202322; + case 2 -> (10*(arg+1)+arg*(arg+1)/2)*100+10+arg; + case 3 -> 400944+arg*10001; + case 4 -> arg==0 ? 4 : 1; + case 5 -> 21+7*arg; + default -> throw new AssertionError(); + }; + } + + @Test public void testEmulatedMemory() throws IOException { + for( String target : new String[]{"riscv","arm"} ) + for( int p=0; p<SOURCES.length; p++ ) { + CodeGen code = new CodeGen(SOURCES[p]).driver(target,"SystemV",null); + RegAllocTestSupport.checkRegisters(code); + int entry=0; + for( var n : code._start._outputs ) + if( n instanceof FunNode fun && "main".equals(fun._name) ) + entry=code._encoding._opStart[fun._nid]; + for( int arg=0; arg<7; arg++ ) { + byte[] image = new byte[1<<20]; + byte[] bits = code._encoding.bits(); + System.arraycopy(bits,0,image,0,bits.length); + long result; + if( target.equals("riscv") ) { + EvalRisc5 cpu = new EvalRisc5(image,1<<16); + cpu._pc=entry; + cpu.regs[riscv.A0]=arg; + assertEquals(target+" program "+p+" arg "+arg,0,cpu.step(100000)); + result=cpu.regs[riscv.A0]; + } else { + EvalArm64 cpu = new EvalArm64(image,1<<16); + cpu._pc=entry; + cpu.regs[arm.X0]=arg; + assertEquals(target+" program "+p+" arg "+arg,0,cpu.step(100000)); + result=cpu.regs[arm.X0]; + } + assertEquals(target+" program "+p+" arg "+arg,expected(p,arg),result); + } + } + } + + @Test public void testNeverMemory() { + // A synthetic exit must retain both precise slices of a no-exit loop. + // Stop at scheduling: the existing allocator cannot coalesce TOP on + // the synthetic scalar return arm. + for( String target : new String[]{"x86_64_v2","riscv","arm"} ) { + CodeGen code = new CodeGen(""" + struct S { int x; int y; }; + S !a=new S; + if( arg ) while( 1 ) { a.x+=arg; a.y+=a.x; } + return a.x+a.y; + """).driver(CodeGen.Phase.LoopTree); + assertNotNull(code._stop.walk(n -> { + if( n instanceof MemMergeNode mem && + mem.in(0) instanceof CProjNode exit && exit.in(0) instanceof NeverNode ) { + var fields = ((TypeMemPtr)Parser.TYPES.get("S"))._obj._fields; + assertTrue(mem.alias(fields[0]._alias) instanceof MemPhiNode); + assertTrue(mem.alias(fields[1]._alias) instanceof MemPhiNode); + return mem; + } + return null; + })); + code.instSelect(target,"SystemV").GCM().localSched(); + } + } + + @Test public void testNativeMemory() throws IOException { + Files.createDirectories(Path.of("build/objs")); + for( int p=0; p<SOURCES.length; p++ ) { + String file="build/objs/memory"+p; + StringBuilder c = new StringBuilder("extern long long memory(long long);\nint main() {\n"); + for( int arg=0; arg<7; arg++ ) + c.append("if(memory(").append(arg).append(")!=").append(expected(p,arg)).append("LL) return ").append(arg+1).append(";\n"); + c.append("return 0;\n}\n"); + Files.writeString(Path.of(file+".c"),c); + TestC.run("val memory = { int arg -> "+SOURCES[p]+" };", + TestC.CALL_CONVENTION,"",file+".c",file,"S","",-1); + } + } +} diff --git a/chapter22/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java b/chapter22/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java index a3b87e90..3f023bb2 100644 --- a/chapter22/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java +++ b/chapter22/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java @@ -185,7 +185,7 @@ public class Evaluator { Object[] body=null; int num; if( type.isAry() ) { - long sz = (Long)val(alloc.in(1)); + long sz = (Long)val(alloc.size()); long n = offToIdx(sz, type); if( n < 0 ) throw new NegativeArraySizeException(""+n); @@ -199,21 +199,17 @@ public class Evaluator { assert elem instanceof TypeMemPtr; } // Length value - body[0] = vall(alloc.in(2+2)); + body[0] = n; } else { body = new Object[num = type._fields.length]; for( int i=0; i<num; i++ ) body[i] = switch( alloc._ptr._obj._fields[i]._type ) { case TypeInteger ti -> 0L; - case TypeFloat tf -> 0; + case TypeFloat tf -> 0D; default -> null; }; } - Object[] mems = new Object[type._fields.length+2]; - // mems[0] is control - mems[1] = new Obj(type,body); // the ref - // mems[2+...] are memory aliases - return mems; + return new Object[]{new Obj(type,body),null}; } private Object load(LoadNode load) { diff --git a/chapter23/README.md b/chapter23/README.md index c8961647..502387eb 100644 --- a/chapter23/README.md +++ b/chapter23/README.md @@ -43,6 +43,12 @@ Also in this chapter we revisit our Types and make some major changes: etc; only the implementation details change. +The lazy memory model from Chapter 10b continues through the new type system. +BulkMemPhi/MemPhi discover aliases during optimization. Each New covers only +instance fields; class-wide (`_one`) fields are excluded from its partial memory. +Field offsets stay symbolic until layout, and Loads retain cyclic field types +and deep-final information while querying the contents of a precise alias. + You can also read [this chapter](https://github.com/SeaOfNodes/Simple/tree/linear-chapter23) in a linear Git revision history on the [linear](https://github.com/SeaOfNodes/Simple/tree/linear) branch and [compare](https://github.com/SeaOfNodes/Simple/compare/linear-chapter22...linear-chapter23) it to the previous chapter. @@ -484,7 +490,7 @@ ranking remains for Chapter 25. Chapter 21's adjacent-copy forwarding also carries forward: after coloring, a sole use in the next instruction can read the source register when its operand mask permits it and no two-address tie is broken. The table below -includes this cleanup. Cohort 22 now also includes the two zero-move C return +includes this cleanup and the lazy memory forward port. Cohort 22 now also includes the two zero-move C return ABI checks added since the earlier audit (26 entries instead of 24). Run `make spill-stats` in this directory. All rows below use **this chapter's @@ -494,11 +500,11 @@ machine graphs are checked separately and do not contribute to the counts. | Program cohort | Compilations | Retained moves | Loop-weighted moves | |---|---:|---:|---:| -| Chapter 20 | 39 | 326 | 445 | -| Chapter 21 | 52 | 439 | 971 | +| Chapter 20 | 39 | 323 | 442 | +| Chapter 21 | 52 | 433 | 965 | | Chapter 22 | 26 | 67 | 67 | -| Chapter 23 | 30 | 81 | 228 | -| **Total** | **147** | **913** | **1,711** | +| Chapter 23 | 30 | 78 | 225 | +| **Total** | **147** | **901** | **1,699** | `_spills` counts retained SplitNodes, including register moves; `_spillScaled` weights them by `8^loopDepth`. These estimate compiler-generated moves, not diff --git a/chapter23/src/main/java/com/seaofnodes/simple/Parser.java b/chapter23/src/main/java/com/seaofnodes/simple/Parser.java index 5b3f4d1a..2dc83905 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/Parser.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/Parser.java @@ -138,7 +138,7 @@ public class Parser { _scope.define(ScopeNode.ARG0, TypeInteger.BOT, false, null, _lexer); ctrl(XCTRL); - _scope.mem(new MemMergeNode(false)); + _scope.mem(con(TypeMem.BOT)); // Parse the sys import _lexer = new Lexer(com.seaofnodes.simple.sys.SYS); @@ -214,7 +214,7 @@ public class Parser { // Build a multi-exit return point for all function returns RegionNode r = new RegionNode( null,null,null).init(); assert r.inProgress(); - PhiNode rmem = new PhiNode(ScopeNode.MEM0,TypeMem.BOT,r,null).init(); + PhiNode rmem = new BulkMemPhiNode(ScopeNode.MEM0,r,null).init(); PhiNode rrez = new PhiNode(ScopeNode.ARG0,Type.BOTTOM,r,null).init(); ReturnNode ret = new ReturnNode(r, rmem, rrez, rpc, fun).init(); fun.setRet(ret); @@ -227,11 +227,8 @@ public class Parser { // external calls. _scope.push(new Kind.Func()); ctrl(fun); // Scope control from function - // Private mem alias tracking per function - MemMergeNode mem = new MemMergeNode(true); - mem.addDef(null); // Alias#0 - mem.addDef(new ParmNode(ScopeNode.MEM0,1,TypeMem.BOT,fun,con(TypeMem.BOT)).peephole()); // All aliases - _scope.mem(mem); + // Unknown callers supply whole memory; field aliases are discovered lazily. + _scope.mem(new ParmNode(ScopeNode.MEM0,1,TypeMem.BOT,fun,con(TypeMem.BOT)).peephole()); // All args, "as-if" called externally for( int i=0; i<ids.length; i++ ) { Type t = sig.arg(i); @@ -246,7 +243,7 @@ public class Parser { // Last expression is the return except for the top-level main if( ctrl()._type==Type.CONTROL ) if( fun.sig() != _code._main ) - fun.addReturn(ctrl(), _scope.mem().merge(), last); + fun.addReturn(ctrl(), _scope.mem(), last); else { fun.setDef(1,XCTRL); // Kill default main _code.addAll(fun._outputs); @@ -627,9 +624,8 @@ public class Parser { */ private Node parseReturn() { var expr = require(parseAsgn(), ";"); - // Need default memory, since it can be lazy, need to force - // a non-lazy Phi - _fun.addReturn(ctrl(), _scope.mem().merge(), expr); + // Force the lazy whole-memory Phi before closing this exit. + _fun.addReturn(ctrl(), _scope.mem(), expr); ctrl(XCTRL); // Kill control return expr; } @@ -1380,31 +1376,31 @@ public class Parser { Field[] fs = obj._fields; if( fs==null ) throw error("Unknown struct type '" + obj._name + "'"); - int len = fs.length; - Node[] ns = new Node[2+len]; - ns[0] = ctrl(); // Control in slot 0 - // Total allocated length in bytes - ns[1] = size; - // Memory aliases for every field - for( int i = 0; i < len; i++ ) - if( !fs[i]._one ) - ns[2+i] = memAlias(fs[i]._alias); - Node nnn = new NewNode(TypeMemPtr.make(obj), ns).peephole().keep(); - for( int i = 0; i < len; i++ ) - if( !fs[i]._one ) - memAlias(fs[i]._alias, new ProjNode(nnn,i+2,memName(fs[i]._alias)).peephole()); - Node ptr = new ProjNode(nnn.unkeep(),1,obj._name).peephole().keep(); - - // Initial nonzero values for every field - for( int i = 0; i < len; i++ ) { - Node val = init.get( i + idx ); + Node prior = mem().keep(); + MemMergeNode input = new MemMergeNode((Node)null); + for( Field f : fs ) if( !f._one ) input.alias(f._alias,prior); + // Allocation zeroes its fields; nonzero initializers become explicit Stores. + Node[] ns = new Node[3]; + ns[0] = ctrl(); + ns[1] = input.peephole(); + ns[2] = size; + Node nnn = new NewNode(TypeMemPtr.make(obj),ns).peephole().keep(); + Node ptr = new ProjNode(nnn,0,obj._name).peephole().keep(); + Node out = new ProjNode(nnn,1,"$mem").peephole().keep(); + MemMergeNode after = new MemMergeNode(prior); + for( Field f : fs ) if( !f._one ) after.alias(f._alias,out); + mem(after.peephole()); + out.unkeep(); + nnn.unkeep(); + prior.unkeep(); + for( int i=0; i<fs.length; i++ ) { + Node val = init.get(idx+i); if( !fs[i]._one && val._type != val._type.makeZero() ) { - Node mem = memAlias(fs[i]._alias); - Node st = new StoreNode(loc(),fs[i]._fname,fs[i]._alias,fs[i]._t,mem,ptr,off(obj,fs[i]._fname),val,true).peephole(); - memAlias(fs[i]._alias,st); + Field f = fs[i]; + Node st = new StoreNode(loc(),f._fname,f._alias,f._t,mem(),ptr,off(obj,f._fname),val,true).peephole(); + mem(new MemMergeNode(mem(),f._alias,st).peephole()); } } - return ptr.unkeep(); } @@ -1430,13 +1426,18 @@ public class Parser { return con(str); } - // We set up memory aliases by inserting special vars in the scope these - // variables are prefixed by $ so they cannot be referenced in Simple code. - // Using vars has the benefit that all the existing machinery of scoping - // and phis work as expected - private Node memAlias(int alias ) { return _scope.mem(alias ); } - private void memAlias(int alias, Node st) { _scope.mem(alias, st); } - public static String memName(int alias) { return ("$"+alias).intern(); } + // Memory is one hidden SSA variable, including across branches and loops. + private Node mem() { return _scope.mem(); } + private void mem(Node n) { _scope.mem(n); } + + private void store(String name, int alias, Type glb, Node ptr, Node off, Node val, boolean init, Node ctrl) { + Node prior = mem().keep(); + Node st = new StoreNode(loc(),name,alias,glb,prior,ptr,off,val,init); + st.setDef(0,ctrl); + st = st.peephole(); + mem(new MemMergeNode(prior,alias,st).peephole()); + prior.unkeep(); + } /** * Parse postfix expression; this can be a field expression, an array @@ -1494,6 +1495,7 @@ public class Parser { if( tf instanceof TypeMemPtr ftmp && ftmp.isFRef() ) tf = ftmp.makeFrom(((TypeMemPtr)(TYPES.get(ftmp._obj._name)))._obj); + expr.keep(); // Field offset; fixed for structs, computed for arrays Node off = (name.equals("[]") // If field is an array body // Array index math @@ -1506,21 +1508,14 @@ public class Parser { Node val = parseAsgn().keep(); Node lift = liftExpr( val, tf, f._final, false ); - Node st = new StoreNode(loc(), name, f._alias, tf, memAlias(f._alias), expr, off.unkeep(), lift, false); - // Arrays include control, as a proxy for a safety range check. - // Structs don't need this; they only need a NPE check which is - // done via the type system. - if( base.isAry() ) st.setDef(0,ctrl()); - memAlias(f._alias, st.peephole()); + store(name,f._alias,tf,expr.unkeep(),off.unkeep(),lift,false,base.isAry() ? ctrl() : null); return val.unkeep(); // "obj.a = expr" returns the expression while updating memory } - // Keep expr for possible store update - expr.keep(); // Load field Node load = f._one && INITS.containsKey( base._name ) ? INITS.get(base._name).in(base.find(name)) - : new LoadNode(loc(),name, f._alias, tf, memAlias(f._alias), expr, off); + : new LoadNode(loc(),name, f._alias, tf, mem(), expr, off); // Arrays include control, as a proxy for a safety range check // Structs don't need this; they only need a NPE check which is // done via the type system. @@ -1530,13 +1525,8 @@ public class Parser { // Check for assign-update, "ptr.fld += expr" or "ary[idx]++" char ch = _lexer.matchOperAssign(); if( ch!=0 ) { - Node op = opAssign(ch,load, tf ); - Node st = new StoreNode(loc(), name, f._alias, tf, memAlias(f._alias), expr.unkeep(), off.unkeep(), op, false); - // Arrays include control, as a proxy for a safety range check. - // Structs don't need this; they only need a NPE check which is - // done via the type system. - if( base.isAry() ) st.setDef(0,ctrl()); - memAlias(f._alias, peep(st)); + Node op = opAssign(ch,load,tf); + store(name,f._alias,tf,expr.unkeep(),off.unkeep(),op,false,base.isAry() ? ctrl() : null); load = postfix(ch) ? load.unkeep() : op; // And use the original loaded value as the result return load; @@ -1647,7 +1637,7 @@ public class Parser { } // Control & memory after parsing args args.set(0,ctrl().keep()); - args.set(1,_scope.mem().merge().keep()); + args.set(1,_scope.mem().keep()); args.push(fcn); // Function pointer // Unkeep them all for( Node arg : args ) @@ -1668,11 +1658,8 @@ public class Parser { call.peephole(); // Rerun peeps after CallEnd, allows early inlining // Control from CallEnd ctrl(new CProjNode(cend,0,ScopeNode.CTRL).peephole()); - // Memory from CallEnd - MemMergeNode mem = new MemMergeNode(true); - mem.addDef(null); // Alias#0 - mem.addDef(new ProjNode(cend,1,ScopeNode.MEM0).peephole()); - _scope.mem(mem); + // A call may update any alias, so its memory result is the new whole state. + _scope.mem(new ProjNode(cend,1,ScopeNode.MEM0).peephole()); // Call result return new ProjNode(cend,2,"#2").peephole(); } diff --git a/chapter23/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java b/chapter23/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java index b8a8d2b3..1d802cae 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java @@ -50,7 +50,7 @@ public abstract class GlobalCodeMotion { var cons = new ArrayList<Node>(); for( Node con : start._outputs ) if( con!=null && !(con instanceof CFGNode) && - (con.isConst() || con instanceof MachNode mach && mach.outregmap()!=null) ) { + (con.isConst() || (con instanceof MachNode mach && mach.outregmap()!=null)) ) { globals.put(con,true); cons.add(con.keep()); // Preserve original inputs while rewiring users. } @@ -172,16 +172,8 @@ public abstract class GlobalCodeMotion { // Loads need their memory inputs' uses also done if( n instanceof MemOpNode ld && ld._isLoad ) - for( Node memuse : ld.mem()._outputs ) - if( late[memuse._nid]==null && - // New makes new memory, never crushes load memory - !(memuse instanceof NewNode) && - // Load-use directly defines memory - (memuse._type instanceof TypeMem || - // Load-use directly defines memory - memuse instanceof CallNode || - // Load-use indirectly defines memory - (memuse._type instanceof TypeTuple tt && tt._types[ld._alias] instanceof TypeMem)) ) + for( Node memuse : ld.antiDeps() ) + if( late[memuse._nid]==null ) continue outer; // All uses done, schedule @@ -193,9 +185,7 @@ public abstract class GlobalCodeMotion { for( Node def : n._inputs ) { if( def==null ) continue; if( late[def._nid]==null ) work.push(def); - for( Node out : def._outputs ) - if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) - work.push(ld); + wakeLoads(def,late,work,new BitSet()); } if( n instanceof LoopNode loop ) for( Node phi : loop._outputs ) @@ -255,23 +245,32 @@ public abstract class GlobalCodeMotion { best instanceof IfNode; } + // An allocation may consume a load's memory through a partial aggregate. + private static void wakeLoads(Node def, CFGNode[] late, WorkList<Node> work, BitSet visit) { + if( visit.get(def._nid) ) return; + visit.set(def._nid); + for( Node out : def._outputs ) + if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) work.push(ld); + if( def instanceof MemMergeNode ) + for( int i=1; i<def.nIns(); i++ ) + if( def.in(i)!=null ) wakeLoads(def.in(i),late,work,visit); + } + private static CFGNode find_anti_dep(CFGNode lca, MemOpNode load, CFGNode early, CFGNode[] late, int[] anti) { // We could skip final-field loads here. // Walk LCA->early, flagging Load's block location choices for( CFGNode cfg=lca; early!=null && cfg!=early.idom(); cfg = cfg.idom() ) anti[cfg._nid] = load._nid; // Walk load->mem uses, looking for Stores causing an anti-dep - for( Node mem : load.mem()._outputs ) { + for( Node mem : load.antiDeps() ) { switch( mem ) { case MemOpNode st: - if( !st._isLoad && load._alias == st._alias ) { - assert late[mem._nid] != null; - lca = anti_dep(load,late[mem._nid],lca,st,anti); - } - break; // Loads do not cause anti-deps on other loads - case CallNode call: - assert late[call._nid]!=null; - lca = anti_dep(load,late[call._nid],lca,call,anti); + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); + break; + case CallNode st: + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); break; case PhiNode phi: // Repeat anti-dep for matching Phi inputs. @@ -280,10 +279,10 @@ public abstract class GlobalCodeMotion { if( phi.in(i)==load.mem() ) lca = anti_dep(load,phi.region().cfg(i),lca,null,anti); break; - case NewNode st: break; - case ReturnNode ret: break; // Load must already be ahead of Return - case MemMergeNode ret: break; // Mem uses now on ScopeMin - case NeverNode never: break; + case NewNode st: + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); + break; default: throw Utils.TODO(); } } @@ -295,7 +294,7 @@ public abstract class GlobalCodeMotion { // Stores are already placed. Constrain the load at that block only. if( anti[stblk._nid]==load._nid ) { lca = stblk.domLCA(lca,null); - if( lca==stblk && st!=null && st._inputs.find(load) == -1 ) + if( lca==stblk && st!=null && !(st instanceof CallNode) && st._inputs.find(load) == -1 ) st.addDef(load); // Same-block load must precede the store. } return lca; diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java new file mode 100644 index 00000000..7d4f1fb8 --- /dev/null +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java @@ -0,0 +1,240 @@ +package com.seaofnodes.simple.node; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.type.Type; +import com.seaofnodes.simple.type.TypeMem; +import java.util.BitSet; + +/** + * A Phi for bulk memory. It covers all aliases except {@link #_aliases}; + * those aliases must be represented by parallel precise MemPhis at the same + * program slice. + */ +public class BulkMemPhiNode extends PhiNode { + + private static final BitSet EMPTY = new BitSet(); + + // Excluded Aliases; empty means NO exclusions. + // Each split clones the exclusions; published sets are never mutated. + public final BitSet _aliases; + + public BulkMemPhiNode(String label, Node... inputs) { + this(label,EMPTY,inputs); + } + + private BulkMemPhiNode(String label, BitSet aliases, Node... inputs) { + super(label,TypeMem.BOT,inputs); + _aliases = aliases; + } + + public boolean isSplit(int alias) { return _aliases.get(alias); } + + public BulkMemPhiNode(BulkMemPhiNode phi) { super(phi); _aliases = phi._aliases; } + + @Override public String label() { return "BulkPhi"+_aliases; } + + @Override + public Type compute() { + Type t = super.compute(); + return t instanceof TypeMem mem ? mem : + t==Type.BOTTOM ? TypeMem.BOT : + TypeMem.TOP; + } + + @Override + public Node idealize() { + Node progress = super.idealize(); + if( progress != null ) return progress; + if( ((RegionNode)region()).inProgress() || region().nIns()<=1 ) + return null; // Input is in-progress + + // "Peek through" a MemMerge that covers this alias set on its default + for( int i=1; i<nIns(); i++ ) + if( in(i) instanceof MemMergeNode mmm && canPeek(mmm) ) { + setDef(i,CodeGen.CODE.add(mmm.in(1))); + return this; + } + + // If any input or output uses or defines a specific alias we cover, + // make a private MemPhi for it, and add it to the excluded list. + for( int i=1; i<nIns(); i++ ) { + int alias = inputAlias(in(i)); + if( alias!=0 ) + return slice(alias); + } + + for( int i=0; i<_outputs.size(); i++ ) { + int alias = outputAlias(_outputs.get(i)); + if( alias!=0 ) + return slice(alias); + } + + return null; + } + + static public boolean checkMem(Node n, int alias) { + for( Node m : n._inputs ) + // Bulk memory supplies this alias + if( m instanceof BulkMemPhiNode bulk ) { + if( bulk._aliases.get(alias) ) + return false; + } else if( m instanceof MemMergeNode mmm && + mmm.in(1) instanceof BulkMemPhiNode bulk && + bulk._aliases.get(alias) && + (alias >= mmm.nIns() || mmm.in(alias)==null) ) + return false; + return true; + } + + + // True if the MemMerge has no precise slice still covered by this Phi. + private boolean canPeek(MemMergeNode mmm) { + return mmm.in(1)!=null && missingAlias(mmm)==0; + } + + // Return an alias required by an input but still covered by this Phi, or 0. + private int inputAlias(Node n) { + return switch(n) { + case ProjNode proj -> 0; + case ParmNode parm -> 0; + case ConstantNode con -> 0; + case MemMergeNode mmm -> missingAlias(mmm); + case MemOpNode mem -> unsplit(mem._alias); + case BulkMemPhiNode bulk -> missingAlias(bulk); + default -> throw new AssertionError("Unexpected memory input "+n); + }; + } + + // Return an alias required by a user but still covered by this Phi, or 0. + private int outputAlias(Node use) { + // This decision depends backwards on an immediate user's alias/set. + // Normal graph-neighbor enqueueing flows from defs to uses, so retain + // an explicit forward dependency to revisit this Phi when the user + // sharpens. + addDepForwards(use); + return switch(use) { + case ScopeNode scope -> 0; + case MemMergeNode mmm -> missingAlias(mmm); + case BulkMemPhiNode bulk -> missingAlias(bulk); + case MemOpNode mem -> unsplit(mem._alias); + case MemPhiNode phi -> unsplit(phi._alias); + default -> { + assert use instanceof ReturnNode || use instanceof CallNode || use instanceof ParmNode + : "Unexpected bulk-memory user "+use; + yield 0; + } + }; + } + + // Alias zero/one is bulk, and an already excluded precise alias cannot + // trigger another split of this Phi. + private int unsplit(int alias) { + if( alias <= 1 ) return 0; + assert !_aliases.get(alias); + return alias; + } + + // First alias excluded by the neighboring bulk Phi but not by this one. + private int missingAlias(BulkMemPhiNode bulk) { + for( int alias = bulk._aliases.nextSetBit(0); alias >= 0; alias = bulk._aliases.nextSetBit(alias+1) ) + if( !_aliases.get(alias) ) + return alias; + return 0; + } + + // Every explicit slot requests that alias, including a partial allocation + // input whose slot still points to this bulk Phi. + private int missingAlias(MemMergeNode mmm) { + for( int alias=2; alias<mmm.nIns(); alias++ ) + if( mmm.in(alias)!=null && + mmm.alias(alias)!=mmm.in(1) && + !_aliases.get(alias) ) + return alias; + return 0; + } + + // Slice out given alias + private MemMergeNode slice( int alias ) { + assert !_aliases.get(alias); + // Inputs are the same; MemPhi will sharpen his own inputs. If a + // parallel precise Phi already exists, reuse it instead of building a + // duplicate for the same Region/alias pair. + Node mem = precisePhi(alias); + + BitSet aliases = ((BitSet)_aliases.clone()); + aliases.set(alias); + BulkMemPhiNode bphi = new BulkMemPhiNode(_label,aliases); + for( int i=0; i<nIns(); i++ ) + bphi.addDef(in(i)); + // Installing bphi creates a new bulk user with a different exclusion + // set. Queue the original memory inputs before bphi itself can peep + // away and hide those neighbor relationships. + for( int i=1; i<bphi.nIns(); i++ ) + CodeGen.CODE.add(bphi.in(i)); + // Do not peephole bphi here, as BulkMemPhi can recursively start a + // second bulk rewrite before this one has finished. Let the worklist + // discover any further splits + bphi.setType(TypeMem.BOT); + Node bulk = bphi; + CodeGen.CODE.add(bulk); + CodeGen.CODE.add(mem); + return aggregate(bulk,alias,mem); + } + + // Build aggregate memory over a bulk Phi, preserving every precise slice + // which the bulk has already excluded. The named alias is newly supplied + // (or replaces its old slice). + MemMergeNode aggregate(Node bulk, int alias, Node precise) { + MemMergeNode mmm = new MemMergeNode(bulk); + for( int old = _aliases.nextSetBit(0); old >= 0; old = _aliases.nextSetBit(old+1) ) + { assert old != alias; mmm.alias(old,precisePhi(old));} + mmm.alias(alias,precise); + assert checkMerge(mmm); + return mmm; + } + + // Find or make the precise Phi parallel to this bulk Phi at the same merge. + private Node precisePhi(int alias) { + MemPhiNode mphi = _findPhi(alias); + if( mphi!=null ) return mphi; + + mphi = new MemPhiNode("$"+alias,alias); + mphi.addDef(region()); + // Due to cycles, must set before calling peephole + mphi.setType(TypeMem.make(alias,Type.BOTTOM)); + for( int i=1; i<nIns(); i++ ) + mphi.addDef(CodeGen.CODE.add(preciseInput(in(i),alias))); + return CodeGen.CODE.add(mphi); + } + + MemPhiNode _findPhi(int alias) { + for( Node use : region()._outputs ) + if( use instanceof MemPhiNode mphi && mphi._alias==alias ) + return mphi; + return null; + } + + // Select an alias already split out of a predecessor's bulk memory. + private Node preciseInput(Node mem, int alias) { + if( mem instanceof MemMergeNode mmm ) + return mmm.alias(alias); + if( mem instanceof BulkMemPhiNode bulk && bulk.isSplit(alias) ) + return bulk.precisePhi(alias); + return mem; + } + + // Every alias excluded by a bulk default must be supplied explicitly. + static boolean checkMerge(MemMergeNode mmm) { + if( !(mmm.in(1) instanceof BulkMemPhiNode bulk) ) return true; + for( int alias = bulk._aliases.nextSetBit(0); alias >= 0; alias = bulk._aliases.nextSetBit(alias+1) ) + if( alias >= mmm.nIns() || mmm.in(alias)==null ) + return false; + return true; + } + + @Override public boolean eq(Node n) { + return _aliases.equals(((BulkMemPhiNode)n)._aliases) && super.eq(n); + } + + @Override int hash() { return _aliases.hashCode(); } +} diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/LoadNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/LoadNode.java index e752c9ce..8b526be2 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/LoadNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/LoadNode.java @@ -1,7 +1,7 @@ package com.seaofnodes.simple.node; -import com.seaofnodes.simple.Parser; import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.Parser; import com.seaofnodes.simple.type.*; import com.seaofnodes.simple.util.Utils; import java.util.BitSet; @@ -24,6 +24,25 @@ public class LoadNode extends MemOpNode { super(loc, name, alias, true, glb, mem, ptr, off); } + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && !clobbered(dep); + } + + // Check only immediate memory users. Writers clobber memory; Phis and + // aggregates might lead to a clobber, so stop rather than search further. + private boolean clobbered(Node dep) { + Node mem = mem(); + dep.addDepForwards(mem); + for( Node use : mem._outputs ) { + if( use==null ) continue; + dep.addDepForwards(use); + if( use instanceof StoreNode || use instanceof PhiNode || + use instanceof MemMergeNode || + use instanceof NewNode || use instanceof CallNode ) return true; + } + return false; + } + // Debugger label @Override public String label() { return "ld_"+mlabel(); } @Override @@ -42,7 +61,7 @@ public class LoadNode extends MemOpNode { t = ary.elem(); // TODO: if offset is known, can peek the constant } // Lift from declared type and memory input - t = t.join(_declaredType).join(mem._t); + t = t.join(_declaredType).join(MemMergeNode.contents(mem(),_alias,this)); if( _declaredType.isFinal() ) t = t.makeRO(); // Deep final applied return t; @@ -50,35 +69,19 @@ public class LoadNode extends MemOpNode { @Override public Node idealize() { - Node ptr = ptr(); - Node mem = mem(); - - if( mem instanceof CastNode cast ) { + if( mem() instanceof CastNode cast ) { setDef(1,cast.in(1)); return this; } - - // Simple Load-after-Store on same address. - if( mem instanceof StoreNode st && - ptr == st.ptr() && off() == st.off() ) { // Must check same object - assert _name.equals(st._name); // Equiv class aliasing is perfect - return extend(st.val()); - } - - // Simple load-after-MemMerge to a known alias can bypass. Happens when inlining. - if( mem instanceof MemMergeNode mem2 ) { - Node memA = mem2.alias(_alias); - for( Node ld : memA._outputs ) - if( ld instanceof LoadNode ) - CodeGen.CODE.add(ld); - setDef(1,memA); + if( mem() instanceof MemMergeNode merge ) { + setDef(1,CodeGen.CODE.add(merge.alias(_alias))); return this; } + Node ptr = ptr(); + Node mem = mem(); - // Simple Load-after-New on same address. - if( mem instanceof ProjNode p && p.in(0) instanceof NewNode nnn && - ptr == nnn.proj(1) ) // Must check same object - return zero(nnn); // Load zero from new + Node win = find(mem,ptr,null,null); + if( win!=null ) return folded(win); // Uplift control to a prior dominating load. for( Node memuse : mem._outputs ) @@ -87,50 +90,7 @@ public class LoadNode extends MemOpNode { cfg0()!=null && cfg0().domLCA(ld.cfg0(),this) == ld.cfg0() ) // Higher control means load is legal earlier return ld; - // Load-after-Store on same address, but bypassing provably unrelated - // stores. This is a more complex superset of the above two peeps. - // "Provably unrelated" is really weak. - if( ptr instanceof ReadOnlyNode ro ) - ptr = ro.in(1); - outer: - while( true ) { - switch( mem ) { - case StoreNode st: - if( ptr == addDep(st.ptr()) && off() == st.off() ) - return extend(castRO(st.val())); // Proved equal - // Can we prove unequal? Offsets do not overlap? - if( !off()._type.join(st.off()._type).isHigh() && // Offsets overlap - !neverAlias(ptr,st.ptr()) ) // And might alias - break outer; // Cannot tell, stop trying - // Pointers cannot overlap - mem = st.mem(); // Proved never equal - break; - case PhiNode phi: - // Assume related - addDep(phi); - break outer; - case ConstantNode top: break outer; // Assume shortly dead - case ProjNode mproj: // Memory projection - switch( mproj.in(0) ) { - case NewNode nnn1: - if( ptr instanceof ProjNode pproj && pproj.in(0) == mproj.in(0) ) - return zero(nnn1); - if( !(ptr instanceof ProjNode pproj && pproj.in(0) instanceof NewNode) ) - break outer; // Cannot tell, ptr not related to New - mem = nnn1.in(nnn1.findAlias(_alias));// Bypass unrelated New - break; - case StartNode start: break outer; - case CallEndNode cend: break outer; // TODO: Bypass no-alias call - default: throw Utils.TODO(); - } - break; - case CastNode cast: mem = cast.in(1); break; - case MemMergeNode merge: mem = merge.alias(_alias); break; - - default: - throw Utils.TODO(); - } - } + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); // Push a Load up through a Phi, as long as it collapses on at least // one arm. If at a Loop, the backedge MUST collapse - else we risk @@ -139,19 +99,27 @@ public class LoadNode extends MemOpNode { // if( pred ) ptr.x = e0; val = pred ? e0 // else ptr.x = e1; : e1; // val = ptr.x; ptr.x = val; - if( mem() instanceof PhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + if( mem() instanceof MemPhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + // Array access control must not be moved across the merge. + in(0)==null && // Offset can be hoisted off() instanceof ConstantNode && // Pointer can be hoisted hoistPtr(ptr,memphi) ) { - // Profit on RHS/Loop backedge - if( profit(memphi,2) || - // Else must not be a loop to count profit on LHS. - (!(memphi.region() instanceof LoopNode) && profit(memphi,1)) ) { - Node ld1 = ld(1); - Node ld2 = ld(2); - return new PhiNode(_name,_declaredType,memphi.region(),ld1,ld2); + // Returning to this memory means no change only if the pointer + // is also loop invariant. A pointer Phi may select another object. + Node stop = memphi.region() instanceof LoopNode && + !(ptr instanceof PhiNode p && p.region()==memphi.region()) ? memphi : null; + if( find(memphi.in(2),ptr,stop,new BitSet())!=null || + (!(memphi.region() instanceof LoopNode) && find(memphi.in(1),ptr,null,new BitSet())!=null) ) { + PhiNode phi = new PhiNode(_name,_declaredType,memphi.region()).keep(); + phi.setType(_type); // Backedge searches may return this value Phi. + for( int i=1; i<memphi.nIns(); i++ ) { + Node p = ptr instanceof PhiNode pp && pp.region()==memphi.region() ? pp.in(i) : ptr; + phi.addDef(load(memphi.in(i),p,stop,phi)); + } + return CodeGen.CODE.add(phi.unkeep()); } } @@ -166,9 +134,67 @@ public class LoadNode extends MemOpNode { return castRO(new ConstantNode(zero).peephole()); } - private Node ld( int idx ) { - Node mem = mem(), ptr = ptr(); - return new LoadNode(_loc,_name,_alias,_declaredType,mem.in(idx),ptr instanceof PhiNode && ptr.in(0)==mem.in(0) ? ptr.in(idx) : ptr, off()).peephole(); + // Search only; no new nodes or rewiring. Return a Store/New that folds, + // the unchanged loop memory, or a merge whose every arm folds. Null fails. + // Register dependencies so a later pointer/offset rewrite retries the query. + private Node find(Node mem, Node ptr, Node stop, BitSet visit) { + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); + while( mem!=null ) { + addDep(mem); + if( mem==stop ) return mem; + switch( mem ) { + case MemMergeNode merge: mem=merge.alias(_alias); break; + case CastNode cast: mem=cast.in(1); break; + case StoreNode st: + addDep(st.ptr()); + if( _alias==st._alias && ptr==st.ptr() && off()==st.off() ) return st; + if( _alias==st._alias && !neverAlias(ptr,st.ptr()) && + !addDep(off())._type.join(addDep(st.off())._type).isHigh() ) return null; + mem=st.mem(); + break; + case ProjNode proj: + if( !(proj.in(0) instanceof NewNode nnn) ) return null; + if( ptr instanceof ProjNode p && p.in(0)==nnn ) return nnn; + if( !(ptr instanceof ProjNode p && p.in(0) instanceof NewNode) ) return null; + mem=nnn.mem(); // Distinct allocations cannot overlap. + break; + case MemPhiNode phi: + addDep(phi.region()); + // Only the original loop closes a successful cycle. Another + // loop or an unfinished merge needs a separate proof. + if( visit==null || !(phi.region() instanceof RegionNode r) || r instanceof LoopNode || + r.inProgress() || phi.inProgress() || r._type!=Type.CONTROL ) return null; + if( visit.get(phi._nid) ) return phi; // Already proved every arm. + for( int i=1; i<phi.nIns(); i++ ) + if( find(phi.in(i),ptr,stop,visit)==null ) return null; + visit.set(phi._nid); + return phi; + default: return null; + } + } + return null; + } + + private Node folded(Node win) { + return win instanceof StoreNode st ? extend(castRO(st.val())) : zero((NewNode)win); + } + + // Materialize only after the profitability search succeeds. The value Phi + // stands for a load which comes back around the loop with unchanged memory. + private Node load(Node mem, Node ptr, Node stop, PhiNode value) { + Node win=find(mem,ptr,stop,new BitSet()); + if( win!=null ) { + if( win==stop ) return value; + if( win instanceof MemPhiNode mp ) { + PhiNode phi=new PhiNode(_name,_declaredType,mp.region()).keep(); + for( int i=1; i<mp.nIns(); i++ ) phi.addDef(load(mp.in(i),ptr,stop,value)); + return phi.unkeep().peephole(); + } + return folded(win).peephole(); + } + LoadNode ld=new LoadNode(_loc,_name,_alias,_declaredType,mem,ptr,off()); + ld.setDef(0,in(0)); + return ld.peephole(); } private static boolean neverAlias( Node ptr1, Node ptr2 ) { @@ -197,16 +223,6 @@ public class LoadNode extends MemOpNode { return false; } - // Profitable if we find a matching Store on this Phi arm. - private boolean profit(PhiNode phi, int idx) { - Node px = phi.in(idx); - if( px==null ) return false; - if( px._type instanceof TypeMem mem && mem._t.isHighOrConst() ) return true; - if( px instanceof StoreNode st1 && ptr()==addDep(st1.ptr() )&& off()==st1.off() ) return true; - addDep(px); - return false; - } - // Read-Only is a deep property, and cannot be cast-away private Node castRO(Node rez) { if( ptr()._type.isFinal() && !rez._type.isFinal() ) @@ -228,4 +244,5 @@ public class LoadNode extends MemOpNode { Node shl = new ShlNode(null,val,shf.keep()).peephole(); return new SarNode(null,shl,shf.unkeep()); } + } diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/LoopNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/LoopNode.java index 6c69a56e..db52c156 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/LoopNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/LoopNode.java @@ -57,11 +57,12 @@ public class LoopNode extends RegionNode { // Now fold control into the exit. Might have 1 valid exit, or an // XCtrl or a bunch of prior NeverNode exits. Node top = new ConstantNode(Type.TOP).peephole(); - Node memout = new MemMergeNode(false); - memout.addDef(f); // placeholder for control + MemMergeNode memout = new MemMergeNode(new ConstantNode(TypeMem.BOT).peephole()); + memout.setDef(0,f); for( Node u : _outputs ) - if( u instanceof PhiNode phi && phi._type.isa(TypeMem.BOT) ) - memout.addDef(phi); + if( u instanceof BulkMemPhiNode ) memout.setDef(1,u); + else if( u instanceof MemPhiNode phi ) memout.alias(phi._alias,phi); + memout.init(); Node ctrl = ret.ctrl(), mem = ret.mem(), expr = ret.expr(); if( ctrl!=null && ctrl._type != Type.XCONTROL ) { @@ -72,7 +73,7 @@ public class LoopNode extends RegionNode { // Nope, insert an aligned exit layer RegionNode r = new RegionNode(_loc,null,ctrl).init(); ctrl = r; r._ltree = stop._ltree; - mem = new PhiNode(r,mem ).init(); + mem = new BulkMemPhiNode(ScopeNode.MEM0,r,mem).init(); expr = new PhiNode(r,expr).init(); } // Append new Never exit diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java index 20aeb454..e56eeb06 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java @@ -5,176 +5,86 @@ import com.seaofnodes.simple.codegen.CodeGen; import com.seaofnodes.simple.type.*; import java.util.*; -/** - * Memory Merge - a merge of many aliases into a "fat memory". All aliases - * are here, but most will be lazy - take the default fat memory. +/** A default slice and explicit alias overrides. A null default means partial + * memory: absent aliases are not covered, rather than having unknown contents. */ public class MemMergeNode extends Node { + public MemMergeNode(MemMergeNode mem) { super(mem); } - /* - * In-Progress means this is being used by the Parser to track memory - * aliases. No optimizations are allowed. When no longer "in progress" - * normal peeps work. - */ - public final boolean _inProgress; - - public MemMergeNode( boolean inProgress) { _type = TypeMem.BOT; _inProgress = inProgress; } - public MemMergeNode(MemMergeNode mem) { super(mem); _inProgress = false; } - - - // If being used by a Scope, this is "in progress" from the Parser. - // Otherwise, it's a memory merge - boolean inProgress() { return _inProgress; } + public MemMergeNode(Node bulk) { super(null,bulk); _type = TypeMem.BOT; } + public MemMergeNode(Node bulk, int alias, Node precise) { + this(bulk); + alias(alias,precise); + } + public Node alias(int alias) { + Node n = alias < nIns() ? in(alias) : null; + assert n!=null || !(in(1) instanceof BulkMemPhiNode bulk) || !bulk.isSplit(alias); + n = n==null ? in(1) : n; + assert n!=null : "Alias not covered: "+alias; + return n; + } - @Override public String label() { return "ALLMEM"; } - @Override public boolean isMem() { return true; } + public void alias(int alias, Node mem) { + assert alias > 1; + while( nIns()<=alias ) addDef(null); + setDef(alias,mem); + } - @Override - public StringBuilder _print1(StringBuilder sb, BitSet visited) { - sb.append(_inProgress ? "Merge[" : "MEM[ "); - for( int j=2; j<nIns(); j++ ) { - sb.append(j); - sb.append(":"); - Node n = in(j); - while( n instanceof ScopeNode loop ) { - sb.append("Lazy_"); - n = loop.mem().alias(j); - } - if( n==null ) sb.append("___ "); - else n._print0(sb, visited).append(" "); + // Scalar contents of one alias. New zeroes the allocated fields; + // join it with the incoming contents for all previously allocated objects. + // Phis use their cached types, so this query does not recurse around loops. + static Type contents(Node mem, int alias, Node dep) { + dep.addDep(mem); + if( mem instanceof CastNode cast ) return contents(cast.in(1),alias,dep); + if( mem instanceof MemMergeNode merge ) + return contents(merge.alias(alias),alias,dep); + if( mem instanceof ProjNode proj && proj.in(0) instanceof NewNode nnn ) { + assert proj._idx==1 && nnn.field(alias)!=null; + dep.addDep(nnn); + return contents(nnn.mem(),alias,dep).meet(nnn.field(alias)._t.makeZero()); } - sb.setLength(sb.length()-1); - return sb.append("]"); + // Function parameters and call results may contain arbitrary heap values. + // Their bulk memory types do not imply empty or zero-filled storage. + if( mem._type==Type.TOP || mem._type==TypeMem.TOP ) return Type.TOP; + if( mem._type instanceof TypeMem mt && mt._alias==alias ) return mt._t; + return Type.BOTTOM; } - // Make a memory merge: no longer a Scope really, tracking memory state but - // not related to the parser in any way. - public Node merge() { - // Force default memory to not be lazy - MemMergeNode merge = new MemMergeNode(false); - for( Node n : _inputs ) - merge.addDef(n); + @Override public String label() { return "MemMerge"; } + @Override public boolean isMem() { return true; } + @Override public Type compute() { return TypeMem.BOT; } + @Override public StringBuilder _print1(StringBuilder sb, BitSet visited) { + sb.append("MEM["); for( int i=1; i<nIns(); i++ ) - merge._mem(i,null); - return merge.peephole(); + if( in(i)!=null ) { + sb.append(i==1 ? "default:" : i+":"); + in(i)._print0(sb,visited).append(" "); + } + return sb.append("]"); } - - @Override public Type compute() { return TypeMem.BOT; } - @Override public Node idealize() { - if( inProgress() ) return null; - - // Fold defaults into the default boolean progress=false, allDefault=true; for( int i=2; i<nIns(); i++ ) { - if( in(i) instanceof CastNode cast ) - { setDef(i,cast.in(1)); progress=true; } - if( in(1) == in(i) ) { setDef(i,null); progress=true; } - else { allDefault=false; } + if( in(i) instanceof CastNode cast ) { setDef(i,cast.in(1)); progress=true; } + if( in(i)!=null && in(i)==in(1) ) { setDef(i,null); progress=true; } + if( in(i) instanceof MemMergeNode mem ) { + setDef(i,CodeGen.CODE.add(mem.alias(i))); + progress=true; + } + if( in(i)!=null ) allDefault=false; } + if( allDefault && in(1)!=null ) return in(1); - // If not merging any memory (all memory is just the default) - if( allDefault ) - return in(1); // Become default memory - - // Collapse stacked all-mem if( in(1) instanceof MemMergeNode mem ) { - // Goal is to swap my default mem with mem's default mem - for( int i=2; i<mem.nIns(); i++ ) { - if( mem.in(i) != null ) { - // deeper default mem has a non-default - if( i>=nIns() || in(i)==null ) - setDefX(i,mem.in(i)); - } - } + for( int i=2; i<mem.nIns(); i++ ) + if( mem.in(i)!=null && (i>=nIns() || in(i)==null) ) + alias(i,mem.in(i)); + CodeGen.CODE.add(mem.in(1)); setDef(1,mem.in(1)); return this; } - + assert BulkMemPhiNode.checkMerge(this); return progress ? this : null; } - - - public Node in( Var v ) { return in(v._idx); } - - public Node alias( int alias ) { - return alias < nIns() && in(alias)!=null ? in(alias) : in(1); - } - - Node alias( int alias, Node st ) { - while( alias >= nIns() ) addDef(null); - return setDef(alias,st); - } - - // Read or update from memory. - // A shared implementation allows us to create lazy phis both during - // lookups and updates; the lazy phi creation is part of chapter 8. - Node _mem( int alias, Node st ) { - // Memory projections are made lazily; if one does not exist - // then it must be START.proj(1) - Node old = alias(alias); - if( old instanceof ScopeNode loop ) { - MemMergeNode loopmem = loop.mem(); - Node memdef = loopmem.alias(alias); - // Lazy phi! - old = memdef instanceof PhiNode phi && loop.ctrl()==phi.region() - // Loop already has a real Phi, use it - ? memdef - // Set real Phi in the loop head - // The phi takes its one input (no backedge yet) from a recursive - // lookup, which might have insert a Phi in every loop nest. - : loopmem.alias(alias, new PhiNode(Parser.memName(alias), TypeMem.make(alias,Type.BOTTOM),loop.ctrl(),loopmem._mem(alias,null),null).peephole() ); - alias(alias,old); - } - // Memory projections are made lazily; expand as needed - return st==null ? old : alias(alias,st); // Not lazy, so this is the answer - } - - - void _merge( MemMergeNode that, RegionNode r) { - int len = Math.max(nIns(),that.nIns()); - for( int i = 1; i < len; i++) - if( alias(i) != that.alias(i) ) { // No need for redundant Phis - // If we are in lazy phi mode we need to a lookup - // by alias as it will trigger a phi creation - Node lhs = this._mem(i,null); - Node rhs = that._mem(i,null); - alias(i, new PhiNode(Parser.memName(i), lhs._type.meet(rhs._type).glb(true), r, lhs, rhs).peephole()); - } - } - - // Fill in the backedge of any inserted Phis - void _endLoopMem( ScopeNode scope, MemMergeNode back, MemMergeNode exit ) { - Node exit_def = exit.alias(1); - for( int i=1; i<nIns(); i++ ) { - if( in(i) instanceof PhiNode phi && phi.region()==scope.ctrl() ) { - assert phi.in(2)==null; - phi.setDef(2,back.alias(i)==scope ? phi : back.alias(i)); // Fill backedge - } - if( exit_def == scope ) // Replace a lazy-phi on the exit path also - exit.alias(i,in(i)); - } - } - - // Now one-time do a useless-phi removal - void _useless( ) { - for( int i=2; i<nIns(); i++ ) { - if( in(i) instanceof PhiNode phi ) { - // Do an eager useless-phi removal - Node in = phi.peephole(); - CodeGen.CODE.addAll(phi._outputs); - phi.moveDepsToWorklist(); - if( in != phi ) { - if( !phi.iskeep() ) // Keeping phi around for parser elsewhere - phi.subsume(in); - setDef(i,in); // Set the update back into Scope - } - } - } - } - - @Override public boolean eq( Node n ) { - return this==n || !_inProgress; - } } diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/MemOpNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/MemOpNode.java index ead82322..65021110 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/MemOpNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/MemOpNode.java @@ -6,6 +6,7 @@ import com.seaofnodes.simple.type.TypeMemPtr; import com.seaofnodes.simple.util.Utils; import java.lang.StringBuilder; import java.util.BitSet; +import java.util.ArrayList; /** * Convenience common base for Load and Store. @@ -78,6 +79,8 @@ public abstract class MemOpNode extends Node { static String mlabel(String name) { return "[]".equals(name) ? "ary" : ("#".equals(name) ? "len" : name); } String mlabel() { return mlabel(_name); } + @Override public boolean isMem() { return !_isLoad; } + public Node mem() { return in(1); } public Node ptr() { return in(2); } public Node off() { return in(3); } @@ -86,6 +89,13 @@ public abstract class MemOpNode extends Node { public StringBuilder _printMach( StringBuilder sb, BitSet visited ) { throw Utils.TODO(); } public int log_size() { return _declaredType.log_size(); } + // Extra conditions for factoring matching memory operations through a Phi. + // The caller has already checked the opcode, input shape and control. + boolean canDrop(MemOpNode other, Node dep) { + dep.addDep(ptr()); + return _alias==other._alias && _declaredType==other._declaredType && err()==null; + } + @Override public boolean eq(Node n) { MemOpNode mem = (MemOpNode)n; // Invariant @@ -106,4 +116,29 @@ public abstract class MemOpNode extends Node { return null; return Parser.error( "Might be null accessing '" + _name + "'",_loc); } + // Memory writers which must follow this read. A packaging node is not a + // write, but can hide New's input, so follow this alias through aggregates. + public ArrayList<Node> antiDeps() { + ArrayList<Node> deps = new ArrayList<>(); + antiDeps(mem(),deps,new BitSet()); + return deps; + } + + private void antiDeps(Node mem, ArrayList<Node> deps, BitSet visit) { + if( visit.get(mem._nid) ) return; + visit.set(mem._nid); + for( Node use : mem._outputs ) { + if( use instanceof MemMergeNode merge ) { + Node slice = _alias<merge.nIns() ? merge.in(_alias) : null; + if( (slice==mem) || (slice==null && merge.in(1)==mem) ) + antiDeps(merge,deps,visit); + } else if( use instanceof CallNode || + (use instanceof MemOpNode st && !st._isLoad && st._alias==_alias) || + (use instanceof NewNode nnn && nnn.field(_alias)!=null) || + (use instanceof MemPhiNode phi && phi._alias==_alias) || + (use instanceof BulkMemPhiNode phi && !phi.isSplit(_alias)) ) { + if( !deps.contains(use) ) deps.add(use); + } + } + } } diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java new file mode 100644 index 00000000..7e54f50f --- /dev/null +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java @@ -0,0 +1,44 @@ +package com.seaofnodes.simple.node; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.type.Type; +import com.seaofnodes.simple.type.TypeMem; + +/** A Phi for one field alias, parallel to a bulk Phi at the same Region. */ +public class MemPhiNode extends PhiNode { + public final int _alias; + + public MemPhiNode(String label, int alias, Node... inputs) { + super(label,TypeMem.make(alias,Type.BOTTOM),inputs); + assert alias > 1; + _alias = alias; + } + + public MemPhiNode(MemPhiNode phi) { super(phi); _alias = phi._alias; } + + @Override public String label() { return "MemPhi_"+_alias; } + + @Override public Type compute() { + assert BulkMemPhiNode.checkMem(this,_alias); + if( !(region() instanceof RegionNode r) ) + return region()._type==Type.XCONTROL ? TypeMem.TOP : _type; + if( r.inProgress() ) return _minType; + Type t = Type.TOP; + for( int i=1; i<nIns(); i++ ) + if( addDep(r.in(i))._type!=Type.XCONTROL ) + t = t.meet(MemMergeNode.contents(in(i),_alias,this)); + return TypeMem.make(_alias,t); + } + + @Override public Node idealize() { + for( int i=1; i<nIns(); i++ ) + if( in(i) instanceof MemMergeNode mem ) { + setDef(i,CodeGen.CODE.add(mem.alias(_alias))); + return this; + } + return super.idealize(); + } + + @Override public boolean eq(Node n) { return _alias==((MemPhiNode)n)._alias && super.eq(n); } + @Override int hash() { return _alias; } +} diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/NewNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/NewNode.java index b0b4103b..49a82a87 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/NewNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/NewNode.java @@ -6,33 +6,35 @@ import com.seaofnodes.simple.util.SB; import java.util.BitSet; -/** - * Allocation! Allocate a chunk of memory, and pre-zero it. - * The inputs include control and size, and ALL aliases being set. - * The output is large tuple, one for every alias plus the created pointer. - * New is expected to be followed by projections for every alias. +/** Allocate a zeroed object. Inputs {ctrl, $mem, size}; + * results {ptr, $mem}. + * The memory input and result cover only the aliases in the allocated struct. */ public class NewNode extends Node implements MultiNode { public final TypeMemPtr _ptr; - public final int _len; public NewNode(TypeMemPtr ptr, Node... nodes) { super(nodes); assert !ptr.nullable(); _ptr = ptr; - _len = ptr._obj._fields.length; - // Control in slot 0 - assert nodes[0]._type==Type.CONTROL || nodes[0]._type == Type.XCONTROL; - // Malloc-length in slot 1 - assert nodes[1]._type instanceof TypeInteger || nodes[1]._type==Type.NIL; - for( int i=0; i<_len; i++ ) - assert ptr._obj._fields[i]._one || nodes[2 + i]._type.isa( TypeMem.BOT ); + assert nodes.length==3; + assert nodes[0]._type==Type.CONTROL || nodes[0]._type==Type.XCONTROL; + assert nodes[1]._type instanceof TypeMem; + assert nodes[2]._type instanceof TypeInteger || nodes[2]._type==Type.NIL; } - public NewNode(NewNode nnn) { super(nnn); _ptr = nnn._ptr; _len = nnn._len; } + public Node mem() { return in(1); } + public Node size() { return in(2); } + + public Field field(int alias) { + for( Field f : _ptr._obj._fields ) + if( !f._one && f._alias==alias ) return f; + return null; + } + + public NewNode(NewNode nnn) { super(nnn); _ptr = nnn._ptr; } - public Node mem (int idx) { return in(idx+2); } @Override public String label() { return "new_"+(_ptr._obj.isAry() ? "ary_"+_ptr._obj._fields[1]._t.str() : _ptr._obj.str()); } @Override @@ -41,38 +43,10 @@ public class NewNode extends Node implements MultiNode { return sb.append(_ptr._obj.str()); } - int findAlias(int alias) { - int idx = _ptr._obj.findAlias(alias); - assert idx!= -1; // Error, caller should be calling - return idx+2; // Skip ctrl, size - } - - // 0 - ctrl - // 1 - byte size - // 2-len+2 - aliases, one per field - // len+2 - 2*len+2 - initial values, one per field - public Node size() { return in(1); } @Override public TypeTuple compute() { - Field[] fs = _ptr._obj._fields; - Type[] ts = new Type[fs.length+2]; - ts[0] = Type.CONTROL; - ts[1] = _ptr; - for( int i=0; i<fs.length; i++ ) { - if( _ptr._obj._fields[i]._one ) { - // Once-only fields use the declared type - ts[i+2] = _ptr._obj._fields[i]._t; - } else { - // Others take from the inputs - Type mt = in(i+2)._type; - TypeMem mem = mt==Type.TOP ? TypeMem.TOP : (TypeMem)mt; - Type tfld = mem._t.meet(mem._t.makeZero()); - Type tfld2 = tfld.join(fs[i]._t ); - ts[i+2] = TypeMem.make(fs[i]._alias,tfld2); - } - } - return TypeTuple.make(ts); + return TypeTuple.make(_ptr,TypeMem.BOT); } @Override @@ -117,9 +91,9 @@ public class NewNode extends Node implements MultiNode { _kills = kills; } public String op() { return "alloc"; } - public RegMask regmap(int i) { return i==1 ? _arg3Mask : null; } + public RegMask regmap(int i) { return i==2 ? _arg3Mask : null; } public RegMask outregmap() { return null; } - public RegMask outregmap(int idx) { return idx==1 ? _retMask : null; } + public RegMask outregmap(int idx) { return idx==0 ? _retMask : null; } public RegMask killmap() { return _kills; } public void asm(CodeGen code, SB sb) { sb.p("#calloc, ").p(code.reg(size())); diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/Node.java b/chapter23/src/main/java/com/seaofnodes/simple/node/Node.java index ab221789..c27270b6 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/Node.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/Node.java @@ -232,6 +232,7 @@ public abstract class Node implements Cloneable { // Error is 'use' does not exist; ok for 'use' to be null. protected boolean delUse( Node use ) { _outputs.del(_outputs.find(use)); + moveDepsToWorklist(); // User-count and anti-dependence queries can now change. return _outputs.isEmpty(); } @@ -305,6 +306,7 @@ public abstract class Node implements Cloneable { n.unlock(); int idx = n._inputs.find(this); n._inputs.set(idx,nnn); + n.moveDepsToWorklist(); // Rewiring can change a dependent query without changing type. nnn.addUse(n); CODE.add(n); CODE.addAll(n._outputs); @@ -552,7 +554,12 @@ public abstract class Node implements Cloneable { * or output of this node, that is, it is at least one step away. The node * being added must benefit from this node being peepholed. */ - <N extends Node> N addDep( N dep ) { + <N extends Node> N addDep( N dep ) { return addDep(dep,false); } + + // Keep an explicit dependency when inspecting an immediate user's shape. + <N extends Node> N addDepForwards(N dep) { return addDep(dep,true); } + + private <N extends Node> N addDep(N dep, boolean forwards) { // Running peepholes during the big assert cannot have side effects // like adding dependencies. if( CODE._midAssert ) return dep; @@ -560,8 +567,8 @@ public abstract class Node implements Cloneable { if( obs != null ) obs.dep(this, dep); if( dep._deps==null ) dep._deps = new Ary<>(Node.class); if( dep._deps .find(this) != -1 ) return dep; // Already on list - if( dep._inputs .find(this) != -1 ) return dep; // No need for deps on immediate neighbors - if( dep._outputs.find(this) != -1 ) return dep; + if( !forwards && dep._inputs .find(this) != -1 ) return dep; // No need for deps on immediate neighbors + if( !forwards && dep._outputs.find(this) != -1 ) return dep; dep._deps.add(this); return dep; } diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/PhiNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/PhiNode.java index a711fe81..8ae94260 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/PhiNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/PhiNode.java @@ -33,6 +33,10 @@ public class PhiNode extends Node { addDef(sample); } + public static PhiNode make(String label, Type type, Node... inputs) { + return type instanceof TypeMem ? new BulkMemPhiNode(label,inputs) : new PhiNode(label,type,inputs); + } + @Override public String label() { return "Phi_"+MemOpNode.mlabel(_label); } @Override @@ -94,6 +98,8 @@ public class PhiNode extends Node { if( live != null ) { if( live._type.isa(_type) ) return live; + // Let the bulk input split this alias before collapsing a precise Phi. + if( live instanceof BulkMemPhiNode ) return null; // Keep the Phi upcast return new CastNode(_type,null,live); } @@ -103,19 +109,9 @@ public class PhiNode extends Node { if( r.in(i)._type == Type.XCONTROL ) return null; - // Simple Phi-after-MemMerge to a known alias can bypass. Happens when inlining. - if( _type instanceof TypeMem tmem && tmem._alias!=1 ) { - for( int i=1; i<nIns(); i++ ) - if( in(i) instanceof MemMergeNode mem ) { - setDef(i,mem.alias(tmem._alias)); - return this; - } - } - - // Generic "pull down op" - Node progress; - if( same_op() && (progress = drop_same_op()) != null ) - return progress; + // Phi(op(a,b,...),op(c,d,...)) -> op(Phi(a,c),Phi(b,d),...). + Node down; + if( same_op() && (down=drop_same_op())!=null ) return down; // If merging Phi(N, cast(N)) - we are losing the cast JOIN effects, so just remove. if( nIns()==3 ) { @@ -149,50 +145,77 @@ public class PhiNode extends Node { // Same op on all Phi paths; all ops have only the Phi as a use. // None have a control input. private boolean same_op() { + Node op = in(1); + if( op instanceof CFGNode || op instanceof ConstantNode || op instanceof PhiNode || + op instanceof ProjNode || op instanceof NewNode || op instanceof ScopeNode || op instanceof MemMergeNode ) return false; + // A bulk Phi must split its aliases before factoring precise Stores. + if( this instanceof BulkMemPhiNode ) return false; + // Bulk splitting currently identifies parallel slices by Region/alias. + // Factoring can introduce another slice at a different memory point; + // wait until that Region's bulk partitioning has finished. + if( op instanceof MemOpNode ) + for( Node use : region()._outputs ) + if( use instanceof BulkMemPhiNode ) { + addDepForwards(use); + return false; + } + Node busy = null; for( int i=1; i<nIns(); i++ ) { - Node op = in(i); - if( in(1).getClass() != op.getClass() || op.in(0)!=null || in(1).nIns() != op.nIns() ) - return false; // Wrong class or CFG bound or mismatched inputs - if( in(1) instanceof MemOpNode mem && mem._alias != ((MemOpNode)op)._alias ) - return false; - if( op.nOuts() > 1 ) { // Too many users, but addDep in case lose users - for( Node out : op._outputs ) - if( out!=null && out!=this ) - addDep(out); - return false; + Node n = in(i); + if( addDep(region().in(i))._type==Type.XCONTROL || + op.getClass()!=n.getClass() || n.nIns()!=op.nIns() || n.in(0)!=null || !op.eq(n) ) return false; + if( n instanceof MemOpNode mem && !mem.canDrop((MemOpNode)op,this) ) return false; + // Moving a Store must remove the old effect, not duplicate it. + // Allow one shared arm: the other arms disappear into the single + // factored operation, usually reducing the total operation count. + addDepForwards(n); + if( n.nOuts()>1 ) { + for( Node use : n._outputs ) + if( use!=null && use!=this ) addDepForwards(use); + if( n instanceof StoreNode || busy!=null ) return false; + busy=n; } - for( int j=1; j<in(1).nIns(); j++ ) - if( op.in(j) instanceof ScopeNode || (op.in(j)==null ^ in(1).in(j)==null) ) - return false; // Lazy Phi input + for( int j=1; j<n.nIns(); j++ ) + if( n.in(j) instanceof ScopeNode || ((n.in(j)==null) != (op.in(j)==null)) ) return false; } return true; } private Node drop_same_op() { - assert !(in(1) instanceof CFGNode); - Node op = in(1); - Node cp = op.copy(); - cp._type = null; // Fresh type - cp.addDef(null); // No control - + Node op = in(1), cp = op.copyEmpty(); + cp._type = null; + cp.addDef(null); for( int j=1; j<op.nIns(); j++ ) { - boolean needsPhi = false; - Node x = op.in(j); // Jth input from sample #1 + Node x = op.in(j); + boolean different = false; for( int i=2; i<nIns(); i++ ) - if( in(i).in(j) != x ) - { needsPhi=true; break; } - if( needsPhi ) { - x = new PhiNode(_label,op.in(j)._type.glb(false)); - x.addDef(region()); - for( int i=1; i<nIns(); i++ ) - x.addDef(in(i).in(j)); - x = x.peephole(); + if( in(i).in(j)!=x ) different=true; + if( different ) { + Node[] ins = new Node[nIns()]; + ins[0] = region(); + Type t = Type.TOP; + for( int i=1; i<nIns(); i++ ) { + ins[i] = in(i).in(j); + t = t.meet(ins[i]._type); + } + // Not accepts both pointers and integers, but a Phi cannot mix them. + if( t==Type.BOTTOM ) { + for( int i=1; i<ins.length; i++ ) addDep(ins[i]); + cp.kill(); + return null; + } + PhiNode phi = j==1 && op instanceof MemOpNode mem + ? new MemPhiNode(_label,mem._alias,ins) + : PhiNode.make(_label,t.glb(false),ins); + x = phi.peephole(); } cp.addDef(x); } - // Test not running backwards, which can happen for e.g. And's - if( cp.compute().isa(compute()) ) - return cp; + // Factoring must not widen the result (e.g. correlated And operands). + if( cp.compute().isa(compute()) ) return cp; + for( int i=1; i<nIns(); i++ ) + for( int j=1; j<in(i).nIns(); j++ ) + if( in(i).in(j)!=null ) addDep(in(i).in(j)); cp.kill(); return null; } diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/RegionNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/RegionNode.java index 42b23633..14ba493e 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/RegionNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/RegionNode.java @@ -48,7 +48,7 @@ public class RegionNode extends CFGNode { if( !hasPhi() && nIns()>3 && // No Phi users, just a control user in(1) instanceof CProjNode p1 && in(2) instanceof CProjNode p2 && - addDep(p1.in(0))==addDep(p2.in(0)) && + addDepForwards(p1).in(0)==addDepForwards(p2).in(0) && p1.in(0) instanceof IfNode iff ) { // Replace with the iff.ctrl directly if( nIns()==3 ) return iff.ctrl(); diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/ReturnNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/ReturnNode.java index 95303994..212bb875 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/ReturnNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/ReturnNode.java @@ -9,15 +9,7 @@ import com.seaofnodes.simple.util.SB; import com.seaofnodes.simple.util.Utils; import java.util.BitSet; -/** - * The Return node has two inputs. The first input is a control node and the - * second is the data node that supplies the return value. - * <p> - * In this presentation, Return functions as a Stop node, since multiple <code>return</code> statements are not possible. - * The Stop node will be introduced in Chapter 6 when we implement <code>if</code> statements. - * <p> - * The Return's output is the value from the data node. - */ +/** Keeps all preceding memory effects and the return value alive: {ctrl, $mem, value, rpc}. */ public class ReturnNode extends CFGNode { public FunNode _fun; @@ -51,7 +43,8 @@ public class ReturnNode extends CFGNode { @Override public Type compute() { if( inProgress () ) return TypeTuple.RET; // In progress - return TypeTuple.make(ctrl()._type,mem()._type,expr()._type); + // Inlining can delete the entry while callers still use this return. + return TypeTuple.make(ctrl()._type,TypeMem.BOT,expr()._type); } @Override public Node idealize() { diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/ScopeNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/ScopeNode.java index 04666956..cddb9f24 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/ScopeNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/ScopeNode.java @@ -12,9 +12,9 @@ import java.util.*; /** * The Scope node is purely a parser helper - it tracks names to nodes with a - * stack of variables. + * stack of variable records. */ -public class ScopeNode extends MemMergeNode { +public class ScopeNode extends Node { /** * The control is a name that binds to the currently active control @@ -24,8 +24,12 @@ public class ScopeNode extends MemMergeNode { public static final String ARG0 = "arg"; public static final String MEM0 = "$mem"; - // All active/live variables in all nested scopes, all run together. - // Maps 1-to-1 to the _inputs array. + public Node in(Var v) { return in(v._idx); } + + @Override public Type compute() { return Type.BOTTOM; } + @Override public Node idealize() { return null; } + + // All active/live variables in all nested scopes, all run together public final Ary<Var> _vars; // Lexical scope is typed: @@ -50,10 +54,10 @@ public class ScopeNode extends MemMergeNode { // A new ScopeNode public ScopeNode() { - super(true); - _vars = new Ary<>(Var .class); - _kinds = new Ary<>(Kind.class); - _guards = new Ary<>(Node.class); + _type = Type.BOTTOM; + _vars = new Ary<>(Var .class); + _kinds = new Ary<>(Kind .class); + _guards = new Ary<>(Node .class); } @Override public String label() { return "Scope"; } @@ -85,7 +89,7 @@ public class ScopeNode extends MemMergeNode { public Node ctrl() { return in(0); } - public MemMergeNode mem() { return (MemMergeNode)in(1); } + public Node mem() { return in(lookup(MEM0)); } public Var var(int i) { return _vars.at(i); } /** @@ -99,7 +103,7 @@ public class ScopeNode extends MemMergeNode { * @return Node that was bound */ public <N extends Node> N ctrl(N n) { return setDef(0,n); } - public Node mem(Node n) { return setDef(1,n); } + public Node mem(Node n) { update(MEM0,n); return n; } public void push( Kind kind ) { kind._lexSize = _vars.size(); @@ -206,12 +210,6 @@ public class ScopeNode extends MemMergeNode { return true; } - // Read from memory - public Node mem( int alias ) { return mem()._mem(alias,null); } - // Write to memory - public void mem( int alias, Node st ) { mem()._mem(alias,st); } - - /** * Lookup a name in all scopes starting from most deeply nested. * @@ -247,7 +245,7 @@ public class ScopeNode extends MemMergeNode { // Set real Phi in the loop head // The phi takes its one input (no backedge yet) from a recursive // lookup, which might have insert a Phi in every loop nest. - : loop.setDef(v._idx,new PhiNode(v._name, v.type(), loop.ctrl(), loop.in(loop.update(v,null)._idx),null).peephole()); + : loop.setDef(v._idx,PhiNode.make(v._name, v.type(), loop.ctrl(), loop.in(loop.update(v,null)._idx),null).peephole()); setDef(v._idx,old); } //assert !v._final || st==null; @@ -285,21 +283,9 @@ public class ScopeNode extends MemMergeNode { n.keep(); dup.addDef(ctrl()); // Control input is just copied - // Memory input is a shallow copy - MemMergeNode memdup = new MemMergeNode(true), mem = mem(); - memdup.addDef(null); - memdup.addDef(loop ? this : mem.in(1)); - for( int i=2; i<mem.nIns(); i++ ) - // For lazy phis on loops we use a sentinel - // that will trigger phi creation on update - memdup.addDef(loop ? this : mem.in(i)); - dup.addDef(memdup); - - // Copy of other inputs - for( int i=2; i<nIns(); i++ ) - // For lazy phis on loops we use a sentinel - // that will trigger phi creation on update - dup.addDef(loop && !var(i)._final ? this : in(i)); + // Memory is one binding, with the same lazy loop Phi as scalar names. + for( int i=1; i<nIns(); i++ ) + dup.addDef(loop && !_vars.at(i)._final ? this : in(i)); return dup; } @@ -313,22 +299,21 @@ public class ScopeNode extends MemMergeNode { */ public RegionNode mergeScopes(ScopeNode that, Parser.Lexer loc) { RegionNode r = ctrl(new RegionNode(loc,null,ctrl(), that.ctrl()).keep()); - mem()._merge(that.mem(),r); - this ._merge(that ,r); + _merge(that,r); that.kill(); // Kill merged scope CodeGen.CODE.add(r); return r.unkeep(); } private void _merge(ScopeNode that, RegionNode r) { - for( int i = 2; i < nIns(); i++) + for( int i = 1; i < nIns(); i++) if( in(i) != that.in(i) ) { // No need for redundant Phis // If we are in lazy phi mode we need to a lookup // by name as it will trigger a phi creation Var v = var(i); Node lhs = this.in(this.update(v,null)); Node rhs = that.in(that.update(v,null)); - setDef(i, new PhiNode(v._name, v.type(), r, lhs, rhs).peephole()); + setDef(i, PhiNode.make(v._name, v.type(), r, lhs, rhs).peephole()); } } @@ -353,23 +338,17 @@ public class ScopeNode extends MemMergeNode { assert ctrl instanceof LoopNode loop && loop.inProgress(); ctrl.setDef(2,back.ctrl()); - mem()._endLoopMem( this, back.mem(), exit.mem() ); - this ._endLoop ( this, back , exit ); + _endLoop(this,back,exit); back.kill(); // Loop backedge is dead // Now one-time do a useless-phi removal - mem()._useless(); - this ._useless(); - - // The exit mem's lazy default value had been the loop top, - // now it goes back to predating the loop. - exit.mem().setDef(1,mem().in(1)); + _useless(); } // Fill in the backedge of any inserted Phis void _endLoop( ScopeNode scope, Node back, Node exit ) { - for( int i=2; i<nIns(); i++ ) { - if( var(i)._final ) continue; // Final vars did not get modified in the loop - if( var(i).type().isHighOrConst() ) continue; // Cannot lift higher than a constant, so no Phi + for( int i=1; i<nIns(); i++ ) { + if( _vars.at(i)._final ) continue; // Final vars did not get modified in the loop + if( _vars.at(i).type().isHighOrConst() ) continue; // Cannot lift higher than a constant, so no Phi if( back.in(i) != scope ) { PhiNode phi = (PhiNode)in(i); assert phi.region()==scope.ctrl() && phi.in(2)==null; @@ -381,6 +360,23 @@ public class ScopeNode extends MemMergeNode { } + // Now one-time do a useless-phi removal + void _useless( ) { + for( int i=1; i<nIns(); i++ ) { + if( in(i) instanceof PhiNode phi ) { + // Do an eager useless-phi removal + Node in = phi.peephole(); + CodeGen.CODE.addAll(phi._outputs); + phi.moveDepsToWorklist(); + if( in != phi ) { + if( !phi.iskeep() ) // Keeping phi around for parser elsewhere + phi.subsume(in); + setDef(i,in); // Set the update back into Scope + } + } + } + } + // Up-casting: using the results of an If to improve a value. // E.g. "if( ptr ) ptr.field;" is legal because ptr is known not-null. public void addGuards( Node ctrl, Node pred, boolean invert ) { diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/StartNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/StartNode.java index 6ed2acd3..186adfb7 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/StartNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/StartNode.java @@ -3,18 +3,11 @@ package com.seaofnodes.simple.node; import com.seaofnodes.simple.codegen.CodeGen; import com.seaofnodes.simple.Parser; import com.seaofnodes.simple.type.*; -import java.util.Arrays; import java.util.BitSet; import java.util.HashSet; import static com.seaofnodes.simple.util.Utils.TODO; -/** - * The Start node represents the start of the function. - * <p> - * Start initially has 1 input (arg) from outside and the initial control. - * In ch10 we also add mem aliases as structs get defined; each field in struct - * adds a distinct alias to Start's tuple. - */ +/** Start supplies control, whole memory, and the argument, in that order. */ public class StartNode extends LoopNode implements MultiNode { final Type _arg; @@ -44,7 +37,7 @@ public class StartNode extends LoopNode implements MultiNode { @Override public TypeTuple compute() { - return TypeTuple.make(Type.CONTROL,TypeMem.TOP,_arg); + return TypeTuple.make(Type.CONTROL,TypeMem.BOT,_arg); } @Override public Node idealize() { return null; } diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/StoreNode.java b/chapter23/src/main/java/com/seaofnodes/simple/node/StoreNode.java index 74c27dc8..ac437788 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/StoreNode.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/StoreNode.java @@ -1,5 +1,6 @@ package com.seaofnodes.simple.node; +import com.seaofnodes.simple.codegen.*; import com.seaofnodes.simple.*; import com.seaofnodes.simple.type.*; import java.util.BitSet; @@ -25,6 +26,10 @@ public class StoreNode extends MemOpNode { } // Debugger label + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && _init==((StoreNode)other)._init; + } + @Override public String label() { return "st_"+mlabel(); } @Override public boolean isMem() { return true; } @@ -38,24 +43,18 @@ public class StoreNode extends MemOpNode { @Override public Type compute() { Type val = val()._type; - Type mem0 = mem()._type; - if( mem0 == Type.TOP ) return TypeMem.TOP; - TypeMem mem = (TypeMem)mem0; // Invariant - if( mem == TypeMem.TOP ) return TypeMem.TOP; - Type t = Type.BOTTOM; // No idea on field contents - // Same alias, lift val to the declared type and then meet into other fields - if( mem._alias == _alias ) { - assert !_declaredType.isFRef(); - // Sharpen memory value; required for narrowing stores where the parser inserts - // zero/sign masking and somebody reads the TypeMem type. - val = val.join(_declaredType); - t = val.meet(mem._t); - } + assert !_declaredType.isFRef(); + Type t = val.join(_declaredType).meet(MemMergeNode.contents(mem(),_alias,this)); return TypeMem.make(_alias,t); } @Override public Node idealize() { + if( mem() instanceof MemMergeNode merge ) { + setDef(1,CodeGen.CODE.add(merge.alias(_alias))); + return this; + } + if( mem() instanceof CastNode cast ) { setDef(1,cast.in(1)); @@ -65,8 +64,8 @@ public class StoreNode extends MemOpNode { // Simple store-after-store on same address. Should pick up the // required init-store being stomped by a first user store. if( mem() instanceof StoreNode st && - ptr()==st.ptr() && // Must check same object - off()==st.off() && // And same offset (could be "same alias" but this handles arrays to same index) + ptr()==st.ptr() && _alias==st._alias && // Must check same object + off()==st.off() && // And same offset ptr()._type instanceof TypeMemPtr && // No bother if weird dead pointers // Must have exactly one use of "this" or you get weird // non-serializable memory effects in the worse case. @@ -76,12 +75,6 @@ public class StoreNode extends MemOpNode { return this; } - // Simple store-after-MemMerge to a known alias can bypass. Happens when inlining. - if( mem() instanceof MemMergeNode mem ) { - setDef(1,mem.alias(_alias)); - return this; - } - // Value is automatically truncated by narrow store if( val() instanceof AndNode and && and.in(2)._type.isConstant() ) { int log = _declaredType.log_size(); diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java index bca3dfa4..c39c5f74 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java @@ -673,6 +673,8 @@ public class arm extends Machine { case NotNode not -> new NotARM(not); case OrNode or -> or(or); case ParmNode parm -> new ParmARM(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read -> new ReadOnlyMach(read); diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java index 48cb5d23..88ec9ebd 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java @@ -345,6 +345,8 @@ public class riscv extends Machine { case NotNode not -> new NotRISC(not); case OrNode or -> or(or); case ParmNode parm -> new ParmRISC(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read-> new ReadOnlyMach(read); diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java index 441cf693..d0fb0081 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java @@ -24,6 +24,11 @@ public class LoadX86 extends MemOpX86 { } static void enc( Encoding enc, Type decl, short dst, short ptr, short idx, int off, int scale ) { + // Allocation may put a full-width integer in an XMM register (or a + // floating value in a GPR). Move its bits using the selected register bank. + if( dst>=x86_64_v2.XMM_OFFSET && decl.log_size()==3 ) decl=TypeFloat.F64; + else if( dst<x86_64_v2.XMM_OFFSET && decl instanceof TypeFloat ) + decl = decl==TypeFloat.F32 ? TypeInteger.U32 : TypeInteger.BOT; if( decl == TypeFloat.F32) enc.add1(0xF3); if( decl == TypeFloat.F64) enc.add1(0xF2); diff --git a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java index 7ac7fc19..cf9288b8 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java @@ -363,6 +363,8 @@ public class x86_64_v2 extends Machine { case NotNode not -> new NotX86(not); case OrNode or -> or(or); case ParmNode parm -> new ParmX86(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read-> new ReadOnlyMach(read); @@ -396,9 +398,8 @@ public class x86_64_v2 extends Machine { if( lhs instanceof LoadNode ld && ld.nOuts() == 1 && ld._declaredType.log_size() >= 3) return new AddMemX86(add, address(ld), ld.ptr(), idx, off, scale, 0, rhs); -// if(rhs instanceof LoadNode ld && ld.nOuts() == 1 && ld._declaredType.log_size() >= 3) { -// throw Utils.TODO(); // Swap load sides -// } + if( rhs instanceof LoadNode ld && ld.nOuts()==1 && ld._declaredType.log_size()>=3 ) + return new AddMemX86(add,address(ld),ld.ptr(),idx,off,scale, 0,lhs); // Attempt a full LEA-style break down. // Returns one of AddX86, AddIX86, LeaX86, or LHS diff --git a/chapter23/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java b/chapter23/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java index 7f7107ec..6f19fa9d 100644 --- a/chapter23/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java +++ b/chapter23/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java @@ -67,7 +67,7 @@ public class SimpleGraphAdapter extends GraphAdapter<Node> { return n instanceof CFGNode ? Role.CTRL : isMem(n) ? Role.MEM : Role.DATA; if( n instanceof RegionNode && i == 0 ) return Role.ASSOC; if( i == 0 || n instanceof RegionNode || n instanceof StopNode ) return Role.CTRL; - if( n instanceof MemMergeNode || i == 1 && (n instanceof MemOpNode || n instanceof ReturnNode || n instanceof CallNode) ) return Role.MEM; + if( n instanceof MemMergeNode || (i == 1 && (n instanceof MemOpNode || n instanceof ReturnNode || n instanceof CallNode || n instanceof NewNode)) ) return Role.MEM; Node def = n.in(i); if( def != null && isMem(def) ) return Role.MEM; return Role.DATA; diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Chapter10Test.java b/chapter23/src/test/java/com/seaofnodes/simple/Chapter10Test.java index 3c5a2913..121e8a22 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/Chapter10Test.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/Chapter10Test.java @@ -2,6 +2,8 @@ package com.seaofnodes.simple; import com.seaofnodes.simple.codegen.CodeGen; import org.junit.Test; +import com.seaofnodes.simple.node.*; +import java.util.BitSet; import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; @@ -468,4 +470,121 @@ return 0; assertEquals("return 0;", code.print()); } + @Test public void testReadBeforeStores() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !a = new S; S !b = new S; + a.x = 11; b.x = 22; + S !p = a; if (arg) p = b; + int before = p.x; + a.y = 55; b.y = 66; + a.x = 33; b.x = 44; + return before*100 + p.x; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals("1133",Eval2.eval(code,0)); + assertEquals("2244",Eval2.eval(code,1)); + } + + static final String NESTED_MEMORY = """ + struct S { int x; int y; int z; }; + S !s = new S; + s.x = 5; s.y = 7; s.z = 11; + while (arg > 0) { + s.x = s.x + 1; + int j = 3; + while (j > 0) { + j = j - 1; + if (j == 1) continue; + s.y = s.y + arg; + if (arg == 2) break; + s.z = s.z + 1; + } + arg = arg - 1; + } + return s; + """; + + @Test public void testMemoryAcrossNestedLoops() { + var code = new CodeGen(NESTED_MEMORY).parse().opto(); + StopNode stop = code._stop; + for (int arg=0; arg<7; arg++) { + assertEquals("S{x="+(5L+arg)+",y="+(7L+arg*(arg+1)-(arg>=2 ? 2 : 0))+ + ",z="+(11L+2*arg-(arg>=2 ? 2 : 0))+"}",Eval2.eval(code,arg)); + } + } + + @Test public void testMemoryAtEarlyReturns() { + var code = new CodeGen(""" + struct S { int x; int y; int z; }; + S !s = new S; + s.x = 3; s.y = 5; s.z = 7; + if (arg) { s.x = 11; s.z = 13; return s; } + s.y = 17; + return s; + """).parse().opto(); + StopNode stop = code._stop; + for (int arg=0; arg<2; arg++) { + assertEquals(arg==0 ? "S{x=3,y=17,z=7}" : "S{x=11,y=5,z=13}",Eval2.eval(code,arg)); + } + } + + @Test public void testKeepLoadsAtMerge() { + var code = new CodeGen(""" + struct S { int x; }; + S !a = new S; S !b = new S; + a.x = arg; b.x = arg+1; + S !p = a; S !q = b; + if (arg<0) { p=b; q=a; } + int v; + if (arg>1) v=p.x; else v=q.x; + return v; + """).parse().opto(); + StopNode stop = code._stop; + // The one-step safety check stops at the memory aggregate. + assertEquals(2,countMemoryNodes(stop,LoadNode.class,new BitSet())); + assertEquals("-1",Eval2.eval(code,-1)); + assertEquals("1",Eval2.eval(code,0)); + assertEquals("3",Eval2.eval(code,3)); + } + + @Test public void testDropStores() { + var code = new CodeGen(""" + struct S { int x; }; + S !s = new S; + if (arg) s.x=arg+1; else s.x=arg+2; + return s; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals(1,countMemoryNodes(stop,StoreNode.class,new BitSet())); + assertEquals("S{x=2}",Eval2.eval(code,0)); + assertEquals("S{x=4}",Eval2.eval(code,3)); + } + + @Test public void testKeepReadsBeforeWrites() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !a = new S; S !b = new S; + a.x=arg; b.x=arg+1; + S !p=a; S !q=b; + if (arg<0) { p=b; q=a; } + int v; + if (arg>1) { v=p.x; p.x=41; p.y=5; } + else { v=q.x; q.x=42; } + return v; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals(2,countMemoryNodes(stop,LoadNode.class,new BitSet())); + assertEquals("-1",Eval2.eval(code,-1)); + assertEquals("1",Eval2.eval(code,0)); + assertEquals("3",Eval2.eval(code,3)); + } + + private static int countMemoryNodes(Node n, Class<?> kind, BitSet seen) { + if (n==null || seen.get(n._nid)) return 0; + seen.set(n._nid); + int cnt=kind.isInstance(n) ? 1 : 0; + for (Node def : n._inputs) cnt+=countMemoryNodes(def,kind,seen); + return cnt; + } } diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Chapter10bTest.java b/chapter23/src/test/java/com/seaofnodes/simple/Chapter10bTest.java new file mode 100644 index 00000000..f2b83cdd --- /dev/null +++ b/chapter23/src/test/java/com/seaofnodes/simple/Chapter10bTest.java @@ -0,0 +1,60 @@ +package com.seaofnodes.simple; + +import com.seaofnodes.simple.codegen.*; +import com.seaofnodes.simple.node.*; +import org.junit.Test; +import static org.junit.Assert.*; + +public class Chapter10bTest { + @Test public void testIndependentField() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !s = new S; + s.x = 42; + while (arg > 0) { s.y = s.y + arg; arg = arg - 1; } + return s.x; + """).parse().opto(); + var stop = code._stop; + assertEquals("return 42;",stop.toString()); + assertEquals("42",Eval2.eval(code,5)); + assertNull(stop.walk(n -> n instanceof LoadNode ? n : null)); + } + + @Test public void testParallelMemoryPhis() { + var code = new CodeGen(""" + struct S { int x; int y; int z; }; + S !a = new S; S !b = new S; + S !p = a; if (arg) p = b; + while (arg > 0) { + a.x = p.x + arg; + a.y = p.y + 2; + a.z = p.z + 3; + arg = arg - 1; + } + return a; + """).parse().opto(); + var stop = code._stop; + int[] phis = {0}; + stop.walk(n -> { + if (n instanceof MemPhiNode) phis[0]++; + if (n instanceof MemMergeNode m && m.in(1) instanceof BulkMemPhiNode b) + for(int a=b._aliases.nextSetBit(0); a>=0; a=b._aliases.nextSetBit(a+1)) + assertTrue(a<m.nIns() && m.in(a)!=null); + assertFalse(n instanceof PhiNode && n.isMem() && + !(n instanceof MemPhiNode) && !(n instanceof BulkMemPhiNode) && !(n instanceof ParmNode)); + return null; + }); + assertTrue(phis[0]>=3); + assertEquals("S{x=1,y=2,z=3}",Eval2.eval(code,4)); + } + + @Test public void testLateSliceWorklist() { + // Seed 97 exposed a store selecting a bulk predecessor without queuing + // it. Keep both the worklist assertion and the final heap check. + for (int seed : new int[]{0,97,123,456}) { + var code = new CodeGen(Chapter10Test.NESTED_MEMORY,com.seaofnodes.simple.type.TypeInteger.BOT,seed,false).parse(); + code.opto(); + assertEquals("S{x=9,y=25,z=17}",Eval2.eval(code,4)); + } + } +} diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Chapter16Test.java b/chapter23/src/test/java/com/seaofnodes/simple/Chapter16Test.java index 310efc45..57ec5508 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/Chapter16Test.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/Chapter16Test.java @@ -253,4 +253,27 @@ return new Square; assertEquals("Square{side=3.0,diag=1.7320508075688772}", Eval2.eval(code, 3)); assertEquals("Square{side=4.0,diag=2.0}", Eval2.eval(code, 4)); } + static final String CONSTRUCTOR_MEMORY = """ + struct S { int x; int y; }; + struct T { int z=arg+40; }; + T !t = new T; + S !a = new S { x=11; y=7; }; + S !b = new S { x=22; y=9; }; + S !p=a; + if (arg) p=b; + int before=p.x; + S !c = new S { + x=p.x+1; + { int i=0; while (i<2) { p.y=p.y+1; i=i+1; } } + y=p.y; + }; + p.x=33; + return before*10000+c.x*100+c.y+t.z; + """; + + @Test public void testConstructorMemory() { + var code = new CodeGen(CONSTRUCTOR_MEMORY).parse().opto(); + assertEquals("111249",Eval2.eval(code,0)); + assertEquals("222352",Eval2.eval(code,1)); + } } diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Chapter17Test.java b/chapter23/src/test/java/com/seaofnodes/simple/Chapter17Test.java index 53c23ecd..37208308 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/Chapter17Test.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/Chapter17Test.java @@ -369,7 +369,7 @@ var b = arg ? new Bar { next = (arg==2) ? new Bar{x=2;}; x=1; }; return b ? b.next ? b.next.x : b.x; // parses "b ? (b.next ? b.next.x : b.x) : 0" """); code.parse().opto(); - assertEquals("return Phi(Region,.x,0);", code.print()); + assertEquals("return Phi(Region,.x,0,.x);", code.print()); assertEquals("0", Eval2.eval(code, 0)); assertEquals("1", Eval2.eval(code, 1)); assertEquals("2", Eval2.eval(code, 2)); diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Chapter18Test.java b/chapter23/src/test/java/com/seaofnodes/simple/Chapter18Test.java index 299d31f9..1c1d011e 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/Chapter18Test.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/Chapter18Test.java @@ -439,4 +439,70 @@ return p.coffee_count; assertEquals("1", Eval2.eval(code, 2)); } + static final String CALL_MEMORY = """ + struct S { int x; int y; }; + val bump = { S s, int d -> int old=s.x; s.x=old+d; return old; }; + S !a = new S { x=10; y=7; }; + S !b = new S { x=20; y=9; }; + S !p=a; if (arg) p=b; + int before=p.x; + int old=bump(p,3); + int after=p.x; + bump(a,5); + return before*1000000+old*10000+after*100+a.x+a.y; + """; + + @Test public void testCallMemory() { + var code = new CodeGen(CALL_MEMORY).parse().opto(); + int[] calls={0}; + code._stop.walk(n -> { if(n instanceof CallNode) calls[0]++; return null; }); + assertEquals(2,calls[0]); // Keep real calls so this checks scheduling across them. + assertEquals("10101325",Eval2.eval(code,0)); + assertEquals("20202322",Eval2.eval(code,1)); + code._stop.walk(n -> { + if(n instanceof CallNode call) { + assertEquals(5,call.nIns()); // No scheduling edges may become extra arguments. + assertTrue(call.fptr()._type instanceof TypeFunPtr); + } + return null; + }); + } + + static final String RECURSIVE_MEMORY = """ + struct S { int x; }; + val rec = { S s, int n -> + if (n==0) return s.x; + int before=s.x; + s.x=before+1; + return before+rec(s,n-1); + }; + S !s=new S { x=10; }; + int value=rec(s,arg); + return value*100+s.x; + """; + + @Test public void testRecursiveMemory() { + var code = new CodeGen(RECURSIVE_MEMORY).parse().opto(); + for(int n=0; n<7; n++) + assertEquals(Long.toString((10L*(n+1)+n*(n+1)/2)*100+10+n),Eval2.eval(code,n)); + } + + static final String INLINE_MEMORY = """ + struct S { int x; int y; }; + val make = { S s -> + S t = new S { x=s.x+1; y=3; }; + s.y=9; + return t; + }; + S !a = new S { x=arg+40; y=7; }; + S t=make(a); + return a.x*10000+a.y*100+t.x+t.y; + """; + + @Test public void testInlineAllocationMemory() { + var code = new CodeGen(INLINE_MEMORY).parse().opto(); + assertNull(code._stop.walk(n -> n instanceof CallNode ? n : null)); + assertEquals("400944",Eval2.eval(code,0)); + assertEquals("420946",Eval2.eval(code,2)); + } } diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Chapter19Test.java b/chapter23/src/test/java/com/seaofnodes/simple/Chapter19Test.java index 93f31e19..7ac16a93 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/Chapter19Test.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/Chapter19Test.java @@ -351,4 +351,72 @@ return sq(arg) + sq(3); code.driver(Phase.LocalSched,"x86_64_v2", "SystemV"); assertEquals("Stop[ return (add,#2,#2); return (mul,Parm_x(sq,i64),x); ]", code.print()); } + + @Test public void testSelectedMemory() { + for( String cpu : new String[]{"x86_64_v2","riscv","arm"} ) + for( String src : new String[]{Chapter10Test.NESTED_MEMORY,Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY,Chapter18Test.RECURSIVE_MEMORY} ) { + var code = new CodeGen(src).parse().opto().typeCheck(); + var aliases = memoryPhis(code._stop); + code.loopTree().instSelect(cpu,"SystemV"); + assertEquals(aliases,memoryPhis(code._stop)); + code.GCM().localSched(); + code._stop.walk(n -> { + if( !(n instanceof CFGNode) && !(n instanceof ProjNode) ) + assertTrue(n.in(0) instanceof CFGNode); + if( n instanceof NewNode nn ) { + assertEquals(2,((TypeTuple)nn._type)._types.length); + assertTrue(nn.mem() instanceof MemMergeNode); + assertNull(nn.mem().in(1)); // Partial allocation input. + nn.cacheRegs(code); + MachNode mach = (MachNode)nn; + assertNull(mach.regmap(1)); + assertNotNull(mach.regmap(2)); + assertNotNull(mach.outregmap(0)); + assertNull(mach.outregmap(1)); + } + if( n instanceof CallNode call ) { + assertEquals(4,call.nIns()); // ctrl, memory, two args; direct target is embedded. + } + return null; + }); + } + } + + private static java.util.ArrayList<String> memoryPhis(Node stop) { + var phis = new java.util.ArrayList<String>(); + stop.walk(n -> { + if( n instanceof MemPhiNode phi ) phis.add("alias:"+phi._alias); + if( n instanceof BulkMemPhiNode phi ) phis.add("bulk:"+phi._aliases); + assertFalse(n instanceof PhiNode && n.isMem() && + !(n instanceof MemPhiNode) && !(n instanceof BulkMemPhiNode) && !(n instanceof ParmNode)); + return null; + }); + java.util.Collections.sort(phis); + return phis; + } + + @Test public void testFoldedReadBeforeWrites() { + var code = new CodeGen(""" + struct S { int x; }; + S !a = new S { x=11; }; S !b = new S { x=22; }; + S !p = a; if( arg ) p = b; + int before = p.x + arg; + a.x=33; b.x=44; + return before; + """).parse().opto().typeCheck().loopTree().instSelect("x86_64_v2","SystemV").GCM().localSched(); + var read = (MemOpNode)code._stop.walk(n -> + n instanceof com.seaofnodes.simple.node.cpus.x86_64_v2.AddMemX86 ? n : null); + assertNotNull(read); + int writes = 0; + for( Node use : read.antiDeps() ) + if( use instanceof MemOpNode st && st.isMem() ) { + assertSame(read.cfg0(),st.cfg0()); + assertTrue(st._inputs.find(read)>=0); + assertTrue(read.cfg0()._outputs.find(read)<st.cfg0()._outputs.find(st)); + assertNull(((MachNode)st).regmap(st._inputs.find(read))); + writes++; + } + assertEquals(1,writes); // The next Store follows this first clobber through memory. + } } diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Chapter20Test.java b/chapter23/src/test/java/com/seaofnodes/simple/Chapter20Test.java index d0c48688..1489b9ac 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/Chapter20Test.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/Chapter20Test.java @@ -162,7 +162,7 @@ for( int i=0; i<ary#-1; i++ ) ary[i+1] += ary[i]; return ary[1] * 1000 + ary[3]; // 1 * 1000 + 6 """; - testTarget(src,"x86_64_v2", "SystemV",7,"return .[];"); + testTarget(src,"x86_64_v2", "SystemV",5,"return .[];"); testTarget(src,"riscv" , "SystemV",7,"return (add,.[],(mul,.[],1000));"); testTarget(src,"arm" , "SystemV",5,"return (add,.[],(mul,.[],1000));"); } @@ -366,8 +366,29 @@ return a; """; testTarget(src,"x86_64_v2", "SystemV",52,null); testTarget(src,"riscv" , "SystemV",44,null); - testTarget(src,"arm" , "SystemV",44,null); + testTarget(src,"arm" , "SystemV",41,null); //assertEquals("int[ 1,2,3,4,5,6,7,8,9,10,11]", Eval2.eval(code, 11)); } + + @Test public void testMemoryAllocation() { + for( String cpu : new String[]{"x86_64_v2","riscv","arm"} ) + for( String src : new String[]{Chapter10Test.NESTED_MEMORY,Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY,Chapter18Test.RECURSIVE_MEMORY, + "int[] !a = new int[3]; a[arg]=arg; return a[0]+1;"} ) { + var code = new CodeGen(src).parse().opto().typeCheck().loopTree() + .instSelect(cpu,"SystemV").GCM().localSched().regAlloc(); + com.seaofnodes.simple.codegen.RegAllocTestSupport.checkRegisters(code); + code._stop.walk(n -> { + // Whole memory, precise Phis, and memory projections need no register. + if( n._type instanceof com.seaofnodes.simple.type.TypeMem ) + assertEquals(-1,code._regAlloc.regnum(n)); + if( n instanceof com.seaofnodes.simple.node.NewNode nn ) { + assertTrue(code._regAlloc.regnum(nn.proj(0))>=0); + assertEquals(-1,code._regAlloc.regnum(nn.proj(1))); + } + return null; + }); + } + } } diff --git a/chapter23/src/test/java/com/seaofnodes/simple/Eval2.java b/chapter23/src/test/java/com/seaofnodes/simple/Eval2.java index f5b78f91..feb67ba9 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/Eval2.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/Eval2.java @@ -314,6 +314,8 @@ public abstract class Eval2 { case TypeFunPtr tfp -> tfp; case TypeMem mem -> "MEM"; case TypeMemPtr tmp -> { + // Unknown-caller inputs are type placeholders, not heap constants. + if( !tmp.isConstant() ) yield tmp; if( tmp._obj.isAry() ) { // Constant array ptr TypeConAry con = (TypeConAry)tmp._obj.field("[]")._t; Object[] xs = new Object[con.len()]; @@ -341,7 +343,7 @@ public abstract class Eval2 { private static Object alloc(NewNode alloc) { TypeStruct type = alloc._ptr._obj; if( type.isAry() ) { - long sz = (Long)val(alloc.in(1)); + long sz = (Long)val(alloc.size()); long x = offToIdx(sz, type); int n = (int)x; if( n!=x || n<0 ) @@ -462,9 +464,9 @@ public abstract class Eval2 { case TypeRPC rpc -> sb.p(x.toString()); case TypeMem mem -> sb.p("$mem"); case TypeTuple tt -> { - if( tt._types.length>1 && tt._types[1] instanceof TypeMemPtr ) + if( tt._types.length>1 && tt._types[0] instanceof TypeMemPtr ) // Assume a NewNode - yield _print( tt._types[1], x, sb, visit ); + yield _print( tt._types[0], x, sb, visit ); throw Utils.TODO(); } case Type tt -> { diff --git a/chapter23/src/test/java/com/seaofnodes/simple/EvalArm64.java b/chapter23/src/test/java/com/seaofnodes/simple/EvalArm64.java index 1ab3aa6c..e0cbdcff 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/EvalArm64.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/EvalArm64.java @@ -542,10 +542,13 @@ public class EvalArm64 { break; } - case 0xD2: { + case 0xD2: // MOVZ + case 0xF2: { // MOVK preserves the other halfwords // movz // get immediate - rval = (ir >> 5) & 0x7FFF; + int shift = ((ir >> 21) & 3)*16; + rval = ((ir >> 5) & 0xFFFFL) << shift; + if( opcode1==0xF2 ) rval |= regs[rdid] & ~(0xFFFFL << shift); break; } diff --git a/chapter23/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java b/chapter23/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java new file mode 100644 index 00000000..0f09b7f0 --- /dev/null +++ b/chapter23/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java @@ -0,0 +1,178 @@ +package com.seaofnodes.simple; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.codegen.RegAllocTestSupport; +import com.seaofnodes.simple.node.*; +import com.seaofnodes.simple.node.cpus.arm.arm; +import com.seaofnodes.simple.node.cpus.riscv.riscv; +import com.seaofnodes.simple.type.TypeMemPtr; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import org.junit.Test; +import static org.junit.Assert.*; + +// Execution regressions are separate from the fixed spill-measurement cohorts. +public class MemoryEncodingTest { + @Test public void testLoopLoadSearch() { + String[] sources = { + """ + u8[] !program=new u8[106]; u8[] !out=new u8[0]; + int sum=0; + for(int i=0; i<arg; i++) { + sum+=program#; + if(i==1) out=new u8[i+1]; + else if(i==3) out=new u8[2]; + } + return sum+out#; + """, + // One possibly aliasing arm must reject the unchanged-memory proof. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; if(arg&1) p=b; + int sum=0; + for(int i=0; i<6; i++) { + if(i==2) p.x=30; + else if(i==4) b.x=40; + sum+=a.x; + } + return sum; + """, + // Unchanged memory does not imply an unchanged loop-carried pointer. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; int sum=0; + for(int i=0; i<6; i++) { + if(i==2) a.x=30; + sum+=p.x; + p=b; if(i&1) p=a; + } + return sum; + """, + // All arms fold, but their values differ: build a value Phi. + """ + struct S { int x; }; S !a=new S; a.x=1; int sum=0; + for(int i=0; i<6; i++) { + sum+=a.x; + if(i&1) a.x=2; + else if(i==2) a.x=3; + else a.x=4; + } + return sum; + """ + }; + for( int p=0; p<sources.length; p++ ) { + CodeGen code=new CodeGen(sources[p]).parse().opto(); + if( p==0 ) assertNull(code._stop.walk(n -> + n instanceof LoadNode ld && ld.ptr() instanceof ProjNode ptr && + ptr.in(0) instanceof NewNode nn && + nn.size()._type==com.seaofnodes.simple.type.TypeInteger.constant(110) ? ld : null)); + for( int arg=0; arg<7; arg++ ) { + long expected=switch(p) { + case 0 -> 106*arg+(arg>=2 ? 2 : 0); + case 1 -> (arg&1)==0 ? 140 : 60; + case 2 -> 130; + default -> 16; + }; + assertEquals(Long.toString(expected),Eval2.eval(code,arg)); + } + } + } + + private static final String[] SOURCES = { + Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY, + Chapter18Test.RECURSIVE_MEMORY, + Chapter18Test.INLINE_MEMORY, + "int[] !a = new int[7]; a[arg]=arg+3; return a[0]+1;", + """ + int[] !a = new int[7]; + for( int i=0; i<7; i++ ) a[i]=i+arg; + for( int i=0; i<6; i++ ) a[i+1]+=a[i]; + return a[6]; + """ + }; + + private static long expected(int program, long arg) { + return switch(program) { + case 0 -> arg==0 ? 111249 : 222351+arg; + case 1 -> arg==0 ? 10101325 : 20202322; + case 2 -> (10*(arg+1)+arg*(arg+1)/2)*100+10+arg; + case 3 -> 400944+arg*10001; + case 4 -> arg==0 ? 4 : 1; + case 5 -> 21+7*arg; + default -> throw new AssertionError(); + }; + } + + @Test public void testEmulatedMemory() throws IOException { + for( String target : new String[]{"riscv","arm"} ) + for( int p=0; p<SOURCES.length; p++ ) { + CodeGen code = new CodeGen(SOURCES[p]).driver(target,"SystemV",null); + RegAllocTestSupport.checkRegisters(code); + int entry=0; + for( var n : code._start._outputs ) + if( n instanceof FunNode fun && "main".equals(fun._name) ) + entry=code._encoding._opStart[fun._nid]; + for( int arg=0; arg<7; arg++ ) { + byte[] image = new byte[1<<20]; + byte[] bits = code._encoding.bits(); + System.arraycopy(bits,0,image,0,bits.length); + long result; + if( target.equals("riscv") ) { + EvalRisc5 cpu = new EvalRisc5(image,1<<16); + cpu._pc=entry; + cpu.regs[riscv.A0]=arg; + assertEquals(target+" program "+p+" arg "+arg,0,cpu.step(100000)); + result=cpu.regs[riscv.A0]; + } else { + EvalArm64 cpu = new EvalArm64(image,1<<16); + cpu._pc=entry; + cpu.regs[arm.X0]=arg; + assertEquals(target+" program "+p+" arg "+arg,0,cpu.step(100000)); + result=cpu.regs[arm.X0]; + } + assertEquals(target+" program "+p+" arg "+arg,expected(p,arg),result); + } + } + } + + @Test public void testNeverMemory() { + // A synthetic exit must retain both precise slices of a no-exit loop. + // Stop at scheduling: the existing allocator cannot coalesce TOP on + // the synthetic scalar return arm. + for( String target : new String[]{"x86_64_v2","riscv","arm"} ) { + CodeGen code = new CodeGen(""" + struct S { int x; int y; }; + S !a=new S; + if( arg ) while( 1 ) { a.x+=arg; a.y+=a.x; } + return a.x+a.y; + """).driver(CodeGen.Phase.LoopTree); + assertNotNull(code._stop.walk(n -> { + if( n instanceof MemMergeNode mem && + mem.in(0) instanceof CProjNode exit && exit.in(0) instanceof NeverNode ) { + var fields = ((TypeMemPtr)Parser.TYPES.get("S"))._obj._fields; + assertTrue(mem.alias(fields[0]._alias) instanceof MemPhiNode); + assertTrue(mem.alias(fields[1]._alias) instanceof MemPhiNode); + return mem; + } + return null; + })); + code.instSelect(target,"SystemV").GCM().localSched(); + } + } + + @Test public void testNativeMemory() throws IOException { + Files.createDirectories(Path.of("build/objs")); + for( int p=0; p<SOURCES.length; p++ ) { + String file="build/objs/memory"+p; + StringBuilder c = new StringBuilder("extern long long memory(long long);\nint main() {\n"); + for( int arg=0; arg<7; arg++ ) + c.append("if(memory(").append(arg).append(")!=").append(expected(p,arg)).append("LL) return ").append(arg+1).append(";\n"); + c.append("return 0;\n}\n"); + Files.writeString(Path.of(file+".c"),c); + TestC.run("val memory = { int arg -> "+SOURCES[p]+" };", + TestC.CALL_CONVENTION,null,"",file+".c",file,"S","",-1); + } + } +} diff --git a/chapter23/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java b/chapter23/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java index 911acc5c..1c116162 100644 --- a/chapter23/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java +++ b/chapter23/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java @@ -184,7 +184,7 @@ public class Evaluator { Object[] body=null; int num; if( type.isAry() ) { - long sz = (Long)val(alloc.in(1)); + long sz = (Long)val(alloc.size()); long n = offToIdx(sz, type); if( n < 0 ) throw new NegativeArraySizeException(""+n); @@ -198,21 +198,17 @@ public class Evaluator { assert elem instanceof TypeMemPtr; } // Length value - body[0] = vall(alloc.in(2+2)); + body[0] = n; } else { body = new Object[num = type._fields.length]; for( int i=0; i<num; i++ ) body[i] = switch( alloc._ptr._obj._fields[i]._t ) { case TypeInteger ti -> 0L; - case TypeFloat tf -> 0; + case TypeFloat tf -> 0D; default -> null; }; } - Object[] mems = new Object[type._fields.length+2]; - // mems[0] is control - mems[1] = new Obj(type,body); // the ref - // mems[2+...] are memory aliases - return mems; + return new Object[]{new Obj(type,body),null}; } private Object load(LoadNode load) { diff --git a/chapter24/README.md b/chapter24/README.md index 58e13f39..59a893d9 100644 --- a/chapter24/README.md +++ b/chapter24/README.md @@ -5,10 +5,12 @@ where `score` in the middle does not have to be repeated. The main compiler change is Interprocedural Sparse Conditional Constant Propagation (SCCP): an optimistic type solve which also discovers call-graph -edges. This snapshot still uses the earlier parser/type and memory machinery; -Chapter 25 currently contains the larger incomplete-types SSA redesign and -separate compilation. Independent correctness fixes will move backward first; -see the [backport review queue](../docs/chapter-backports.md). +edges. Memory follows Chapter 10b's lazy partitioning: BulkMemPhi represents +whole memory, MemPhi merges precise aliases, and allocations consume and produce +partial memory for their instance fields. During SCCP, memory Phis ignore +unreachable inputs and allocation results stay high until control reaches them. +Chapter 25 retains the larger incomplete-types SSA redesign, private constructor +memory, and separate compilation; see the [backport review queue](../docs/chapter-backports.md). You can also read [this chapter](https://github.com/SeaOfNodes/Simple/tree/linear-chapter24) in a linear Git revision history on the [linear](https://github.com/SeaOfNodes/Simple/tree/linear) branch and [compare](https://github.com/SeaOfNodes/Simple/compare/linear-chapter23...linear-chapter24) it to the previous chapter. @@ -299,22 +301,22 @@ both the cheaper first attempt and the mandatory fallback. Chapter 21's adjacent-copy forwarding also carries forward: after coloring, a sole use in the next instruction can read the source register when its operand mask permits it and no two-address tie is broken. The table below -includes this cleanup. Cohort 22 now also includes the two zero-move C return +includes this cleanup and the lazy memory forward port. Cohort 22 now also includes the two zero-move C return ABI checks added since the earlier audit (26 entries instead of 24). Run `make spill-stats`. All rows use **Chapter 24's compiler**, optimizer seed 123, and the same source/target combinations as the earlier cohort tables. These Windows results combine x86 SystemV/Win64 and RISC-V/ARM SystemV, -remeasured on 2026-09-28 after the function-pointer lifetime backport. +remeasured after the lazy memory forward port. | Program cohort | Compilations | Retained moves | Loop-weighted moves | |---|---:|---:|---:| -| Chapter 20 | 39 | 319 | 431 | -| Chapter 21 | 52 | 425 | 957 | +| Chapter 20 | 39 | 320 | 439 | +| Chapter 21 | 52 | 420 | 952 | | Chapter 22 | 26 | 67 | 67 | -| Chapter 23 | 30 | 81 | 228 | -| Chapter 24 | 67 | 418 | 1,286 | -| **Total** | **214** | **1,310** | **2,969** | +| Chapter 23 | 30 | 78 | 225 | +| Chapter 24 | 67 | 422 | 1,325 | +| **Total** | **214** | **1,307** | **3,008** | `_spills` counts retained SplitNodes, including register moves. `_spillScaled` weights those moves by `8^loopDepth`; it is a cost estimate, not measured memory diff --git a/chapter24/src/main/java/com/seaofnodes/simple/Parser.java b/chapter24/src/main/java/com/seaofnodes/simple/Parser.java index 14af367d..432ec137 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/Parser.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/Parser.java @@ -138,7 +138,7 @@ public class Parser { _scope.define(ScopeNode.ARG0, TypeInteger.BOT, false, null, _lexer); ctrl(XCTRL); - _scope.mem(new MemMergeNode(false)); + _scope.mem(con(TypeMem.BOT)); // Parse the sys import _lexer = new Lexer(sys.SYS); @@ -222,7 +222,7 @@ public class Parser { // Build a multi-exit return point for all function returns RegionNode r = new RegionNode( null,null,null).init(); assert r.inProgress(); - PhiNode rmem = new PhiNode(ScopeNode.MEM0,TypeMem.BOT,r,null).init(); + PhiNode rmem = new BulkMemPhiNode(ScopeNode.MEM0,r,null).init(); PhiNode rrez = new PhiNode(ScopeNode.ARG0,Type.BOTTOM,r,null).init(); ReturnNode ret = new ReturnNode(r, rmem, rrez, rpc, fun).init(); fun.setRet(ret); @@ -235,11 +235,8 @@ public class Parser { // external calls. _scope.push(new Kind.Func()); ctrl(fun); // Scope control from function - // Private mem alias tracking per function - MemMergeNode mem = new MemMergeNode(true); - mem.addDef(null); // Alias#0 - mem.addDef(new ParmNode(ScopeNode.MEM0,1,TypeMem.BOT,fun,con(TypeMem.BOT)).peephole()); // All aliases - _scope.mem(mem); + // Unknown callers supply whole memory; field aliases are discovered lazily. + _scope.mem(new ParmNode(ScopeNode.MEM0,1,TypeMem.BOT,fun,con(TypeMem.BOT)).peephole()); // All args, "as-if" called externally for( int i=0; i<ids.length; i++ ) { Type t = sig.arg(i); @@ -254,7 +251,7 @@ public class Parser { // Last expression is the return except for the top-level main if( ctrl()._type==Type.CONTROL ) if( fun.sig() != _code._main ) - fun.addReturn(ctrl(), _scope.mem().merge(), last); + fun.addReturn(ctrl(), _scope.mem(), last); else { fun.setDef(1,XCTRL); // Kill default main _code.addAll(fun._outputs); @@ -634,9 +631,8 @@ public class Parser { */ private Node parseReturn() { var expr = require(parseAsgn(), ";"); - // Need default memory, since it can be lazy, need to force - // a non-lazy Phi - _fun.addReturn(ctrl(), _scope.mem().merge(), expr); + // Force the lazy whole-memory Phi before closing this exit. + _fun.addReturn(ctrl(), _scope.mem(), expr); ctrl(XCTRL); // Kill control return expr; } @@ -1438,31 +1434,31 @@ public class Parser { Field[] fs = obj._fields; if( fs==null ) throw error("Unknown struct type '" + obj._name + "'"); - int len = fs.length; - Node[] ns = new Node[2+len]; - ns[0] = ctrl(); // Control in slot 0 - // Total allocated length in bytes - ns[1] = size; - // Memory aliases for every field - for( int i = 0; i < len; i++ ) - if( !fs[i]._one ) - ns[2+i] = memAlias(fs[i]._alias); - Node nnn = new NewNode(TypeMemPtr.make(obj), ns).peephole().keep(); - for( int i = 0; i < len; i++ ) - if( !fs[i]._one ) - memAlias(fs[i]._alias, new ProjNode(nnn,i+2,memName(fs[i]._alias)).peephole()); - Node ptr = new ProjNode(nnn.unkeep(),1,obj._name).peephole().keep(); - - // Initial nonzero values for every field - for( int i = 0; i < len; i++ ) { - Node val = init.get( i + idx ); + Node prior = mem().keep(); + MemMergeNode input = new MemMergeNode((Node)null); + for( Field f : fs ) if( !f._one ) input.alias(f._alias,prior); + // Allocation zeroes its fields; nonzero initializers become explicit Stores. + Node[] ns = new Node[3]; + ns[0] = ctrl(); + ns[1] = input.peephole(); + ns[2] = size; + Node nnn = new NewNode(TypeMemPtr.make(obj),ns).peephole().keep(); + Node ptr = new ProjNode(nnn,0,obj._name).peephole().keep(); + Node out = new ProjNode(nnn,1,"$mem").peephole().keep(); + MemMergeNode after = new MemMergeNode(prior); + for( Field f : fs ) if( !f._one ) after.alias(f._alias,out); + mem(after.peephole()); + out.unkeep(); + nnn.unkeep(); + prior.unkeep(); + for( int i=0; i<fs.length; i++ ) { + Node val = init.get(idx+i); if( !fs[i]._one && val._type != val._type.makeZero() ) { - Node mem = memAlias(fs[i]._alias); - Node st = new StoreNode(loc(),fs[i]._fname,fs[i]._alias,fs[i]._t,mem,ptr,off(obj,fs[i]._fname),val,true).peephole(); - memAlias(fs[i]._alias,st); + Field f = fs[i]; + Node st = new StoreNode(loc(),f._fname,f._alias,f._t,mem(),ptr,off(obj,f._fname),val,true).peephole(); + mem(new MemMergeNode(mem(),f._alias,st).peephole()); } } - return ptr.unkeep(); } @@ -1488,13 +1484,18 @@ public class Parser { return con(str); } - // We set up memory aliases by inserting special vars in the scope these - // variables are prefixed by $ so they cannot be referenced in Simple code. - // Using vars has the benefit that all the existing machinery of scoping - // and phis work as expected - private Node memAlias(int alias ) { return _scope.mem(alias ); } - private void memAlias(int alias, Node st) { _scope.mem(alias, st); } - public static String memName(int alias) { return ("$"+alias).intern(); } + // Memory is one hidden SSA variable, including across branches and loops. + private Node mem() { return _scope.mem(); } + private void mem(Node n) { _scope.mem(n); } + + private void store(String name, int alias, Type glb, Node ptr, Node off, Node val, boolean init, Node ctrl) { + Node prior = mem().keep(); + Node st = new StoreNode(loc(),name,alias,glb,prior,ptr,off,val,init); + st.setDef(0,ctrl); + st = st.peephole(); + mem(new MemMergeNode(prior,alias,st).peephole()); + prior.unkeep(); + } /** * Parse postfix expression; this can be a field expression, an array @@ -1554,6 +1555,7 @@ public class Parser { if( base.isAry() && tf instanceof TypeConAry con ) tf = con.elem(); + expr.keep(); // Field offset; fixed for structs, computed for arrays Node off = (name.equals("[]") // If field is an array body // Array index math @@ -1566,21 +1568,14 @@ public class Parser { Node val = parseAsgn().keep(); Node lift = liftExpr( val, tf, f._final, false ); - Node st = new StoreNode(loc(), name, f._alias, tf, memAlias(f._alias), expr, off.unkeep(), lift, false); - // Arrays include control, as a proxy for a safety range check. - // Structs don't need this; they only need a NPE check which is - // done via the type system. - if( base.isAry() ) st.setDef(0,ctrl()); - memAlias(f._alias, st.peephole()); + store(name,f._alias,tf,expr.unkeep(),off.unkeep(),lift,false,base.isAry() ? ctrl() : null); return val.unkeep(); // "obj.a = expr" returns the expression while updating memory } - // Keep expr for possible store update - expr.keep(); // Load field Node load = f._one && INITS.containsKey( base._name ) ? INITS.get(base._name).in(base.find(name)) - : new LoadNode(loc(),name, f._alias, tf, memAlias(f._alias), expr, off); + : new LoadNode(loc(),name, f._alias, tf, mem(), expr, off); // Arrays include control, as a proxy for a safety range check // Structs don't need this; they only need a NPE check which is // done via the type system. @@ -1590,13 +1585,8 @@ public class Parser { // Check for assign-update, "ptr.fld += expr" or "ary[idx]++" char ch = _lexer.matchOperAssign(); if( ch!=0 ) { - Node op = opAssign(ch,load, tf ); - Node st = new StoreNode(loc(), name, f._alias, tf, memAlias(f._alias), expr.unkeep(), off.unkeep(), op, false); - // Arrays include control, as a proxy for a safety range check. - // Structs don't need this; they only need a NPE check which is - // done via the type system. - if( base.isAry() ) st.setDef(0,ctrl()); - memAlias(f._alias, peep(st)); + Node op = opAssign(ch,load,tf); + store(name,f._alias,tf,expr.unkeep(),off.unkeep(),op,false,base.isAry() ? ctrl() : null); load = postfix(ch) ? load.unkeep() : op; // And use the original loaded value as the result return load; @@ -1707,7 +1697,7 @@ public class Parser { } // Control & memory after parsing args args.set(0,ctrl().keep()); - args.set(1,_scope.mem().merge().keep()); + args.set(1,_scope.mem().keep()); args.push(fcn); // Function pointer // Unkeep them all for( Node arg : args ) @@ -1727,11 +1717,8 @@ public class Parser { call.peephole(); // Rerun peeps after CallEnd, allows early inlining // Control from CallEnd ctrl(new CProjNode(cend,0,ScopeNode.CTRL).peephole()); - // Memory from CallEnd - MemMergeNode mem = new MemMergeNode(true); - mem.addDef(null); // Alias#0 - mem.addDef(new ProjNode(cend,1,ScopeNode.MEM0).peephole()); - _scope.mem(mem); + // A call may update any alias, so its memory result is the new whole state. + _scope.mem(new ProjNode(cend,1,ScopeNode.MEM0).peephole()); // Call result return new ProjNode(cend,2,"#2").peephole(); } diff --git a/chapter24/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java b/chapter24/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java index b8a8d2b3..1d802cae 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java @@ -50,7 +50,7 @@ public abstract class GlobalCodeMotion { var cons = new ArrayList<Node>(); for( Node con : start._outputs ) if( con!=null && !(con instanceof CFGNode) && - (con.isConst() || con instanceof MachNode mach && mach.outregmap()!=null) ) { + (con.isConst() || (con instanceof MachNode mach && mach.outregmap()!=null)) ) { globals.put(con,true); cons.add(con.keep()); // Preserve original inputs while rewiring users. } @@ -172,16 +172,8 @@ public abstract class GlobalCodeMotion { // Loads need their memory inputs' uses also done if( n instanceof MemOpNode ld && ld._isLoad ) - for( Node memuse : ld.mem()._outputs ) - if( late[memuse._nid]==null && - // New makes new memory, never crushes load memory - !(memuse instanceof NewNode) && - // Load-use directly defines memory - (memuse._type instanceof TypeMem || - // Load-use directly defines memory - memuse instanceof CallNode || - // Load-use indirectly defines memory - (memuse._type instanceof TypeTuple tt && tt._types[ld._alias] instanceof TypeMem)) ) + for( Node memuse : ld.antiDeps() ) + if( late[memuse._nid]==null ) continue outer; // All uses done, schedule @@ -193,9 +185,7 @@ public abstract class GlobalCodeMotion { for( Node def : n._inputs ) { if( def==null ) continue; if( late[def._nid]==null ) work.push(def); - for( Node out : def._outputs ) - if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) - work.push(ld); + wakeLoads(def,late,work,new BitSet()); } if( n instanceof LoopNode loop ) for( Node phi : loop._outputs ) @@ -255,23 +245,32 @@ public abstract class GlobalCodeMotion { best instanceof IfNode; } + // An allocation may consume a load's memory through a partial aggregate. + private static void wakeLoads(Node def, CFGNode[] late, WorkList<Node> work, BitSet visit) { + if( visit.get(def._nid) ) return; + visit.set(def._nid); + for( Node out : def._outputs ) + if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) work.push(ld); + if( def instanceof MemMergeNode ) + for( int i=1; i<def.nIns(); i++ ) + if( def.in(i)!=null ) wakeLoads(def.in(i),late,work,visit); + } + private static CFGNode find_anti_dep(CFGNode lca, MemOpNode load, CFGNode early, CFGNode[] late, int[] anti) { // We could skip final-field loads here. // Walk LCA->early, flagging Load's block location choices for( CFGNode cfg=lca; early!=null && cfg!=early.idom(); cfg = cfg.idom() ) anti[cfg._nid] = load._nid; // Walk load->mem uses, looking for Stores causing an anti-dep - for( Node mem : load.mem()._outputs ) { + for( Node mem : load.antiDeps() ) { switch( mem ) { case MemOpNode st: - if( !st._isLoad && load._alias == st._alias ) { - assert late[mem._nid] != null; - lca = anti_dep(load,late[mem._nid],lca,st,anti); - } - break; // Loads do not cause anti-deps on other loads - case CallNode call: - assert late[call._nid]!=null; - lca = anti_dep(load,late[call._nid],lca,call,anti); + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); + break; + case CallNode st: + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); break; case PhiNode phi: // Repeat anti-dep for matching Phi inputs. @@ -280,10 +279,10 @@ public abstract class GlobalCodeMotion { if( phi.in(i)==load.mem() ) lca = anti_dep(load,phi.region().cfg(i),lca,null,anti); break; - case NewNode st: break; - case ReturnNode ret: break; // Load must already be ahead of Return - case MemMergeNode ret: break; // Mem uses now on ScopeMin - case NeverNode never: break; + case NewNode st: + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); + break; default: throw Utils.TODO(); } } @@ -295,7 +294,7 @@ public abstract class GlobalCodeMotion { // Stores are already placed. Constrain the load at that block only. if( anti[stblk._nid]==load._nid ) { lca = stblk.domLCA(lca,null); - if( lca==stblk && st!=null && st._inputs.find(load) == -1 ) + if( lca==stblk && st!=null && !(st instanceof CallNode) && st._inputs.find(load) == -1 ) st.addDef(load); // Same-block load must precede the store. } return lca; diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java new file mode 100644 index 00000000..7d4f1fb8 --- /dev/null +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java @@ -0,0 +1,240 @@ +package com.seaofnodes.simple.node; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.type.Type; +import com.seaofnodes.simple.type.TypeMem; +import java.util.BitSet; + +/** + * A Phi for bulk memory. It covers all aliases except {@link #_aliases}; + * those aliases must be represented by parallel precise MemPhis at the same + * program slice. + */ +public class BulkMemPhiNode extends PhiNode { + + private static final BitSet EMPTY = new BitSet(); + + // Excluded Aliases; empty means NO exclusions. + // Each split clones the exclusions; published sets are never mutated. + public final BitSet _aliases; + + public BulkMemPhiNode(String label, Node... inputs) { + this(label,EMPTY,inputs); + } + + private BulkMemPhiNode(String label, BitSet aliases, Node... inputs) { + super(label,TypeMem.BOT,inputs); + _aliases = aliases; + } + + public boolean isSplit(int alias) { return _aliases.get(alias); } + + public BulkMemPhiNode(BulkMemPhiNode phi) { super(phi); _aliases = phi._aliases; } + + @Override public String label() { return "BulkPhi"+_aliases; } + + @Override + public Type compute() { + Type t = super.compute(); + return t instanceof TypeMem mem ? mem : + t==Type.BOTTOM ? TypeMem.BOT : + TypeMem.TOP; + } + + @Override + public Node idealize() { + Node progress = super.idealize(); + if( progress != null ) return progress; + if( ((RegionNode)region()).inProgress() || region().nIns()<=1 ) + return null; // Input is in-progress + + // "Peek through" a MemMerge that covers this alias set on its default + for( int i=1; i<nIns(); i++ ) + if( in(i) instanceof MemMergeNode mmm && canPeek(mmm) ) { + setDef(i,CodeGen.CODE.add(mmm.in(1))); + return this; + } + + // If any input or output uses or defines a specific alias we cover, + // make a private MemPhi for it, and add it to the excluded list. + for( int i=1; i<nIns(); i++ ) { + int alias = inputAlias(in(i)); + if( alias!=0 ) + return slice(alias); + } + + for( int i=0; i<_outputs.size(); i++ ) { + int alias = outputAlias(_outputs.get(i)); + if( alias!=0 ) + return slice(alias); + } + + return null; + } + + static public boolean checkMem(Node n, int alias) { + for( Node m : n._inputs ) + // Bulk memory supplies this alias + if( m instanceof BulkMemPhiNode bulk ) { + if( bulk._aliases.get(alias) ) + return false; + } else if( m instanceof MemMergeNode mmm && + mmm.in(1) instanceof BulkMemPhiNode bulk && + bulk._aliases.get(alias) && + (alias >= mmm.nIns() || mmm.in(alias)==null) ) + return false; + return true; + } + + + // True if the MemMerge has no precise slice still covered by this Phi. + private boolean canPeek(MemMergeNode mmm) { + return mmm.in(1)!=null && missingAlias(mmm)==0; + } + + // Return an alias required by an input but still covered by this Phi, or 0. + private int inputAlias(Node n) { + return switch(n) { + case ProjNode proj -> 0; + case ParmNode parm -> 0; + case ConstantNode con -> 0; + case MemMergeNode mmm -> missingAlias(mmm); + case MemOpNode mem -> unsplit(mem._alias); + case BulkMemPhiNode bulk -> missingAlias(bulk); + default -> throw new AssertionError("Unexpected memory input "+n); + }; + } + + // Return an alias required by a user but still covered by this Phi, or 0. + private int outputAlias(Node use) { + // This decision depends backwards on an immediate user's alias/set. + // Normal graph-neighbor enqueueing flows from defs to uses, so retain + // an explicit forward dependency to revisit this Phi when the user + // sharpens. + addDepForwards(use); + return switch(use) { + case ScopeNode scope -> 0; + case MemMergeNode mmm -> missingAlias(mmm); + case BulkMemPhiNode bulk -> missingAlias(bulk); + case MemOpNode mem -> unsplit(mem._alias); + case MemPhiNode phi -> unsplit(phi._alias); + default -> { + assert use instanceof ReturnNode || use instanceof CallNode || use instanceof ParmNode + : "Unexpected bulk-memory user "+use; + yield 0; + } + }; + } + + // Alias zero/one is bulk, and an already excluded precise alias cannot + // trigger another split of this Phi. + private int unsplit(int alias) { + if( alias <= 1 ) return 0; + assert !_aliases.get(alias); + return alias; + } + + // First alias excluded by the neighboring bulk Phi but not by this one. + private int missingAlias(BulkMemPhiNode bulk) { + for( int alias = bulk._aliases.nextSetBit(0); alias >= 0; alias = bulk._aliases.nextSetBit(alias+1) ) + if( !_aliases.get(alias) ) + return alias; + return 0; + } + + // Every explicit slot requests that alias, including a partial allocation + // input whose slot still points to this bulk Phi. + private int missingAlias(MemMergeNode mmm) { + for( int alias=2; alias<mmm.nIns(); alias++ ) + if( mmm.in(alias)!=null && + mmm.alias(alias)!=mmm.in(1) && + !_aliases.get(alias) ) + return alias; + return 0; + } + + // Slice out given alias + private MemMergeNode slice( int alias ) { + assert !_aliases.get(alias); + // Inputs are the same; MemPhi will sharpen his own inputs. If a + // parallel precise Phi already exists, reuse it instead of building a + // duplicate for the same Region/alias pair. + Node mem = precisePhi(alias); + + BitSet aliases = ((BitSet)_aliases.clone()); + aliases.set(alias); + BulkMemPhiNode bphi = new BulkMemPhiNode(_label,aliases); + for( int i=0; i<nIns(); i++ ) + bphi.addDef(in(i)); + // Installing bphi creates a new bulk user with a different exclusion + // set. Queue the original memory inputs before bphi itself can peep + // away and hide those neighbor relationships. + for( int i=1; i<bphi.nIns(); i++ ) + CodeGen.CODE.add(bphi.in(i)); + // Do not peephole bphi here, as BulkMemPhi can recursively start a + // second bulk rewrite before this one has finished. Let the worklist + // discover any further splits + bphi.setType(TypeMem.BOT); + Node bulk = bphi; + CodeGen.CODE.add(bulk); + CodeGen.CODE.add(mem); + return aggregate(bulk,alias,mem); + } + + // Build aggregate memory over a bulk Phi, preserving every precise slice + // which the bulk has already excluded. The named alias is newly supplied + // (or replaces its old slice). + MemMergeNode aggregate(Node bulk, int alias, Node precise) { + MemMergeNode mmm = new MemMergeNode(bulk); + for( int old = _aliases.nextSetBit(0); old >= 0; old = _aliases.nextSetBit(old+1) ) + { assert old != alias; mmm.alias(old,precisePhi(old));} + mmm.alias(alias,precise); + assert checkMerge(mmm); + return mmm; + } + + // Find or make the precise Phi parallel to this bulk Phi at the same merge. + private Node precisePhi(int alias) { + MemPhiNode mphi = _findPhi(alias); + if( mphi!=null ) return mphi; + + mphi = new MemPhiNode("$"+alias,alias); + mphi.addDef(region()); + // Due to cycles, must set before calling peephole + mphi.setType(TypeMem.make(alias,Type.BOTTOM)); + for( int i=1; i<nIns(); i++ ) + mphi.addDef(CodeGen.CODE.add(preciseInput(in(i),alias))); + return CodeGen.CODE.add(mphi); + } + + MemPhiNode _findPhi(int alias) { + for( Node use : region()._outputs ) + if( use instanceof MemPhiNode mphi && mphi._alias==alias ) + return mphi; + return null; + } + + // Select an alias already split out of a predecessor's bulk memory. + private Node preciseInput(Node mem, int alias) { + if( mem instanceof MemMergeNode mmm ) + return mmm.alias(alias); + if( mem instanceof BulkMemPhiNode bulk && bulk.isSplit(alias) ) + return bulk.precisePhi(alias); + return mem; + } + + // Every alias excluded by a bulk default must be supplied explicitly. + static boolean checkMerge(MemMergeNode mmm) { + if( !(mmm.in(1) instanceof BulkMemPhiNode bulk) ) return true; + for( int alias = bulk._aliases.nextSetBit(0); alias >= 0; alias = bulk._aliases.nextSetBit(alias+1) ) + if( alias >= mmm.nIns() || mmm.in(alias)==null ) + return false; + return true; + } + + @Override public boolean eq(Node n) { + return _aliases.equals(((BulkMemPhiNode)n)._aliases) && super.eq(n); + } + + @Override int hash() { return _aliases.hashCode(); } +} diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/LoadNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/LoadNode.java index 5f28a39b..12095868 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/LoadNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/LoadNode.java @@ -1,7 +1,7 @@ package com.seaofnodes.simple.node; -import com.seaofnodes.simple.Parser; import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.Parser; import com.seaofnodes.simple.type.*; import com.seaofnodes.simple.util.Utils; import java.util.BitSet; @@ -24,6 +24,25 @@ public class LoadNode extends MemOpNode { super(loc, name, alias, true, glb, mem, ptr, off); } + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && !clobbered(dep); + } + + // Check only immediate memory users. Writers clobber memory; Phis and + // aggregates might lead to a clobber, so stop rather than search further. + private boolean clobbered(Node dep) { + Node mem = mem(); + dep.addDepForwards(mem); + for( Node use : mem._outputs ) { + if( use==null ) continue; + dep.addDepForwards(use); + if( use instanceof StoreNode || use instanceof PhiNode || + use instanceof MemMergeNode || + use instanceof NewNode || use instanceof CallNode ) return true; + } + return false; + } + // Debugger label @Override public String label() { return "ld_"+mlabel(); } @Override @@ -52,7 +71,7 @@ public class LoadNode extends MemOpNode { if( t instanceof TypeConAry ary ) t = ary.elem(); // TODO: if offset is known, can peek the constant // Lift from declared type and memory input - t = t.join(mem._t); + t = t.join(MemMergeNode.contents(mem(),_alias,this)); if( _declaredType.isFinal() ) t = t.makeRO(); // Deep final applied // Pinch between declared type @@ -62,35 +81,19 @@ public class LoadNode extends MemOpNode { @Override public Node idealize() { - Node ptr = ptr(); - Node mem = mem(); - - if( mem instanceof CastNode cast ) { + if( mem() instanceof CastNode cast ) { setDef(1,cast.in(1)); return this; } - - // Simple Load-after-Store on same address. - if( mem instanceof StoreNode st && - ptr == st.ptr() && off() == st.off() ) { // Must check same object - assert _name.equals(st._name); // Equiv class aliasing is perfect - return extend(st.val()); - } - - // Simple load-after-MemMerge to a known alias can bypass. Happens when inlining. - if( mem instanceof MemMergeNode mem2 ) { - Node memA = mem2.alias(_alias); - for( Node ld : memA._outputs ) - if( ld instanceof LoadNode ) - CodeGen.CODE.add(ld); - setDef(1,memA); + if( mem() instanceof MemMergeNode merge ) { + setDef(1,CodeGen.CODE.add(merge.alias(_alias))); return this; } + Node ptr = ptr(); + Node mem = mem(); - // Simple Load-after-New on same address. - if( mem instanceof ProjNode p && p.in(0) instanceof NewNode nnn && - ptr == nnn.proj(1) ) // Must check same object - return zero(nnn); // Load zero from new + Node win = find(mem,ptr,null,null); + if( win!=null ) return folded(win); // Uplift control to a prior dominating load. for( Node memuse : mem._outputs ) @@ -99,50 +102,7 @@ public class LoadNode extends MemOpNode { cfg0()!=null && cfg0().domLCA(ld.cfg0(),this) == ld.cfg0() ) // Higher control means load is legal earlier return ld; - // Load-after-Store on same address, but bypassing provably unrelated - // stores. This is a more complex superset of the above two peeps. - // "Provably unrelated" is really weak. - if( ptr instanceof ReadOnlyNode ro ) - ptr = ro.in(1); - outer: - while( true ) { - switch( mem ) { - case StoreNode st: - if( ptr == addDep(st.ptr()) && off() == st.off() ) - return extend(castRO(st.val())); // Proved equal - // Can we prove unequal? Offsets do not overlap? - if( !off()._type.join(st.off()._type).isHigh() && // Offsets overlap - !neverAlias(ptr,st.ptr()) ) // And might alias - break outer; // Cannot tell, stop trying - // Pointers cannot overlap - mem = st.mem(); // Proved never equal - break; - case PhiNode phi: - // Assume related - addDep(phi); - break outer; - case ConstantNode top: break outer; // Assume shortly dead - case ProjNode mproj: // Memory projection - switch( mproj.in(0) ) { - case NewNode nnn1: - if( ptr instanceof ProjNode pproj && pproj.in(0) == mproj.in(0) ) - return zero(nnn1); - if( !(ptr instanceof ProjNode pproj && pproj.in(0) instanceof NewNode) ) - break outer; // Cannot tell, ptr not related to New - mem = nnn1.in(nnn1.findAlias(_alias));// Bypass unrelated New - break; - case StartNode start: break outer; - case CallEndNode cend: break outer; // TODO: Bypass no-alias call - default: throw Utils.TODO(); - } - break; - case CastNode cast: mem = cast.in(1); break; - case MemMergeNode merge: mem = merge.alias(_alias); break; - - default: - throw Utils.TODO(); - } - } + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); // Push a Load up through a Phi, as long as it collapses on at least // one arm. If at a Loop, the backedge MUST collapse - else we risk @@ -151,19 +111,27 @@ public class LoadNode extends MemOpNode { // if( pred ) ptr.x = e0; val = pred ? e0 // else ptr.x = e1; : e1; // val = ptr.x; ptr.x = val; - if( mem() instanceof PhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + if( mem() instanceof MemPhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + // Array access control must not be moved across the merge. + in(0)==null && // Offset can be hoisted off() instanceof ConstantNode && // Pointer can be hoisted hoistPtr(ptr,memphi) ) { - // Profit on RHS/Loop backedge - if( profit(memphi,2) || - // Else must not be a loop to count profit on LHS. - (!(memphi.region() instanceof LoopNode) && profit(memphi,1)) ) { - Node ld1 = ld(1); - Node ld2 = ld(2); - return new PhiNode(_name,_declaredType,memphi.region(),ld1,ld2); + // Returning to this memory means no change only if the pointer + // is also loop invariant. A pointer Phi may select another object. + Node stop = memphi.region() instanceof LoopNode && + !(ptr instanceof PhiNode p && p.region()==memphi.region()) ? memphi : null; + if( find(memphi.in(2),ptr,stop,new BitSet())!=null || + (!(memphi.region() instanceof LoopNode) && find(memphi.in(1),ptr,null,new BitSet())!=null) ) { + PhiNode phi = new PhiNode(_name,_declaredType,memphi.region()).keep(); + phi.setType(_type); // Backedge searches may return this value Phi. + for( int i=1; i<memphi.nIns(); i++ ) { + Node p = ptr instanceof PhiNode pp && pp.region()==memphi.region() ? pp.in(i) : ptr; + phi.addDef(load(memphi.in(i),p,stop,phi)); + } + return CodeGen.CODE.add(phi.unkeep()); } } @@ -178,9 +146,67 @@ public class LoadNode extends MemOpNode { return castRO(new ConstantNode(zero).peephole()); } - private Node ld( int idx ) { - Node mem = mem(), ptr = ptr(); - return new LoadNode(_loc,_name,_alias,_declaredType,mem.in(idx),ptr instanceof PhiNode && ptr.in(0)==mem.in(0) ? ptr.in(idx) : ptr, off()).peephole(); + // Search only; no new nodes or rewiring. Return a Store/New that folds, + // the unchanged loop memory, or a merge whose every arm folds. Null fails. + // Register dependencies so a later pointer/offset rewrite retries the query. + private Node find(Node mem, Node ptr, Node stop, BitSet visit) { + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); + while( mem!=null ) { + addDep(mem); + if( mem==stop ) return mem; + switch( mem ) { + case MemMergeNode merge: mem=merge.alias(_alias); break; + case CastNode cast: mem=cast.in(1); break; + case StoreNode st: + addDep(st.ptr()); + if( _alias==st._alias && ptr==st.ptr() && off()==st.off() ) return st; + if( _alias==st._alias && !neverAlias(ptr,st.ptr()) && + !addDep(off())._type.join(addDep(st.off())._type).isHigh() ) return null; + mem=st.mem(); + break; + case ProjNode proj: + if( !(proj.in(0) instanceof NewNode nnn) ) return null; + if( ptr instanceof ProjNode p && p.in(0)==nnn ) return nnn; + if( !(ptr instanceof ProjNode p && p.in(0) instanceof NewNode) ) return null; + mem=nnn.mem(); // Distinct allocations cannot overlap. + break; + case MemPhiNode phi: + addDep(phi.region()); + // Only the original loop closes a successful cycle. Another + // loop or an unfinished merge needs a separate proof. + if( visit==null || !(phi.region() instanceof RegionNode r) || r instanceof LoopNode || + r.inProgress() || phi.inProgress() || r._type!=Type.CONTROL ) return null; + if( visit.get(phi._nid) ) return phi; // Already proved every arm. + for( int i=1; i<phi.nIns(); i++ ) + if( find(phi.in(i),ptr,stop,visit)==null ) return null; + visit.set(phi._nid); + return phi; + default: return null; + } + } + return null; + } + + private Node folded(Node win) { + return win instanceof StoreNode st ? extend(castRO(st.val())) : zero((NewNode)win); + } + + // Materialize only after the profitability search succeeds. The value Phi + // stands for a load which comes back around the loop with unchanged memory. + private Node load(Node mem, Node ptr, Node stop, PhiNode value) { + Node win=find(mem,ptr,stop,new BitSet()); + if( win!=null ) { + if( win==stop ) return value; + if( win instanceof MemPhiNode mp ) { + PhiNode phi=new PhiNode(_name,_declaredType,mp.region()).keep(); + for( int i=1; i<mp.nIns(); i++ ) phi.addDef(load(mp.in(i),ptr,stop,value)); + return phi.unkeep().peephole(); + } + return folded(win).peephole(); + } + LoadNode ld=new LoadNode(_loc,_name,_alias,_declaredType,mem,ptr,off()); + ld.setDef(0,in(0)); + return ld.peephole(); } private static boolean neverAlias( Node ptr1, Node ptr2 ) { @@ -209,16 +235,6 @@ public class LoadNode extends MemOpNode { return false; } - // Profitable if we find a matching Store on this Phi arm. - private boolean profit(PhiNode phi, int idx) { - Node px = phi.in(idx); - if( px==null ) return false; - if( px._type instanceof TypeMem mem && mem._t.isHighOrConst() ) return true; - if( px instanceof StoreNode st1 && ptr()==addDep(st1.ptr() )&& off()==st1.off() ) return true; - addDep(px); - return false; - } - // Read-Only is a deep property, and cannot be cast-away private Node castRO(Node rez) { if( ptr()._type.isFinal() && !rez._type.isFinal() ) @@ -240,4 +256,5 @@ public class LoadNode extends MemOpNode { Node shl = new ShlNode(null,val,shf.keep()).peephole(); return new SarNode(null,shl,shf.unkeep()); } + } diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/LoopNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/LoopNode.java index 6c69a56e..db52c156 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/LoopNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/LoopNode.java @@ -57,11 +57,12 @@ public class LoopNode extends RegionNode { // Now fold control into the exit. Might have 1 valid exit, or an // XCtrl or a bunch of prior NeverNode exits. Node top = new ConstantNode(Type.TOP).peephole(); - Node memout = new MemMergeNode(false); - memout.addDef(f); // placeholder for control + MemMergeNode memout = new MemMergeNode(new ConstantNode(TypeMem.BOT).peephole()); + memout.setDef(0,f); for( Node u : _outputs ) - if( u instanceof PhiNode phi && phi._type.isa(TypeMem.BOT) ) - memout.addDef(phi); + if( u instanceof BulkMemPhiNode ) memout.setDef(1,u); + else if( u instanceof MemPhiNode phi ) memout.alias(phi._alias,phi); + memout.init(); Node ctrl = ret.ctrl(), mem = ret.mem(), expr = ret.expr(); if( ctrl!=null && ctrl._type != Type.XCONTROL ) { @@ -72,7 +73,7 @@ public class LoopNode extends RegionNode { // Nope, insert an aligned exit layer RegionNode r = new RegionNode(_loc,null,ctrl).init(); ctrl = r; r._ltree = stop._ltree; - mem = new PhiNode(r,mem ).init(); + mem = new BulkMemPhiNode(ScopeNode.MEM0,r,mem).init(); expr = new PhiNode(r,expr).init(); } // Append new Never exit diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java index 2a399b97..2894556c 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/MemMergeNode.java @@ -5,58 +5,62 @@ import com.seaofnodes.simple.codegen.CodeGen; import com.seaofnodes.simple.type.*; import java.util.*; -/** - * Memory Merge - a merge of many aliases into a "fat memory". All aliases - * are here, but most will be lazy - take the default fat memory. +/** A default slice and explicit alias overrides. A null default means partial + * memory: absent aliases are not covered, rather than having unknown contents. */ public class MemMergeNode extends Node { + public MemMergeNode(MemMergeNode mem) { super(mem); } - /* - * In-Progress means this is being used by the Parser to track memory - * aliases. No optimizations are allowed. When no longer "in progress" - * normal peeps work. - */ - public final boolean _inProgress; - - public MemMergeNode( boolean inProgress) { _type = TypeMem.BOT; _inProgress = inProgress; } - public MemMergeNode(MemMergeNode mem) { super(mem); _inProgress = false; } - - - // If being used by a Scope, this is "in progress" from the Parser. - // Otherwise, it's a memory merge - boolean inProgress() { return _inProgress; } + public MemMergeNode(Node bulk) { super(null,bulk); _type = TypeMem.BOT; } + public MemMergeNode(Node bulk, int alias, Node precise) { + this(bulk); + alias(alias,precise); + } + public Node alias(int alias) { + Node n = alias < nIns() ? in(alias) : null; + assert n!=null || !(in(1) instanceof BulkMemPhiNode bulk) || !bulk.isSplit(alias); + n = n==null ? in(1) : n; + assert n!=null : "Alias not covered: "+alias; + return n; + } - @Override public String label() { return "ALLMEM"; } - @Override public boolean isMem() { return true; } + public void alias(int alias, Node mem) { + assert alias > 1; + while( nIns()<=alias ) addDef(null); + setDef(alias,mem); + } - @Override - public StringBuilder _print1(StringBuilder sb, BitSet visited) { - sb.append(_inProgress ? "Merge[" : "MEM[ "); - for( int j=2; j<nIns(); j++ ) { - sb.append(j); - sb.append(":"); - Node n = in(j); - while( n instanceof ScopeNode loop ) { - sb.append("Lazy_"); - n = loop.mem().alias(j); - } - if( n==null ) sb.append("___ "); - else n._print0(sb, visited).append(" "); + // Scalar contents of one alias. New zeroes the allocated fields; + // join it with the incoming contents for all previously allocated objects. + // Phis use their cached types, so this query does not recurse around loops. + static Type contents(Node mem, int alias, Node dep) { + dep.addDep(mem); + if( mem instanceof CastNode cast ) return contents(cast.in(1),alias,dep); + if( mem instanceof MemMergeNode merge ) + return contents(merge.alias(alias),alias,dep); + if( mem instanceof ProjNode proj && proj.in(0) instanceof NewNode nnn ) { + assert proj._idx==1 && nnn.field(alias)!=null; + dep.addDep(nnn); + if( dep.addDep(nnn.in(0))._type.isHigh() ) return Type.TOP; + return contents(nnn.mem(),alias,dep).meet(nnn.field(alias)._t.makeZero()); } - sb.setLength(sb.length()-1); - return sb.append("]"); + // Function parameters and call results may contain arbitrary heap values. + // Their bulk memory types do not imply empty or zero-filled storage. + if( mem._type==Type.TOP || mem._type==TypeMem.TOP ) return Type.TOP; + if( mem._type instanceof TypeMem mt && mt._alias==alias ) return mt._t; + return Type.BOTTOM; } - // Make a memory merge: no longer a Scope really, tracking memory state but - // not related to the parser in any way. - public Node merge() { - // Force default memory to not be lazy - MemMergeNode merge = new MemMergeNode(false); - for( Node n : _inputs ) - merge.addDef(n); + @Override public String label() { return "MemMerge"; } + @Override public boolean isMem() { return true; } + @Override public StringBuilder _print1(StringBuilder sb, BitSet visited) { + sb.append("MEM["); for( int i=1; i<nIns(); i++ ) - merge._mem(i,null); - return merge.peephole(); + if( in(i)!=null ) { + sb.append(i==1 ? "default:" : i+":"); + in(i)._print0(sb,visited).append(" "); + } + return sb.append("]"); } @@ -68,118 +72,27 @@ public class MemMergeNode extends Node { } @Override public Node idealize() { - if( inProgress() ) return null; - - // Fold defaults into the default boolean progress=false, allDefault=true; for( int i=2; i<nIns(); i++ ) { - if( in(i) instanceof CastNode cast ) - { setDef(i,cast.in(1)); progress=true; } - if( in(1) == in(i) ) { setDef(i,null); progress=true; } - else { allDefault=false; } + if( in(i) instanceof CastNode cast ) { setDef(i,cast.in(1)); progress=true; } + if( in(i)!=null && in(i)==in(1) ) { setDef(i,null); progress=true; } + if( in(i) instanceof MemMergeNode mem ) { + setDef(i,CodeGen.CODE.add(mem.alias(i))); + progress=true; + } + if( in(i)!=null ) allDefault=false; } + if( allDefault && in(1)!=null ) return in(1); - // If not merging any memory (all memory is just the default) - if( allDefault ) - return in(1); // Become default memory - - // Collapse stacked all-mem if( in(1) instanceof MemMergeNode mem ) { - // Goal is to swap my default mem with mem's default mem - for( int i=2; i<mem.nIns(); i++ ) { - if( mem.in(i) != null ) { - // deeper default mem has a non-default - if( i>=nIns() || in(i)==null ) - setDefX(i,mem.in(i)); - } - } + for( int i=2; i<mem.nIns(); i++ ) + if( mem.in(i)!=null && (i>=nIns() || in(i)==null) ) + alias(i,mem.in(i)); + CodeGen.CODE.add(mem.in(1)); setDef(1,mem.in(1)); return this; } - + assert BulkMemPhiNode.checkMerge(this); return progress ? this : null; } - - - public Node in( Var v ) { return in(v._idx); } - - public Node alias( int alias ) { - return alias < nIns() && in(alias)!=null ? in(alias) : in(1); - } - - Node alias( int alias, Node st ) { - while( alias >= nIns() ) addDef(null); - return setDef(alias,st); - } - - // Read or update from memory. - // A shared implementation allows us to create lazy phis both during - // lookups and updates; the lazy phi creation is part of chapter 8. - Node _mem( int alias, Node st ) { - // Memory projections are made lazily; if one does not exist - // then it must be START.proj(1) - Node old = alias(alias); - if( old instanceof ScopeNode loop ) { - MemMergeNode loopmem = loop.mem(); - Node memdef = loopmem.alias(alias); - // Lazy phi! - old = memdef instanceof PhiNode phi && loop.ctrl()==phi.region() - // Loop already has a real Phi, use it - ? memdef - // Set real Phi in the loop head - // The phi takes its one input (no backedge yet) from a recursive - // lookup, which might have insert a Phi in every loop nest. - : loopmem.alias(alias, new PhiNode(Parser.memName(alias), TypeMem.make(alias,Type.BOTTOM),loop.ctrl(),loopmem._mem(alias,null),null).peephole() ); - alias(alias,old); - } - // Memory projections are made lazily; expand as needed - return st==null ? old : alias(alias,st); // Not lazy, so this is the answer - } - - - void _merge( MemMergeNode that, RegionNode r) { - int len = Math.max(nIns(),that.nIns()); - for( int i = 1; i < len; i++) - if( alias(i) != that.alias(i) ) { // No need for redundant Phis - // If we are in lazy phi mode we need to a lookup - // by alias as it will trigger a phi creation - Node lhs = this._mem(i,null); - Node rhs = that._mem(i,null); - alias(i, new PhiNode(Parser.memName(i), lhs._type.meet(rhs._type).glb(true), r, lhs, rhs).peephole()); - } - } - - // Fill in the backedge of any inserted Phis - void _endLoopMem( ScopeNode scope, MemMergeNode back, MemMergeNode exit ) { - Node exit_def = exit.alias(1); - for( int i=1; i<nIns(); i++ ) { - if( in(i) instanceof PhiNode phi && phi.region()==scope.ctrl() ) { - assert phi.in(2)==null; - phi.setDef(2,back.alias(i)==scope ? phi : back.alias(i)); // Fill backedge - } - if( exit_def == scope ) // Replace a lazy-phi on the exit path also - exit.alias(i,in(i)); - } - } - - // Now one-time do a useless-phi removal - void _useless( ) { - for( int i=2; i<nIns(); i++ ) { - if( in(i) instanceof PhiNode phi ) { - // Do an eager useless-phi removal - Node in = phi.peephole(); - CodeGen.CODE.addAll(phi._outputs); - phi.moveDepsToWorklist(); - if( in != phi ) { - if( !phi.iskeep() ) // Keeping phi around for parser elsewhere - phi.subsume(in); - setDef(i,in); // Set the update back into Scope - } - } - } - } - - @Override public boolean eq( Node n ) { - return this==n || !_inProgress; - } } diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/MemOpNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/MemOpNode.java index ead82322..65021110 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/MemOpNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/MemOpNode.java @@ -6,6 +6,7 @@ import com.seaofnodes.simple.type.TypeMemPtr; import com.seaofnodes.simple.util.Utils; import java.lang.StringBuilder; import java.util.BitSet; +import java.util.ArrayList; /** * Convenience common base for Load and Store. @@ -78,6 +79,8 @@ public abstract class MemOpNode extends Node { static String mlabel(String name) { return "[]".equals(name) ? "ary" : ("#".equals(name) ? "len" : name); } String mlabel() { return mlabel(_name); } + @Override public boolean isMem() { return !_isLoad; } + public Node mem() { return in(1); } public Node ptr() { return in(2); } public Node off() { return in(3); } @@ -86,6 +89,13 @@ public abstract class MemOpNode extends Node { public StringBuilder _printMach( StringBuilder sb, BitSet visited ) { throw Utils.TODO(); } public int log_size() { return _declaredType.log_size(); } + // Extra conditions for factoring matching memory operations through a Phi. + // The caller has already checked the opcode, input shape and control. + boolean canDrop(MemOpNode other, Node dep) { + dep.addDep(ptr()); + return _alias==other._alias && _declaredType==other._declaredType && err()==null; + } + @Override public boolean eq(Node n) { MemOpNode mem = (MemOpNode)n; // Invariant @@ -106,4 +116,29 @@ public abstract class MemOpNode extends Node { return null; return Parser.error( "Might be null accessing '" + _name + "'",_loc); } + // Memory writers which must follow this read. A packaging node is not a + // write, but can hide New's input, so follow this alias through aggregates. + public ArrayList<Node> antiDeps() { + ArrayList<Node> deps = new ArrayList<>(); + antiDeps(mem(),deps,new BitSet()); + return deps; + } + + private void antiDeps(Node mem, ArrayList<Node> deps, BitSet visit) { + if( visit.get(mem._nid) ) return; + visit.set(mem._nid); + for( Node use : mem._outputs ) { + if( use instanceof MemMergeNode merge ) { + Node slice = _alias<merge.nIns() ? merge.in(_alias) : null; + if( (slice==mem) || (slice==null && merge.in(1)==mem) ) + antiDeps(merge,deps,visit); + } else if( use instanceof CallNode || + (use instanceof MemOpNode st && !st._isLoad && st._alias==_alias) || + (use instanceof NewNode nnn && nnn.field(_alias)!=null) || + (use instanceof MemPhiNode phi && phi._alias==_alias) || + (use instanceof BulkMemPhiNode phi && !phi.isSplit(_alias)) ) { + if( !deps.contains(use) ) deps.add(use); + } + } + } } diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java new file mode 100644 index 00000000..1c354d5c --- /dev/null +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/MemPhiNode.java @@ -0,0 +1,44 @@ +package com.seaofnodes.simple.node; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.type.Type; +import com.seaofnodes.simple.type.TypeMem; + +/** A Phi for one field alias, parallel to a bulk Phi at the same Region. */ +public class MemPhiNode extends PhiNode { + public final int _alias; + + public MemPhiNode(String label, int alias, Node... inputs) { + super(label,TypeMem.make(alias,Type.BOTTOM),inputs); + assert alias > 1; + _alias = alias; + } + + public MemPhiNode(MemPhiNode phi) { super(phi); _alias = phi._alias; } + + @Override public String label() { return "MemPhi_"+_alias; } + + @Override public Type compute() { + assert BulkMemPhiNode.checkMem(this,_alias); + if( !(region() instanceof RegionNode r) ) + return (region()._type==Type.XCONTROL || region()._type==Type.TOP) ? TypeMem.TOP : _type; + if( r.inProgress() ) return r instanceof LoopNode ? _minType : TypeMem.BOT; + Type t = Type.TOP; + for( int i=1; i<nIns(); i++ ) + if( addDep(r.in(i))._type!=Type.XCONTROL && r.in(i)._type!=Type.TOP ) + t = t.meet(MemMergeNode.contents(in(i),_alias,this)); + return TypeMem.make(_alias,t); + } + + @Override public Node idealize() { + for( int i=1; i<nIns(); i++ ) + if( in(i) instanceof MemMergeNode mem ) { + setDef(i,CodeGen.CODE.add(mem.alias(_alias))); + return this; + } + return super.idealize(); + } + + @Override public boolean eq(Node n) { return _alias==((MemPhiNode)n)._alias && super.eq(n); } + @Override int hash() { return _alias; } +} diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/NewNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/NewNode.java index 1a15439e..5d4d5bf0 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/NewNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/NewNode.java @@ -7,33 +7,35 @@ import com.seaofnodes.simple.util.Utils; import java.util.BitSet; -/** - * Allocation! Allocate a chunk of memory, and pre-zero it. - * The inputs include control and size, and ALL aliases being set. - * The output is large tuple, one for every alias plus the created pointer. - * New is expected to be followed by projections for every alias. +/** Allocate a zeroed object. Inputs {ctrl, $mem, size}; + * results {ptr, $mem}. + * The memory input and result cover only the aliases in the allocated struct. */ public class NewNode extends Node implements MultiNode { public final TypeMemPtr _ptr; - public final int _len; public NewNode(TypeMemPtr ptr, Node... nodes) { super(nodes); assert !ptr.nullable(); _ptr = ptr; - _len = ptr._obj._fields.length; - // Control in slot 0 - assert nodes[0]._type==Type.CONTROL || nodes[0]._type == Type.XCONTROL; - // Malloc-length in slot 1 - assert nodes[1]._type instanceof TypeInteger || nodes[1]._type==Type.NIL; - for( int i=0; i<_len; i++ ) - assert ptr._obj._fields[i]._one || nodes[2 + i]._type.isa( TypeMem.BOT ); + assert nodes.length==3; + assert nodes[0]._type==Type.CONTROL || nodes[0]._type==Type.XCONTROL; + assert nodes[1]._type instanceof TypeMem; + assert nodes[2]._type instanceof TypeInteger || nodes[2]._type==Type.NIL; } - public NewNode(NewNode nnn) { super(nnn); _ptr = nnn._ptr; _len = nnn._len; } + public Node mem() { return in(1); } + public Node size() { return in(2); } + + public Field field(int alias) { + for( Field f : _ptr._obj._fields ) + if( !f._one && f._alias==alias ) return f; + return null; + } + + public NewNode(NewNode nnn) { super(nnn); _ptr = nnn._ptr; } - public Node mem (int idx) { return in(idx+2); } @Override public String label() { return "new_"+(_ptr._obj.isAry() ? "ary_"+_ptr._obj._fields[1]._t.str() : _ptr._obj.str()); } @Override @@ -42,46 +44,12 @@ public class NewNode extends Node implements MultiNode { return sb.append(_ptr._obj.str()); } - int findAlias(int alias) { - int idx = _ptr._obj.findAlias(alias); - assert idx!= -1; // Error, caller should be calling - return idx+2; // Skip ctrl, size - } - - // 0 - ctrl - // 1 - byte size - // 2-len+2 - aliases, one per field - // len+2 - 2*len+2 - initial values, one per field - public Node size() { return in(1); } @Override public TypeTuple compute() { - Field[] fs = _ptr._obj._fields; - Type[] ts = new Type[fs.length+2]; - if( in(0)._type.isHigh() ) { - ts[0] = Type.XCONTROL; - ts[1] = _ptr.makeHigh(); - for( int i=0; i<fs.length; i++ ) - ts[i+2] = Type.TOP; - } else { - ts[0] = Type.CONTROL; - ts[1] = _ptr; - for( int i=0; i<fs.length; i++ ) { - if( _ptr._obj._fields[i]._one ) { - // Once-only fields use the declared type - ts[i+2] = _ptr._obj._fields[i]._t; - } else { - // Others take from the inputs - Type mt = in(i+2)._type; - TypeMem mem = mt==Type.TOP ? TypeMem.TOP : (TypeMem)mt; - //Type tfld = mem._t.meet(mem._t.makeZero()); - Type tfld = mem._t; - Type tfld2 = tfld.join(fs[i]._t ); - ts[i+2] = TypeMem.make(fs[i]._alias,tfld2); - } - } - } - return TypeTuple.make(ts); + return in(0)._type.isHigh() + ? TypeTuple.make(_ptr.makeHigh(),TypeMem.TOP) + : TypeTuple.make(_ptr,TypeMem.BOT); } @Override @@ -126,9 +94,9 @@ public class NewNode extends Node implements MultiNode { _kills = kills; } public String op() { return "alloc"; } - public RegMask regmap(int i) { return i==1 ? _arg3Mask : null; } + public RegMask regmap(int i) { return i==2 ? _arg3Mask : null; } public RegMask outregmap() { return null; } - public RegMask outregmap(int idx) { return idx==1 ? _retMask : null; } + public RegMask outregmap(int idx) { return idx==0 ? _retMask : null; } public RegMask killmap() { return _kills; } public void asm(CodeGen code, SB sb) { sb.p("#calloc, ").p(code.reg(size())); diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/Node.java b/chapter24/src/main/java/com/seaofnodes/simple/node/Node.java index c746d7d5..48e3d63f 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/Node.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/Node.java @@ -233,6 +233,7 @@ public abstract class Node implements Cloneable { // Error is 'use' does not exist; ok for 'use' to be null. public boolean delUse( Node use ) { _outputs.del(_outputs.find(use)); + moveDepsToWorklist(); // User-count and anti-dependence queries can now change. return _outputs.isEmpty(); } @@ -309,6 +310,7 @@ public abstract class Node implements Cloneable { n.unlock(); int idx = n._inputs.find(this); n._inputs.set(idx,nnn); + n.moveDepsToWorklist(); // Rewiring can change a dependent query without changing type. nnn.addUse(n); CODE.add(n); CODE.addAll(n._outputs); @@ -556,7 +558,12 @@ public abstract class Node implements Cloneable { * or output of this node, that is, it is at least one step away. The node * being added must benefit from this node being peepholed. */ - <N extends Node> N addDep( N dep ) { + <N extends Node> N addDep( N dep ) { return addDep(dep,false); } + + // Keep an explicit dependency when inspecting an immediate user's shape. + <N extends Node> N addDepForwards(N dep) { return addDep(dep,true); } + + private <N extends Node> N addDep(N dep, boolean forwards) { // Running peepholes during the big assert cannot have side effects // like adding dependencies. if( CODE._midAssert ) return dep; @@ -564,8 +571,8 @@ public abstract class Node implements Cloneable { if( obs != null ) obs.dep(this, dep); if( dep._deps==null ) dep._deps = new Ary<>(Node.class); if( dep._deps .find(this) != -1 ) return dep; // Already on list - if( dep._inputs .find(this) != -1 ) return dep; // No need for deps on immediate neighbors - if( dep._outputs.find(this) != -1 ) return dep; + if( !forwards && dep._inputs .find(this) != -1 ) return dep; // No need for deps on immediate neighbors + if( !forwards && dep._outputs.find(this) != -1 ) return dep; dep._deps.add(this); return dep; } diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/PhiNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/PhiNode.java index cfeb1404..8d8a276c 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/PhiNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/PhiNode.java @@ -35,6 +35,10 @@ public class PhiNode extends Node { addDef(sample); } + public static PhiNode make(String label, Type type, Node... inputs) { + return type instanceof TypeMem ? new BulkMemPhiNode(label,inputs) : new PhiNode(label,type,inputs); + } + @Override public String label() { return "Phi_"+MemOpNode.mlabel(_label); } @Override @@ -114,6 +118,8 @@ public class PhiNode extends Node { if( live != null ) { if( live._type.isa(_type) ) return live; + // Let the bulk input split this alias before collapsing a precise Phi. + if( live instanceof BulkMemPhiNode ) return null; // Keep the Phi upcast return new CastNode(_type,null,live); } @@ -123,19 +129,9 @@ public class PhiNode extends Node { if( r.in(i)._type == Type.XCONTROL ) return null; - // Simple Phi-after-MemMerge to a known alias can bypass. Happens when inlining. - if( _type instanceof TypeMem tmem && tmem._alias!=1 ) { - for( int i=1; i<nIns(); i++ ) - if( in(i) instanceof MemMergeNode mem ) { - setDef(i,mem.alias(tmem._alias)); - return this; - } - } - - // Generic "pull down op" - Node progress; - if( same_op() && (progress = drop_same_op()) != null ) - return progress; + // Phi(op(a,b,...),op(c,d,...)) -> op(Phi(a,c),Phi(b,d),...). + Node down; + if( same_op() && (down=drop_same_op())!=null ) return down; // If merging Phi(N, cast(N)) - we are losing the cast JOIN effects, so just remove. if( nIns()==3 ) { @@ -170,50 +166,77 @@ public class PhiNode extends Node { // Same op on all Phi paths; all ops have only the Phi as a use. // None have a control input. private boolean same_op() { + Node op = in(1); + if( op instanceof CFGNode || op instanceof ConstantNode || op instanceof PhiNode || + op instanceof ProjNode || op instanceof NewNode || op instanceof ScopeNode || op instanceof MemMergeNode ) return false; + // A bulk Phi must split its aliases before factoring precise Stores. + if( this instanceof BulkMemPhiNode ) return false; + // Bulk splitting currently identifies parallel slices by Region/alias. + // Factoring can introduce another slice at a different memory point; + // wait until that Region's bulk partitioning has finished. + if( op instanceof MemOpNode ) + for( Node use : region()._outputs ) + if( use instanceof BulkMemPhiNode ) { + addDepForwards(use); + return false; + } + Node busy = null; for( int i=1; i<nIns(); i++ ) { - Node op = in(i); - if( in(1).getClass() != op.getClass() || op.in(0)!=null || in(1).nIns() != op.nIns() ) - return false; // Wrong class or CFG bound or mismatched inputs - if( in(1) instanceof MemOpNode mem && mem._alias != ((MemOpNode)op)._alias ) - return false; - if( op.nOuts() > 1 ) { // Too many users, but addDep in case lose users - for( Node out : op._outputs ) - if( out!=null && out!=this ) - addDep(out); - return false; + Node n = in(i); + if( addDep(region().in(i))._type==Type.XCONTROL || + op.getClass()!=n.getClass() || n.nIns()!=op.nIns() || n.in(0)!=null || !op.eq(n) ) return false; + if( n instanceof MemOpNode mem && !mem.canDrop((MemOpNode)op,this) ) return false; + // Moving a Store must remove the old effect, not duplicate it. + // Allow one shared arm: the other arms disappear into the single + // factored operation, usually reducing the total operation count. + addDepForwards(n); + if( n.nOuts()>1 ) { + for( Node use : n._outputs ) + if( use!=null && use!=this ) addDepForwards(use); + if( n instanceof StoreNode || busy!=null ) return false; + busy=n; } - for( int j=1; j<in(1).nIns(); j++ ) - if( op.in(j) instanceof ScopeNode || (op.in(j)==null ^ in(1).in(j)==null) ) - return false; // Lazy Phi input + for( int j=1; j<n.nIns(); j++ ) + if( n.in(j) instanceof ScopeNode || ((n.in(j)==null) != (op.in(j)==null)) ) return false; } return true; } private Node drop_same_op() { - assert !(in(1) instanceof CFGNode); - Node op = in(1); - Node cp = op.copy(); - cp._type = null; // Fresh type - cp.addDef(null); // No control - + Node op = in(1), cp = op.copyEmpty(); + cp._type = null; + cp.addDef(null); for( int j=1; j<op.nIns(); j++ ) { - boolean needsPhi = false; - Node x = op.in(j); // Jth input from sample #1 + Node x = op.in(j); + boolean different = false; for( int i=2; i<nIns(); i++ ) - if( in(i).in(j) != x ) - { needsPhi=true; break; } - if( needsPhi ) { - x = new PhiNode(_label,op.in(j)._type.glb(false)); - x.addDef(region()); - for( int i=1; i<nIns(); i++ ) - x.addDef(in(i).in(j)); - x = x.peephole(); + if( in(i).in(j)!=x ) different=true; + if( different ) { + Node[] ins = new Node[nIns()]; + ins[0] = region(); + Type t = Type.TOP; + for( int i=1; i<nIns(); i++ ) { + ins[i] = in(i).in(j); + t = t.meet(ins[i]._type); + } + // Not accepts both pointers and integers, but a Phi cannot mix them. + if( t==Type.BOTTOM ) { + for( int i=1; i<ins.length; i++ ) addDep(ins[i]); + cp.kill(); + return null; + } + PhiNode phi = j==1 && op instanceof MemOpNode mem + ? new MemPhiNode(_label,mem._alias,ins) + : PhiNode.make(_label,t.glb(false),ins); + x = phi.peephole(); } cp.addDef(x); } - // Test not running backwards, which can happen for e.g. And's - if( cp.compute().isa(compute()) ) - return cp; + // Factoring must not widen the result (e.g. correlated And operands). + if( cp.compute().isa(compute()) ) return cp; + for( int i=1; i<nIns(); i++ ) + for( int j=1; j<in(i).nIns(); j++ ) + if( in(i).in(j)!=null ) addDep(in(i).in(j)); cp.kill(); return null; } diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/RegionNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/RegionNode.java index e9c958ac..1ffbcd60 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/RegionNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/RegionNode.java @@ -46,10 +46,10 @@ public class RegionNode extends CFGNode { // If a CFG diamond with no merging, delete: "if( pred ) {} else {};" if( !hasPhi() && nIns()>3 && // No Phi users, just a control user - in(1) instanceof CProjNode p1 && - in(2) instanceof CProjNode p2 && - addDep(p1.in(0))==addDep(p2.in(0)) && - p1.in(0) instanceof IfNode iff ) { + in(1) instanceof CProjNode p1 && + in(2) instanceof CProjNode p2 && + addDepForwards(p1).in(0)==addDepForwards(p2).in(0) && + p1.in(0) instanceof IfNode iff ) { // Replace with the iff.ctrl directly if( nIns()==3 ) return iff.ctrl(); // Just delete the path for fat Regions diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/ReturnNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/ReturnNode.java index 80b3fe4f..ce6d15bb 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/ReturnNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/ReturnNode.java @@ -9,15 +9,7 @@ import com.seaofnodes.simple.util.SB; import com.seaofnodes.simple.util.Utils; import java.util.BitSet; -/** - * The Return node has two inputs. The first input is a control node and the - * second is the data node that supplies the return value. - * <p> - * In this presentation, Return functions as a Stop node, since multiple <code>return</code> statements are not possible. - * The Stop node will be introduced in Chapter 6 when we implement <code>if</code> statements. - * <p> - * The Return's output is the value from the data node. - */ +/** Keeps all preceding memory effects and the return value alive: {ctrl, $mem, value, rpc}. */ public class ReturnNode extends CFGNode { public FunNode _fun; @@ -51,7 +43,8 @@ public class ReturnNode extends CFGNode { @Override public Type compute() { if( inProgress () ) return TypeTuple.RET; // In progress - return TypeTuple.make(ctrl()._type,mem()._type,expr()._type); + // Inlining can delete the entry while callers still use this return. + return TypeTuple.make(ctrl()._type,TypeMem.BOT,expr()._type); } @Override public Node idealize() { diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/ScopeNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/ScopeNode.java index 04666956..cddb9f24 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/ScopeNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/ScopeNode.java @@ -12,9 +12,9 @@ import java.util.*; /** * The Scope node is purely a parser helper - it tracks names to nodes with a - * stack of variables. + * stack of variable records. */ -public class ScopeNode extends MemMergeNode { +public class ScopeNode extends Node { /** * The control is a name that binds to the currently active control @@ -24,8 +24,12 @@ public class ScopeNode extends MemMergeNode { public static final String ARG0 = "arg"; public static final String MEM0 = "$mem"; - // All active/live variables in all nested scopes, all run together. - // Maps 1-to-1 to the _inputs array. + public Node in(Var v) { return in(v._idx); } + + @Override public Type compute() { return Type.BOTTOM; } + @Override public Node idealize() { return null; } + + // All active/live variables in all nested scopes, all run together public final Ary<Var> _vars; // Lexical scope is typed: @@ -50,10 +54,10 @@ public class ScopeNode extends MemMergeNode { // A new ScopeNode public ScopeNode() { - super(true); - _vars = new Ary<>(Var .class); - _kinds = new Ary<>(Kind.class); - _guards = new Ary<>(Node.class); + _type = Type.BOTTOM; + _vars = new Ary<>(Var .class); + _kinds = new Ary<>(Kind .class); + _guards = new Ary<>(Node .class); } @Override public String label() { return "Scope"; } @@ -85,7 +89,7 @@ public class ScopeNode extends MemMergeNode { public Node ctrl() { return in(0); } - public MemMergeNode mem() { return (MemMergeNode)in(1); } + public Node mem() { return in(lookup(MEM0)); } public Var var(int i) { return _vars.at(i); } /** @@ -99,7 +103,7 @@ public class ScopeNode extends MemMergeNode { * @return Node that was bound */ public <N extends Node> N ctrl(N n) { return setDef(0,n); } - public Node mem(Node n) { return setDef(1,n); } + public Node mem(Node n) { update(MEM0,n); return n; } public void push( Kind kind ) { kind._lexSize = _vars.size(); @@ -206,12 +210,6 @@ public class ScopeNode extends MemMergeNode { return true; } - // Read from memory - public Node mem( int alias ) { return mem()._mem(alias,null); } - // Write to memory - public void mem( int alias, Node st ) { mem()._mem(alias,st); } - - /** * Lookup a name in all scopes starting from most deeply nested. * @@ -247,7 +245,7 @@ public class ScopeNode extends MemMergeNode { // Set real Phi in the loop head // The phi takes its one input (no backedge yet) from a recursive // lookup, which might have insert a Phi in every loop nest. - : loop.setDef(v._idx,new PhiNode(v._name, v.type(), loop.ctrl(), loop.in(loop.update(v,null)._idx),null).peephole()); + : loop.setDef(v._idx,PhiNode.make(v._name, v.type(), loop.ctrl(), loop.in(loop.update(v,null)._idx),null).peephole()); setDef(v._idx,old); } //assert !v._final || st==null; @@ -285,21 +283,9 @@ public class ScopeNode extends MemMergeNode { n.keep(); dup.addDef(ctrl()); // Control input is just copied - // Memory input is a shallow copy - MemMergeNode memdup = new MemMergeNode(true), mem = mem(); - memdup.addDef(null); - memdup.addDef(loop ? this : mem.in(1)); - for( int i=2; i<mem.nIns(); i++ ) - // For lazy phis on loops we use a sentinel - // that will trigger phi creation on update - memdup.addDef(loop ? this : mem.in(i)); - dup.addDef(memdup); - - // Copy of other inputs - for( int i=2; i<nIns(); i++ ) - // For lazy phis on loops we use a sentinel - // that will trigger phi creation on update - dup.addDef(loop && !var(i)._final ? this : in(i)); + // Memory is one binding, with the same lazy loop Phi as scalar names. + for( int i=1; i<nIns(); i++ ) + dup.addDef(loop && !_vars.at(i)._final ? this : in(i)); return dup; } @@ -313,22 +299,21 @@ public class ScopeNode extends MemMergeNode { */ public RegionNode mergeScopes(ScopeNode that, Parser.Lexer loc) { RegionNode r = ctrl(new RegionNode(loc,null,ctrl(), that.ctrl()).keep()); - mem()._merge(that.mem(),r); - this ._merge(that ,r); + _merge(that,r); that.kill(); // Kill merged scope CodeGen.CODE.add(r); return r.unkeep(); } private void _merge(ScopeNode that, RegionNode r) { - for( int i = 2; i < nIns(); i++) + for( int i = 1; i < nIns(); i++) if( in(i) != that.in(i) ) { // No need for redundant Phis // If we are in lazy phi mode we need to a lookup // by name as it will trigger a phi creation Var v = var(i); Node lhs = this.in(this.update(v,null)); Node rhs = that.in(that.update(v,null)); - setDef(i, new PhiNode(v._name, v.type(), r, lhs, rhs).peephole()); + setDef(i, PhiNode.make(v._name, v.type(), r, lhs, rhs).peephole()); } } @@ -353,23 +338,17 @@ public class ScopeNode extends MemMergeNode { assert ctrl instanceof LoopNode loop && loop.inProgress(); ctrl.setDef(2,back.ctrl()); - mem()._endLoopMem( this, back.mem(), exit.mem() ); - this ._endLoop ( this, back , exit ); + _endLoop(this,back,exit); back.kill(); // Loop backedge is dead // Now one-time do a useless-phi removal - mem()._useless(); - this ._useless(); - - // The exit mem's lazy default value had been the loop top, - // now it goes back to predating the loop. - exit.mem().setDef(1,mem().in(1)); + _useless(); } // Fill in the backedge of any inserted Phis void _endLoop( ScopeNode scope, Node back, Node exit ) { - for( int i=2; i<nIns(); i++ ) { - if( var(i)._final ) continue; // Final vars did not get modified in the loop - if( var(i).type().isHighOrConst() ) continue; // Cannot lift higher than a constant, so no Phi + for( int i=1; i<nIns(); i++ ) { + if( _vars.at(i)._final ) continue; // Final vars did not get modified in the loop + if( _vars.at(i).type().isHighOrConst() ) continue; // Cannot lift higher than a constant, so no Phi if( back.in(i) != scope ) { PhiNode phi = (PhiNode)in(i); assert phi.region()==scope.ctrl() && phi.in(2)==null; @@ -381,6 +360,23 @@ public class ScopeNode extends MemMergeNode { } + // Now one-time do a useless-phi removal + void _useless( ) { + for( int i=1; i<nIns(); i++ ) { + if( in(i) instanceof PhiNode phi ) { + // Do an eager useless-phi removal + Node in = phi.peephole(); + CodeGen.CODE.addAll(phi._outputs); + phi.moveDepsToWorklist(); + if( in != phi ) { + if( !phi.iskeep() ) // Keeping phi around for parser elsewhere + phi.subsume(in); + setDef(i,in); // Set the update back into Scope + } + } + } + } + // Up-casting: using the results of an If to improve a value. // E.g. "if( ptr ) ptr.field;" is legal because ptr is known not-null. public void addGuards( Node ctrl, Node pred, boolean invert ) { diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/StartNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/StartNode.java index 6ed2acd3..186adfb7 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/StartNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/StartNode.java @@ -3,18 +3,11 @@ package com.seaofnodes.simple.node; import com.seaofnodes.simple.codegen.CodeGen; import com.seaofnodes.simple.Parser; import com.seaofnodes.simple.type.*; -import java.util.Arrays; import java.util.BitSet; import java.util.HashSet; import static com.seaofnodes.simple.util.Utils.TODO; -/** - * The Start node represents the start of the function. - * <p> - * Start initially has 1 input (arg) from outside and the initial control. - * In ch10 we also add mem aliases as structs get defined; each field in struct - * adds a distinct alias to Start's tuple. - */ +/** Start supplies control, whole memory, and the argument, in that order. */ public class StartNode extends LoopNode implements MultiNode { final Type _arg; @@ -44,7 +37,7 @@ public class StartNode extends LoopNode implements MultiNode { @Override public TypeTuple compute() { - return TypeTuple.make(Type.CONTROL,TypeMem.TOP,_arg); + return TypeTuple.make(Type.CONTROL,TypeMem.BOT,_arg); } @Override public Node idealize() { return null; } diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/StoreNode.java b/chapter24/src/main/java/com/seaofnodes/simple/node/StoreNode.java index 09257730..7a75688c 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/StoreNode.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/StoreNode.java @@ -1,5 +1,6 @@ package com.seaofnodes.simple.node; +import com.seaofnodes.simple.codegen.*; import com.seaofnodes.simple.*; import com.seaofnodes.simple.type.*; import java.util.BitSet; @@ -25,6 +26,10 @@ public class StoreNode extends MemOpNode { } // Debugger label + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && _init==((StoreNode)other)._init; + } + @Override public String label() { return "st_"+mlabel(); } @Override public boolean isMem() { return true; } @@ -38,24 +43,18 @@ public class StoreNode extends MemOpNode { @Override public Type compute() { Type val = val()._type; - Type mem0 = mem()._type; - if( mem0 == Type.TOP ) return TypeMem.TOP; - TypeMem mem = (TypeMem)mem0; // Invariant - if( mem == TypeMem.TOP ) return TypeMem.TOP; - Type t = Type.BOTTOM; // No idea on field contents - // Same alias, lift val to the declared type and then meet into other fields - if( mem._alias == _alias ) { - assert !_declaredType.isFRef(); - // Sharpen memory value; required for narrowing stores where the parser inserts - // zero/sign masking and somebody reads the TypeMem type. - val = val.join(_declaredType); - t = val.meet(mem._t); - } + assert !_declaredType.isFRef(); + Type t = val.join(_declaredType).meet(MemMergeNode.contents(mem(),_alias,this)); return TypeMem.make(_alias,t); } @Override public Node idealize() { + if( mem() instanceof MemMergeNode merge ) { + setDef(1,CodeGen.CODE.add(merge.alias(_alias))); + return this; + } + if( mem() instanceof CastNode cast ) { setDef(1,cast.in(1)); @@ -65,8 +64,8 @@ public class StoreNode extends MemOpNode { // Simple store-after-store on same address. Should pick up the // required init-store being stomped by a first user store. if( mem() instanceof StoreNode st && - ptr()==st.ptr() && // Must check same object - off()==st.off() && // And same offset (could be "same alias" but this handles arrays to same index) + ptr()==st.ptr() && _alias==st._alias && // Must check same object + off()==st.off() && // And same offset ptr()._type instanceof TypeMemPtr && // No bother if weird dead pointers // Must have exactly one use of "this" or you get weird // non-serializable memory effects in the worse case. @@ -76,12 +75,6 @@ public class StoreNode extends MemOpNode { return this; } - // Simple store-after-MemMerge to a known alias can bypass. Happens when inlining. - if( mem() instanceof MemMergeNode mem ) { - setDef(1,mem.alias(_alias)); - return this; - } - // Value is automatically truncated by narrow store if( val() instanceof AndNode and && and.in(2)._type.isConstant() ) { int log = _declaredType.log_size(); diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java index 70034227..eab72947 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/arm/arm.java @@ -677,6 +677,8 @@ public class arm extends Machine { case NotNode not -> new NotARM(not); case OrNode or -> or(or); case ParmNode parm -> new ParmARM(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read -> new ReadOnlyMach(read); diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java index 48cb5d23..88ec9ebd 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/riscv/riscv.java @@ -345,6 +345,8 @@ public class riscv extends Machine { case NotNode not -> new NotRISC(not); case OrNode or -> or(or); case ParmNode parm -> new ParmRISC(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read-> new ReadOnlyMach(read); diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java index 441cf693..d0fb0081 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java @@ -24,6 +24,11 @@ public class LoadX86 extends MemOpX86 { } static void enc( Encoding enc, Type decl, short dst, short ptr, short idx, int off, int scale ) { + // Allocation may put a full-width integer in an XMM register (or a + // floating value in a GPR). Move its bits using the selected register bank. + if( dst>=x86_64_v2.XMM_OFFSET && decl.log_size()==3 ) decl=TypeFloat.F64; + else if( dst<x86_64_v2.XMM_OFFSET && decl instanceof TypeFloat ) + decl = decl==TypeFloat.F32 ? TypeInteger.U32 : TypeInteger.BOT; if( decl == TypeFloat.F32) enc.add1(0xF3); if( decl == TypeFloat.F64) enc.add1(0xF2); diff --git a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java index 7ac7fc19..cf9288b8 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/x86_64_v2.java @@ -363,6 +363,8 @@ public class x86_64_v2 extends Machine { case NotNode not -> new NotX86(not); case OrNode or -> or(or); case ParmNode parm -> new ParmX86(parm); + case BulkMemPhiNode phi -> new BulkMemPhiNode(phi); + case MemPhiNode phi -> new MemPhiNode(phi); case PhiNode phi -> new PhiNode(phi); case ProjNode prj -> prj(prj); case ReadOnlyNode read-> new ReadOnlyMach(read); @@ -396,9 +398,8 @@ public class x86_64_v2 extends Machine { if( lhs instanceof LoadNode ld && ld.nOuts() == 1 && ld._declaredType.log_size() >= 3) return new AddMemX86(add, address(ld), ld.ptr(), idx, off, scale, 0, rhs); -// if(rhs instanceof LoadNode ld && ld.nOuts() == 1 && ld._declaredType.log_size() >= 3) { -// throw Utils.TODO(); // Swap load sides -// } + if( rhs instanceof LoadNode ld && ld.nOuts()==1 && ld._declaredType.log_size()>=3 ) + return new AddMemX86(add,address(ld),ld.ptr(),idx,off,scale, 0,lhs); // Attempt a full LEA-style break down. // Returns one of AddX86, AddIX86, LeaX86, or LHS diff --git a/chapter24/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java b/chapter24/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java index c895a0cb..4417ccd1 100644 --- a/chapter24/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java +++ b/chapter24/src/main/java/com/seaofnodes/simple/print/SimpleGraphAdapter.java @@ -67,7 +67,7 @@ public class SimpleGraphAdapter extends GraphAdapter<Node> { return n instanceof CFGNode ? Role.CTRL : isMem(n) ? Role.MEM : Role.DATA; if( n instanceof RegionNode && i == 0 ) return Role.ASSOC; if( i == 0 || n instanceof RegionNode || n instanceof StopNode ) return Role.CTRL; - if( n instanceof MemMergeNode || i == 1 && (n instanceof MemOpNode || n instanceof ReturnNode || n instanceof CallNode) ) return Role.MEM; + if( n instanceof MemMergeNode || (i == 1 && (n instanceof MemOpNode || n instanceof ReturnNode || n instanceof CallNode || n instanceof NewNode)) ) return Role.MEM; Node def = n.in(i); if( def != null && isMem(def) ) return Role.MEM; return Role.DATA; diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter10Test.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter10Test.java index 3c5a2913..121e8a22 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Chapter10Test.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter10Test.java @@ -2,6 +2,8 @@ package com.seaofnodes.simple; import com.seaofnodes.simple.codegen.CodeGen; import org.junit.Test; +import com.seaofnodes.simple.node.*; +import java.util.BitSet; import static org.junit.Assert.assertEquals; import static org.junit.Assert.fail; @@ -468,4 +470,121 @@ return 0; assertEquals("return 0;", code.print()); } + @Test public void testReadBeforeStores() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !a = new S; S !b = new S; + a.x = 11; b.x = 22; + S !p = a; if (arg) p = b; + int before = p.x; + a.y = 55; b.y = 66; + a.x = 33; b.x = 44; + return before*100 + p.x; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals("1133",Eval2.eval(code,0)); + assertEquals("2244",Eval2.eval(code,1)); + } + + static final String NESTED_MEMORY = """ + struct S { int x; int y; int z; }; + S !s = new S; + s.x = 5; s.y = 7; s.z = 11; + while (arg > 0) { + s.x = s.x + 1; + int j = 3; + while (j > 0) { + j = j - 1; + if (j == 1) continue; + s.y = s.y + arg; + if (arg == 2) break; + s.z = s.z + 1; + } + arg = arg - 1; + } + return s; + """; + + @Test public void testMemoryAcrossNestedLoops() { + var code = new CodeGen(NESTED_MEMORY).parse().opto(); + StopNode stop = code._stop; + for (int arg=0; arg<7; arg++) { + assertEquals("S{x="+(5L+arg)+",y="+(7L+arg*(arg+1)-(arg>=2 ? 2 : 0))+ + ",z="+(11L+2*arg-(arg>=2 ? 2 : 0))+"}",Eval2.eval(code,arg)); + } + } + + @Test public void testMemoryAtEarlyReturns() { + var code = new CodeGen(""" + struct S { int x; int y; int z; }; + S !s = new S; + s.x = 3; s.y = 5; s.z = 7; + if (arg) { s.x = 11; s.z = 13; return s; } + s.y = 17; + return s; + """).parse().opto(); + StopNode stop = code._stop; + for (int arg=0; arg<2; arg++) { + assertEquals(arg==0 ? "S{x=3,y=17,z=7}" : "S{x=11,y=5,z=13}",Eval2.eval(code,arg)); + } + } + + @Test public void testKeepLoadsAtMerge() { + var code = new CodeGen(""" + struct S { int x; }; + S !a = new S; S !b = new S; + a.x = arg; b.x = arg+1; + S !p = a; S !q = b; + if (arg<0) { p=b; q=a; } + int v; + if (arg>1) v=p.x; else v=q.x; + return v; + """).parse().opto(); + StopNode stop = code._stop; + // The one-step safety check stops at the memory aggregate. + assertEquals(2,countMemoryNodes(stop,LoadNode.class,new BitSet())); + assertEquals("-1",Eval2.eval(code,-1)); + assertEquals("1",Eval2.eval(code,0)); + assertEquals("3",Eval2.eval(code,3)); + } + + @Test public void testDropStores() { + var code = new CodeGen(""" + struct S { int x; }; + S !s = new S; + if (arg) s.x=arg+1; else s.x=arg+2; + return s; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals(1,countMemoryNodes(stop,StoreNode.class,new BitSet())); + assertEquals("S{x=2}",Eval2.eval(code,0)); + assertEquals("S{x=4}",Eval2.eval(code,3)); + } + + @Test public void testKeepReadsBeforeWrites() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !a = new S; S !b = new S; + a.x=arg; b.x=arg+1; + S !p=a; S !q=b; + if (arg<0) { p=b; q=a; } + int v; + if (arg>1) { v=p.x; p.x=41; p.y=5; } + else { v=q.x; q.x=42; } + return v; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals(2,countMemoryNodes(stop,LoadNode.class,new BitSet())); + assertEquals("-1",Eval2.eval(code,-1)); + assertEquals("1",Eval2.eval(code,0)); + assertEquals("3",Eval2.eval(code,3)); + } + + private static int countMemoryNodes(Node n, Class<?> kind, BitSet seen) { + if (n==null || seen.get(n._nid)) return 0; + seen.set(n._nid); + int cnt=kind.isInstance(n) ? 1 : 0; + for (Node def : n._inputs) cnt+=countMemoryNodes(def,kind,seen); + return cnt; + } } diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter10bTest.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter10bTest.java new file mode 100644 index 00000000..f2b83cdd --- /dev/null +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter10bTest.java @@ -0,0 +1,60 @@ +package com.seaofnodes.simple; + +import com.seaofnodes.simple.codegen.*; +import com.seaofnodes.simple.node.*; +import org.junit.Test; +import static org.junit.Assert.*; + +public class Chapter10bTest { + @Test public void testIndependentField() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !s = new S; + s.x = 42; + while (arg > 0) { s.y = s.y + arg; arg = arg - 1; } + return s.x; + """).parse().opto(); + var stop = code._stop; + assertEquals("return 42;",stop.toString()); + assertEquals("42",Eval2.eval(code,5)); + assertNull(stop.walk(n -> n instanceof LoadNode ? n : null)); + } + + @Test public void testParallelMemoryPhis() { + var code = new CodeGen(""" + struct S { int x; int y; int z; }; + S !a = new S; S !b = new S; + S !p = a; if (arg) p = b; + while (arg > 0) { + a.x = p.x + arg; + a.y = p.y + 2; + a.z = p.z + 3; + arg = arg - 1; + } + return a; + """).parse().opto(); + var stop = code._stop; + int[] phis = {0}; + stop.walk(n -> { + if (n instanceof MemPhiNode) phis[0]++; + if (n instanceof MemMergeNode m && m.in(1) instanceof BulkMemPhiNode b) + for(int a=b._aliases.nextSetBit(0); a>=0; a=b._aliases.nextSetBit(a+1)) + assertTrue(a<m.nIns() && m.in(a)!=null); + assertFalse(n instanceof PhiNode && n.isMem() && + !(n instanceof MemPhiNode) && !(n instanceof BulkMemPhiNode) && !(n instanceof ParmNode)); + return null; + }); + assertTrue(phis[0]>=3); + assertEquals("S{x=1,y=2,z=3}",Eval2.eval(code,4)); + } + + @Test public void testLateSliceWorklist() { + // Seed 97 exposed a store selecting a bulk predecessor without queuing + // it. Keep both the worklist assertion and the final heap check. + for (int seed : new int[]{0,97,123,456}) { + var code = new CodeGen(Chapter10Test.NESTED_MEMORY,com.seaofnodes.simple.type.TypeInteger.BOT,seed,false).parse(); + code.opto(); + assertEquals("S{x=9,y=25,z=17}",Eval2.eval(code,4)); + } + } +} diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter16Test.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter16Test.java index 310efc45..57ec5508 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Chapter16Test.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter16Test.java @@ -253,4 +253,27 @@ return new Square; assertEquals("Square{side=3.0,diag=1.7320508075688772}", Eval2.eval(code, 3)); assertEquals("Square{side=4.0,diag=2.0}", Eval2.eval(code, 4)); } + static final String CONSTRUCTOR_MEMORY = """ + struct S { int x; int y; }; + struct T { int z=arg+40; }; + T !t = new T; + S !a = new S { x=11; y=7; }; + S !b = new S { x=22; y=9; }; + S !p=a; + if (arg) p=b; + int before=p.x; + S !c = new S { + x=p.x+1; + { int i=0; while (i<2) { p.y=p.y+1; i=i+1; } } + y=p.y; + }; + p.x=33; + return before*10000+c.x*100+c.y+t.z; + """; + + @Test public void testConstructorMemory() { + var code = new CodeGen(CONSTRUCTOR_MEMORY).parse().opto(); + assertEquals("111249",Eval2.eval(code,0)); + assertEquals("222352",Eval2.eval(code,1)); + } } diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter17Test.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter17Test.java index 53c23ecd..37208308 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Chapter17Test.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter17Test.java @@ -369,7 +369,7 @@ var b = arg ? new Bar { next = (arg==2) ? new Bar{x=2;}; x=1; }; return b ? b.next ? b.next.x : b.x; // parses "b ? (b.next ? b.next.x : b.x) : 0" """); code.parse().opto(); - assertEquals("return Phi(Region,.x,0);", code.print()); + assertEquals("return Phi(Region,.x,0,.x);", code.print()); assertEquals("0", Eval2.eval(code, 0)); assertEquals("1", Eval2.eval(code, 1)); assertEquals("2", Eval2.eval(code, 2)); diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter18Test.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter18Test.java index 479e9e62..6e027312 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Chapter18Test.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter18Test.java @@ -439,4 +439,70 @@ return p.coffee_count; assertEquals("1", Eval2.eval(code, 2)); } + static final String CALL_MEMORY = """ + struct S { int x; int y; }; + val bump = { S s, int d -> int old=s.x; s.x=old+d; return old; }; + S !a = new S { x=10; y=7; }; + S !b = new S { x=20; y=9; }; + S !p=a; if (arg) p=b; + int before=p.x; + int old=bump(p,3); + int after=p.x; + bump(a,5); + return before*1000000+old*10000+after*100+a.x+a.y; + """; + + @Test public void testCallMemory() { + var code = new CodeGen(CALL_MEMORY).parse().opto(); + int[] calls={0}; + code._stop.walk(n -> { if(n instanceof CallNode) calls[0]++; return null; }); + assertEquals(2,calls[0]); // Keep real calls so this checks scheduling across them. + assertEquals("10101325",Eval2.eval(code,0)); + assertEquals("20202322",Eval2.eval(code,1)); + code._stop.walk(n -> { + if(n instanceof CallNode call) { + assertEquals(5,call.nIns()); // No scheduling edges may become extra arguments. + assertTrue(call.fptr()._type instanceof TypeFunPtr); + } + return null; + }); + } + + static final String RECURSIVE_MEMORY = """ + struct S { int x; }; + val rec = { S s, int n -> + if (n==0) return s.x; + int before=s.x; + s.x=before+1; + return before+rec(s,n-1); + }; + S !s=new S { x=10; }; + int value=rec(s,arg); + return value*100+s.x; + """; + + @Test public void testRecursiveMemory() { + var code = new CodeGen(RECURSIVE_MEMORY).parse().opto(); + for(int n=0; n<7; n++) + assertEquals(Long.toString((10L*(n+1)+n*(n+1)/2)*100+10+n),Eval2.eval(code,n)); + } + + static final String INLINE_MEMORY = """ + struct S { int x; int y; }; + val make = { S s -> + S t = new S { x=s.x+1; y=3; }; + s.y=9; + return t; + }; + S !a = new S { x=arg+40; y=7; }; + S t=make(a); + return a.x*10000+a.y*100+t.x+t.y; + """; + + @Test public void testInlineAllocationMemory() { + var code = new CodeGen(INLINE_MEMORY).parse().opto(); + assertNull(code._stop.walk(n -> n instanceof CallNode ? n : null)); + assertEquals("400944",Eval2.eval(code,0)); + assertEquals("420946",Eval2.eval(code,2)); + } } diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter19Test.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter19Test.java index df94e97c..841d9298 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Chapter19Test.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter19Test.java @@ -426,4 +426,72 @@ return sq(arg) + sq(3); code.driver(Phase.LocalSched,"x86_64_v2", "SystemV"); assertEquals("Stop[ return (add,#2,#2); return (mul,Parm_x(sq,i64),x); ]", code.print()); } + + @Test public void testSelectedMemory() { + for( String cpu : new String[]{"x86_64_v2","riscv","arm"} ) + for( String src : new String[]{Chapter10Test.NESTED_MEMORY,Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY,Chapter18Test.RECURSIVE_MEMORY} ) { + var code = new CodeGen(src).parse().opto().typeCheck(); + var aliases = memoryPhis(code._stop); + code.loopTree().instSelect(cpu,"SystemV"); + assertEquals(aliases,memoryPhis(code._stop)); + code.GCM().localSched(); + code._stop.walk(n -> { + if( !(n instanceof CFGNode) && !(n instanceof ProjNode) ) + assertTrue(n.in(0) instanceof CFGNode); + if( n instanceof NewNode nn ) { + assertEquals(2,((TypeTuple)nn._type)._types.length); + assertTrue(nn.mem() instanceof MemMergeNode); + assertNull(nn.mem().in(1)); // Partial allocation input. + nn.cacheRegs(code); + MachNode mach = (MachNode)nn; + assertNull(mach.regmap(1)); + assertNotNull(mach.regmap(2)); + assertNotNull(mach.outregmap(0)); + assertNull(mach.outregmap(1)); + } + if( n instanceof CallNode call ) { + assertEquals(4,call.nIns()); // ctrl, memory, two args; direct target is embedded. + } + return null; + }); + } + } + + private static java.util.ArrayList<String> memoryPhis(Node stop) { + var phis = new java.util.ArrayList<String>(); + stop.walk(n -> { + if( n instanceof MemPhiNode phi ) phis.add("alias:"+phi._alias); + if( n instanceof BulkMemPhiNode phi ) phis.add("bulk:"+phi._aliases); + assertFalse(n instanceof PhiNode && n.isMem() && + !(n instanceof MemPhiNode) && !(n instanceof BulkMemPhiNode) && !(n instanceof ParmNode)); + return null; + }); + java.util.Collections.sort(phis); + return phis; + } + + @Test public void testFoldedReadBeforeWrites() { + var code = new CodeGen(""" + struct S { int x; }; + S !a = new S { x=11; }; S !b = new S { x=22; }; + S !p = a; if( arg ) p = b; + int before = p.x + arg; + a.x=33; b.x=44; + return before; + """).parse().opto().typeCheck().loopTree().instSelect("x86_64_v2","SystemV").GCM().localSched(); + var read = (MemOpNode)code._stop.walk(n -> + n instanceof com.seaofnodes.simple.node.cpus.x86_64_v2.AddMemX86 ? n : null); + assertNotNull(read); + int writes = 0; + for( Node use : read.antiDeps() ) + if( use instanceof MemOpNode st && st.isMem() ) { + assertSame(read.cfg0(),st.cfg0()); + assertTrue(st._inputs.find(read)>=0); + assertTrue(read.cfg0()._outputs.find(read)<st.cfg0()._outputs.find(st)); + assertNull(((MachNode)st).regmap(st._inputs.find(read))); + writes++; + } + assertEquals(1,writes); // The next Store follows this first clobber through memory. + } } diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter20Test.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter20Test.java index 5639f4f6..2035ef9b 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Chapter20Test.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter20Test.java @@ -162,7 +162,7 @@ for( int i=0; i<ary#-1; i++ ) ary[i+1] += ary[i]; return ary[1] * 1000 + ary[3]; // 1 * 1000 + 6 """; - testTarget(src,"x86_64_v2", "SystemV",7,"return .[];"); + testTarget(src,"x86_64_v2", "SystemV",5,"return .[];"); testTarget(src,"riscv" , "SystemV",7,"return (add,.[],(mul,.[],1000));"); testTarget(src,"arm" , "SystemV",5,"return (add,.[],(mul,.[],1000));"); } @@ -205,7 +205,7 @@ s.cs[0] = 67; // C s.cs[1] = 108; // l hashCode(s); """; - testTarget(src,"x86_64_v2", "SystemV",0,null); + testTarget(src,"x86_64_v2", "SystemV",9,null); testTarget(src,"riscv" , "SystemV",0,null); testTarget(src,"arm" , "SystemV",0,null); } @@ -366,8 +366,29 @@ return a; """; testTarget(src,"x86_64_v2", "SystemV",52,null); testTarget(src,"riscv" , "SystemV",44,null); - testTarget(src,"arm" , "SystemV",44,null); + testTarget(src,"arm" , "SystemV",41,null); //assertEquals("int[ 1,2,3,4,5,6,7,8,9,10,11]", Eval2.eval(code, 11)); } + + @Test public void testMemoryAllocation() { + for( String cpu : new String[]{"x86_64_v2","riscv","arm"} ) + for( String src : new String[]{Chapter10Test.NESTED_MEMORY,Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY,Chapter18Test.RECURSIVE_MEMORY, + "int[] !a = new int[3]; a[arg]=arg; return a[0]+1;"} ) { + var code = new CodeGen(src).parse().opto().typeCheck().loopTree() + .instSelect(cpu,"SystemV").GCM().localSched().regAlloc(); + com.seaofnodes.simple.codegen.RegAllocTestSupport.checkRegisters(code); + code._stop.walk(n -> { + // Whole memory, precise Phis, and memory projections need no register. + if( n._type instanceof com.seaofnodes.simple.type.TypeMem ) + assertEquals(-1,code._regAlloc.regnum(n)); + if( n instanceof com.seaofnodes.simple.node.NewNode nn ) { + assertTrue(code._regAlloc.regnum(nn.proj(0))>=0); + assertEquals(-1,code._regAlloc.regnum(nn.proj(1))); + } + return null; + }); + } + } } diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Chapter24Test.java b/chapter24/src/test/java/com/seaofnodes/simple/Chapter24Test.java index 93c778c8..b98a95b7 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Chapter24Test.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Chapter24Test.java @@ -516,7 +516,7 @@ var stack11 = { int x -> """ return 0 < arg < arg+1 < 4; """; - TestC.run(src, "stacked_r_11x", null, stack11,0); + TestC.run(src, "stacked_r_11x", null, stack11,2); // Evaluate on RISC5 emulator EvalRisc5 R5 = TestRisc5.build("stacked_r_11", src2, 1, 2, false); @@ -525,7 +525,7 @@ return 0 < arg < arg+1 < 4; assertEquals(1,R5.regs[riscv.A0]); // Evaluate on ARM emulator - EvalArm64 arm = TestArm64.build("stacked_r_11", src2, 1, 2, false); + EvalArm64 arm = TestArm64.build("stacked_r_11", src2, 1, 0, false); trap = arm.step(100); assertEquals(0,trap); assertEquals(1,arm.regs[0]); diff --git a/chapter24/src/test/java/com/seaofnodes/simple/Eval2.java b/chapter24/src/test/java/com/seaofnodes/simple/Eval2.java index fc826e04..e0d56399 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/Eval2.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/Eval2.java @@ -314,6 +314,8 @@ public abstract class Eval2 { case TypeFunPtr tfp -> tfp; case TypeMem mem -> "MEM"; case TypeMemPtr tmp -> { + // Unknown-caller inputs are type placeholders, not heap constants. + if( !tmp.isConstant() ) yield tmp; if( tmp._obj.isAry() ) { // Constant array ptr Type elem = tmp._obj.field("[]")._t; if( elem instanceof TypeConAry con ) { @@ -350,7 +352,7 @@ public abstract class Eval2 { private static Object alloc(NewNode alloc) { TypeStruct type = alloc._ptr._obj; if( type.isAry() ) { - long sz = (Long)val(alloc.in(1)); + long sz = (Long)val(alloc.size()); long x = offToIdx(sz, type); int n = (int)x; if( n!=x || n<0 ) @@ -471,9 +473,9 @@ public abstract class Eval2 { case TypeRPC rpc -> sb.p(x.toString()); case TypeMem mem -> sb.p("$mem"); case TypeTuple tt -> { - if( tt._types.length>1 && tt._types[1] instanceof TypeMemPtr ) + if( tt._types.length>1 && tt._types[0] instanceof TypeMemPtr ) // Assume a NewNode - yield _print( tt._types[1], x, sb, visit ); + yield _print( tt._types[0], x, sb, visit ); throw Utils.TODO(); } case Type tt -> { diff --git a/chapter24/src/test/java/com/seaofnodes/simple/EvalArm64.java b/chapter24/src/test/java/com/seaofnodes/simple/EvalArm64.java index bd425908..c24b47bb 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/EvalArm64.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/EvalArm64.java @@ -549,10 +549,13 @@ public class EvalArm64 { break; } - case 0xD2: { + case 0xD2: // MOVZ + case 0xF2: { // MOVK preserves the other halfwords // movz // get immediate - rval = (ir >> 5) & 0x7FFF; + int shift = ((ir >> 21) & 3)*16; + rval = ((ir >> 5) & 0xFFFFL) << shift; + if( opcode1==0xF2 ) rval |= regs[rdid] & ~(0xFFFFL << shift); break; } diff --git a/chapter24/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java b/chapter24/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java new file mode 100644 index 00000000..cd1c972d --- /dev/null +++ b/chapter24/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java @@ -0,0 +1,177 @@ +package com.seaofnodes.simple; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.codegen.RegAllocTestSupport; +import com.seaofnodes.simple.node.*; +import com.seaofnodes.simple.node.cpus.arm.arm; +import com.seaofnodes.simple.node.cpus.riscv.riscv; +import com.seaofnodes.simple.type.TypeMemPtr; +import java.io.IOException; +import java.nio.file.Files; +import java.nio.file.Path; +import org.junit.Test; +import static org.junit.Assert.*; + +// Execution regressions are separate from the fixed spill-measurement cohorts. +public class MemoryEncodingTest { + @Test public void testLoopLoadSearch() { + String[] sources = { + """ + u8[] !program=new u8[106]; u8[] !out=new u8[0]; + int sum=0; + for(int i=0; i<arg; i++) { + sum+=program#; + if(i==1) out=new u8[i+1]; + else if(i==3) out=new u8[2]; + } + return sum+out#; + """, + // One possibly aliasing arm must reject the unchanged-memory proof. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; if(arg&1) p=b; + int sum=0; + for(int i=0; i<6; i++) { + if(i==2) p.x=30; + else if(i==4) b.x=40; + sum+=a.x; + } + return sum; + """, + // Unchanged memory does not imply an unchanged loop-carried pointer. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; int sum=0; + for(int i=0; i<6; i++) { + if(i==2) a.x=30; + sum+=p.x; + p=b; if(i&1) p=a; + } + return sum; + """, + // All arms fold, but their values differ: build a value Phi. + """ + struct S { int x; }; S !a=new S; a.x=1; int sum=0; + for(int i=0; i<6; i++) { + sum+=a.x; + if(i&1) a.x=2; + else if(i==2) a.x=3; + else a.x=4; + } + return sum; + """ + }; + for( int p=0; p<sources.length; p++ ) { + CodeGen code=new CodeGen(sources[p]).parse().opto(); + if( p==0 ) assertNull(code._stop.walk(n -> + n instanceof LoadNode ld && ld.ptr() instanceof ProjNode ptr && + ptr.in(0) instanceof NewNode nn && + nn.size()._type==com.seaofnodes.simple.type.TypeInteger.constant(110) ? ld : null)); + for( int arg=0; arg<7; arg++ ) { + long expected=switch(p) { + case 0 -> 106*arg+(arg>=2 ? 2 : 0); + case 1 -> (arg&1)==0 ? 140 : 60; + case 2 -> 130; + default -> 16; + }; + assertEquals(Long.toString(expected),Eval2.eval(code,arg)); + } + } + } + + private static final String[] SOURCES = { + Chapter16Test.CONSTRUCTOR_MEMORY, + Chapter18Test.CALL_MEMORY, + Chapter18Test.RECURSIVE_MEMORY, + Chapter18Test.INLINE_MEMORY, + "int[] !a = new int[7]; a[arg]=arg+3; return a[0]+1;", + """ + int[] !a = new int[7]; + for( int i=0; i<7; i++ ) a[i]=i+arg; + for( int i=0; i<6; i++ ) a[i+1]+=a[i]; + return a[6]; + """ + }; + + private static long expected(int program, long arg) { + return switch(program) { + case 0 -> arg==0 ? 111249 : 222351+arg; + case 1 -> arg==0 ? 10101325 : 20202322; + case 2 -> (10*(arg+1)+arg*(arg+1)/2)*100+10+arg; + case 3 -> 400944+arg*10001; + case 4 -> arg==0 ? 4 : 1; + case 5 -> 21+7*arg; + default -> throw new AssertionError(); + }; + } + + @Test public void testEmulatedMemory() throws IOException { + for( String target : new String[]{"riscv","arm"} ) + for( int p=0; p<SOURCES.length; p++ ) { + CodeGen code = new RegAllocTestSupport.CheckedCodeGen(SOURCES[p]).driver(target,"SystemV",null); + int entry=0; + for( var n : code._start._outputs ) + if( n instanceof FunNode fun && "main".equals(fun._name) ) + entry=code._encoding._opStart[fun._nid]; + for( int arg=0; arg<7; arg++ ) { + byte[] image = new byte[1<<20]; + byte[] bits = code._encoding.bits(); + System.arraycopy(bits,0,image,0,bits.length); + long result; + if( target.equals("riscv") ) { + EvalRisc5 cpu = new EvalRisc5(image,1<<16); + cpu._pc=entry; + cpu.regs[riscv.A0]=arg; + assertEquals(target+" program "+p+" arg "+arg,0,cpu.step(100000)); + result=cpu.regs[riscv.A0]; + } else { + EvalArm64 cpu = new EvalArm64(image,1<<16); + cpu._pc=entry; + cpu.regs[arm.X0]=arg; + assertEquals(target+" program "+p+" arg "+arg,0,cpu.step(100000)); + result=cpu.regs[arm.X0]; + } + assertEquals(target+" program "+p+" arg "+arg,expected(p,arg),result); + } + } + } + + @Test public void testNeverMemory() { + // A synthetic exit must retain both precise slices of a no-exit loop. + // Stop at scheduling: the existing allocator cannot coalesce TOP on + // the synthetic scalar return arm. + for( String target : new String[]{"x86_64_v2","riscv","arm"} ) { + CodeGen code = new CodeGen(""" + struct S { int x; int y; }; + S !a=new S; + if( arg ) while( 1 ) { a.x+=arg; a.y+=a.x; } + return a.x+a.y; + """).driver(CodeGen.Phase.LoopTree); + assertNotNull(code._stop.walk(n -> { + if( n instanceof MemMergeNode mem && + mem.in(0) instanceof CProjNode exit && exit.in(0) instanceof NeverNode ) { + var fields = ((TypeMemPtr)Parser.TYPES.get("S"))._obj._fields; + assertTrue(mem.alias(fields[0]._alias) instanceof MemPhiNode); + assertTrue(mem.alias(fields[1]._alias) instanceof MemPhiNode); + return mem; + } + return null; + })); + code.instSelect(target,"SystemV").GCM().localSched(); + } + } + + @Test public void testNativeMemory() throws IOException { + Files.createDirectories(Path.of("build/objs")); + for( int p=0; p<SOURCES.length; p++ ) { + String file="build/objs/memory"+p; + StringBuilder c = new StringBuilder("extern long long memory(long long);\nint main() {\n"); + for( int arg=0; arg<7; arg++ ) + c.append("if(memory(").append(arg).append(")!=").append(expected(p,arg)).append("LL) return ").append(arg+1).append(";\n"); + c.append("return 0;\n}\n"); + Files.writeString(Path.of(file+".c"),c); + TestC.run("val memory = { int arg -> "+SOURCES[p]+" };", + TestC.CALL_CONVENTION,null,"",file+".c",file,"S","",-1); + } + } +} diff --git a/chapter24/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java b/chapter24/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java index 911acc5c..1c116162 100644 --- a/chapter24/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java +++ b/chapter24/src/test/java/com/seaofnodes/simple/evaluator/Evaluator.java @@ -184,7 +184,7 @@ public class Evaluator { Object[] body=null; int num; if( type.isAry() ) { - long sz = (Long)val(alloc.in(1)); + long sz = (Long)val(alloc.size()); long n = offToIdx(sz, type); if( n < 0 ) throw new NegativeArraySizeException(""+n); @@ -198,21 +198,17 @@ public class Evaluator { assert elem instanceof TypeMemPtr; } // Length value - body[0] = vall(alloc.in(2+2)); + body[0] = n; } else { body = new Object[num = type._fields.length]; for( int i=0; i<num; i++ ) body[i] = switch( alloc._ptr._obj._fields[i]._t ) { case TypeInteger ti -> 0L; - case TypeFloat tf -> 0; + case TypeFloat tf -> 0D; default -> null; }; } - Object[] mems = new Object[type._fields.length+2]; - // mems[0] is control - mems[1] = new Obj(type,body); // the ref - // mems[2+...] are memory aliases - return mems; + return new Object[]{new Obj(type,body),null}; } private Object load(LoadNode load) { diff --git a/chapter25/README.md b/chapter25/README.md index 4feef80f..b7925a1f 100644 --- a/chapter25/README.md +++ b/chapter25/README.md @@ -9,6 +9,16 @@ These are the contents of the current snapshot. Independent correctness fixes will move to earlier chapters first; moving the larger SSA redesign backward, or splitting this chapter, is deferred. See the [backport queue](../docs/chapter-backports.md). +The memory work from Chapters 21-24 carries forward here: Load search proves +all arms of a merge before moving a read out of a loop, Phi factoring checks +memory clobbers, and scheduling follows aliases through memory aggregates. +The search follows `EscapeNode` to private or public memory and waits for +unresolved Store aliases. Array-length reads have no element bounds check; +array-element reads retain control. Integer value Phis use the existing typed +conversion machinery to preserve the load's width across loop widening. +New still produces private constructor memory, with publication handled by +Escape; the earlier chapters' allocation-input design does not replace it. + You can also read [this chapter](https://github.com/SeaOfNodes/Simple/tree/linear-chapter25) in the [linear history](https://github.com/SeaOfNodes/Simple/tree/linear) and [compare it with Chapter 24](https://github.com/SeaOfNodes/Simple/compare/linear-chapter24...linear-chapter25). @@ -168,10 +178,12 @@ used only by the next instruction is removed when the operand accepts its source register and is not tied to the instruction's result. The table below is the **earlier audit**, before adjacent-copy forwarding. -The current replay has ten String-constructor parse failures, also reproduced +The replay has ten String-constructor parse failures, also reproduced with the original allocator. On the identical 216 successful allocations, -forwarding reduces 2,563 / 5,496 moves / weighted moves to 2,494 / 5,399. -This is a partial comparison, not a replacement complete-suite total. The +adjacent-copy forwarding reduced 2,563 / 5,496 moves / weighted moves to +2,494 / 5,399. The subsequent memory corrections produce 2,488 / 5,379 on +those same 216 successful allocations. These are partial comparisons, not +replacement complete-suite totals. The current cohort 25 has 14 allocations, including later C return ABI checks; the historical table has ten. Native cohort-25 checks and fresh system-library encoding pass in both runs. See the [backport record](../docs/chapter-backports.md) diff --git a/chapter25/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java b/chapter25/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java index ecd23c1e..484bcaa9 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/codegen/GlobalCodeMotion.java @@ -172,17 +172,9 @@ public abstract class GlobalCodeMotion { // Loads need their memory inputs' uses also done if( n instanceof MemOpNode ld && ld._isLoad ) - for( Node memuse : ld.mem()._outputs ) - if( late[memuse._nid]==null && - // New makes new memory, never crushes load memory - !(memuse instanceof NewNode) && - // Load-use directly defines memory - (memuse._type instanceof TypeMem || - // Load-use directly defines memory - memuse instanceof CallNode) ) { - assert !( memuse._type instanceof TypeTuple tt && tt._types[ld._alias] instanceof TypeMem ); + for( Node memuse : ld.antiDeps() ) + if( late[memuse._nid]==null ) continue outer; - } // All uses done, schedule _doSchedLate(n,ns,late,anti); @@ -193,9 +185,7 @@ public abstract class GlobalCodeMotion { for( Node def : n._inputs ) { if( def==null ) continue; if( late[def._nid]==null ) work.push(def); - for( Node out : def._outputs ) - if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) - work.push(ld); + wakeLoads(def,late,work,new BitSet()); } if( n instanceof LoopNode loop ) for( Node phi : loop._outputs ) @@ -255,25 +245,32 @@ public abstract class GlobalCodeMotion { best instanceof IfNode; } + // A writer may consume a load's memory through an aggregate. + private static void wakeLoads(Node def, CFGNode[] late, WorkList<Node> work, BitSet visit) { + if( visit.get(def._nid) ) return; + visit.set(def._nid); + for( Node out : def._outputs ) + if( out instanceof MemOpNode ld && ld._isLoad && late[ld._nid]==null ) work.push(ld); + if( def instanceof MemMergeNode ) + for( int i=1; i<def.nIns(); i++ ) + if( def.in(i)!=null ) wakeLoads(def.in(i),late,work,visit); + } + private static CFGNode find_anti_dep(CFGNode lca, MemOpNode load, CFGNode early, CFGNode[] late, int[] anti) { // We could skip final-field loads here. // Walk LCA->early, flagging Load's block location choices for( CFGNode cfg=lca; cfg!=early.idom(); cfg = cfg.idom() ) anti[cfg._nid] = load._nid; // Walk load->mem uses, looking for Stores causing an anti-dep - for( Node mem : load.mem()._outputs ) { + for( Node mem : load.antiDeps() ) { switch( mem ) { case MemOpNode st: - if( !st._isLoad && load._alias == st._alias ) { - assert late[mem._nid] != null; - lca = anti_dep(load,late[mem._nid],lca,st,anti); - } - break; // Loads do not cause anti-deps on other loads - case EscapeNode esc: - if( load._alias == esc.fld()._alias ) { - assert late[mem._nid] != null; - lca = anti_dep(load,late[mem._nid],lca,mem,anti); - } + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); + break; + case EscapeNode st: + assert late[st._nid]!=null; + lca = anti_dep(load,late[st._nid],lca,st,anti); break; case CallNode call: assert late[call._nid]!=null; @@ -286,8 +283,6 @@ public abstract class GlobalCodeMotion { if( phi.in(i)==load.mem() ) lca = anti_dep(load,phi.region().cfg(i),lca,null,anti); break; - case ReturnNode ret: break; // Load must already be ahead of Return - case MemMergeNode ret: break; // Mem uses now on ScopeMin default: throw Utils.TODO("Should not reach here"); } } @@ -299,7 +294,7 @@ public abstract class GlobalCodeMotion { // Stores are already placed. Constrain the load at that block only. if( anti[stblk._nid]==load._nid ) { lca = stblk.domLCA(lca,null); - if( lca==stblk && st!=null && st._inputs.find(load) == -1 ) + if( lca==stblk && st!=null && !(st instanceof CallNode) && st._inputs.find(load) == -1 ) st.addDef(load); // Same-block load must precede the store. } return lca; diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java b/chapter25/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java index cabf7d69..82c4af10 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/BulkMemPhiNode.java @@ -253,7 +253,7 @@ public class BulkMemPhiNode extends PhiNode { mphi.setType(TypeMem.BOT.makeFrom(alias)); for( int i=1; i<nIns(); i++ ) mphi.addDef(CodeGen.CODE.add(preciseInput(in(i),alias))); - return (MemPhiNode)mphi.peephole(); + return CodeGen.CODE.add(mphi); } MemPhiNode _findPhi(int alias) { diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/LoadNode.java b/chapter25/src/main/java/com/seaofnodes/simple/node/LoadNode.java index 193fbe17..85ca42cd 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/LoadNode.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/LoadNode.java @@ -31,6 +31,25 @@ public class LoadNode extends MemOpNode { } @Override public Tag serialTag() { return Tag.Load; } + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && !clobbered(dep); + } + + // Check only immediate memory users. Writers clobber memory; Phis and + // aggregates might lead to a clobber, so stop rather than search further. + private boolean clobbered(Node dep) { + Node mem = mem(); + dep.addDepForwards(mem); + for( Node use : mem._outputs ) { + if( use==null ) continue; + dep.addDepForwards(use); + if( use instanceof StoreNode || use instanceof PhiNode || + use instanceof MemMergeNode || + use instanceof NewNode || use instanceof EscapeNode || use instanceof CallNode ) return true; + } + return false; + } + // Debugger label @Override public String label() { return "ld_"+mlabel(); } @Override @@ -89,10 +108,10 @@ public class LoadNode extends MemOpNode { Node mem = mem(); Node ptr = ptr(); - // Loads into structs do not need a ctrl edge, as null-ptr checking is - // baked into the type system. Loads into arrays DO need the ctrl - // edge, at least until proper range-checking is in place. - if( in(0)!=null && ptr._type instanceof TypeMemPtr tmp && !tmp._obj.isAry() ) { + // Null checks are handled by types. Array elements retain control for + // range checks, but reading an array's length has no index to check. + if( in(0)!=null && ptr._type instanceof TypeMemPtr tmp && + (!tmp._obj.isAry() || _name.equals("#")) ) { setDef(0,null); return this; } @@ -112,19 +131,6 @@ public class LoadNode extends MemOpNode { if( _alias == 1 ) return null; - // External storage is shared even through different class namespaces. - boolean external = ptr._type instanceof TypeMemPtr tmp && - (fld=tmp._obj.field(_name)) != null && fld._extern; - - // Simple Load-after-Store on same address. - if( mem instanceof StoreNode st && - ptr == st.nnptr() && - off() == st.off() && // Must check same object - st.val()._type.isa(_type) ) { - assert _name.equals(st._name); // Equiv class aliasing is perfect - return extend(st.val()); - } - if( mem instanceof MemMergeNode merge ) { setDef(1,merge.alias(_alias)); // In support of uplifting control to a prior dominating load, if @@ -144,67 +150,9 @@ public class LoadNode extends MemOpNode { addDep(ld)._type.isa(_type) ) // and not rolling backwards return ld; - // Load-after-Store on same address, but bypassing provably unrelated - // stores. This is a more complex superset of the above two peeps. - // "Provably unrelated" is really weak. - if( ptr instanceof ReadOnlyNode ro ) - ptr = ro.in(1); - outer: - while( true ) { - switch( mem ) { - case StoreNode st: - if( ptr == addDep(st.ptr()) && off() == st.off() ) - return extend(castRO(st.val())); // Proved equal - // Can we prove unequal? Offsets do not overlap? - if( !off()._type.join(st.off()._type).isHigh() && // Offsets overlap - (external || !neverAlias(ptr,st.ptr())) ) { // And might alias - addDep( off()); // Offsets can fold, proving unequal - addDep(st.off()); // Offsets can fold, proving unequal - break outer; // Cannot tell, stop trying - } - // Pointers cannot overlap - mem = st.mem(); // Proved never equal - break; - case PhiNode phi: - // Assume related - addDep(phi); - break outer; - case ConstantNode con: - // Load from constant memory - if( con._con instanceof TypeMem tmem ) - throw Utils.TODO("need to see a test case"); - // return ConstantNode.make(tmem._t); - break outer; // Assume shortly dead - case ProjNode mproj: // Memory projection - switch( mproj.in(0) ) { - case NewNode nnn1: - if( external ) break outer; // Class allocation does not initialize C storage - // Direct load from fresh zero/default-filled allocation. - Type decl = declaredType(); - assert decl!=Type.BOTTOM; - Type zero = decl.makeZero(); - assert zero.isa(_type); // Catch bug uninitialized null field - return ConstantNode.make(zero); - case CallEndNode cend: addDep(mproj); break outer; // TODO: Bypass no-alias call - default: throw Utils.TODO("Should not reach here"); - } - case MemMergeNode merge: mem = merge.alias(_alias); break; - case EscapeNode esc: - if( esc.self()==ptr ) // Proved equal - { mem = esc.priv(); break; } - // Mal-formed, dying - if( !(esc.self().in(0) instanceof NewNode) ) - break outer; - // Two NewNodes are always unequal - if( ptr.in(0) instanceof NewNode ) - { mem = esc.pub(); break; } - // TODO: Can we prove unequal? - break outer; - - default: - throw Utils.TODO(); - } - } + Node win = find(mem,ptr,null,null); + if( win!=null ) return folded(win); + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); // Push a Load up through a Phi, as long as it collapses on at least // one arm. If at a Loop, the backedge MUST collapse - else we risk @@ -213,34 +161,119 @@ public class LoadNode extends MemOpNode { // if( pred ) ptr.x = e0; val = pred ? e0 // else ptr.x = e1; : e1; // val = ptr.x; ptr.x = val; - if( mem instanceof PhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + if( mem() instanceof MemPhiNode memphi && memphi.region()._type == Type.CONTROL && memphi.nIns()== 3 && + // Array access control must not be moved across the merge. + in(0)==null && // Offset can be hoisted off() instanceof ConstantNode && - // Not control dependent - in(0)==null && // Pointer can be hoisted hoistPtr(ptr,memphi) ) { - // Profit on RHS/Loop backedge - if( profit(memphi,2) || - // Else must not be a loop to count profit on LHS. - (!(memphi.region() instanceof LoopNode) && profit(memphi,1)) ) { - Node ld1 = ld(1); - Node ld2 = ld(2); - PhiNode phi = new PhiNode(_name, memphi.region(),ld1,ld2); - phi.setType(phi.compute().join(_type)); - return phi; + // Returning to this memory means no change only if the pointer + // is also loop invariant. A pointer Phi may select another object. + Node stop = memphi.region() instanceof LoopNode && + !(ptr instanceof PhiNode p && p.region()==memphi.region()) ? memphi : null; + if( find(memphi.in(2),ptr,stop,new BitSet())!=null || + (!(memphi.region() instanceof LoopNode) && find(memphi.in(1),ptr,null,new BitSet())!=null) ) { + // Integer loop Phis widen to i64. Keep the load's declared + // width at the result, as for ordinary typed loop variables. + if( _type instanceof TypeInteger && !declaredType().isa(_type) ) return null; + PhiNode phi = new PhiNode(_name,memphi.region()).keep(); + phi.setType(_type instanceof TypeInteger ? TypeInteger.BOT : _type); + for( int i=1; i<memphi.nIns(); i++ ) { + Node p = ptr instanceof PhiNode pp && pp.region()==memphi.region() ? pp.in(i) : ptr; + phi.addDef(load(memphi.in(i),p,stop,phi)); + } + CodeGen.CODE.add(phi.unkeep()); + return _type instanceof TypeInteger ? new ConvertNode(declaredType(),phi) : phi; } } return null; } - private Node ld( int idx ) { - Node mem = mem(), ptr = ptr(); - assert in(0)==null; - assert !(ptr instanceof PhiNode && ptr.in(0)==mem.in(0)); // If fails, need to use pre-merged ptr at same phi - return new LoadNode(_loc,_name,_alias,null,mem.in(idx),ptr, off()).peephole(); + // Load a flavored zero from a New + private Node zero(NewNode nnn) { + Type zero = declaredType().makeZero(); + assert zero.isa(_type); // Catch an uninitialized non-null field + return castRO(new ConstantNode(zero).peephole()); + } + + // Search only; no new nodes or rewiring. Return a Store/New that folds, + // the unchanged loop memory, or a merge whose every arm folds. Null fails. + // Register dependencies so a later pointer/offset rewrite retries the query. + private Node find(Node mem, Node ptr, Node stop, BitSet visit) { + if( ptr instanceof ReadOnlyNode ro ) ptr=ro.in(1); + Field fld; + boolean external = ptr._type instanceof TypeMemPtr tmp && + (fld=tmp._obj.field(_name))!=null && fld._extern; + while( mem!=null ) { + addDep(mem); + if( mem==stop ) return mem; + switch( mem ) { + case MemMergeNode merge: mem=merge.alias(_alias); break; + case StoreNode st: + // Alias 1 is still unresolved, not a proven different field. + if( st._alias==1 ) return null; + addDep(st.ptr()); + if( _alias==st._alias && (ptr==st.ptr() || ptr==st.nnptr()) && off()==st.off() ) { + // A full-width load cannot narrow the forwarded value. + if( declaredType() instanceof TypeInteger ti && ti.log_size()==3 && + !addDep(st.val())._type.isa(_type) ) return null; + return st; + } + if( (external || _alias==st._alias) && (external || !neverAlias(ptr,st.ptr())) && + !addDep(off())._type.join(addDep(st.off())._type).isHigh() ) return null; + mem=st.mem(); + break; + case ProjNode proj: + // New's memory is private; public memory is joined at Escape. + if( external || !(proj.in(0) instanceof NewNode nnn) ) return null; + if( declaredType()==Type.BOTTOM ) return null; + return nnn; + case EscapeNode esc: + if( external ) return null; + addDep(esc.self()); + if( esc.self()==ptr ) { mem=esc.priv(); break; } + if( neverAlias(ptr,esc.self()) ) { mem=esc.pub(); break; } + return null; + case MemPhiNode phi: + addDep(phi.region()); + // Only the original loop closes a successful cycle. Another + // loop or an unfinished merge needs a separate proof. + if( visit==null || !(phi.region() instanceof RegionNode r) || r instanceof LoopNode || + r.inProgress() || phi.inProgress() || r._type!=Type.CONTROL ) return null; + if( visit.get(phi._nid) ) return phi; // Already proved every arm. + for( int i=1; i<phi.nIns(); i++ ) + if( find(phi.in(i),ptr,stop,visit)==null ) return null; + visit.set(phi._nid); + return phi; + default: return null; + } + } + return null; + } + + private Node folded(Node win) { + return win instanceof StoreNode st ? extend(castRO(st.val())) : zero((NewNode)win); + } + + // Materialize only after the profitability search succeeds. The value Phi + // stands for a load which comes back around the loop with unchanged memory. + private Node load(Node mem, Node ptr, Node stop, PhiNode value) { + Node win=find(mem,ptr,stop,new BitSet()); + if( win!=null ) { + if( win==stop ) return value; + if( win instanceof MemPhiNode mp ) { + PhiNode phi=new PhiNode(_name,mp.region()).keep(); + for( int i=1; i<mp.nIns(); i++ ) phi.addDef(load(mp.in(i),ptr,stop,value)); + return CodeGen.CODE.add(phi.unkeep().peephole()); + } + Node fold=folded(win); + if( fold!=null ) return fold.peephole(); + } + LoadNode ld=new LoadNode(_loc,_name,_alias,in(0),mem,ptr,off()); + return ld.peephole(); } private static boolean neverAlias( Node ptr1, Node ptr2 ) { @@ -269,21 +302,6 @@ public class LoadNode extends MemOpNode { return false; } - // Profitable if we find a matching Store on this Phi arm. - private boolean profit(PhiNode memphi, int idx) { - Node px = memphi.in(idx); - if( px==null ) return false; - // Memory is collapsing, no bother - if( px._type instanceof TypeMem mem && mem._t.isHighOrConst() ) - return false; - if( px instanceof StoreNode st1 && ptr()==addDep(st1.nnptr() ) && off()==st1.off() && addDep(st1.val())._type.isa(_type) ) - // To avoid cyclic pushing a Load up then down, getting here means - // the load *must* match against the Store - return true; - addDep(px); - return false; - } - // Read-Only is a deep property, and cannot be cast-away private Node castRO(Node rez) { if( ptr()._type.isFinal() && !rez._type.isFinal() ) diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/MemOpNode.java b/chapter25/src/main/java/com/seaofnodes/simple/node/MemOpNode.java index 441edc3e..de7adede 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/MemOpNode.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/MemOpNode.java @@ -8,6 +8,7 @@ import com.seaofnodes.simple.util.BAOS; import com.seaofnodes.simple.util.Utils; import java.lang.StringBuilder; import java.util.BitSet; +import java.util.ArrayList; import java.util.HashMap; import java.util.IdentityHashMap; @@ -101,6 +102,13 @@ public abstract class MemOpNode extends TypeNode { public StringBuilder _printMach( StringBuilder sb, BitSet visited ) { throw Utils.TODO(); } + // Extra conditions for factoring matching memory operations through a Phi. + // The caller has already checked the opcode, input shape and control. + boolean canDrop(MemOpNode other, Node dep) { + dep.addDep(ptr()); + return _alias>1 && _alias==other._alias && declaredType()==other.declaredType() && err()==null; + } + @Override public boolean eq(Node n) { MemOpNode mem = (MemOpNode)n; // Invariant @@ -128,4 +136,30 @@ public abstract class MemOpNode extends TypeNode { return null; } + + // Memory writers which must follow this read. A packaging node is not a + // write; follow this alias through aggregates to its writers. + public ArrayList<Node> antiDeps() { + ArrayList<Node> deps = new ArrayList<>(); + antiDeps(mem(),deps,new BitSet()); + return deps; + } + + private void antiDeps(Node mem, ArrayList<Node> deps, BitSet visit) { + if( visit.get(mem._nid) ) return; + visit.set(mem._nid); + for( Node use : mem._outputs ) { + if( use instanceof MemMergeNode merge ) { + Node slice = _alias<merge.nIns() ? merge.in(_alias) : null; + if( (slice==mem) || (slice==null && merge.in(1)==mem) ) + antiDeps(merge,deps,visit); + } else if( use instanceof CallNode || + (use instanceof MemOpNode st && !st._isLoad && st._alias==_alias) || + (use instanceof EscapeNode esc && esc.fld()._alias==_alias) || + (use instanceof MemPhiNode phi && phi._alias==_alias) || + (use instanceof BulkMemPhiNode phi && !phi.isSplit(_alias)) ) { + if( !deps.contains(use) ) deps.add(use); + } + } + } } diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/Node.java b/chapter25/src/main/java/com/seaofnodes/simple/node/Node.java index 59c9f8e3..8842db78 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/Node.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/Node.java @@ -306,6 +306,7 @@ public abstract class Node implements Cloneable { // Error is 'use' does not exist; ok for 'use' to be null. public boolean delUse( Node use ) { _outputs.del(_outputs.find(use)); + moveDepsToWorklist(); // User-count and anti-dependence queries can now change. return _outputs.isEmpty(); } diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/PhiNode.java b/chapter25/src/main/java/com/seaofnodes/simple/node/PhiNode.java index a4c73472..9d8adb15 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/PhiNode.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/PhiNode.java @@ -154,67 +154,77 @@ public class PhiNode extends Node { // Same op on all Phi paths; all ops have only the Phi as a use. // None have a control input. private boolean same_op() { - Node busy=null; - if( in(1) instanceof ConstantNode ) - return false; - for( int i=1; i<nIns(); i++ ) { - Node op = in(i); - if( in(1).getClass() != op.getClass() || op.in(0)!=null || in(1).nIns() != op.nIns() ) - return false; // Wrong class or CFG bound or mismatched inputs - if( in(1) instanceof MemOpNode mem ) { - // Mismatched aliases - if( mem._alias != ((MemOpNode)op)._alias ) return false; - // Load is clobbered somewhere, and can not be pulled forward past the Phi? - if( mem instanceof LoadNode ) - for( Node use : op.outs() ) - if( use instanceof StoreNode ) - return false; - } - if( in(1) instanceof EscapeNode ) - return false; - if( in(1) instanceof MemMergeNode ) - return false; // Have to keep aliases straight - if( op.nOuts() > 1 ) { - if( busy==null ) busy = op; - else { // Too many users, but addDep in case lose users - for( Node out : op._outputs ) - if( out!=null && out!=this ) - addDep(out); - for( Node out : busy._outputs ) - if( out!=null && out!=this ) - addDep(out); + Node op = in(1); + if( op instanceof EscapeNode || op instanceof CFGNode || op instanceof ConstantNode || op instanceof PhiNode || + op instanceof ProjNode || op instanceof NewNode || op instanceof ScopeNode || op instanceof MemMergeNode ) return false; + // A bulk Phi must split its aliases before factoring precise Stores. + if( this instanceof BulkMemPhiNode ) return false; + // Bulk splitting currently identifies parallel slices by Region/alias. + // Factoring can introduce another slice at a different memory point; + // wait until that Region's bulk partitioning has finished. + if( op instanceof MemOpNode ) + for( Node use : region()._outputs ) + if( use instanceof BulkMemPhiNode ) { + addDepForwards(use); return false; } + Node busy = null; + for( int i=1; i<nIns(); i++ ) { + Node n = in(i); + if( addDep(region().in(i))._type==Type.XCONTROL || + op.getClass()!=n.getClass() || n.nIns()!=op.nIns() || n.in(0)!=null || !op.eq(n) ) return false; + if( n instanceof MemOpNode mem && !mem.canDrop((MemOpNode)op,this) ) return false; + // Moving a Store must remove the old effect, not duplicate it. + // Allow one shared arm: the other arms disappear into the single + // factored operation, usually reducing the total operation count. + addDepForwards(n); + if( n.nOuts()>1 ) { + for( Node use : n._outputs ) + if( use!=null && use!=this ) addDepForwards(use); + if( n instanceof StoreNode || busy!=null ) return false; + busy=n; } + for( int j=1; j<n.nIns(); j++ ) + if( n.in(j) instanceof ScopeNode || ((n.in(j)==null) != (op.in(j)==null)) ) return false; } return true; } private Node drop_same_op() { - assert !(in(1) instanceof CFGNode); - Node op = in(1); - Node cp = op.copy(); - cp._type = null; // Fresh type - cp.addDef(null); // No control - + Node op = in(1), cp = op.copyEmpty(); + cp._type = null; + cp.addDef(null); for( int j=1; j<op.nIns(); j++ ) { - boolean needsPhi = false; - Node x = op.in(j); // Jth input from sample #1 + Node x = op.in(j); + boolean different = false; for( int i=2; i<nIns(); i++ ) - if( in(i).in(j) != x ) - { needsPhi=true; break; } - if( needsPhi ) { - x = make(_label,op.in(j)._type); - x.addDef(region()); - for( int i=1; i<nIns(); i++ ) - x.addDef(in(i).in(j)); - x = x.peephole(); + if( in(i).in(j)!=x ) different=true; + if( different ) { + Node[] ins = new Node[nIns()]; + ins[0] = region(); + Type t = Type.TOP; + for( int i=1; i<nIns(); i++ ) { + ins[i] = in(i).in(j); + t = t.meet(ins[i]._type); + } + // Do not introduce an operand Phi mixing incompatible type families. + if( t==Type.BOTTOM ) { + for( int i=1; i<ins.length; i++ ) addDep(ins[i]); + cp.kill(); + return null; + } + PhiNode phi = j==1 && op instanceof MemOpNode mem + ? new MemPhiNode(_label,mem._alias,ins) + : PhiNode.make(_label,t,ins); + x = phi.peephole(); } cp.addDef(x); } - // Test not running backwards, which can happen for e.g. And's - if( cp.compute().isa(compute()) ) - return cp; + // Factoring must not widen the result (e.g. correlated And operands). + if( cp.compute().isa(compute()) ) return cp; + for( int i=1; i<nIns(); i++ ) + for( int j=1; j<in(i).nIns(); j++ ) + if( in(i).in(j)!=null ) addDep(in(i).in(j)); cp.kill(); return null; } diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/RegionNode.java b/chapter25/src/main/java/com/seaofnodes/simple/node/RegionNode.java index 32434850..a886183e 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/RegionNode.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/RegionNode.java @@ -48,7 +48,7 @@ public class RegionNode extends CFGNode { if( !hasPhi() && nIns()>3 && // No Phi users, just a control user in(1) instanceof CProjNode p1 && in(2) instanceof CProjNode p2 && - addDep(p1.in(0))==addDep(p2.in(0)) && + addDepForwards(p1).in(0)==addDepForwards(p2).in(0) && p1.in(0) instanceof IfNode iff ) { // Replace with the iff.ctrl directly if( nIns()==3 ) return iff.ctrl(); diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/StoreNode.java b/chapter25/src/main/java/com/seaofnodes/simple/node/StoreNode.java index 2683ddf5..fe379c80 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/StoreNode.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/StoreNode.java @@ -49,6 +49,10 @@ public class StoreNode extends MemOpNode { baos.write(_size); } + @Override boolean canDrop(MemOpNode other, Node dep) { + return super.canDrop(other,dep) && _init==((StoreNode)other)._init; + } + // Debugger label @Override public String label() { return "st_"+mlabel(); } @Override public boolean isMem() { return true; } diff --git a/chapter25/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java b/chapter25/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java index c55002c1..aca0eea1 100644 --- a/chapter25/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java +++ b/chapter25/src/main/java/com/seaofnodes/simple/node/cpus/x86_64_v2/LoadX86.java @@ -24,6 +24,11 @@ public class LoadX86 extends MemOpX86 { } static void enc( Encoding enc, Type decl, short dst, short ptr, short idx, int off, int scale ) { + // Allocation may put a full-width integer in an XMM register (or a + // floating value in a GPR). Move its bits using the selected register bank. + if( dst>=x86_64_v2.XMM_OFFSET && decl.log_size()==3 ) decl=TypeFloat.F64; + else if( dst<x86_64_v2.XMM_OFFSET && decl instanceof TypeFloat ) + decl = decl==TypeFloat.F32 ? TypeInteger.U32 : TypeInteger.BOT; if( decl == TypeFloat.F32) enc.add1(0xF3); if( decl == TypeFloat.F64) enc.add1(0xF2); diff --git a/chapter25/src/test/java/com/seaofnodes/simple/Chapter11Test.java b/chapter25/src/test/java/com/seaofnodes/simple/Chapter11Test.java index 2e661901..d515df8d 100644 --- a/chapter25/src/test/java/com/seaofnodes/simple/Chapter11Test.java +++ b/chapter25/src/test/java/com/seaofnodes/simple/Chapter11Test.java @@ -152,6 +152,8 @@ return v.f; """); code.parse().opto(); assertEquals("return Phi(Loop,0,2);", code.print()); + for( int arg : new int[]{-1,0,1,3,100} ) + assertEquals("0",Eval2.eval(code,arg)); } @Test diff --git a/chapter25/src/test/java/com/seaofnodes/simple/Chapter20Test.java b/chapter25/src/test/java/com/seaofnodes/simple/Chapter20Test.java index ce604cf1..e39c09e7 100644 --- a/chapter25/src/test/java/com/seaofnodes/simple/Chapter20Test.java +++ b/chapter25/src/test/java/com/seaofnodes/simple/Chapter20Test.java @@ -183,7 +183,7 @@ for( int i=0; i<ary#-1; i++ ) return ary[1] * 1000 + ary[3]; // 1 * 1000 + 6 """; testCPU(src,"x86_64_v2", "SystemV",-1,"return mov(.[]);"); - testCPU(src,"riscv" , "SystemV", 7,"return (add,.[],(mul,.[],1000));"); + testCPU(src,"riscv" , "SystemV", 9,"return (add,.[],(mul,.[],1000));"); testCPU(src,"arm" , "SystemV", 5,"return (add,.[],(mul,.[],1000));"); } diff --git a/chapter25/src/test/java/com/seaofnodes/simple/Chapter21Test.java b/chapter25/src/test/java/com/seaofnodes/simple/Chapter21Test.java index b88e8af9..08da0654 100644 --- a/chapter25/src/test/java/com/seaofnodes/simple/Chapter21Test.java +++ b/chapter25/src/test/java/com/seaofnodes/simple/Chapter21Test.java @@ -279,7 +279,7 @@ for( int i=0; i<ary#-1; i++ ) return ary[1] * 1000 + ary[3]; // 1 * 1000 + 6 """; testCPU(src,"x86_64_v2", "SystemV",-1,"return mov(.[]);"); - testCPU(src,"riscv" , "SystemV", 7,"return (add,.[],(mul,.[],1000));"); + testCPU(src,"riscv" , "SystemV", 9,"return (add,.[],(mul,.[],1000));"); testCPU(src,"arm" , "SystemV", 5,"return (add,.[],(mul,.[],1000));"); } @@ -500,7 +500,7 @@ val sieve = { int N -> // Evaluate on ARM5 emulator; expect return of an array of primes in // the simulated heap. - EvalArm64 A5 = TestArm64.build("sieve", src, 100, 105, false); + EvalArm64 A5 = TestArm64.build("sieve", src, 100, 88, false); int trap_arm = A5.step(10000); assertEquals(0, trap_arm); int ary_arm = (int)A5.regs[arm.X0]; diff --git a/chapter25/src/test/java/com/seaofnodes/simple/EvalArm64.java b/chapter25/src/test/java/com/seaofnodes/simple/EvalArm64.java index 86fcf96f..1503adb5 100644 --- a/chapter25/src/test/java/com/seaofnodes/simple/EvalArm64.java +++ b/chapter25/src/test/java/com/seaofnodes/simple/EvalArm64.java @@ -544,10 +544,13 @@ public class EvalArm64 { break; } - case 0xD2: { + case 0xD2: // MOVZ + case 0xF2: { // MOVK preserves the other halfwords // movz // get immediate - rval = (ir >> 5) & 0x7FFF; + int shift = ((ir >> 21) & 3)*16; + rval = ((ir >> 5) & 0xFFFFL) << shift; + if( opcode1==0xF2 ) rval |= regs[rdid] & ~(0xFFFFL << shift); break; } diff --git a/chapter25/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java b/chapter25/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java new file mode 100644 index 00000000..2786b403 --- /dev/null +++ b/chapter25/src/test/java/com/seaofnodes/simple/MemoryEncodingTest.java @@ -0,0 +1,104 @@ +package com.seaofnodes.simple; + +import com.seaofnodes.simple.codegen.CodeGen; +import com.seaofnodes.simple.node.*; +import java.util.BitSet; +import org.junit.Test; +import static org.junit.Assert.*; + +// Execution regressions are separate from the fixed spill-measurement cohorts. +public class MemoryEncodingTest { + @Test public void testLoopLoadSearch() { + String[] sources = { + """ + u8[] !program=new u8[106]; u8[] !out=new u8[0]; + int sum=0; + for(int i=0; i<arg; i++) { + sum+=program#; + if(i==1) out=new u8[i+1]; + else if(i==3) out=new u8[2]; + } + return sum+out#; + """, + // One possibly aliasing arm must reject the unchanged-memory proof. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; if(arg&1) p=b; + int sum=0; + for(int i=0; i<6; i++) { + if(i==2) p.x=30; + else if(i==4) b.x=40; + sum+=a.x; + } + return sum; + """, + // Unchanged memory does not imply an unchanged loop-carried pointer. + """ + struct S { int x; }; S !a=new S; S !b=new S; + a.x=10; b.x=20; S !p=a; int sum=0; + for(int i=0; i<6; i++) { + if(i==2) a.x=30; + sum+=p.x; + p=b; if(i&1) p=a; + } + return sum; + """, + // All arms fold, but their values differ: build a value Phi. + """ + struct S { int x; }; S !a=new S; a.x=1; int sum=0; + for(int i=0; i<6; i++) { + sum+=a.x; + if(i&1) a.x=2; + else if(i==2) a.x=3; + else a.x=4; + } + return sum; + """ + }; + // Exercise both unresolved Store aliases and nested value-Phi cleanup. + for( long seed : new long[]{0,9,126} ) + for( int p=0; p<sources.length; p++ ) { + CodeGen code=new CodeGen(sources[p],seed,true).parse().opto(); + if( p==0 ) assertNull(code._stop.walk(n -> + n instanceof LoadNode ld && ld.ptr() instanceof ProjNode ptr && + ptr.in(0) instanceof NewNode nn && + nn.size()._type==com.seaofnodes.simple.type.TypeInteger.constant(110) ? ld : null)); + for( int arg=0; arg<7; arg++ ) { + long expected=switch(p) { + case 0 -> 106*arg+(arg>=2 ? 2 : 0); + case 1 -> (arg&1)==0 ? 140 : 60; + case 2 -> 130; + default -> 16; + }; + assertEquals(Long.toString(expected),Eval2.eval(code,arg)); + } + } + } + + @Test public void testKeepReadsBeforeWrites() { + var code = new CodeGen(""" + struct S { int x; int y; }; + S !a = new S; S !b = new S; + a.x=arg; b.x=arg+1; + S !p=a; S !q=b; + if (arg<0) { p=b; q=a; } + int v; + if (arg>1) { v=p.x; p.x=41; p.y=5; } + else { v=q.x; q.x=42; } + return v; + """).parse().opto(); + StopNode stop = code._stop; + assertEquals(2,countMemoryNodes(stop,LoadNode.class,new BitSet())); + assertEquals("-1",Eval2.eval(code,-1)); + assertEquals("1",Eval2.eval(code,0)); + assertEquals("3",Eval2.eval(code,3)); + } + + private static int countMemoryNodes(Node n, Class<?> kind, BitSet seen) { + if (n==null || seen.get(n._nid)) return 0; + seen.set(n._nid); + int cnt=kind.isInstance(n) ? 1 : 0; + for (Node def : n._inputs) cnt+=countMemoryNodes(def,kind,seen); + return cnt; + } +} diff --git a/docs/chapter-backports.md b/docs/chapter-backports.md index 7269babc..b50bd25d 100644 --- a/docs/chapter-backports.md +++ b/docs/chapter-backports.md @@ -22,17 +22,6 @@ Splitting Chapter 25 is deferred, not an instruction to renumber. ## Pending corrections -- **Forward Chapter 10b memory partitioning through 22-24.** Cliff reviewed - and committed the port through 20 in `f01ee052`. Chapter 21 is implemented - and tested; stop here for review of encoding and execution. Parameters, calls, and returns carry whole memory; aliases split - lazily within functions. - Keep private constructor memory, escape tracking, and incomplete-type - inference in 25. Carry generalized Phi factoring, its one-step Load guard, - and dependency fixes with the next ports; later snapshots already have parts - of that rewrite. Ideal Calls must keep their last input as the function pointer; direct machine - calls embed that target. Scheduling must not append anti-dependence edges to - either argument list. - ### Chapter 15 allocation design (implemented) Chapters 12-14 retain explicit initialization Stores. Chapter 15 replaces them @@ -132,7 +121,7 @@ other addend; the existing constant selector materializes it. Keep Chapter 20's inlining-safe Return typing and the dependency registrations on a rejected Phi factoring attempt. Those fixes predate this memory port. -### Chapter 21 encoding (implemented, review checkpoint) +### Chapter 21 encoding (reviewed) All three targets retain the memory Phi subclasses and New's pointer slot 0, memory slot 1, and size input 2. Chapter 21 caches New's register masks in the @@ -195,6 +184,45 @@ Logs: `build/load-search21-tests5.log`, `build/load-search21-seeds.log`, `build/load-search21-loop-seeds.log`, `build/brain-variants-search.log`, and `build/load-search21-spills-complete.log`. +### Chapters 22-24 lazy memory (implemented, review checkpoint) + +The forward port now reaches Chapter 24. Parameters, calls, and returns carry +whole memory; BulkMemPhi/MemPhi discover aliases within functions. New consumes +partial memory for its instance fields and produces `{ptr, $mem}`. The Chapter +21 loop Load search, one-step Load factoring guard, dependency fixes, and +alias-aware scheduling/evaluation carry through all three snapshots. + +Chapter 22 preserves C calls and its default-main handling. Chapter 23 keeps +cyclic field types, deep-final Loads, and symbolic field offsets; class-wide +fields do not participate in an allocation's memory coverage. Phi factoring +rejects operand Phis mixing pointers and integers, even for boolean operations. +A precise Phi waits for its bulk input to split instead of hiding that request +behind an upcast. Eval2 keeps unknown-caller pointer placeholders distinct from +actual heap constants. + +Chapter 24 retains its optimistic solver: MemPhi ignores TOP/XCONTROL arms, +New returns high results while its control is high, and alias contents register +a dependency on that control. The existing integer widening and call-graph +solver remain intact. Register checks run before encoding adds untyped branches. +Private constructor memory, escape tracking, and incomplete-type SSA stay in 25. +The separate pretty-printer consolidation remains pending. + +### Chapter 25 memory follow-through (implemented, same review batch) + +The memory representation was already present. The remaining port carries the +pure loop Load search, generalized Phi factoring with the one-step Load guard, +aggregate-aware scheduling, Call operand protection, and dependency wakeups. +It also carries the x86 register-bank Load encoding and ARM MOVZ/MOVK emulator +corrections. Bulk splitting queues new precise Phis instead of assuming an +eager fold still returns a MemPhi. + +The search preserves private constructor memory and Escape publication. A Store +whose alias is still 1 blocks the proof; external storage cannot be bypassed +using allocation identity. Length reads can drop control, while indexed array +reads retain it. Integer loop value Phis use Convert to preserve the declared +load width, and every new nested value Phi is queued for optimization. The +constructor, escape, serialization, and incomplete-type architecture stays intact. + ### Adjacent-copy forwarding: Chapters 21-25 Chapter 21 now removes a post-color Split when its sole use is the immediately @@ -469,6 +497,38 @@ The top-level runner accepts explicit chapter lists, e.g. ## Validation record +- **Chapter 25 memory follow-through.** Full Make validation passes **482 tests + plus the 17-seed fuzzer wrapper**, and the release jar and system library build. + The loop-search and read-before-write regressions fail against the original LoadNode and pass with the port. A + scratch run across **100 optimizer seeds / 3,100 evaluations** checks the + invariant length, a possibly aliasing arm, a changing loop pointer, differing + stored values at a multiway merge, and reads before writes. Seeds 0 and 9 are + retained in the regression for unresolved Store aliases and nested Phi queueing. + The historical spill replay retains exactly its **ten known String-constructor + parse failures**, with no spill-golden or native-check failures. The same 216 + successful allocations total **2,488 moves / 5,379 weighted moves**, compared + with 2,494 / 5,399 before this port; this is not a complete-suite total. + Logs: `build/mem25-negative.log`, `build/mem25-seeds.log`, + `build/mem25-final2.log` (spill replay), and `build/mem25-final4.log` + (full tests and release). + +- **Chapters 22-24 lazy memory.** Full Make suites pass **454 / 476 / 506 + tests**, respectively, plus each chapter's fuzzer wrapper. Release jars build + for all three chapters. Forwarded checks cover nested memory, constructors, + calls and recursion, allocation registers, native x86 and emulated RISC-V/ARM + execution, no-exit loop memory, and the loop Load search. Existing Chapter 23 + null-guard tests cover the mixed pointer/integer factoring rejection; Chapter + 24's BubbleSort workload covers the precise-Phi/bulk-input collapse guard. + Spill reports pass all **117 / 147 / 214** compilation entries, including + runtime and allocation checks, with no golden failures. Current raw/weighted + totals are **823 / 1,474**, **901 / 1,699**, and **1,307 / 3,008**; prior + totals were 826 / 1,470, 913 / 1,711, and 1,310 / 2,969. These include graph + changes outside allocation, so they are not allocator-only comparisons. + The chapter README tables contain the current cohort breakdowns. + Logs: `build/mem22-final.log`, `build/mem22-release.log`, + `build/mem23-final2.log`, `build/mem23-release.log`, and + `build/mem24-final3.log`. + - **Chapter 21 lazy memory and encoding.** Baseline: **412 tests plus the fuzzer wrapper**. Final: **431 tests plus the fuzzer wrapper**. Make release and spill-stats pass. The 16 forwarded memory checks retain scheduling and @@ -486,7 +546,7 @@ The top-level runner accepts explicit chapter lists, e.g. Logs: `build/memory21-baseline.log`, `build/memory21-final.log`, `build/memory21-seeds.log`, `build/memory21-spills-baseline.log`, `build/memory21-spills-final.log`, `build/memory21-spills-ablation.log`, - and `build/memory21-release.log`. Chapters 22+ are unchanged for review. + and `build/memory21-release.log`. The later 22-24 port is recorded above. - **Chapter 20 lazy memory and register allocation.** Baseline: 380 tests plus the fuzzer wrapper. Final: **396 tests plus the fuzzer wrapper**, with the diff --git a/skills/chapter25-codex-notes.md b/skills/chapter25-codex-notes.md index bcc24a33..d4f9bf2c 100644 --- a/skills/chapter25-codex-notes.md +++ b/skills/chapter25-codex-notes.md @@ -44,9 +44,11 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m - Chapter 10 is split into `chapter10a` (one bulk memory chain) and `chapter10b` (lazy graph partitioning). Both are standalone snapshots. Cliff reviewed and - committed the forward port through Chapter 20. Chapter 21 now carries lazy - memory through encoding and execution on all three targets. Stop for implementation - review at Chapter 21; leave 22-24 unchanged. See the concrete boundary notes + committed the forward port through Chapter 21. Chapters 22-24 now carry lazy + memory through cyclic types and SCCP; related corrections now reach 25. + The 22-25 batch is ready for review. + Keep Chapter 25's separate constructor-memory and incomplete-type design. + See the concrete boundary notes in `docs/chapter-backports.md`. GCM readiness and anti-dependency checks must both filter by alias and ignore MemMerge as a clobber; keep the evaluator's same alias filtering too. MemMerge still needs ordinary data-dependency placement. @@ -131,7 +133,27 @@ For shared viewer work, read [the graph viewer notes](graph-viewer-codex-notes.m adjacent-copy forwarding removes them. Measure executed memory traffic too. Carry this search with subsequent memory ports. -- Generalized Phi factoring now starts in 10a and is forwarded through 21. +- Chapters 23-24 exclude class-wide (`Field._one`) fields from New's alias + coverage and keep symbolic field offsets until layout. Alias contents must + preserve cyclic field types and readonly information. Phi factoring must not + introduce a pointer/integer operand Phi even when the result is boolean. + A precise Phi must let its bulk input split before collapsing to it; inserting + a scalar-style upcast would hide the alias request from the bulk Phi. + Chapter 24 MemPhi skips both TOP and XCONTROL predecessor arms during SCCP; + New's outputs and alias-contents query stay high while its control is high. + Check registers before encoding, which may add untyped branch nodes. + +- Chapter 25 Load search follows Escape to private/public memory; New has no + public-memory input to bypass. Alias 1 on a Store is unresolved, so it blocks + the search. External storage cannot use allocation disjointness. Array length + Loads can drop control, while element Loads retain it. New integer value Phis + start at i64 and use Convert at the result to preserve the declared width; + decline hoisting when that would lose a sharper inferred range. Queue every + new value Phi, including nested merges, and queue new precise memory Phis + instead of eagerly folding them while bulk splitting is still in progress. + User-count dependencies must wake from Node.delUse, including unkeep. + +- Generalized Phi factoring now starts in 10a and is forwarded through 25. Memory-specific eligibility lives in `MemOpNode.canDrop`, with virtual Load and Store checks. Load owns `clobbered`: all these chapters check only immediate memory users, without recursion or a visited set. Stop at Stores, |
