macOS Force-Balance / xzone-malloc Crash — and the Automatic Workaround

On macOS, computing a relative-weight ForceBalanceReg's D0 scale with a large direct matrix factorization, then running the zone phase in the same process, crashes with a SIGILL deep inside the system allocator. This is an upstream platform bug (Apple libmalloc / Julia), not a Sentinel memory error.

Sentinel works around it automatically. On macOS, resolve_fb_weights! computes D0 via a condensed, factorization-free solve (unpreconditioned GMRES on the pressure-condensed stiffness) instead of the direct factorization. No above-threshold factorization runs, so the heap is never corrupted and the run completes normally. Relative force balance "just works" on macOS. Linux keeps the faster direct solve.

The upstream bug (characterization)

A run with ForceBalanceReg(frac; relative=true) (the FORGE default), computed the old (direct) way, SIGILLs at the start of the zone phase, immediately after the weight-resolution line ForceBalanceReg: relative weights resolved, D0/S0 = …. The macOS crash reporter itself then deadlocks in malloc, so Julia prints an empty backtrace and the process exits 137. The real stack (in ~/Library/Logs/DiagnosticReports/julia-*.ips) points into xzone-malloc internals — e.g. _xzm_segment_group_*, _xzm_xzone_find_and_malloc_from_freelist_chunk, _os_unfair_lock_recursive_abort.

The trigger is the sequence "a large SuiteSparse/UMFPACK factorization + solve (the D0 global solve) followed by any zone-phase solve, in the same process." It is a size threshold on the factorization (survives ≤47k DOF, crashes ≥95k DOF at the same density) — not concurrency, scale spread, or u/p structure. Isolation matrix (T18), computed the direct way:

Configuration	Result
relative FB → direct resolve + zone phase	crash (deterministic)
absolute FB (no resolve) + zone phase	clean
resolve only, no zone phase	clean
zone phase only, no FB	clean

Two independent memory-error detectors clear the code:

• Linux valgrind: zero invalid reads/writes/frees over the full workload, correct objective.

• macOS Guard Malloc (DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib): the identical repro that SIGILLs 100 % under the default allocator runs to completion cleanly — swapping xzone out for guard pages eliminates the crash.

Neither MallocNanoZone=0 nor forcing allocator compaction (malloc_zone_pressure_relief) after the resolve step prevents it. The fault is in macOS xzone-malloc's own segment/lock machinery, tripped by this specific allocation pattern. Linux/DGX runs are unaffected.

The workaround (what macOS does now)

The D0 global solve is the only above-threshold factorization in the whole FB workload — every zone solve is small (below threshold). So Sentinel computes D0 without a large factorization:

• Condensed stiffness (condense_pressure=true): eliminating the elemental pressure DOFs is exact and removes the saddle-point indefiniteness that makes the uncondensed system unusable for an unpreconditioned Krylov solve. (The uncondensed saddle solved iteratively gives ~36 % D0 error — useless; condensation is what makes the iterative solve viable.) The condensed stiffness can still be ill-conditioned, so accuracy is obtained with a tight solver tolerance rather than a loose one (see below).

• Factorization-free solve (FactorizationFreeSolver = unpreconditioned GMRES): no lu, no ILU — nothing above the crash threshold is factorized.

D0 only needs ~1 % accuracy (it anchors a forgiving relative weight), but on the ill-conditioned condensed system the GMRES residual is a poor proxy for solution accuracy, so the solver uses a tight rtol (not a loose one) to stay safely past the convergence plateau. Cost: a one-time resolve (seconds to a few minutes, depending on problem size and GMRES convergence), negligible against a minutes-to-hours run.

Never precondition the D0 solve with ILU

An ILU (or any lu) is the factorization that poisons the heap. The D0 iterative solver must stay unpreconditioned (or use only a Jacobi diagonal — not a factorization). Do not route it through KrylovSolver, which builds ILU.

Accuracy

Pressure condensation is exact, so the condensed-iterative D0 equals the direct D0 up to the GMRES tolerance, and the resolved D0/S0 is required to match the direct value within ~1 %. Direct reference values: T18 5.764497533422361e-6, MGH 3.93393930289679e-7. This equivalence is verified against a direct solve in the unit test suite; the full-system T18/MGH match is checked as a release-validation gate.

Performance — why the resolve is a slow, one-time cost

The condensed system is ill-conditioned (off-diagonal Schur κ/μ coupling), so unpreconditioned GMRES does not reach rtol on production meshes — it runs to itmax (a one-time resolve of minutes). The resolved D0/S0 scalar is accurate anyway (it is far more forgiving than the raw GMRES residual), so this is correct, just slow — and negligible against a minutes-to-hours inversion.

There is no faster factorization-free option (measured, 2026-07-21):

• Jacobi (diagonal) preconditioner — no measurable convergence benefit on T18/MGH (the ill-conditioning is off-diagonal, not diagonal). Kept as a selectable precond=:jacobi option but not the default.

• Zone-decomposed D0 (consolidating small, below-threshold per-zone solves) — ~9% inaccurate on T18, because zone-boundary data comes from the measured field rather than the global solution; it fails the ~1% accuracy gate.

• ILU-quality preconditioning would converge, but an ILU is a factorization and would re-poison xzone-malloc — forbidden.

Do not re-attempt Jacobi or zone-decomposition for speed without new evidence.

Override — force the direct path (version-recheck lever)

Set SENTINEL_ALLOW_MACOS_FB_RELATIVE=1 to force the old direct D0 path on macOS. This is the lever to re-test whether a newer macOS or Julia has fixed the upstream bug: with it set, run the real relative-FB workload; if it no longer SIGILLs, the platform bug is fixed and the macOS branch can eventually be removed (restoring the faster direct solve everywhere). With the override set on a still-affected system, macOS relative FB is expected to SIGILL.

Recheck note

Confirmed still-present on macOS 26.5.1 (arm64, Apple Silicon) under both Julia 1.11.5 and 1.12.6 as of 2026-07-20 (the 1.11→1.12 bump also moved SuiteSparse_jll 7.7.0 → 7.8.3; the crash signature is identical, pointing at Apple's libmalloc rather than a Julia-version issue). Re-verify periodically with the override recipe above; update this date when you do.

Upstream

Tracked as a Julia/Apple-libmalloc platform bug. Reproduces deterministically via the isolation matrix above. Report drafted; not yet filed (pending a maintainer channel — Apple Feedback Assistant and/or JuliaLang/julia).