0640 — The shared loop re-claims instead of ending
Edit this pageContext. [0638] settled that a fatal exit of the shared reconcile loop logs a declared Error and does not restart, on the reasoning that “a spawned worker returning fatal would replay” and “the next join starts a fresh loop”. Building it showed that reasoning is wrong in both directions. It ships a regression: today a fatal returns from SystemManager.Run, the caller’s process exits, and the orchestrator restarts it, so upkeep resumes on its own. Under a shared loop nothing returns to a Consume by design, so with no restart a single fatal leaves that process’s upkeep dead for its whole lifetime — and for vulkan manager run, whose entire job is the manager, the process would sit up and healthy doing nothing at all. “The next join starts a fresh loop” does not save either case: a long-lived consumer joins once at Consume and never again, and a dedicated manager process has exactly one caller. If every consumer in a deployment hit one transient fatal, upkeep would stop everywhere until someone restarted pods — the failure [0635] set out to remove. The replay worry is real but is what a backoff is for, and the codebase already answers it that way everywhere else: the tick runner backs off per failure and escalates to a declared Error past its curve rather than giving up.
Decision. The loop re-claims. reconcile runs manager.Runner.Run in a loop bounded only by the loop context: a life that ends on its own logs VK0065 with the error, the attempt, and the delay, waits out SystemManagerConfig.RunRetry (the per-loop curve, defaulting like every other), and claims the row again. The misconfiguration class keeps its fast, loud failure through a different door: join resolves the system owner and builds the runner synchronously, so a deployment with no registered system fails the caller — the CLI still exits immediately on the errors that mean “this cannot work”, while runtime faults become a retried Error line. This also removes the state [0638] would have needed: s.stop != nil now means exactly “a loop is running”, because the loop’s lifetime is the callers’ lifetime and nothing else ends it, so there is no dead-loop bookkeeping and no join that has to notice one.
Consequences. Upkeep self-heals in place: a worker row fixed while consumers run is picked up on the next life with no restart, and the attempt attribute is the operator’s signal for the difference between a blip and a worker that fails every time it starts. vulkan manager run no longer exits on a runtime fatal, which is a real change for anyone whose alerting watches for the process dying — the VK0065 Error line is what to watch instead, and its docs page says so. Supersedes [0638]‘s clause “with no automatic restart — a spawned worker returning fatal would replay — and the next join starts a fresh loop”; the rest of 0638 stands, including that no caller receives the fatal. Verified live under -race: two Run calls on one client share one loop and one instance row, the loop outlives the caller that started it, the last one out releases the claim rather than letting it expire, and a later Run on the same client starts a fresh loop. Rejected: returning the fatal to every blocked caller, which tears down consumers for a deployment-wide fault that is not their business; returning it only to explicit RunManager callers, which rebuilds the two-models split [0638] existed to collapse; and putting the retry in the consumer’s adapter, which would spread one policy across the callers instead of keeping it beside the loop.
Amended. [0641] deletes the shared loop: the re-claim decision here stands, now running inside each caller’s own Run, and the stop != nil observation is moot.