A runtime instrumentation stack for measuring perturbation dynamics, characterizing branchpoint geometry, and conducting falsifiable closed-loop control experiments during autoregressive generation — without modifying model weights.
Foundation rebuild status (2026-07-19). A code-and-claims audit rebuilt Observer around Observe → Perturb → Compare → Prove, with Act/controller work downstream and paused. The active implementation now distinguishes consumed tokens from predicted next tokens, records authoritative loaded-model provenance, creates independent same-context one-step branchpoint forks, runs matched clean/perturbed recovery branches, and validates diagnostics against simple baselines and known synthetic trajectories.
This audit also reopens two v2 interpretations. First, the F31 AUROCs came
from continued controller shadow/active trajectories. After the first token
difference, later rows had different contexts, so their labels measured
shifted/cascading divergence rather than independent local flippability.
Second, the historical BASE → PERTURB → REASK protocol changed context and
continued only the perturbed branch; its perturbation effects remain
observations, but its recovery ratios and regime labels are provisional.
The historical numbers below are retained as an audit record, not erased.
The current contract is documented in
docs/OBSERVER_FOUNDATIONS.md.
The repaired protocols were then calibrated in 42 Qwen3-1.7B runs across three seeds and two prompts. At relative additive magnitude 0.30, unique whole-run prompt-slice AUROC reached 0.831 on sourdough and 0.886 on water cycle. Matched hysteresis showed near-zero and order-one endpoint propagation at the same magnitude on different seeds. These new findings are F32/F33 below; they do not resume the controller.
A subsequent 280-run Q2 sweep captured paired deltas at every downstream block and final RMSNorm. It found that raw L2 amplification, normalized structural disturbance, and output disruption are not interchangeable. Final RMSNorm erases radial final-layer scaling but preserves directional additive change. F34 records the scoped result. A subsequent 60-run Q3 suite continued verified clean and perturbed branches across ten prompts. It observed improve, degrade, and tie outcomes under frozen task rubrics, but a clean pre-flip predictor reached only 0.571 mean whole-prompt-held-out AUROC, below the prespecified 0.70 gate. F35 records that measured negative result. A preregistered 90-run M3R-2 replication then used 15 entirely new prompts, higher-resolution frozen rubrics, and a compact regularized predictor. It reached 0.596 mean AUROC across 13 valid prompt holdouts, still below 0.70, and the earlier procedural direction did not replicate. F36 records the strengthened negative result; controller work remains paused.
v2 update (2026-04-19). The first version of this paper presented closed-loop stability control as the central contribution. Subsequent experiments falsified the original controller thesis on Qwen3-1.7B: scaling interventions at the final layer have zero effect (absorbed by RMSNorm), additive interventions are either silent no-ops or over-actuating depending on magnitude, and acting earlier in the residual stream cascades destructively. Most fundamentally, the divergence signal that drives the controller was found to measure token-level prose surprise — word-starts, semantic transitions, structural boundaries — rather than dynamical instability in any control-theoretic sense.
What survives is a repairable instrument and a set of observed perturbation effects. The July foundation audit replaces the old branchpoint and hysteresis interpretations with matched protocols. The historical 0.82–0.86 AUROCs are no longer presented as evidence that local flippability is predictable; F31 and Q1 are reopened for calibration with independent one-step counterfactuals.
Sections rewritten in v2: §6 (spectral diagnostics, methodology corrected), §9 (controller empirical evaluation, replaced with falsification arc), §10 (experimental results, replaced with the Qwen3-1.7B controller arc), §11 (new — branchpoint hijacking), §12 (new — mapping program), §14 (limitations updated), §15 (conclusion rewritten). The July 2026 correction subsequently rewrote §3 architecture, §7 hysteresis, §11 F31, §12 status, §13 reproducibility, §14 limitations, and §15 conclusion around the repaired foundation.
We present observer, an open-source runtime stack for studying
perturbation dynamics in autoregressive language models. Its active foundation is
Observe → Perturb → Compare → Prove: canonical token-time telemetry,
independent local counterfactual forks, persistent propagation experiments,
matched recovery, and provenance-rich artifacts. The historical
divergence field is a compatibility alias for a descriptive
local VAR(1) prediction error. Simple hidden-velocity, hidden-acceleration,
logit-entropy, and top-1-margin baselines are emitted beside the advanced
probes.
We use this instrument to falsify our own initial closed-loop stability control hypothesis on Qwen3-1.7B and report what the experiments actually showed: the divergence signal correlates with token-level prose surprise (word-starts, structural boundaries, semantic transitions) rather than dynamical instability in a control-theoretic sense. Closed-loop control over this signal is not effective on the tested model. We then report an observed effect from the same apparatus: additive interventions can produce token and trajectory changes. However, a later audit found that the reported 0.82 and 0.86 AUROCs used shifted/cascading labels from continued controller trajectories and therefore do not establish predictable local branchpoints. We preserve that negative methodology lesson, describe the repaired matched protocols, and reopen the mapping questions with explicit stop conditions and controller-return criteria.
The dominant paradigm in mechanistic interpretability — sparse autoencoders, circuit discovery, logit lens analysis — answers the question "what does this model compute?" It is fundamentally a post-hoc analytical approach. The field has produced significant understanding of model internals, but has largely deferred a different class of question:
Can we detect when generation is destabilizing, in real time, and do something about it?
This is the question observer is built to answer. It is closer in spirit to control engineering than to interpretability research: rather than analyzing a system's internal structure, we treat the model as a dynamical system and ask whether we can build a feedback loop around it.
The practical stakes are not abstract. High-stakes deployments of language models — in agentic settings, long-horizon tasks, adversarial environments — require some answer to the question of whether generation has gone off course and whether that course can be corrected. The current state of the art is largely output-level heuristics: does the text look wrong? Observer proposes that the answer should be visible in the hidden trajectory before it surfaces in the output, and that a runtime controller can act on that signal.
Scope caveat (v1, retained for context): Observer was framed as a research instrument. The original paper hedged that the divergence signal measured trajectory instability and that empirical validation of downstream correlates was "the necessary next step."
Update (v2): we performed that validation. The divergence signal does not measure dynamical instability in the control-theoretic sense it was framed as measuring; it measures token-level prose surprise. See §10 for the falsification arc and §11 for the observed perturbation effects and reopened local-branchpoint question that emerged from the same apparatus.
Observer occupies a space adjacent to several lines of existing work, without directly duplicating any of them.
TransformerLens (Nanda, 2022) provides the dominant toolkit for mechanistic interpretability research: model loading, hook-based activation capture and modification, and a large community of research built on its abstractions. It is an exploration tool — excellent for research notebooks and circuit analysis, not designed around systematic experimental protocols or recovery measurement.
pyvene (Wu et al., 2024) formalizes interventions as first-class serializable primitives, enabling composable intervention specifications across locations, granularity, and sequence position. It is an execution library: it provides the mechanics of intervention without opinions about experimental design, hysteresis, or recovery.
nnsight provides a Pythonic interface for local and remote model execution, including NDIF infrastructure. Observer's current optional nnsight adapter supports local execution only. Remote execution is intentionally rejected until Observer has a trace-based backend that can preserve its event and intervention contracts honestly.
The Representation Engineering paper (Zou et al., 2023) demonstrated that model behavioral tendencies can be read from and written to activation space via linear probes and steering vectors. The Inference-Time Intervention paper (Li et al., 2023) applied shifted activations at inference time, improving TruthfulQA performance from 32.5% to 65.1%. Neither line of work focused on recovery dynamics or closed-loop feedback.
Recent work on LLM output consistency (Raj et al., 2023; Huang et al., 2023) characterizes stability at the output level — how often does the same model produce the same answer across runs? Observer operates at a different layer: activation-level perturbation dynamics within a single generation, not output-level consistency across generations.
LinEAS (Rodriguez et al., NeurIPS 2025; arXiv:2503.10679) trains activation steering end-to-end with a global distributional loss, showing that locally tuned maps produce unintended downstream shifts when applied out-of-sample. Observer's adaptive controller is designed to detect and respond to such downstream cascades in real time.
FASB (Cheng et al., 2025; arXiv:2508.17621) dynamically determines intervention necessity and strength by tracking internal states during generation, with a backtracking mechanism to correct deviated tokens. Observer shares the adaptive framing but adds deterministic branchpointing and explicit recovery measurement, quantifying whether the trajectory recovered or remained shifted after intervention ended.
Grant et al. (2025; arXiv:2511.04638) provide a theoretical treatment of how causal interventions can push representations off the model's natural manifold, distinguishing benign null-space divergences from pernicious ones that activate dormant pathways. Observer's PLASTIC and DIVERGENT regime classifications can be interpreted through this taxonomy, offering empirical runtime signatures for divergence types their framework characterizes theoretically.
HARP (Hu et al., 2025; arXiv:2509.11536) decomposes hidden state space into semantic and reasoning subspaces via SVD of the unembedding layer, achieving AUROC 92.8% on TriviaQA hallucination detection. Observer's windowed SVD probe tracks effective rank dynamically within a generation rather than using static subspace decomposition for classification, a complementary signal.
HALT (Shapiro, Taneja, and Goel, Feb 2026; arXiv:2602.02888) treats token log-probability sequences as a time series for lightweight hallucination detection without requiring internal model access. Observer's VAR(1) predictor applies a related time-series framing to hidden state trajectories, a white-box signal that feeds an active intervention loop rather than a post-hoc detector.
Observer's active contribution is experimental discipline around runtime intervention: canonical causal events, deterministic same-context local forks, persistent propagation experiments, equal matched-recovery branches, and provenance that makes matching claims checkable. Existing tools execute interventions; Observer separates what changed immediately, what propagated, what persisted, and what recovered. Closed-loop action remains a downstream, paused research direction.
Observer's active architecture follows four evidence stages. Historical versioned directories remain in the repository for reproduction, but they are not the current protocol contract.
Record the consumed token, pre/post-intervention hidden state, logits, predicted next token, simple baselines, and advanced token-time diagnostics without assigning unsupported semantic meaning.
Use independent one-step same-context forks for local sensitivity, or continued matched branches for propagation. Do not mix those labels.
Measure hidden and logit distances, token flips, exposure endpoints, and equal intervention-free matched recovery continuations.
Bind every result to loaded-model identity, exact config hash, context fingerprints, protocol-validity fields, structured events, zero controls, and tests. Act/controller work is downstream.
PROMPT │ ▼ [ SeedCache: build_seed_cache() ] │ past_key_values snapshot │ next_token_logits │ seed_hidden @ intervention_layer │ ├──────────────────────────┐ ▼ ▼ [ CLEAN one-step fork ] [ PERTURBED one-step fork ] same consumed token same consumed token same sampling RNG intervention active │ │ └──────────┬───────────────┘ ▼ [ Local causal comparison ] hidden cosine / relative L2 KL / JS on logits argmax and sampled-token flip discard perturbed fork advance clean trajectory only
The central design problem in intervention experiments is confounding. A naive implementation runs the baseline and intervention branches from separate forward passes over the same prompt. This introduces at minimum: different random number generator states at the point of token sampling (even under greedy decoding, CUDA operations can have ordering nondeterminism), and potentially different attention mask states depending on the batching implementation.
The SeedCache resolves this by running the prompt exactly once, then cloning the resulting model state for both branches:
# Run prompt once, snapshot pre-generation state
def build_seed_cache(model, tokenizer, device, prompt, layer) -> SeedCache:
hook = _HiddenCaptureHook()
handle = layers[layer].register_forward_hook(hook)
with torch.no_grad():
outputs = model(input_ids, use_cache=True, return_dict=True)
handle.remove()
return SeedCache(
past_key_values = outputs.past_key_values, # full KV cache
next_token_logits = outputs.logits[:,-1,:], # first token dist
seed_hidden = hook.captured, # hidden @ layer
fingerprint = compute_cache_fingerprint(...) # checksum
)
# Both branches start from identical state
baseline_cache = seed_cache.clone()
intervention_cache = seed_cache.clone()
# SeedCache.clone() deep-copies past_key_values via clone_past_key_values()
# handles DynamicCache, legacy tuple-of-tuples, and generic objectscache.pyThe fingerprint hashes the first-layer key-cache shape, dtype, and a deterministic value sample. Experiments use it with cache sequence length to verify both forks share a common origin. This is the kind of rigor that most published intervention papers treat as an implementation detail but actually matters for result validity.
Why this matters: Without a shared branchpoint, local sensitivity measurements conflate intervention effects with different initial state. Recovery additionally requires two aligned continuations, as specified in §7.
The signal historically called divergence is a
per-token held-out local prediction error from a VAR(1)
model fit on a sliding window of projected hidden states. Canonical
diagnostics now expose local_prediction_error and retain
divergence as a compatibility alias.
The hidden state ht ∈ ℝD (where D is the model's hidden dimension, typically 4096–8192) is projected to a fixed low-dimensional space via a deterministic Rademacher matrix:
The Rademacher projection preserves inner products in expectation (Johnson-Lindenstrauss), reduces the regression problem from D-dimensional to k-dimensional (k=64), and is computed once per hidden dimension via a seeded RNG — making it reproducible across runs and comparable across model families with different hidden sizes.
A first-order vector autoregressive model is fit on the sliding window W = {zt-n, ..., zt-1} via ridge regression:
Critically, the matrix A is fit on the window excluding the newest state zt. The prediction ẑt = zt-1 · A is then compared to the actual observed zt. This is a held-out evaluation: the model is never trained on the transition it is asked to predict. This matters because in-sample VAR(1) error on a short window would collapse toward zero regardless of actual trajectory instability.
The per-token scalar divergence combines normalized L2 error and cosine distance with a symmetric denominator to avoid blow-ups when projected norms are near zero:
When the hidden trajectory is locally predictable, the VAR(1) fit is good and divergence is low. When generation dynamics shift — through perturbation, distributional shift in the prompt context, or internal instability — the held-out prediction error increases. The signal is cheap: one matrix multiply per token in 64-dimensional space.
def step(self, hidden: torch.Tensor) -> float:
z = self._project(hidden) # (D,) → (64,)
self._window.add(z) # FIFO buffer, maxlen=8
if len(self._window) < 3:
return 0.0
states = self._window.matrix() # (T, 64)
train = states[:-1, :] # exclude newest
A = _fit_var1_ridge(train) # fit on T-1 transitions
pred = states[-2, :] @ A # predict from t-1
actual = states[-1, :] # held-out: actual t
return _divergence(pred, actual)["combined"]predictor.pyThe divergence signal is the primary input to the controller, but the V1.5 observability runner and the adaptive controller also compute three supplementary diagnostics that provide corroborating signal and richer telemetry for offline analysis.
v2 correction. The original spectral module FFT'd the flattened hidden-state vector along the feature-index axis and reported entropy, flatness, centroid, and band fractions over that spectrum. The v1 paper acknowledged that "the feature index is not a temporal axis" but defended the metrics as a stable characterization of activation energy distribution. This defense does not survive a permutation test: neuron ordering in transformer hidden states is arbitrary (a function of weight initialization, not semantics), and any neuron-axis FFT summary is a function of that arbitrary ordering. Permuting neurons changes every reported metric; the underlying activation is unchanged.
The v2 implementation rewrites this module as a token-time spectral probe:
hidden states are accumulated into a sliding window of shape [T, D]
and the FFT is taken along the time axis (dim=0). Per-frequency power is then
averaged across the D dimensions, producing a scalar trajectory spectrum. This
captures real structure — slow drift vs. high-frequency oscillation in
activation patterns across generation steps — and is invariant to neuron
permutation. A built-in self-test reports a non-zero permutation-change ratio
whenever the window has at least 8 tokens, confirming the time axis is in fact
what's being analyzed.
The corrected metrics, computed on the time-axis trajectory spectrum, are:
| Metric | Description |
|---|---|
spectral_entropy | Normalized Shannon entropy of the time-axis power spectrum. High = energy spread across slow and fast trajectory frequencies. |
spectral_flatness | Geometric mean / arithmetic mean of power. Approaches 1.0 for white-noise trajectories, 0.0 for tonally pure ones. |
centroid | Normalized frequency centroid ∈ [0,1]. High centroid = trajectory dominated by step-to-step oscillation rather than slow drift. |
high_frac | Fraction of power in the upper 20% of trajectory frequencies. |
rolloff_85 | Normalized frequency below which 85% of cumulative power falls. |
permutation_change | (new in v2) Self-test ratio comparing the spectrum of the actual trajectory to the spectrum of a randomly time-permuted version of the same window. Should be > 0 — confirms time-axis behavior. If a future regression makes this near zero we know the module has reverted to neuron-axis behavior. |
Empirically, permutation_change turned out to be the strongest single
feature for predicting branchpoint flippability on Qwen3-1.7B in §11.5 — a feature
that was conceived as a methodology self-test ended up carrying real signal about
trajectory geometry.
A window of hidden vectors {ht-w, ..., ht} ∈ ℝW×D is stacked into a matrix X and its singular value decomposition computed via the Gram trick: eigenvalues of XXT (a W×W matrix with small W) yield the squared singular values without requiring the full D×D computation. An SVD of a single vector returns only the vector norm — uninformative. The windowed approach captures the local rank structure of the trajectory: whether the model is moving through a low-dimensional manifold or exploring higher-dimensional space.
Effective rank is computed as exp(H(p)) where p is the normalized singular value distribution — the exponential of the entropy of squared singular values. A drop in effective rank signals that the trajectory is collapsing onto a lower-dimensional subspace, a potential precursor to repetition or mode collapse.
At three probed layers (early / mid / late), the velocity norm vt = ||htL − ht-1L||2 is tracked over a sliding window. Mean velocity defines stiffness; the linear slope of velocity over the window defines stiffness trend. Elasticity = 1/(1 + stiffness) provides a bounded stability score in (0,1]. This is a diagnostic proxy, not a physical quantity.
The active protocol is matched-exposure-recovery-v2. Clean and
perturbed trajectories begin from clones of the same SeedCache, advance in
lockstep with paired sampling randomness, and receive identical phase
instructions.
Phase 1: MATCHED EXPOSURE ───────────────────────────────────────────────────── SeedCache.clone() → CLEAN branch (intervention always off) SeedCache.clone() → PERTURBED branch (noise active only in window) Same requested decisions, same sampling RNG Phase 2: MATCHED RECOVERY ───────────────────────────────────────────────────── Continue BOTH exposure endpoints No new instruction on either branch Intervention disabled on both branches Same requested recovery length, same sampling RNG Metrics: initial intervention distance = first active paired decision active-window peak = largest active paired distance (may include accumulation) propagated distance = paired distance after the active window residual distance = final paired recovery distance recovery = 1 - residual / propagated
Each aligned decision records hidden cosine distance, relative hidden L2, logit Jensen–Shannon and KL divergence, argmax flip, and paired sampled-token flip. The primary paired distance is the maximum of hidden cosine, relative hidden L2, and JS for that step.
A recovery value and regime are emitted only if the exposure includes a
post-intervention decision, its endpoint exceeds the configured propagation
floor, and both recovery traces have their full matched length. Otherwise
the run is explicitly invalid or
no-propagation.
Valid recovery values use the following descriptive regimes:
R > 0.8. Most propagated separation decays during matched recovery.
0.4 < R ≤ 0.8. Some propagated separation remains.
0 ≤ R ≤ 0.4. Most propagated separation persists over this window.
R < 0. Paired separation grows during the matched recovery window.
These labels describe a measured window; they are not material properties of a model. The historical three-stage re-ask results remain visible elsewhere in this paper, but their recovery/regime interpretation is provisional because that protocol changed context and lacked a matched clean recovery branch.
The intervention engine is the core experimental workhorse. It runs baseline and intervention branches from a shared SeedCache, captures full hidden trajectories from both, and computes a rich set of comparison metrics.
| Type | Operation | Parameters |
|---|---|---|
| additive | Add a unit random vector scaled by magnitude to last-token hidden state. | magnitude, seed |
| projection | Project out a random k-dimensional subspace: h ← h (I − QQT) | subspace_dim, seed |
| scaling | Multiply last-token hidden state by scalar s. | scale |
| sae | Steer along SAE decoder column for a specified feature index. | sae_repo, feature_idx, strength |
Hooks are registered with register_forward_hook and removed in finally
blocks. Critically, the intervention is applied before the hook captures the hidden state —
so the captured tensor reflects what downstream layers actually receive, not the pre-intervention value.
This is the correct ordering that many published implementations miss.
The TrajectoryComparison object implements a layered fallback strategy for
computing per-token distances between branches:
The primary metric is cosine distance on actual hidden vectors (preferred). If hooks fail to attach and hidden vectors are unavailable, it falls back to Jensen-Shannon divergence on the logit distributions. If logits are also unavailable, it falls back to normalized L2 on hidden norms. The code documents this explicitly: "hidden_norm alone is not sufficient — the same norm can hide large vector changes."
Recovery is computed over the post-intervention window: deviation_during (mean primary metric during active intervention), final_distance (primary metric at final token), recovery_ratio = (deviation_during − final_distance) / deviation_during, and convergence_rate (negative slope of primary metric over post-intervention tokens via linear fit).
The adaptive controller closes the loop: per-token diagnostics drive a proportional scaling intervention that damps the hidden state when the composite score exceeds a threshold.
v2 status. The architecture in this section is unchanged from v1, and the implementation runs as described. What changed is the empirical story. The controller is no longer presented as a working component with interesting attractor-selection behavior; it is presented as a falsifiable hypothesis that we falsified. §10 reports the failure modes (silent no-op at L=−1 with scaling, over-actuation at higher additive magnitudes, destructive cascade when acting earlier in the stack). The controller code remains in the repository as research scaffolding for a future redesign — the criteria under which controller research would resume are listed in §11.7.
The spectral and SVD terms are gated — they only contribute when they exceed a baseline level (spectral entropy above 0.75, high-frequency fraction above 0.30), to avoid penalizing normal variation. The rank delta term detects sudden changes in trajectory dimensionality.
A 3-token moving average of the score is computed. When the smoothed score exceeds a threshold, the controller applies a scaling intervention to the last-token hidden state at the monitored layer, then enters a cooldown period during which the scale is held and further threshold evaluations are suppressed:
| Status | Condition | Scale Applied | Cooldown |
|---|---|---|---|
| STABLE | avg_score ≤ 0.55 | 1.0 (no intervention) | — |
| WARNING | 0.55 < avg_score ≤ 0.85 | 0.90 | 3 tokens |
| CRITICAL | avg_score > 0.85 | 0.75 | 6 tokens |
| COOLDOWN | Post-intervention hold | Held from trigger | Counting down |
The scaling intervention multiplies the hidden state: ht ← s · ht. This reduces the magnitude of the current representation, which typically reduces the entropy of the downstream logit distribution and pulls the model toward its modal behavior. The mechanism is simple and its effects are legible — a deliberate choice given that the controller is a research instrument, not a production component.
When --shadow is set, the controller observes and logs its decisions
but does not apply the scaling hook. This allows calibration of threshold and weight
parameters on a given model and prompt distribution before active deployment.
The separation of observation and actuation is explicit in the code:
if (scale_used < 1.0) and (not shadow): hook.set_active(True).
v2 replacement. The v1 §10 reported two experiment families on
Qwen2.5-7B that, in retrospect, lacked controls we now know are required: per-step
intervention-applied counts, decoupled measure/act layers, and pair-level
shadow/active comparison. The v1 headline ("controller aggressiveness determines
which attractor the model lands in") was reported as a controller property but is
consistent with two alternative explanations we could not rule out at the time:
Qwen2.5-specific scaling behavior, and prompt-seed-specific basin topology
independent of the controller. The v1 results are preserved in the repository's
RESEARCH_CONTROLLER.md archive for historical context.
In v2 we ran a longer arc on Qwen3-1.7B (28 layers) with explicit controls. We report it here in the order the experiments happened, including the negative results, because the negative results determine the structure of the rest of the paper. The model used throughout this section is Qwen3-1.7B; generalization to other architectures is treated as out of scope for the current mapping program (one cross-model scope-check is reported in §11.5).
On the prompt "Write step-by-step instructions for baking sourdough bread.", the closed-loop controller at its v1 design defaults (act_layer=−1, intervention_type= scaling, scale_warn=0.90, scale_crit=0.75) produced essentially the same per-token divergence as shadow mode. avg_raw_div: 0.765 (shadow) vs 0.768 (active). Warning counts: 8 vs 7. Critical counts: 1 vs 1. With 5 seeds × 1 prompt the difference is well within seed variance. This was the first signal that something in the pipeline did not work as advertised.
A diagnostic stress run isolated the cause. With scale=0.5 (halving the
final-layer hidden state) we measured logit_kl_mean_during = 0.0000 across 5 seeds.
With scale=2.0 (doubling), also 0.0000. token_match_rate = 1.000 in
both cases. The scaling intervention was a true no-op at L=−1.
The mechanism is structural: Qwen3 places an RMSNorm between the last transformer block and the LM head. Scaling the input to that norm by any constant factor is erased — the norm rescales to unit variance, the LM head sees an essentially identical input, and the argmax is unchanged. The closed loop in F4 was firing the controller (the trigger was active), the scaling intervention was applied, and the intervention had no downstream effect. The closed loop was open at the actuator.
A 4-way intervention-type comparison at the same layer (additive, scaling@0.5,
scaling@2.0, projection-onto-64-dim-subspace) clarified what does work. Additive
perturbation with relative magnitude 1.0 produced
logit_kl_mean_during = 10.40 ± 2.78 across 5 seeds (DSR = 3.73), with
token_match_rate = 0.145 — that is, 85% of generated tokens differed from baseline.
Projection produced larger logit shifts but always landed in a runaway regime.
Scaling at any magnitude produced exactly zero. The conclusion was that
at L=−1 on Qwen3-1.7B, additive perturbation is the only intervention class
that reliably reaches the LM head decision distribution.
Returning to the F4 setup with additive replacing scaling and a random seeded direction: shadow vs. active avg_raw_div = 0.6732 vs 0.6728 (Δ = +0.0004 across 5 seeds). Per-seed, the active runs split — some flipped a few tokens and improved divergence, others flipped tokens and worsened it. The expected value of a random direction was zero, and that is what the data showed. This was the first direct evidence that the controller's actuator was now reaching the model (token_match_rate ≠ 1.000 on most seeds) but its effect was undirected.
Two follow-up experiments replaced the random additive direction with a drift-
opposing one. The implementation maintains a reference hidden state (either an
EMA of recent hidden states or a frozen anchor from the first N clean
tokens), computes drift = h_current − h_reference at each step,
normalizes, and injects −β·drift_direction as the corrective additive
delta. The intuition matches a textbook proportional controller pulling toward a
setpoint.
EMA reference (F23): avg_raw_div 0.6547 vs shadow 0.6732 — a 2.74% improvement, 0.106 shadow-stdevs. Anchor reference (F24): 0.6537, an additional 0.001 improvement. Both improvements are concentrated in a single seed where the controller's first intervention coincided with a token-flip that diverted the trajectory into a more coherent basin (the F25 phenomenon, characterized below). Other seeds were unchanged or slightly worse.
Per-step trace analysis of F23/F24 revealed that the small aggregate improvements were not closed-loop control. On one of the F24 seeds, the controller's first intervention at step 7 flipped a single token (a leading whitespace became "Use"), and the subsequent generation entered an entirely different output basin — a coherent recipe with explicit ingredients ("100g flour, 100g water, 100g…") in place of shadow's degenerate numbered-list stub ("1. Prepare 2. Mix 3. Let…"). On other seeds with already-coherent baselines, the controller fired repeatedly, flipped no tokens, and active output was character-identical to shadow.
This pattern is reproducible: small final-layer additive perturbations can flip individual tokens at branchpoints where the LM head's top-2 logit margin is small, and the resulting trajectory enters a different attractor. The visible "controller helps" effect is one such hijack landing in a better basin. The visible "controller does nothing" effect is the controller firing in regions where its perturbation is smaller than the local logit margin. The controller is not stabilizing trajectories. It is occasionally redirecting them at branchpoints, and whether the new basin is better or worse is a property of the basin, not the controller.
A first attempt to fix F25 by moving the actuation layer one step back (act_layer
= −2) produced what looked like a clean 1.7σ improvement on aggregate avg_raw_div.
Inspection of intervention_applied counts in events.jsonl showed the
controller fired exactly 1 time across 5 active runs. The "improvement" was
entirely an artifact of moving measure_layer from −1 to −2 in the
same step (the original implementation required them to be equal). Divergence at
L=−2 is naturally lower than at L=−1, and the comparison was apples-to-oranges.
The controller was a spectator.
Resolving this required a small code change: decouple measure_layer
from act_layer in the runtime engine, with separate forward hooks for
measurement (capture-only) and actuation (modify). Once decoupled, the layer-move
hypothesis could be tested honestly.
With measure_layer=−1 held fixed (identical signal to shadow) and
act_layer ∈ {−1, −2, −3}, on the same 5-seed sourdough suite:
| Cell | avg_raw_div | Δ vs shadow | interventions fired |
|---|---|---|---|
| SHADOW | 0.6732 ± 0.174 | — | 0 |
| ACT_L−1 | 0.6537 | +0.020 (+0.11σ) | 19 |
| ACT_L−2 | 0.7057 | −0.033 (−0.19σ) | 22 |
| ACT_L−3 | 0.6918 | −0.019 (−0.11σ) | 25 |
Acting one layer back makes things worse, not better. Acting two layers back is also worse. The controller is firing at similar rates in all three configurations (~20 interventions per 5-seed suite), so this is not a "intervention never fires" artifact. The interpretation is that perturbations earlier in the residual stream cascade through subsequent attention and MLP layers, accumulating drift rather than steering it. This closes out the layer-placement direction of controller redesign.
The cumulative result of the controller arc. Across nine findings (F4, F17, F18, F21, F22, F23, F24, F26, F27), every controller variant either (a) does nothing because the actuator is absorbed by an intervening norm layer, (b) does nothing because the perturbation is smaller than local logit margins, (c) opportunistically hijacks one branchpoint per seed and otherwise does nothing, or (d) is destabilizing rather than stabilizing.
The simplest explanation that fits all of these is that the divergence signal we are measuring does not measure what closed-loop stability control needs it to measure. Inspection of the highest-divergence steps in observe runs (replicated from v1's "divergence spikes at structural boundaries" finding, which we confirm) shows that the signal spikes at word-starts, punctuation, the transitions between numbered list items, and the boundaries between semantic units. These are normal features of well-formed prose, not symptoms of trajectory destabilization. Closed-loop intervention on this signal is therefore a controller fighting prose structure.
The instrument is sound. The trigger signal is not what it was framed as. The naive controller redesign space (varying intervention class, magnitude, direction, layer, reference rule) is exhausted within the observe-run regime we can support on commodity hardware.
The controller arc was a falsified hypothesis. Its runs still contain a narrower observation worth preserving: additive interventions can change generated tokens and redirect continued trajectories, whereas final-layer scaling can be absorbed by RMSNorm. We historically called this branchpoint hijacking. The July audit separates that observed trajectory effect from the stronger, reopened claim that local same-context flippability is predictable.
A small number of additive perturbations applied during a stress run can produce one or more single-token flips at positions where the LM head's argmax is margin-vulnerable. Once a single token has been flipped, subsequent tokens are drawn from a different conditional distribution, and the trajectory enters a different attractor with its own local dynamics. The visible footprint of a successful hijack on the F23/F24 sourdough runs is one token of difference between shadow and active output, followed by a continuation that is structurally and semantically distinct.
A scope check on TinyLlama-1.1B (Llama family architecture, vs. Qwen-family for
Qwen3-1.7B) using the same configuration confirmed that the flip mechanism
generalizes. On 3 of 5 seeds with non-degenerate generation, the active cell
flipped tokens vs shadow with the controller firing 9–10 times per seed. However,
on TinyLlama every hijacked seed landed in a worse basin: avg_raw_div was higher
in active than shadow on all three (e.g., seed 2: 1.006 → 1.066, output became
"Pleaseincludeingredients,measurements,bakingtime…" with whitespace
tokens dropped). This contrasts with Qwen3 sourdough seed 2, where the same
mechanism landed the trajectory in a coherent recipe basin.
The corrected conclusion is narrower: continued additive interventions changed outputs in both tested model families, and the consequence differed. A calibrated independent local-fork suite is still required before calling the local mechanism architecture-general.
The intended question remains important: given a clean decision context, can we predict whether a small intervention at that exact context flips the next token? The historical analysis attempted to answer it using paired controller shadow/active runs and labeled a row positive whenever the two continued trajectories emitted different tokens at the same ordinal step.
Two methodology defects matter. An early version used intervention-downstream active features, a direct label leak. Switching to shadow features fixed that leak but not the label: once the active run emits a different token, every later active forward is conditioned on a different context. Later token differences are cascading consequences of an earlier flip, not independent answers to the same-context local counterfactual.
The historical shadow-feature analysis produced the following numbers. They are preserved for provenance and as an audit lesson, not accepted as local flippability performance:
| Prompt | Pairs | Step rows | Held-out AUROC |
|---|---|---|---|
| Sourdough (procedural) | 12 | 576 | 0.82 |
| Water cycle (descriptive) | 5 | 240 | 0.86 |
Both historical slices exceeded 0.80, but those thresholds cannot close Q1 because the target variable did not match Q1's causal event. The reported feature associations were:
| Feature | Sourdough sign | Water cycle sign | Mechanistic interpretation |
|---|---|---|---|
step_idx | + (later) | + (later, AUROC 0.80 alone) | Baseline divergence accumulates with sequence length; later tokens have more chances to be at margin. |
spectral.permutation_change | + (high) | − (low) | Time-axis spectral structure; sign flips between procedural and descriptive prompts. |
layer_stiffness.−1.elasticity | + (low velocity) | + | Slow-moving final-layer activations are at branchpoints more often. |
svd.top1_energy_frac | + (concentrated) | — | Trajectory concentrated in dominant direction — local instability. |
spectral.total_power | − (high) | − | High-energy steps are less flippable. |
These associations are hypotheses only. In particular,
step_idx may predict the increasing chance that two already
divergent trajectories remain different, rather than local causal
vulnerability. Prompt-class sign differences likewise require replication
with independent local forks.
Corrected status. F31 and Q1 are reopened. The active
branchpoints runner now creates an independent clean and
perturbed one-step fork at every verified shared context, pairs sampling
randomness, discards the perturbed fork, and advances only clean
generation. The replacement analyzer accepts only those artifacts, splits
by whole run, fits preprocessing on training runs only, and reports
precision/recall beside AUROC. F32 records the subsequent target-model
recalibration.
The repaired M1R suite ran independent layer-27 one-step forks on Qwen3-1.7B across three seeds, two prompts, and relative additive magnitudes 0, 0.15, and 0.30. All zero controls were exact no-ops. At magnitude 0.30, sourdough produced 18 argmax flips in 141 decisions and water cycle produced 14 in 141.
| Prompt | Unique held-out splits | AUROC mean | Range | Precision | Recall |
|---|---|---|---|---|---|
| Sourdough | 3 | 0.831 | 0.773–0.886 | 0.233 | 0.178 |
| Water cycle | 3 | 0.886 | 0.855–0.913 | 0.689 | 0.444 |
Both prompt slices clear Q1's prespecified AUROC threshold. The recurring clean predictors are top-1 margin, logit entropy, layer stiffness, and spectral structure. This is a scoped positive result, not a universal or controller-ready detector: fixed-threshold precision/recall remain uneven and each prompt has only three unique leave-one-run-out assignments.
MHR ran 24 matched layer-13 exposure/recovery experiments using the same prompts and seeds at magnitudes 0, 0.10, 0.20, and 0.30. Every recovery pair matched and every zero control remained exactly zero. At magnitude 0.30, propagated endpoint distance ranged from 0.000987 to 0.940433 on sourdough and from 0.887970 to 1.305816 on water cycle. The same nominal intervention can therefore be nearly absorbed or trigger an order-one basin switch depending on the sampled trajectory.
Absolute initial, propagated, and residual distances are the primary evidence. Endpoint calibration did not identify the downstream layer where amplification occurred; F34 records the subsequent Q2 trace. No bounded-cascade controller layer was established by F33.
Q2 ran 280 Qwen3-1.7B cells across five seeds, two prompts, layers 6/13/20/27, additive magnitudes 0/0.10/0.20/0.30, and scaling factors 1.0/0.5/1.5. At each of 4,200 decisions, independent clean and perturbed one-step forks shared the same cache and RNG. The instrument captured every downstream transformer block and final RMSNorm, discarded the perturbed fork, and advanced only clean generation. All 80 identity-control runs were exact no-ops.
| Additive magnitude 0.30 | L6 | L13 | L20 | L27 |
|---|---|---|---|---|
| Peak raw-L2 amplification | 8.662× | 4.214× | 1.684× | 1.000× |
| Final relative delta | 0.0699 | 0.0975 | 0.1929 | 0.2800 |
| Mean logit KL | 0.0347 | 0.0227 | 0.0629 | 0.0861 |
| Argmax flips | 10/150 | 9/150 | 12/150 | 18/150 |
Earlier injections grew more in absolute L2 because residual-stream scale grows through the network, while their relative and angular disturbance usually contracted. Historical F27's “earlier layer = worse” is therefore not a universal local law; repeated intervention and context cascade remain plausible causes of the controller-era result.
Final-layer scale 0.5 was removed exactly by RMSNorm. Scale 1.5 was reduced to relative delta 7.54e-8 and mean KL 1.33e-8, with zero argmax flips. Directional additive magnitude 0.30 survived with relative delta 0.2800 and 18/150 flips. Scaling direction and prompt also mattered materially. Layer 13 had the lowest aggregate mean KL in every non-control setting, but did not uniformly minimize prompt-slice flips. Q2 thus has a scoped mechanistic answer, not a controller-ready bounded-layer result.
M3R rebuilt the historical controller-dependent basin proposal as
causal-branch-continuation-v1. It ran 60 Qwen3-1.7B cells:
ten prompts spanning factual, procedural, creative, reasoning, and code;
three seeds; and layer-27 relative additive magnitudes 0 and 0.30. Pass one
mapped 2,820 independent same-context local forks and selected at most one
early, middle, and late episode without reading continuation outcomes. Pass
two reset and exactly replayed the clean path, then continued clean and
perturbed branches for 48 tokens with the intervention disabled and paired
sampling randomness.
All 60 run hashes validated, and all 90 selected identity-control episodes were exact ties. The active runs contained 69 selected sampled-token flips and 21 no-flips. Under deterministic prompt-specific rubrics whose scorer did not receive branch identity, sampled flips produced 31 improvements, 11 degradations, and 27 ties. These labels mean higher or lower declared task-rubric score, not universal semantic quality.
A predictor restricted to clean pre-flip/context features used 42 non-tie sampled-flip rows and held out each complete prompt. Five prompts contained both labels; mean held-out AUROC was 0.571 (range 0.000–0.889), below the prespecified 0.70 gate. The procedural slice produced 12 improve, 0 degrade, and 6 tie outcomes across two prompts and three seeds, but both procedural holdouts lacked negative examples. That slice is a replication target, not a replacement success criterion; F36 later tested it on fresh prompts and it did not replicate. Q3 is therefore a measured negative and the controller remains paused.
M3R-2 froze its complete design before seeing replication data: 15 prompts disjoint from M3R, three seeds, exact magnitude-zero controls, magnitude 0.30 at layer 27, 64 mapped tokens, 96 intervention-free continuation tokens, and at most five outcome-blind episodes. The predictor was limited to 23 declared clean pre-flip/context fields plus prompt class, with L2 regularization 0.05, whole-prompt holdout, a minimum of five valid splits, and the unchanged 0.70 AUROC gate. Code outputs were parsed statically and never executed.
All 90 run hashes and all 45 identity controls validated. The protocol mapped 5,670 local branchpoints and retained 217 active episodes: 195 sampled-token flips and 22 no-flips. Sampled flips produced 67 improve, 49 degrade, and 79 tie outcomes under the task-specific rubrics. There were 116 eligible binary rows and 13 valid held-out prompts; mean held-out AUROC was 0.596 (range 0.000–1.000), below 0.70. Two prompts lacking degradation rows remained visible as invalid splits.
The original M3R procedural slice was 12 improve, 0 degrade, and 6 tie. Three fresh procedural prompts produced 16 improve, 19 degrade, and 9 tie: the procedural pattern did not replicate. M3R-2 therefore strengthens the measured-negative Q3 conclusion without changing the goalpost. These labels remain operational rubric outcomes, not universal semantic quality. Controller work remains paused.
Following the controller arc's falsification, the project's center of gravity shifts from "build a closed-loop controller" to "map the geometry of trajectory sensitivity, persistence, and branchpoint behavior in Qwen3-1.7B." This is a deliberately scoped agenda — single model, mechanistic questions, explicit stop conditions per question — designed to keep the work falsifiable and finite.
| Question | Stop condition | Status | |
|---|---|---|---|
| Q1 | Branchpoint geometry: when are tokens flippable? | Independent same-context forks; precision ≥0.7 and recall ≥0.5, or whole-run held-out AUROC ≥0.80, across ≥2 prompts. | Scoped positive at magnitude 0.30 (F32) |
| Q2 | Perturbation propagation: how does an injected delta evolve through the residual stream? | Per-layer propagation curves that mechanistically explain F27. | Scoped mechanistic answer (F34); controller criterion not met |
| Q3 | Basin structure: when does a flip improve vs. degrade output? | Pre-flip generation feature predicts improve-vs-degrade with AUROC ≥ 0.7 on Qwen3-1.7B. | Replicated negative: M3R-2 mean prompt-held-out AUROC 0.596 (F36) |
The closed-loop controller research direction is paused, not abandoned. We explicitly define the conditions under which it would be reasonable to reopen: the controller returns to active investigation when any two of the following become true.
act_layer instead of the current trial-and-error).If two of these land, a controller redesign experiment using the new trigger, layer, and gate becomes worth running. If none land within the mapping program, the observed perturbation findings (F25/F29) plus any newly validated Q1–Q3 results stand on their own as an interpretability contribution and the controller stays paused.
What does not justify reopening the controller is more tuning of the
existing design space. The combinations
(scaling | additive-random | additive-EMA-opposing | additive-anchor-opposing)
× (L=−1, L=−2, L=−3) × (magnitude ∈ {0.3, 0.6, 0.8, 1.2})
have all been tested and are recorded in §10 and the
RESEARCH_CONTROLLER.md archive. Reopening this space without new
information from the mapping work would be blind tuning.
Observer is designed to produce artifacts that can support publishable claims, not just exploratory analysis. The compute environment for the reported experiments is a single NVIDIA H200 GPU via RunPod.
Every run produces a config hash (SHA-256 of the exact final sorted-key configuration) and authoritative loaded-model identity. Protocols that claim matched contexts also record cache fingerprints and sequence lengths. The full configuration, trajectory data, protocol-validity fields, and computed metrics are written to structured artifacts.
The included REPRODUCIBILITY.md specifies a reporting checklist for public claims:
pin commit hash in every figure caption; report model key, backend, seed, and intervention settings;
run at least 3 seeds per comparison; report mean + confidence interval, not best run;
publish raw results.json used for plots. This is the standard that is routinely
absent from published intervention work.
CI installs the package in editable mode, runs the pytest contract suite,
compiles the active package/scripts, and lints with Ruff. The tests cover
installation, event semantics, provenance, semantic layer resolution,
same-context local forks, matched recovery, run-grouped analysis, and
synthetic diagnostic validation. The repository keeps two top-level research documents:
RESEARCH.md tracks the active mapping program and is the entry point
for any new session, and RESEARCH_CONTROLLER.md archives the
completed controller arc with full F-numbered findings so subsequent agents can
cite established facts without re-deriving them. docs/RESEARCH_WORKFLOW.md
documents the experiment handoff protocol both documents follow.
Several limitations constrain v2's claims, in addition to the central limitation documented in §10 (the divergence trigger does not measure what closed-loop stability needs it to measure).
Target-model calibration is narrow. F32/F33 cover Qwen3-1.7B, three seeds, two prompts, one branchpoint layer, one hysteresis layer, and the reported magnitudes. F34 adds five seeds, four injection depths, and two intervention families, but still only two prompts. F35 adds ten prompts across five declared classes, but still only three seeds, one model, one active magnitude, and at most three selected episodes per run. F36 adds 15 entirely new prompts and at most five selected episodes, but remains one model, one active magnitude, and three seeds. Cross-model claims remain open.
Prompt-class diversity. The repaired F32 suite reran one procedural and one descriptive prompt with valid local labels, but two prompts are not enough to establish a general prompt-class law.
Sampling. v1 noted that all generation used greedy argmax. v2
added temperature/top-p/top-k sampling support, but a subtle finding emerged: with
matched randomness, torch.multinomial can produce the same drawn
token from slightly different conditional distributions. The repaired
protocols preserve paired randomness to isolate the intervention and report
KL/JS plus argmax flips so distribution shifts remain visible even when the
sampled token is unchanged.
Basin-outcome labels are operational and uneven. F35 rebuilt the causal continuation protocol and used frozen prompt-specific evaluators, but those scores are not universal semantic-quality judgments. Only 42 sampled-flip episodes had a non-tie binary outcome; four prompt holdouts lacked one class, and the palindrome code rubric tied every selected episode. F36's frozen v2 rubrics increased this to 116 binary rows and 13 valid prompts, yet mean holdout AUROC remained 0.596. Its static code contract can parse an extracted function without executing it, but does not certify runtime correctness or general code quality.
Asserted controller weights. The 70/15/10/5 weighting in the v1 composite score (§9) was a design choice, not derived from empirical optimization. In v2 this is moot — the composite-score-driven controller is paused as a class — but if controller research resumes the weighting choice should be re-derived from the new trigger signal rather than carried forward.
Architecture coverage. Layer discovery currently handles Llama/
Qwen-style (model.model.layers), GPT-2/GPT-J
(transformer.h), GPT-NeoX (gpt_neox.layers), and
encoder-decoder (model.decoder.layers). Models tested in this paper:
Qwen3-1.7B (28 layers, primary) and TinyLlama-1.1B-Chat (22 layers, scope check
only). Falcon, Mistral (sliding window attention), Gemma, Phi, and Mamba would
require additional handling and are out of scope for this preprint.
VAR(1) window constraints. With window size 8, the VAR(1) model is fit on 7 transitions in 64-dimensional space. The ridge regularization (λ=0.01) stabilizes the regression, but statistical power of the prediction error signal is limited, particularly in the first few tokens before the window fills. This is part of why F28's "divergence measures prose surprise" framing makes sense: the predictor is fitting short-window local trajectory dynamics, which are legitimately disrupted at semantic-unit boundaries in normal generation.
This project was developed by an independent researcher without formal ML or software engineering training, with no prior programming experience, and without institutional funding. Implementation was carried out through iterative AI-assisted coding workflows over evenings and weekends on rented compute.
We report this as methodological context. The research questions, experimental decisions, and acceptance criteria were set by the author; AI assistants provided implementation support for code generation and revision. All code, runs, and claims were human-reviewed against run artifacts before inclusion in this paper.
v1 listed three planned validation experiments. v2 reports what happened to each.
v1 Experiment A (minimal downstream correlation). Status: partially answered, with a different answer than expected. The intended test was "do divergence statistics differ between correct and incorrect outputs." What we found instead is that divergence reliably spikes at structural boundaries in well-formed prose — paragraph breaks, semantic transitions, numbered list markers, word-starts. Most "high divergence" steps are not associated with incorrect output; they are associated with normal writing. This redirected the research from "use divergence as a hallucination detector" to "characterize what divergence actually measures" (F28).
v1 Experiment B (attractor-basin replication). Status: did not replicate as a controller-property claim. v1's headline result on Qwen2.5-7B (controller aggressiveness selects which incorrect-claim attractor the model lands in) could not be reproduced on Qwen3-1.7B because scaling at L=−1 has zero effect on Qwen3 (F17/F22). The basin-selection phenomenon may still be real on Qwen2.5-7B; we cannot confirm or deny without a re-run with the measurement controls v2 added. We do not currently plan that re-run because the mechanistic finding (F25 branchpoint hijacking) supersedes it as a more general and more measurable phenomenon.
v1 Experiment C (signal baseline comparison). Status: mechanics implemented; semantic comparison pending. Observer now emits hidden velocity, hidden acceleration, logit entropy, and top-1 margin beside local VAR(1) prediction error, temporal spectral, and windowed-SVD signals. A deterministic validator checks the actual probes on constant, linear-drift, oscillatory, and shock trajectories. The historical F31 AUROCs cannot answer which signal predicts local flippability because their labels were cascading; that head-to-head comparison belongs in the repaired calibration suite.
Foundation-first calibration and Q3 replication complete. M1R, MHR, Q2 propagation, M3R causal basin mapping, and M3R-2 prompt-breadth replication are complete on Qwen3-1.7B. The frozen pre-flip predictor missed its Q3 gate twice, and the original procedural direction did not replicate. Any further basin experiment should begin with a new preregistered mechanistic question—not post-hoc feature tuning or controller redesign. No controller rows are accepted as causal branchpoint or basin-outcome labels.
Observer started as an attempt to build a closed-loop stability controller for autoregressive language model generation: detect destabilization in real time, apply proportional damping, observe recovery. The instrument we built does most of what we set out to build — deterministic branchpointing, per-token telemetry, hooked interventions, real-time controller logic — but the central control claim did not survive contact with our own validation experiments. The trigger signal we had been calling "trajectory instability" turned out to measure token-level prose surprise: word-starts, semantic transitions, structural boundaries. A controller built on that signal is, in effect, fighting normal writing.
Reporting the falsification matters. The v1 paper presented closed-loop control as a working contribution, with a striking attractor-selection result on Qwen2.5-7B that, we now suspect, lacked the controls to rule out simpler explanations. The v2 work makes the central claim falsifiable, runs the falsification, and reports it. That is the value of building a research instrument before claiming a research result with it.
What remains is a repaired instrument, observed perturbation effects, and a sharper open question. The active runtime now supports canonical events, authoritative provenance, independent local branchpoint forks, matched exposure/recovery traces, direct diagnostic baselines, and deterministic validation, per-layer propagation capture, exact clean replay, and paired branch continuation with declared outcome rubrics. The historical AUROC predictor is not retained as a positive finding because its target labels were cascading. A repaired local predictor did clear a scoped Q1 AUROC gate, while the Q3 better-versus-worse predictor missed its prompt-held-out gate in M3R and again in the larger M3R-2 replication. The mapping program in §12 records these results with explicit criteria for reopening controller work.
The control theory framing remains intentional, but its meaning has changed. An observer in the control engineering sense estimates internal state from external outputs. The observer here is now best understood as exactly that — a state estimator and characterization tool for transformer trajectories — without the active feedback loop that the v1 framing claimed and v2 falsified. Whether a different trigger signal (one that actually correlates with downstream output failure) could rebuild a working controller is an open question this work has not answered, and is the central question for any v3.
This project was developed by an independent researcher without formal ML or
software engineering training and with no prior programming experience, using
AI-assisted implementation workflows and rented compute. We include this as
methodological context. The decision to invest in falsifying the v1 central
claim — rather than continuing to tune it — was set by the author and
implemented against artifacts that were jointly reviewed before being treated
as evidence. All code, runs, and claims were human-reviewed against generated
artifacts. The complete F-numbered evidence chain (F1–F36) is preserved in
RESEARCH_CONTROLLER.md, RESEARCH.md, and
docs/OBSERVER_FOUNDATIONS.md in the
repository, with full per-experiment run identifiers so any claim in this paper
can be traced to its underlying run artifacts.
Repository: github.com/aeon0199/observer
License: MIT. Cite via CITATION.cff.
Selected references: Nanda et al. (2022) TransformerLens. Wu et al. (2024) pyvene. Zou et al. (2023) Representation Engineering. Li et al. (2023) Inference-Time Intervention. Raj et al. (2023); Huang et al. (2023). Rodriguez et al. (2025) LinEAS (arXiv:2503.10679). Cheng et al. (2025) FASB (arXiv:2508.17621). Grant et al. (2025) (arXiv:2511.04638). Hu et al. (2025) HARP (arXiv:2509.11536). Shapiro et al. (2026) HALT (arXiv:2602.02888). Johnson & Lindenstrauss (1984) Extensions of Lipschitz mappings into a Hilbert space.