OMN-1: interpreting spectra through ablation

For W = UΣVᵀ, singular values describe the strength of linear action along different directions. A large condition number indicates a wide ratio between largest and smallest singular values. It does not, by itself, measure reasoning quality or identify the effect on predictions.

The internal audit reports inspection of 196 weight matrices in OMN-Mini-1.5B-Base v2. Attention projections showed wider singular-value ranges than feed-forward projections. These are weight measurements, not direct measurements of hidden-state rank.

The intervention selected layers.27.attn.o_proj, with 2,048 singular directions. Rank-K approximations retained the largest K singular values, replacing the original matrix before evaluation.

The record reports baseline perplexity of 6.11. Keeping 1,024 directions retained approximately 92.8% of squared-singular-value energy and produced perplexity 6.48. Keeping 512 directions produced 7.02. These are historical recorded values, not experiments rerun for this page.

Removing the selected tail as a group affected predictions on the evaluation sequences. A small share of weight energy was not equivalent to no functional contribution.

The intervention does not show that every removed direction was individually necessary, all layers have full effective rank, or the architecture is immune to representation collapse. It does not demonstrate a reasoning advantage over another model.

This public summary concerns one checkpoint, one projection and the documented evaluation setup. It does not supply independent replication or confidence intervals. Dataset composition, numerical reconstruction and target selection matter.

Further study should include multiple layers and checkpoints, reconstruction controls and task-level evaluation alongside perplexity.

A separate internal experiment reports perplexity changing from 12.97 to 692.08 after clamping learned query/key normalization gains to 1.0. Its baseline belongs to that experiment and must not be conflated with the truncation baseline.

This shows sensitivity to the learned gains in that checkpoint. It does not establish why the gains developed, identify an optimal attention temperature or predict the performance of a separately trained model with different normalization.

Pair structural diagnostics with interventions and downstream evaluation. Neither attractive spectra nor extreme condition numbers substitute for testing what a model does.

These records guide further research; they are not mathematical guarantees of reasoning, privacy or production reliability.