OMN
Model architecture and training research
OMN (Orthogonal Manifold Network) is Aethr Intelligence’s model architecture and training research programme. OMN-1 is a dense, decoder-only Transformer implementation combining parallel residual blocks, grouped-query attention, query/key RMSNorm, SwiGLU and rotary position embeddings with a Muon-plus-AdamW training recipe.
The name does not imply that model weights are constrained to an orthogonal manifold: Muon approximately orthogonalizes selected optimizer updates. The programme builds on established methods and studies their interaction. Efficiency, reasoning and specialization are research goals; comparative superiority has not been established.
Representation Health Across Layers
How do weight spectra and functional behaviour change during training? We examine diagnostics alongside targeted ablations without equating matrix rank with intelligence.
Parallel Residual Design
How does a shared normalized input for attention and feed-forward branches affect training and performance under matched experimental conditions?
Attention and Context Evaluation
How do query/key normalization and rotary embeddings affect attention behaviour and retrieval as sequence length increases? Configured context must be tested, not assumed.
Muon + AdamW
OMN-1 applies Muon to selected two-dimensional weight matrices and AdamW to embeddings and normalization parameters. Newton–Schulz iterations approximately orthogonalize momentum-derived updates, not the model weights. This does not constrain weights to a Stiefel manifold.
Parallel residual blocks
Attention and the SwiGLU feed-forward branch receive the same RMS-normalized input. Their outputs are added to the residual stream. Parallel describes the dependency graph; simultaneous hardware execution and speedups require measurement.
Query/key normalization
The reference implementation applies RMSNorm to query and key vectors and uses grouped-query attention. Learned feature-wise gains are shared across heads, rather than independently learned per-head scalars. Normalization alone does not guarantee bounded logits or stable attention.
Spectra and functional ablations
Internal analysis examined weight-matrix spectra and a targeted truncation experiment. Removing singular directions from one attention output matrix increased perplexity on the recorded evaluation. This motivates broader tests; it does not prove full-rank representations or that every parameter is necessary.
Rotary position embeddings
The reference configuration uses rotary position embeddings with a base of 500,000 and a maximum of 32,768 positions. A configured limit is not a demonstrated retrieval capability. Historical long-context tests showed limitations that require further training and evaluation.
Implementation notes and current limitations.
The historical prototype and its analyses are not a production model release. Read the architectural definition, recorded ablations and context limitations in our technical notes.