Aethr Research / Architecture
Our foundation-model architecture for studying how intelligence is built.
OMN is AETHR's proprietary foundation-model architecture and the experimental ground for our model research. It lets us question the defaults, rebuild the pieces that matter, and turn new architectural ideas into the foundations of future AETHR models.
What OMN means
OMN — Orthogonal Manifold Network
OMN is AETHR's foundation-model architecture built around a simple idea: different parts of a model have different structure, and the architecture should respect that structure rather than treating the entire network as one homogeneous object.
Its design brings architectural choices, attention stabilization, and parameter geometry into the research loop itself — so the model can be trained, measured, and evolved from the foundation upward.
The research thesis
Structure before scale.
Before asking what happens when a model gets larger, OMN asks whether the parameters already inside the model are being represented, optimized, and stabilized in the right way.
Modern language models have demonstrated that scale is extraordinarily powerful. But parameter count alone does not determine how effectively a model learns. The geometry of its weights, the dynamics of attention, the path information takes through the network, and the optimizer acting on each parameter all shape what training can produce.
OMN began from the premise that these choices should remain part of the research. Rather than treating the Transformer as a fixed recipe and changing only its size, OMN revisits several of the assumptions underneath that recipe — how different classes of parameters should be optimized, how attention regulates its own scale, and how information moves through each block.
Better use of structure before simply adding more parameters.
OMN treats parameter efficiency as a research question rather than an assumption. The aim is to understand whether a model can make better use of a given parameter budget when its optimization strategy follows the structure of its parameters and its attention dynamics are allowed to adapt during training.
This is why OMN's defining ideas are not isolated features. They follow from one thesis: the internal structure of the model deserves as much attention as its external scale.
Why Orthogonal Manifold Network?
The name reflects one of the central ideas behind OMN: large parts of a Transformer are matrix-valued objects, and their optimization need not be treated as ordinary unconstrained vector optimization.
In OMN, the major 2D transformer weight matrices follow the Muon optimization path, where Newton–Schulz iterations produce orthogonalized updates informed by matrix geometry. Vector-like parameters remain on the AdamW path in ordinary Euclidean parameter space.
This separation gives OMN its dual-manifold optimization philosophy: different parameter structures are allowed to follow different geometric assumptions during learning.
Built to make the foundation itself more learnable, stable, and open to experimentation.
01
Geometry should inform optimization
Not every parameter is the same kind of object.
A transformer contains large matrices, vectors, gains, normalization parameters, and embeddings. OMN does not assume that one optimizer geometry is necessarily appropriate for all of them. Its dual-manifold training system routes 2D transformer matrices through Muon while vector-like and shared parameters remain on AdamW.
Different structures. Different optimization paths.
02
Attention should be able to adapt
Attention should find its own scale.
OMN places QK-Norm directly inside the attention mechanism, giving query and key representations learned normalization behaviour during training. The first checkpoint provided unusually strong evidence that this mechanism became structurally important: when the learned gains were clamped back toward their initial regime, model perplexity deteriorated dramatically.
Stability is learned as part of the model, not treated only as an external setting.
03
The information path should stay legible
Simpler paths make better experiments.
OMN uses a parallel residual structure in which attention and the feed-forward network operate from a shared normalized representation. This reduces unnecessary architectural branching and gives the research programme a cleaner surface for asking causal questions about what individual components are doing.
An architecture should be powerful enough to learn and clear enough to investigate.
04
Capacity must become learned capability
Configuration is not capability.
OMN can be configured for long contexts, but the first checkpoint demonstrated why architecture alone is not enough. Without sufficient exposure to genuinely coherent long sequences during training, configured context does not automatically become usable model capability. That finding now informs how future OMN-based models are intended to be trained.
Architectural capacity matters only when training teaches the model how to use it.
How OMN is built
Why OMN
The foundation should remain part of the research.
Most model development begins after the architecture has already been chosen. OMN gives AETHR a foundation we can investigate directly — from optimization and attention behaviour to specialization and long-context training.
The purpose is not novelty for its own sake. OMN exists so architectural decisions remain open to measurement, ablation, and revision as our understanding improves.
Design principles
A shared residual path
OMN uses a parallel residual design in which attention and feed-forward computation operate from a shared normalized representation. The goal is a simpler information path and a cleaner experimental surface for architectural study.
Attention that stabilizes itself
QK-Norm is integrated directly into the attention path, allowing learned normalization behaviour to shape how attention logits evolve during training.
Optimization matched to parameter geometry
OMN separates matrix-valued transformer weights from vector-like parameters, allowing different optimizers to be applied where their geometry differs.
Architecture built for experimentation
The implementation keeps core components explicit and measurable so architectural changes can be evaluated causally rather than treated as opaque configuration choices.
Reference block
One OMN decoder block: a shared normalized representation feeds parallel attention and feed-forward branches, recombined into the residual stream and stacked into the next layer.
Optimization
One model. Two parameter geometries.
OMN separates its parameters according to the geometry of the objects being optimized. Vector-like parameters follow the familiar Euclidean optimization path through AdamW, while 2D transformer weight matrices are assigned to Muon, where updates are orthogonalized through Newton–Schulz iterations and exploit the geometry of matrix-valued parameters.
This gives OMN a dual-manifold optimization system rather than a single global update rule. The matrix path is designed around orthogonal, Stiefel-like geometry; the vector path remains Euclidean. In the first OMN checkpoint, roughly 82.5% of parameters followed the Muon matrix path, while the remaining parameters — including vector-like tensors and the tied embedding/LM-head path — remained on AdamW.
The idea is simple: parameters with fundamentally different structure should not necessarily be optimized as if they were the same object.
~82.5% of parameters → Muon matrix path · ~17.5% → AdamW path
Read the optimization designDesigned as a system, not a collection of defaults.
OMN combines familiar transformer components with a deliberately different training and architectural structure. The emphasis is on how those pieces interact: parallel residual computation, learned attention normalization, grouped-query attention, and optimizer assignment based on parameter geometry.
Standard approach
Single optimizer across most parameters
OMN
Matrix weights and vector-like parameters follow different optimization paths
Standard approach
Attention scaling largely fixed by configuration
OMN
QK normalization gains are learned and become part of the model's internal operating regime
Standard approach
Architecture treated primarily as a configuration choice
OMN
Architecture is treated as an experimental variable to be measured and revised
Empirical findings
What the model taught us.
QK-Norm became load-bearing.
When learned QK-Norm gains were clamped toward their initialization, perplexity degraded dramatically. The experiment showed that the normalization behaviour was not decorative — the model had incorporated it deeply into how attention operated.
Read the ablationSmall singular directions were not simply dead weight.
Post-training spectral analysis showed extreme conditioning in some attention matrices. Causal truncation experiments indicated that some small singular directions still carried functional information rather than representing trivial collapse.
Read the spectral analysisConfigured context was not demonstrated context.
Although the architecture was configured for long context, the data pipeline limited exposure to genuinely coherent long documents. This separated theoretical context capacity from capability actually learned during pretraining.
Read the context evaluationArchitecture is only the beginning.
OMN is the foundation beneath AETHR's model research. The lessons from the first checkpoint — what worked, what failed, and what became unexpectedly important — now inform the development of Darwin, AETHR's emerging foundation-model family.
- OMN · Architecture
- Experimentation · Training + Ablation
- Learnings · Architecture + Optimization
- Darwin · Model family
Technical depth