OMN-1: architecture and optimizer design

This note describes the OMN-1 implementation and the OMN-Mini-1.5B-Base v2 checkpoint discussed in our internal post-pretraining analysis. The record reports approximately 193 billion pretraining tokens. That historical run is distinct from later code revisions and forthcoming model releases. No new training or benchmark run was performed for this note.

OMN is Aethr’s architecture and training research programme. OMN-1 uses established Transformer building blocks; its identity lies in the implementation, configuration and training recipe investigated together. This is not a claim that attention, RMSNorm, SwiGLU or Muon originated at Aethr.

For hidden state x, a block computes n = RMSNorm(x), then y = x + Attention(n) + FFN(n). Both branches receive the same normalized input, unlike a sequential block where the feed-forward branch follows the attention residual update.

The reference configuration has 28 blocks, width 2,048, intermediate width 5,504 and head dimension 128. Grouped-query attention uses 16 query heads and 4 key/value heads. The feed-forward network uses SwiGLU. Input embeddings and the output vocabulary projection share weights over 128,256 tokens.

Parallel residual describes dependencies, not a measured latency improvement or a guarantee that the PyTorch implementation launches both branches concurrently.

Query and key vectors receive learned RMS normalization before RoPE and attention. The gains are trainable feature-wise parameters shared across heads in the reference implementation, not independent scalar head temperatures. They do not establish a fixed logit ceiling or freedom from hallucination.

RoPE theta is 500,000 and maximum configured positions are 32,768. FP32 rotary calculations address numerical precision. Neither that choice nor the configured limit proves reliable 32K retrieval.

Selected non-embedding matrix parameters use Muon; embeddings, normalization parameters and remaining parameters use AdamW. Embeddings are matrices too, so the split is not simply “2D versus 1D.”

Muon applies a finite Newton–Schulz iteration to approximately orthogonalize a momentum-derived update before applying it to a weight matrix. Adding such an update does not keep the weight itself orthogonal. Orthogonal initialization likewise does not impose a persistent Stiefel-manifold constraint.

We therefore describe the recipe as Muon plus AdamW. It does not imply full representational rank, guaranteed generalization or protection against training-data extraction.

Architectural efficiency and reasoning gains require matched comparisons controlling data, compute, scale and evaluation settings. Successful training alone cannot establish which component caused a capability.

Industrial specialization is a development direction for models built on OMN. The text architecture itself is not a sensor model, simulator or industrial controller; those functions require appropriate tools, training and evaluation.