Aethr Research / Evaluation
Evaluations & Technical Reports
Research claims should be traceable to evidence.
AETHR uses evaluations, ablations, implementation audits, and technical reports to separate what a model is designed to do from what it has actually demonstrated. This page is the evidence layer of our research programme: the methods we use to inspect our models, the reports we publish, the limitations we retain, and the standards we expect a result to meet before it becomes a public claim.
Evidence discipline
We separate what was built from what was demonstrated.
A model architecture can contain a mechanism without proving that the mechanism is beneficial. A training configuration can specify a capability without proving that the trained checkpoint learned it. A historical experiment can contain an interesting result without having sufficient provenance to support a public claim. AETHR therefore treats evidence status as part of the research itself — design intent is not the same as evidence.
Implemented
Established directly from code or configuration. Example: OMN's parallel residual topology or optimizer routing.
Observed
Measured directly from an artifact or checkpoint. Example: parameter count or learned Q/K gain values.
Reported
Recorded in historical analysis but not independently reproduced from preserved raw outputs.
Interpreted
A conclusion or hypothesis derived from measurements.
Uncertain
An unresolved question where existing evidence is insufficient.
Planned
A future experiment, evaluation, or qualification activity.
AETHR does not collapse all six into “result.”
Evaluation surface
A model is more than a benchmark score.
01
Architecture
02
Optimization
03
Training system
04
Model behaviour
05
Interventions & ablations
06
Deployment reliability
Publications
Current research reports
Every entry below is a real, live publication — except where explicitly marked in preparation, in which case it is not clickable. No fabricated dates, no placeholder reports.
Method
From experiment to public claim
A result should not become a headline merely because a script produced a number. Where possible, AETHR preserves the full provenance chain behind every published claim:
- 01Question
- 02Implementation
- 03Controlled experiment
- 04Raw artifact
- 05Reproduction / audit
- 06Interpretation
- 07Public claim
Concretely: code revision, model and checkpoint identity, tokenizer identity, evaluation dataset revision, raw outputs, configuration, random seed, metric definition, plotting source, and known limitations.
Standard
What a technical result should carry with it
- Model identity
- Checkpoint hash
- Source identity
- Revision / fingerprint
- Data identity
- Dataset and revision
- Evaluation protocol
- Prompting / scoring configuration
- Raw results
- Preserved artifact
- Metric definition
- Exact calculation
- Limitations
- Known boundaries
- Status
- Implemented / observed / reproduced
Not every historical OMN experiment meets this standard yet. Some early experiments predate it and are being requalified before being treated as publication-grade evidence. That honesty is intentional.
Maturity
Not every interesting result belongs on a benchmark chart.
The OMN audit uncovered results with absent output files, divergent baselines across evaluation panels, conflicting spectral summaries, incomplete benchmark provenance, and configured capability confused with demonstrated capability. Those results remain internal until they can be regenerated or properly qualified.
The purpose of evaluation infrastructure is not to protect impressive numbers. It is to make incorrect conclusions harder to preserve. AETHR would rather narrow a claim than preserve a number we can no longer defend.
Future direction
Where the evaluation programme goes next
None of the following is claimed as complete. Each is an active direction:
Architecture ablations
Matched experiments
Optimizer routing, QK-Norm, block topology, normalization — each against controlled baselines that do not yet exist.
Scaling
Across Darwin scales
How OMN-derived principles behave across Darwin Type-S and Type-M.
Long-context qualification
Separated measurements
Configured context, training exposure, retrieval, and generation stability — measured independently, never conflated.
Specialization
Domain models
Evaluated against base models, domain baselines, retrieval/tool systems, and deterministic engineering checks.
Agentic / tool use
Bounded behavior
Tool correctness, argument correctness, permission boundaries, abstention, recovery.
Industrial deployment
Consequential evaluation
Evidence traceability, physical consistency, uncertainty, false-confidence rate, unsafe recommendation rate, human escalation.
Systems
In consequential systems, correctness cannot come from the model alone.
For future Industrial Intelligence deployments, some questions should be answered by deterministic engineering or validated software rather than generative inference. The intended architecture: the model reasons, interprets, and proposes; tools calculate, retrieve, and verify; policies bound allowed actions; the human retains authority. Evaluation must therefore assess the complete system, not only the language model.
Honesty
What this page does not imply
AETHR does not currently claim:
- exhaustive safety evaluation
- frontier-model certification
- complete benchmark coverage
- verified robustness across every deployment environment
- production qualification for autonomous industrial control
- universal reproducibility of every historical OMN experiment
This page documents the evaluation standards AETHR is establishing and the evidence that currently meets them.