Token Superposition Training: Nous Research Speeds LLM Pre-Training 2.5x in 2026

In a breakthrough that promises to reshape the economics of large language model development, Nous Research has released a new pre-training method called Token Superposition Training (TST). The technique can reduce wall-clock training time by up to 2.5 times across models ranging from 270 million to 10 billion parameters, all without changing the underlying architecture, tokenizer, optimizer, or inference-time behavior.

How Token Superposition Training Works

According to a technical report published by Nous Research and covered by MarkTechPost, TST operates in two distinct phases. In Phase 1, the method averages contiguous token embeddings into 'bags,' effectively compressing the sequence length and allowing the model to process far more training data in the same wall-clock time. In Phase 2, the model reverts to standard next-token prediction, fine-tuning its representations on the full, uncompressed data.

The approach was validated at four scales: 270 million, 600 million, 3 billion dense, and a 10 billion-parameter mixture-of-experts (MoE) architecture. At each scale, the models trained with TST matched or exceeded the performance of models trained with conventional methods, while requiring significantly less time.

Superposition Principles Behind the Method

The concept of superposition in neural networks has been gaining traction in academic research. A 2019 paper on arXiv, titled "Superposition of many models into one," explored how a single neural network could host multiple, distinct computational pathways simultaneously. That foundational work laid the theoretical groundwork for techniques like TST, which effectively superimposes multiple token representations into compressed 'bags' during early training.

More recently, a NeurIPS 2025 oral paper titled "Superposition Yields Robust Neural Scaling," authored by Yizhou Liu, Ziming Liu, and Jeff Gore, demonstrated that superposition principles can explain and improve neural scaling laws in LLMs. The paper, available on OpenReview, argues that superposition allows models to learn more robust representations per parameter, directly supporting the empirical results Nous Research has now achieved.

"The scaling laws we observed in our experiments align closely with the theoretical predictions from the superposition literature," a Nous Research spokesperson told reporters. "Token Superposition Training is essentially a practical implementation of those theoretical insights, optimized for production-grade model training."

The method does not modify the model architecture itself. Instead, it changes only the data processing pipeline during pre-training. This means that any existing LLM architecture—dense or MoE—can adopt TST without requiring architectural redesign or retraining of downstream components.

Impact on LLM Pre-Training Speed and Efficiency

The implications of Token Superposition Training for AI model acceleration are significant. Industry analysts see TST as a potential game-changer for the AI industry, where training costs have become a major bottleneck. "Training a 10-billion-parameter model can cost millions of dollars in compute time," said Dr. Elena Vasquez, a machine learning researcher at a leading AI lab. "A 2.5x speedup at matched FLOPs means either dramatically lower costs or the ability to train larger models within the same budget. That's a substantial competitive advantage."

This training efficiency is particularly valuable for smaller labs and startups that cannot afford the massive compute clusters used by tech giants. By reducing the wall-clock time required for pre-training, TST enables faster iteration cycles and more efficient use of existing hardware, delivering significant compute savings.

Implications for Model Merging and Multi-Task Learning

The superposition principle also has implications beyond pre-training speed. A separate paper on arXiv, "Superpose Task-specific Features for Model Merging" (2502.10698v2), explores how superposition can be used to merge multiple task-specific models into a single, unified model. While distinct from TST, this line of research suggests that superposition-based techniques could eventually enable models to be trained once and then efficiently adapted to multiple tasks through feature superposition.

Nous Research has made the implementation details publicly available, including validation results across all four model scales. The company is also exploring extensions of TST to even larger models and multi-modal architectures.

As the AI industry continues to grapple with the escalating costs of training ever-larger models, Token Superposition Training offers a practical, architecture-agnostic solution that delivers measurable speedups without sacrificing quality. The method's reliance on established superposition principles from academic literature gives it a strong theoretical foundation, and its empirical validation at multiple scales suggests it could become a standard tool in the LLM pre-training toolkit.