Semi-Dual Attention: A Structure-Preserving Attention Mechanism Derived from Semi-Dual Optimization Theory

Resumo

Since the introduction of the Transformer architecture, attention mechanisms have become a core component of sequence modeling and large-scale pre-trained models. However, almost all existing mainstream attention methods are based on a "primal" modeling perspective, that is, assigning weights by directly normalizing the similarity between the query and the key. While this design is computationally efficient, it theoretically implies the risk of unconstrained amplification of attention weights and lacks explicit control over the globally optimal structure. This paper proposes a novel attention mechanism—Semi-Dual Attention (SDA)—which for the first time systematically introduces semi-dual optimization theory into Transformer attention modeling. Unlike existing methods, SDA treats the query vector as the primal variable for explicit optimization and the key vector as the implicit dual variable, and constrains the attention score through the semi-dual envelope. Thus, without destroying the original Transformer structure, it introduces a controllable, stable, and theoretically explainable attention regulation mechanism. Controlled experiments on the WikiText-2 language modeling task show that, under strictly constrained settings with extremely small parameter sizes and a very limited number of training epochs, the SDA Transformer performs in agreement with the standard Transformer, validating its structural safety and general substitutability. More importantly, this paper demonstrates from an optimization theory perspective that SDA possesses superior stability and generalization potential in larger-scale and more complex tasks, laying a solid theoretical foundation for subsequent large-scale models and cross-task transfer.