ActTail is a research method from February 2026 for running a large language model with less arithmetic. For every word it writes, a model multiplies its input by huge tables of numbers. Activation sparsity skips the smallest input values, which saves work but costs quality. ActTail decides for each part of the model how much to skip, and in the authors' tests quality suffered less than with equal skipping everywhere.
What is activation sparsity?
In a transformer, each layer multiplies its input, a vector of activations, by weight matrices called projections: Q, K, V and O in the attention block, gate, up and down in the feed-forward block. Activation sparsity sets the smallest activations to zero, so the matching weights do not have to be loaded or multiplied during inference. TEAL (Liu et al., 2024) does this without retraining: it keeps the values with the largest magnitude. It reached 40 to 50% sparsity across the model with minimal quality loss, and its kernels decoded up to 1.53 times faster at 40% sparsity and 1.8 times faster at 50%. TEAL also tunes the sparsity of each matrix within a transformer block with a greedy search, at the same level for every block.
What does ActTail change?
ActTail, by Wenwen Hou, Xinyuan Song and Shiwei Liu (arXiv, February 2026), sets a budget for every projection across the whole model from a property of its weights. For each weight matrix it computes the heavy-tail exponent α of its empirical spectral density (the distribution of eigenvalues of its correlation matrix). In Heavy-Tailed Self-Regularization (HT-SR) theory, heavier tails indicate a better-trained layer. A linear formula turns the α values into sparsity ratios, scaled so that the model as a whole meets the target, such as 80%. The allocation depends only on the weights, so it is computed once. During inference each projection then keeps its largest inputs up to its own budget. In the paper's analysis of the result, attention projections, especially Q and O, get lower sparsity and the feed-forward projections higher.
How much does it help?
The authors compared ActTail with TEAL's TopK version at one uniform sparsity for all projections, and they measured only quality. At 80% sparsity, perplexity on WikiText2 (lower is better) was 17.76 for LLaMA-2-13B with ActTail against 29.66 with TEAL's uniform allocation, 40.1% lower. The gain was 21.8% for LLaMA-2-7B (28.43 against 36.37), 9.4% for Mistral-7B and 1.8% for Qwen1.5-7B. At 60% sparsity the two methods were close: 5.86 against 5.94 on LLaMA-2-7B. Both stay far from the full model, which scores 4.57 on LLaMA-2-13B without sparsity. On six downstream tasks, including MMLU and ARC, ActTail improved the average score at 80% sparsity by 8.0% relative to uniform sparsity on LLaMA-2-13B.
Where are the limits?
- The paper reports no speed measurements, so it does not show whether uneven budgets run as fast as uniform ones.
- The baseline is uniform TopK sparsity. The paper does not compare ActTail with TEAL's own greedy allocation per block.
- The tests cover four models of 7 and 13 billion parameters, released in 2023 and 2024.
- The paper is inconsistent about the direction. Its theory section and conclusion say heavier-tailed projections can be sparsified more aggressively, while its analysis of the allocation shows those projections receiving lower sparsity.
- At the high sparsity where ActTail helps most, quality is still much lower than without sparsity.
Example
Suppose a company runs Mistral-7B on its own server and wants faster answers. With uniform 70% sparsity, every projection drops 70% of its inputs. With ActTail, some projections drop more and others less, averaging the same 70%. In the paper's test on Mistral-7B, that change lowered perplexity by 1.8% at 70% sparsity and by 9.4% at 80%, compared with uniform TopK.