Sequential Attention: Making AI models leaner and faster without sacrificing accuracy

Google Research introduced Sequential Attention as a subset selection algorithm for making large scale machine learning models more efficient. The algorithm uses a greedy mechanism to sequentially select the best next component to add to the model during a single training process. This integration allows application to large models with minimal overhead. The method addresses the NP hard nature of feature selection by treating it as a sequential process rather than a one shot weighting. It maintains a set of selected candidates and uses attention scores to evaluate the importance of remaining candidates in context. Researchers noted that this adaptive approach captures high order non linear interactions missed by simpler filter methods. Sequential Attention achieved state of the art results across neural network benchmarks and proved mathematically equivalent to the Orthogonal Matching Pursuit algorithm in linear regression cases. An extension called SequentialAttention++ applies the framework to structured neural network pruning by removing blocks of weights. The approach showed significant gains in model compression and efficiency on hardware accelerators without sacrificing accuracy in tasks such as ImageNet classification. Applications include optimizing feature embedding layers in large embedding models for recommender systems. The researchers said the technique provides quality gains and efficiency savings in these models.