Seasonal influenza viruses undergo continuous antigenic drift due to mutations in the hemagglutinin (HA) protein, rendering vaccines ineffective and posing a significant global public health challenge. Existing computational models can predict antigenic relationships but generally lack interpretability, making it difficult to reveal the molecular determinants driving antigenic changes. To address this, we developed PREDAC-Transformer, an end-to-end deep learning framework that integrates sequence, physicochemical, and evolutionary features, and employs a self-attention mechanism to capture long-range dependencies relevant to antigenicity. More importantly, we introduce the integrated antigenicity score, which combines attention-based attribution with information-theoretic metrics to provide continuous quantification and ranking of the antigenic contributions of individual amino acid site. Our results demonstrate that PREDAC-Transformer not only significantly improves the accuracy of antigenic relationship prediction but also successfully recapitulates major historical antigenic cluster transitions. Using integrated antigenicity score, we systematically identified two classes of key sites: global key sites with sustained impact on antigenic evolution, and cluster-transition determining sites that drive cluster transitions. These sites include most canonical epitopes and reveal additional functional residues previously overlooked, which may influence immune escape via cooperative effects or glycosylation. Collectively, these findings advance our understanding of influenza antigenic evolution and provide novel insights for refining computational models. PREDAC-Transformer achieves high-precision prediction while attributing antigenic differences to individual residues, thereby linking viral genomic variation, antigenic change, and public health decision-making. This framework has the potential to reduce experimental burdens in influenza surveillance and assist in vaccine strain recommendation, thereby supporting global influenza control efforts.