Highly pathogenic avian influenza (HPAI) H5 viruses have recently been documented in mammals including humans, posing a major threat to global public health. To prevent a potential H5 pandemic, it is critical to elucidate the antigenic evolutionary pattern and identify key drivers underlying its evolution. In this work, we construct a comprehensive antigenic map of H5 influenza viruses spanning their evolutionary history and classified three antigenic clusters with no cross-neutralization. The first corresponds to ancestral clades, the second to 2.3.4.4* clades being predominant since 2010, and the third to 2.3.4.4 h clade. Despite the gradually increasing genetic distances from ancestral to 2.3.4.4* to 2.3.4.4 h, their antigenic evolution does not follow the same progressive pattern: the antigenic distance between 2.3.4.4 h and ancestral is smaller than that between 2.3.4.4* and ancestral. This divergence is associated with two distinct mutation patterns at six key amino acid positions: (1) persistent mutations at positions 88 (N > R > S), 199 (D > N > S), and 205 (K > N > D), and (2) reversible mutations at positions 131 (Q > L > Q), 139 (S > P > S), and 289 (N > H > N). These findings not only reveal the antigenic evolution mechanism of H5 influenza, but also provide important guidance for vaccine strain selection and broad-spectrum vaccine development.