The antigenic drift of viral glycoproteins must be balanced by purifying selection pressure to maintain functionality. Understanding these evolutionary processes is key to predicting and combating viral evolution but is primarily based on influenza A(H3N2), which may limit generalisability. By characterising the influenza B virus haemagglutinin (HA) over 8 decades of circulation in humans, we found continuous genetic diversification, punctuated with antigenic changes that did not follow a linear path in antigenic space. Antigenic change is primarily underpinned by re-occurring mutations and deletions at positions 136, 150, 162-165, 197 and 203. These residues form complex epistatic networks that modulate the antigenic impact of mutation recycling. They also generate permissive backbones on which immune escape can emerge with limited replicative fitness cost. Our study identifies critical similarities and differences with A(H3N2) evolution and demonstrates the role of epistasis in balancing antigenic novelty with viral fitness. Our findings and genetic, antigenic and phenotypic datasets support the development of genotype-to-phenotype prediction tools, but such predictions need to capture the complex outcomes of epistasis.