Influenza B viruses are responsible for significant disease burden caused by viruses of both the Yamagata- and Victoria-lineage. Since the circulating patterns of influenza B viruses in different countries vary we investigated molecular properties and evolution dynamics of influenza B viruses circulating in Germany between 1996 and 2020. A change of the dominant lineage occurred in Germany in seven seasons in over past 25 years. A total of 676 sequences of hemagglutinin coding domain 1 (HA1) and 516 sequences of neuraminidase (NA) genes of Yamagata- and Victoria-lineage viruses were analyzed using time-scaled phylogenetic tree. Phylogenetic analysis demonstrated that Yamagata-lineage viruses are more diverse than the Victoria-lineage viruses and could be divided into nine genetic groups whereas Victoria-lineage viruses presented six genetic groups. Comparative phylogenetic analyses of both the HA and NA segments together revealed a number of inter-lineage as well as inter- and intra-clade reassortants. We identified key amino acid substitutions in major HA epitopes such as in four antigenic sites and receptor-binding sites (RBS) and in the regions close to them, with most substitutions in the 120-loop of both lineage viruses. Altogether, seventeen substitutions were fixed over time within the Yamagata-lineage with twelve of them in the antigenic sites. Thirteen substitutions were identified within the Victoria-lineage, with eleven of them in the antigenic sites. Moreover, all Victoria-lineage viruses of the 2017/2018 season were characterized by a deletion of two amino acids at the position 162-163 in the antigenic site of HA1. The viruses with triple deletion Δ162-164 were found in Germany since season 2018/2019. We highlighted the interplay between substitutions in the glycosylation sites and RBS and antigenic epitope during HA evolution. The results obtained underscore the need for continuous monitoring of circulating influenza B viruses. Early detection of strains with genetic and antigenic variation is essential to predict the circulation patterns for the following season. Such information is important for the development of optimal vaccines and strategies for prevention and control of influenza.