Influenza A virus (IAV) is a major public health problem and a pandemic threat. Its evolution is largely driven by diversifying positive selection so that relative fitness of different amino acid variants changes with time due to changes in herd immunity or genomic context, and novel amino acid variants attain fitness advantage. Here, we hypothesize that diversifying selection also has another manifestation: the fitness associated with a particular amino acid variant should decline with time since its origin, as the herd immunity adapts to it. By tracing the evolution of antigenic sites at IAV surface proteins, we show that an amino acid variant becomes progressively more likely to become replaced by another variant with time since its origin-a phenomenon we call "senescence." Senescence is particularly pronounced at experimentally validated antigenic sites, implying that it is largely driven by host immunity. By contrast, at internal sites, existing variants become more favorable with time, probably due to arising contingent mutations at other epistatically interacting sites. Our findings reveal a previously undescribed facet of adaptive evolution and suggest approaches for prediction of evolutionary dynamics of pathogens.