Avian influenza H5N1 clade 2.3.4.4b viruses caused a global panzootic and, unexpectedly, widespread outbreaks in dairy cattle, therefore representing a pandemic threat. To inform control strategies, it is critical to determine whether the potential to adapt to bovine cells is a general feature of H5N1 viruses, is specific to viruses of clade 2.3.4.4b, or narrowly restricted to some genotypes within this clade. Using a large panel of recombinant viruses representing >60 years of H5N1 history and other IAVs for comparison, we demonstrate replicative fitness in bovine cells is: (i) highly variable across 2.3.4.4b genotypes, (ii) limited in viruses predating the global expansion of this clade, (iii) determined by the internal gene cassette, and (iv) not restricted to udder epithelial cells. Mutations in the PB2 polymerase subunit emerge as key determinants of adaptation, although their phenotypic effects are context dependent. Bovine B3.13 and some avian genotypes exhibit enhanced modulation of bovine interferon-induced antiviral responses, determined by at least PB2, nucleoprotein, and the non-structural protein NS1. Our results highlight the polygenic nature of IAV host range, and reveal that the replication fitness in bovine cells, and likely their potential to adapt to cattle, varies greatly during the evolutionary trajectory of H5N1 viruses.