Highly pathogenic avian influenza (HPAI) H5 viruses have posed a substantial pandemic threat through repeated human infection since their emergence in China in 1996. Nationwide control measures, including vaccination of poultry, were implemented in 2005, leading to a sharp reduction in H5N1 virus outbreaks. In 2008, novel non-N1 subtype (H5Nx) viruses emerged, gradually replacing the dominant H5N1 subtype and causing global outbreaks. The cause of this major shift in the ecology of HPAI H5 viruses remains unknown. Here, we show that major H5N1 virus lineages underwent population bottlenecks in 2006, followed by a recovery in virus populations between 2007 and 2009. Our analyses indicate that control measures, not competition from H5Nx viruses, were responsible for the H5N1 decline, with an H5N1 lineage capable of infecting poultry and wild birds experiencing a less severe population bottleneck due to circulation in unaffected wild birds. We show that H5Nx viruses emerged during the successful suppression of H5N1 virus populations in poultry, providing an opportunity for antigenically distinct H5Nx viruses to propagate. Avian influenza vaccination programs would benefit from universal vaccines targeting a wider diversity of influenza viruses to prevent the emergence of novel subtypes. IMPORTANCE A major shift in the ecology of highly pathogenic avian influenza (HPAI) H5 viruses occurred from 2008 to 2014, when viruses with non-N1 neuraminidase genes (termed H5Nx viruses) emerged and caused global H5 virus outbreaks. Here, we demonstrate that nationwide control measures, including vaccination in China, successfully suppressed H5N1 populations in poultry, providing an opportunity for antigenically distinct H5Nx viruses to emerge. In particular, we show that the widespread use of H5N1 vaccines likely conferred a fitness advantage to H5Nx viruses due to the antigenic mismatch of the neuraminidase genes. These results indicate that avian influenza vaccination programs would benefit from universal vaccines that target a wider diversity of influenza viruses to prevent potential emergence of novel subtypes.