H5N1 highly pathogenic avian influenza viruses have spread globally and pose a pandemic risk. Prior studies suggest that early life exposures to group 1 influenza viruses (H1N1 and H2N2) prime antibodies that cross-react to the hemagglutinin of H5N1, which is also a group 1 virus. However, less is known about how immune history affects antibody responses against the H5N1 neuraminidase (NA). We measured NA inhibition antibodies against multiple H5N1 viruses using sera from 155 individuals born between 1927 and 2016. Individuals likely primed in childhood with H1N1 viruses possessed higher levels of antibodies that cross-react with the NA of H5N1 viruses compared to those primed with H2N2 or H3N2 viruses. While young children rarely possessed cross-reactive N1 antibodies, childhood infections with contemporary H1N1, but not H3N2, viruses elicited them. We also measured antibodies against an H5N5 virus (A6 genotype) that recently caused a fatal infection in the United States. Consistent with the lack of circulation of N5 viruses in humans, we found low levels of antibodies against the N5 NA. Our data suggest that immune history greatly impacts the generation of cross-reactive NA antibodies, and that reassortment with other NAs may increase the risk of H5 infection of humans.