The evolutionary dynamics of seasonal influenza A viruses (IAVs) have been well characterized at the population level, with antigenic drift known to be a major force in driving strain turnover. The evolution of IAV populations at the within-host level, however, is still less well characterized. Improving our understanding of within-host IAV evolution has the potential to shed light on the sources of new strains, including new antigenic variants, at the population level. Existing studies have pointed towards the role that stochastic processes play in shaping within-host viral evolution in acute infections of both humans and pigs. Here, we first apply a population genetic model called the ‘Beta-with-Spikes’ approximation to longitudinal intrahost Single Nucleotide Variant (iSNV) frequency data to quantify the extent of genetic drift acting on IAV populations at the within-host scale. We estimate a small effective population size for human IAV infections (?Ne=49?, 95% confidence interval: [28, 84]) and show that the observed iSNV dynamics are consistent with a Wright-Fisher model using various summary statistics. Using a diffusion approximation approach, we then further show that sampling noise is small relative to the magnitude of genetic drift in this dataset. We then apply similar analyses to the swine IAV dataset, arriving again at a very small effective viral population size estimate. However, we find that features of the swine IAV data cannot be consistently accounted for with a basic Wright-Fisher model of evolution and that sampling noise (in the broadest sense) can better account for the iSNV frequency changes observed in the swine IAV data. Our findings on IAV evolution within acutely infected humans contribute to a growing number of studies that point towards the important role of genetic drift in shaping patterns of genetic diversity in this host. Our findings also raise questions about what processes (e.g., spatial within-host compartmentalization, ecological superinfection) may impede our ability to quantify the strength of genetic drift acting on IAV populations in acutely infected swine.