Co-infection with avian influenza virus (AIV) is common in poultry populations, primarily due to the diversity of viral subtypes and the practice of mixed-species farming. However, the replication dynamics of different AIV subtypes within distinct avian host-species remain unclear. In this study, we isolated and identified H5N1 and H5N8 AIVs from a naturally co-infected duck and characterized their biological characteristic regarding co-infection under both in vitro and in vivo conditions. Phylogenetic analysis revealed the HA genes of both H5 subtype AIVs belonged to clade 2.3.2.1d, sharing identical segments excluding differences in the M and NA genes. The H5N1 virus exhibited significantly higher infectivity and replication than the H5N8 virus in chicken embryo fibroblasts (CEF) and duck embryo fibroblasts (DEF). Serial passages of co-infection in vitro revealed that H5N1 virus became dominant in the first passage in both CEF and DEF cells. In vivo, both viruses caused high mortality in SPF chickens, while nonlethal in SPF ducks. In co-infected SPF chickens, H5N1 virus exhibited a pronounced replication advantage and higher viral loads. However, compared with single infections, co-infection delayed death time and reduced replication capacity. Conversely, in co-infected SPF ducks, H5N8 virus exhibited a significant replication advantage; however, compared to single infections, the replication capacity of H5N8 virus was reduced while H5N1 virus was maintained. These findings suggest that host-species influence the replication fitness and dominance of AIVs during co-infection, highlighting the importance of enhanced epidemiological surveillance and reducing mixed-species farming to minimize reassortment and evolution of high-risk AIVs.