Novel bat H17N10 and H18N11 influenza A viruses (IAVs) are incapable of reassortment with conventional IAVs during co-infection. To date, the underlying mechanisms that inhibit bat and conventional IAV reassortment remain poorly understood. Herein, we used the bat influenza M gene in the PR8 H1N1 virus genetic background to determine the molecular basis that restricts reassortment of segment 7. Our results showed that NEP and M1 from bat H17N10 and H18N11 can interact with PR8 M1 and NEP, resulting in mediating PR8 viral ribonucleoprotein (vRNP) nuclear export and formation of virus-like particles with single vRNP. Further studies demonstrated that the incompatible packaging signals (PSs) of H17N10 or H18N11 M segment led to the failure to rescue recombinant viruses in the PR8 genetic background. Recombinant PR8 viruses (rPR8psH18M and rPR8psH17M) containing bat influenza M coding region flanked with the PR8 M PSs were rescued but displayed lower replication in contrast to the parental PR8 virus, which is due to a low efficiency of recombinant virus uncoating correlating with the functions of the bat M2. Our studies reveal molecular mechanisms of the M gene that hinder reassortment between bat and conventional IAVs, which will help to understand the biology of novel bat IAVs.