Epidemiological and clinical evidence indicates that humans infected with the 1918 pandemic H1N1 influenza virus and highly pathogenic avian H5N1 influenza viruses often displayed severe lung pathology. High viral load and extensive infiltration of macrophages are the hallmarks of highly pathogenic (HP) influenza viral infections. However, it remains unclear what biological mechanisms primarily determine the observed difference in the kinetics of viral load and macrophages between HP and low pathogenic (LP) viral infections, and how the mechanistic differences are associated with viral pathogenicity. In this study, we develop a mathematical model of viral dynamics that includes the dynamics of different macrophage populations and interferon. We fit the model to in vivo kinetic data of viral load and macrophage level from BALB/c mice infected with an HP or LP strain of H1N1/H5N1 virus using Bayesian inference. Our primary finding is that HP viruses have a higher viral infection rate, a lower interferon production rate and a lower macrophage recruitment rate compared to LP viruses, which are strongly associated with more severe tissue damage (quantified by a higher percentage of epithelial cell loss). We also quantify the relative contribution of macrophages to viral clearance and find that macrophages do not play a dominant role in the direct clearance of free viruses although their role in mediating immune responses such as interferon production is crucial. Our work provides new insight into the mechanisms that convey the observed difference in viral and macrophage kinetics between HP and LP infections and establishes an improved model-fitting framework to enhance the analysis of new data on viral pathogenicity.