Cell tropism, or the preference of a virus for particular cell types, has major implications for viral transmission, pathogenesis, and evolution. An increase in viral fitness --- increased within-host replication, also leading to increased transmission between hosts --- can result from a virus changing its cell tropism. This is illustrated in the context of influenza, where adaptation to infect cells expressing $\alpha$2-6 linked sialic acid receptors enhances human-to-human transmissibility. Target cell populations differ not only in abundance but also in intrinsic properties such as susceptibility, viral production, and interferon responses, rendering the relationship between tropism and viral fitness multi-faceted and complex. Understanding how different cell tropisms quantitatively change fitness remains an important open question in virology and quantitative biology. Here, we present a within-host mathematical model that incorporates distinct target cell types differing in key properties, and examine how cell tropism affects viral fitness, as measured by metrics such as peak viral load, infection duration, or total virus produced. Our analysis reveals that tradeoffs may arise when cell types differ by multiple characteristics. We further demonstrate that model parameters describing heterogeneity between cell types can be more accurately inferred when cell type proportions are measured alongside viral load. Our findings provide a framework for assessing the links between viral evolution, cell tropism, and within-host fitness, and motivate the design of experiments to collect quantitative data on between-cell heterogeneity.