In order to minimize the impact of influenza on public health, accurate early forecasting is essential. Various deep-learning-based models have been proposed to predict future influenza occurrences by capturing temporal/regional patterns from past occurrence time-series data. However, the prediction performance of these unimodal approaches is limited because they extract knowledge only from collected data, and users cannot input contextual information and domain knowledge to them. Recently, large language models (LLMs) have demonstrated the potential to improve prediction accuracy by linking contextual text information to time-series predictions. In this paper, we propose MIFlu, a multimodal influenza forecasting scheme that can fuse contextual text information to time-series influenza occurrences using two LLMs. It first extracts text embeddings from the user´s text prompts that contain contextual information using a text-embedding LLM. Then, MIFlu fuses the text embeddings and time-series embeddings and uses the fused embeddings to predict future occurrences using a forecasting LLM. In extensive experiments using public national/regional influenza datasets, MIFlu outperforms other predictive models, improving prediction performance by up to 26.2% compared to state-of-the-art models. We also analyze the effect of various textual input embedders, hyperparameters, and the amount of training data on forecasting accuracy.