Influenza is a common respiratory disease that can cause human illness and death. Timely and accurate prediction of disease risk is of great importance for public health management and prevention. The influenza data belong to typical spatiotemporal data in that influenza transmission is influenced by regional and temporal interactions. Many existing methods only use the historical time series information for prediction, which ignores the effect of spatial correlations of neighboring regions and temporal correlations of different time periods. Mining spatiotemporal information for risk prediction is a significant and challenging issue. In this paper, we propose a new end-to-end spatiotemporal deep neural network structure for influenza risk prediction. The proposed model mainly consists of two parts. The first stage is the spatiotemporal feature extraction stage where two-stream convolutional and recurrent neural networks are constructed to extract the different regions and time granularity information. Then, a dynamically parametric-based fusion method is adopted to integrate the two-stream features and making predictions. In our work, we demonstrate that our method, tested on two influenza-like illness (ILI) datasets (US-HHS and SZ-HIC), achieved the best performance across all evaluation metrics. The results imply that our method has outstanding performance for spatiotemporal feature extraction and enables accurate predictions compared to other well-known influenza forecasting models.