Influenza A H5N1 viruses remain a substantial threat to global public health. In particular, the expanding genetic diversity of H5N1 viruses and the associated risk for human adaptation underscore the importance of better understanding host immune responses that may protect against disease or infection. Although much emphasis has been placed on investigating early virus-host interactions and the induction of innate immune responses, little is known of the consequent adaptive immune response to H5N1 virus infection. In this review, we describe the H5N1 virus-specific and cross-reactive antibody and T cell responses in humans and animal models. Data from limited studies suggest that although initially robust, there is substantial waning of the serum antibody responses in survivors of H5N1 virus infection. Characterization of monoclonal antibodies generated from memory B cells of survivors of H5N1 virus infection has provided an understanding of the fine specificity of the human antibody response to H5N1 virus infection and identified strategies for immunotherapy. Human T cell responses induced by infection with seasonal influenza viruses are directed to relatively conserved internal proteins and cross-react with the H5N1 subtype. A role for T cell-based heterosubtypic immunity against H5N1 viruses is suggested in animal studies. Further studies on adaptive immune responses to H5N1 virus infection in both humans and animals are needed to inform the design of optimal immunological treatment and prevention modalities.