Abstract: Objectives: In recent years, many deep learning-based classification models have been proposed. Because the size of histopathology images is extremely large, current methods usually divide it into many small patches of the same size and then build the classification model. The models first extract the features of patches and then use pooling methods etc. to integrate patch-level features into slide-level features for classification, in which the extraction of patch-level features and independent with the integration and classification process, making it impossible to adapt the feature extractor with the feedback of classification results. Therefore, they can not obtain the feature which is most useful for the whole slide image (WSI) classification. The objective of the paper is to solve the problems faced by above methods. Methods: This paper proposed an end-to-end attention pooling method based on a patch-scoring model. Firstly, we build a patch scoring model to score each patch with the multiple instance learning methods, selecting patches based on the scores. At last, we utilized an attention-pooling module to integrate the features of selected patches to build a classification model. Besides, according to the character of the histopathology image, we proposed to use a customized squared average function to normalize the attention weights to focus more on the high scored patches in positive WSI, making the classification results more interpretable. Results: On the public available dataset CAMELYON16, the proposed achieved a F1 score at 0.644. And it achieved a F1 score at 0.593 at the public available BACH dataset. When the new normalization function was used, the average attention weights difference between positive patches and negative patches is 0.017, which is much higher than when softmax function was used. Conclusions: We can conclude from the experiments that the end-to-end attention pooling method is a better method for breast histopathology image classification. The proposed normalization function can make the attention module more interpretable.