End-to-End Attention Pooling for Histopathology Image Classification

Objectives In recent years, many deep learning-based classification models have been proposed. Because the size of histopathology images is extremely large, current methods usually divide it into many small patches of the same size and then build the classification model. The models first extract the features of patches and then use pooling methods etc. to integrate patch-level features into slide-level features for classification, in which the extraction of patch-level features and independent with the integration and classification process, making it impossible to adapt the feature extractor with the feedback of classification results. Therefore, they can not obtain the feature which is most useful for the whole slide image (WSI) classification. The objective of the paper is to solve the problems faced by above methods.

Methods We propose an end-to-end attention pooling method based on a patch-scoring model. First, we build a patch scoring model to score each patch with the multiple instance learning methods, selecting patches based on the scores. Second, we utilize an attention-pooling module to integrate the features of selected patches to build a classification model. Besides, according to the character of the histopathology image, we propose to use a customized squared average function to normalize the attention weights to focus more on the high scored patches in positive WSI, making the classification results more interpretable.

Results On the public available dataset CAMELYON16, the proposed method achieves F1 score at 0.644. And it achieves F1 score at 0.593 at the public available BACH dataset. When the new normalization function is used, the average attention weights difference between positive patches and negative patches is 0.017, which is much higher than when softmax function was used.

Conclusions We can conclude from the experiments that the end-to-end attention pooling method is a better method for breast histopathology image classification. The proposed normalization function can make the attention module more interpretable.