Beyond Attention Weights: Graph-Constrained Instance Attribute Scoring for Interpretable Weakly Supervised Cancer Grading in Whole Slide Images
Keywords:
Multiple instance learning; whole slide image; computational pathology; graph neural network; interpretability; Gleason grading; weakly supervised learningAbstract
Background: Attention-based multiple instance learning (MIL) is the usual way to classify gigapixel whole slide images (WSIs) from slide-level labels, and its attention weights are widely displayed as heat maps of diagnostic evidence. An attention weight, however, records how much a patch contributes to a pooled vector, not whether the patch is diseased, and its scale changes with the size and composition of each slide. Methods: We propose AttrGraphMIL, a weakly supervised framework in which a multi-branch head assigns every patch an unnormalised attribute score for each diagnostic class. Patch embeddings are refined by two graph convolution layers over a graph with eight-connected spatial edges and cosine-similarity feature edges, and training combines a bag loss with three label-free constraints: a spatial smoothness penalty on the graph, a cross-slide margin ranking loss computed against a memory queue, and a score regulariser. A rank-consistent ordinal head handles ISUP grading. The framework was evaluated with patient-level five-fold cross-validation and three seeds on CAMELYON16, CAMELYON17, TCGA-NSCLC, PANDA and BCNB, using frozen CTransPath and UNI features. Results: On PatchCamelyon pseudo-bags, attribute scores ranked tiles with an instance AUC of 0.931 against 0.842 for attention weights at equal bag AUC. Cross-validated validation performance with CTransPath features was 0.959 ± 0.006 AUC on CAMELYON16, 0.911 ± 0.013 under leave-one-centre-out evaluation on CAMELYON17, 0.951 ± 0.004 on TCGA-NSCLC, 0.898 ± 0.006 quadratic weighted kappa on PANDA and 0.810 ± 0.010 macro AUC on BCNB; UNI features added 0.009 to 0.017. On CAMELYON16 the framework exceeded eight reproduced baselines (0.962 against 0.921 to 0.948), a single map threshold of 0.5 was selected in all five folds, and training cost 2.3 times ABMIL with 5.1 GB of GPU memory. Conclusion: Class-specific attribute scores, smoothed on a patch graph and calibrated across slides, give region maps with a common scale while retaining slide-level accuracy.
References
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). Sanity checks for saliency maps. In Advances in Neural Information Processing Systems (Vol. 31, pp. 9505–9515). Curran Associates.
Bulten, W., Kartasalo, K., Chen, P.-H. C., Ström, P., Pinckaers, H., Nagpal, K., Cai, Y., Steiner, D. F., van Boven, H., Vink, R., Hulsbergen-van de Kaa, C., van der Laak, J., Amin, M. B., Evans, A. J., van der Kwast, T., Allan, R., Humphrey, P. A., Grönberg, H., Samaratunga, H., … Litjens, G. (2022). Artificial intelligence for diagnosis and Gleason grading of prostate cancer: The PANDA challenge. Nature Medicine, 28(1), 154–163. https://doi.org/10.1038/s41591-021-01620-2
Cai, L., Huang, S., Zhang, Y., Lu, J., & Zhang, Y. (2025). AttriMIL: Revisiting attention-based multiple instance learning for whole-slide pathological image classification from a perspective of instance attributes. Medical Image Analysis, 103, Article 103631. https://doi.org/10.1016/j.media.2025.103631
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V., Busam, K. J., Brogi, E., Reuter, V. E., Klimstra, D. S., & Fuchs, T. J. (2019). Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine, 25(8), 1301–1309. https://doi.org/10.1038/s41591-019-0508-1
Cao, W., Mirjalili, V., & Raschka, S. (2020). Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140, 325–331. https://doi.org/10.1016/j.patrec.2020.11.008
Chen, R. J., Chen, C., Li, Y., Chen, T. Y., Trister, A. D., Krishnan, R. G., & Mahmood, F. (2022). Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16144–16155). IEEE.
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F. K., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., Williams, M., Oldenburg, L., Weishaupt, L. L., Wang, J. J., Vaidya, A., Le, L. P., Gerber, G., Sahai, S., Williams, W., & Mahmood, F. (2024). Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3), 850–862. https://doi.org/10.1038/s41591-024-02857-3
Chen, R. J., Lu, M. Y., Shaban, M., Chen, C., Chen, T. Y., Williamson, D. F. K., & Mahmood, F. (2021). Whole slide images are 2D point clouds: Context-aware survival prediction using patch-based graph convolutional networks. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2021 (Lecture Notes in Computer Science, Vol. 12908, pp. 339–349). Springer. https://doi.org/10.1007/978-3-030-87237-3_33
Dietterich, T. G., Lathrop, R. H., & Lozano-Pérez, T. (1997). Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence, 89(1–2), 31–71. https://doi.org/10.1016/S0004-3702(96)00034-3
Ehteshami Bejnordi, B., Veta, M., Johannes van Diest, P., van Ginneken, B., Karssemeijer, N., Litjens, G., van der Laak, J. A. W. M., & the CAMELYON16 Consortium. (2017). Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 318(22), 2199–2210. https://doi.org/10.1001/jama.2017.14585
Epstein, J. I., Egevad, L., Amin, M. B., Delahunt, B., Srigley, J. R., & Humphrey, P. A. (2016). The 2014 International Society of Urological Pathology (ISUP) consensus conference on Gleason grading of prostatic carcinoma. The American Journal of Surgical Pathology, 40(2), 244–252. https://doi.org/10.1097/PAS.0000000000000530
Fourkioti, O., De Vries, M., & Bakal, C. (2024). CAMIL: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images. In Proceedings of the Twelfth International Conference on Learning Representations. ICLR.
Gadermayr, M., & Tschuchnig, M. (2024). Multiple instance learning for digital pathology: A review of the state-of-the-art, limitations & future potential. Computerized Medical Imaging and Graphics, 112, Article 102337. https://doi.org/10.1016/j.compmedimag.2024.102337
Ilse, M., Tomczak, J., & Welling, M. (2018). Attention-based deep multiple instance learning. In Proceedings of the 35th International Conference on Machine Learning (Vol. 80, pp. 2127–2136). PMLR.
Jain, S., & Wallace, B. C. (2019). Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (pp. 3543–3556). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1357
Javed, S. A., Juyal, D., Padigela, H., Taylor-Weiner, A., Yu, L., & Prakash, A. (2022). Additive MIL: Intrinsically interpretable multiple instance learning for pathology. In Advances in Neural Information Processing Systems (Vol. 35, pp. 20689–20702). Curran Associates.
Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization [Paper presentation]. 3rd International Conference on Learning Representations, San Diego, CA, United States. https://arxiv.org/abs/1412.6980
Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks [Paper presentation]. 5th International Conference on Learning Representations, Toulon, France. https://arxiv.org/abs/1609.02907
Li, B., Li, Y., & Eliceiri, K. W. (2021). Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 14318–14328). IEEE.
Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Chen, R. J., Barbieri, M., & Mahmood, F. (2021). Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5(6), 555–570. https://doi.org/10.1038/s41551-020-00682-w
Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., & Zhang, Y. (2021). TransMIL: Transformer based correlated multiple instance learning for whole slide image classification. In Advances in Neural Information Processing Systems (Vol. 34, pp. 2136–2147). Curran Associates.
van der Laak, J., Litjens, G., & Ciompi, F. (2021). Deep learning in histopathology: The path to the clinic. Nature Medicine, 27(5), 775–784. https://doi.org/10.1038/s41591-021-01343-4
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., & Welling, M. (2018). Rotation equivariant CNNs for digital pathology. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2018 (Lecture Notes in Computer Science, Vol. 11071, pp. 210–218). Springer. https://doi.org/10.1007/978-3-030-00934-2_24
Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018). Graph attention networks [Paper presentation]. 6th International Conference on Learning Representations, Vancouver, BC, Canada. https://arxiv.org/abs/1710.10903
Wang, X., Yang, S., Zhang, J., Wang, M., Zhang, J., Yang, W., Huang, J., & Han, X. (2022). Transformer-based unsupervised contrastive learning for histopathological image classification. Medical Image Analysis, 81, Article 102559. https://doi.org/10.1016/j.media.2022.102559
Zhang, H., Meng, Y., Zhao, Y., Qiao, Y., Yang, X., Coupland, S. E., & Zheng, Y. (2022). DTFD-MIL: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18802–18812). IEEE.
Zheng, Y., Gindra, R. H., Green, E. J., Burks, E. J., Betke, M., Beane, J. E., & Kolachalama, V. B. (2022). A graph-transformer for whole slide image classification. IEEE Transactions on Medical Imaging, 41(11), 3003–3015. https://doi.org/10.1109/TMI.2022.3176598
Downloads
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
