Deep learning applied to histopathology: We apply deep learning methods to analyze tissue images and to integrate them with multi-omic and clinical data. Taking advantage of the multi-resolution nature of the whole slide H&E images, we introduced a customized deep convolutional neural network architecture, Panoptes, to integrate features of different magnifications (Hong et al., Cell Reports Medicine 2021). We have applied Panoptes to predict subtypes and molecular features for endometrial cancer, STK11 mutations in lung adenocarcinoma, as well as immune response, the CpG island methylator phenotype and telomere length in glioblastoma. We have also integrated histopathology and proteogenomics at a pan-cancer level (Wang et al., Cell Reports Medicine 2023).

Spatial omics: We have developed a method, CANVAS, for characterization of tissue heterogeneity through segmentation-free representation learning and applied it to high-dimensional spatial omics data (Tan et al., bioRxiv 2024). CANVAS uses a self-supervised machine learning approach based on vision transformers and masked imaging modeling, where a large portion of the image is removed and the remaining image is used to reconstruct the original image. We are applying CANVAS to analyze mass cytometry and mass spectrometry imaging data of lung cancer, and to multiplexed immunoflourescence data of endometrial and cervical cancer.