NEST: Spatially-mapped cell-cell communication patterns using a deep learning-based attention mechanism

Dysregulation of communication between cells mediates complex diseases such as cancer and diabetes. However, detecting cell-cell communication (CCC) at scale remains one of the greatest challenges in transcriptomics. While gene expression measured with single-cell RNA sequencing and spatial transcriptomics reinvigorated computational approaches to detecting CCC, most existing methods exhibit high false positive rates, do not integrate spatial proximity of ligand-receptor interactions, and cannot detect CCC between individual cells. We overcome these challenges by presenting NEST (NEural network on Spatial Transcriptomics), which uses a graph attention network paired with an unsupervised contrastive learning approach to decipher patterns of communication while retaining the strength of each signal. We introduce new synthetic benchmarking experiments which demonstrate how NEST outperforms existing tools and detects biologically-relevant CCC along with directionality and confidence across spot- and cell-based technologies measuring several different tissues and diseases. In our applications, NEST identifies T-cell homing signals in human lymph nodes, aggressive cancer CCC in lung adenocarcinoma, and discovers new patterns of communication that act as relay networks in pancreatic cancer. Beyond two-dimensional data, we also highlight NESTs ability to detect CCC in three-dimensional spatial transcriptomic data.