ProClust provides reliable statistical significance of clusters for single-cell transcriptomic data

Abstract

Clustering of single-cell RNA sequencing (scRNA-seq) data is widely used to identify cell populations. However, overclustering can partition a single cell population into multiple clusters, potentially leading to spurious identification of cell subpopulations. Assessing the statistical significance of clusters is therefore needed to support their biological interpretation. We propose ProClust, a statistical test for clustering significance in scRNA-seq data. ProClust defines a single cluster as a log-concave distribution and adopts a projection-based approach to quantify clustering uncertainty through p-values. These projections can also provide interpretable visual evidence for individual clusterings. ProClust merges clusters when there is insufficient evidence to reject the single-cluster null hypothesis, refining an initial overclustering without prespecifying the final number of clusters. Across parametric and PBMC-based semi-synthetic simulations, ProClust avoided false subdivisions under single-population null settings while consistently recovering the underlying population structure across different numbers of true populations. In real scRNA-seq datasets from the Human Lung Cell Atlas, Tabula Muris, and Hydra, ProClust produced clustering solutions that showed overall agreement with reference annotations while retaining supported substructure and merging groups with insufficient evidence for separation. Together, these results demonstrate that ProClust provides a statistically principled and interpretable framework for refining an initial overclustering and distinguishing supported cellular heterogeneity. The ProClust R package is available at https://github.com/Anonym027/ProClust.