HR-FRGS: A Novel Biomarker Discovery Protocol using Dual Layer Hypergraph Learning for NGS RNA Sequence Data

Abstract

Biomarker discovery from high-dimensional RNA sequencing data remains challenging. Conventional methods such as differential expression analysis, pairwise protein-protein interaction networks, and weighted gene co-expression network analysis suffer from false-positive interactions, transitivity-driven noise, and arbitrary thresholds for feature selection. Addressing these limitations, this study proposed a novel framework, Hypergraph Regularised Fuzzy Rough Gene Selection (HR-FRGS), for knowledge-driven biomarker discovery. This pipeline integrates information-rich protein clustering with entropy filtering and PPI-pruned co-expression interactions to form a dual-layer hypergraph. A hypergraph-based scoring method that combines Random Walk with Restart diffusion and Hypergraph Betweenness Centrality is used to identify genes that are central in the biological network. Furthermore an autonomous fuzzy rough set selection eliminates arbitrary threshold dependency for gene selection. HR-FRGS was applied to four TCGA cancer cohorts: Lung Adenocarcinoma, Head and Neck Cancer, Kidney Clear Cell Carcinoma, and Colon Cancer, which reduced over 60,000 transcripts to compact biomarker panels. Validation using classical Machine Learning algorithms with cross-validation demonstrated that biomarker panels matched or exceeded the classification performance of the full transcriptome (AUC > 0.99, MCC > 0.94) and confirmed generalizability (MCC $\geq$ 0.893). Benchmarking against baseline methods and an ablation study showed the contribution of major parts of the algorithm. Functional enrichment analysis captures established pan-cancer hallmarks and cohort-specific oncogenic mechanisms, confirming the biological relevance of the identified genes.