MethylQUEEN: A Methylation Encoded DNA Foundation Model

Abstract

DNA 5-methylcytosine (5mC) modification plays a pivotal role in many biological processes, yet 5mC information and pattern hidden behind remains to be explored. Here, we develop Methylation Language Model based on Quintuple Bidirectional Transformer (MethylQUEEN), a novel pre-trained DNA methylation foundation model capable of sensing methylation states and covering the genome-wide methylation landscape. Through tailored methylation-prone pre-training, MethylQUEEN effectively captured epigenetics information hidden within the DNA sequences: it accurately traces DNAs tissue-of-origin, and successfully recovers the expression profile through methylation states. Integrative analysis on MethylQUEENs attention scores also enables us to reveal the unique methylation status of a tissue for precise disease detection, and identifying key regulatory 5mC sites for disease intervention. As a result, MethylQUEEN signifies a new paradigm in methylation analysis for various biological problems. Besides, our study demonstrates the effectiveness of directly integrating methylation information into pre-training, offering new perspectives and methodologies for a range of methylation-related biological processes. It serves as an initial exploration for the development of more comprehensive epigenomic models.