This is an extensive and continuously updated compilation of graph SSL literature categorized by the knowledge-based taxonomy, proposed by our paper ๐A Survey on Self-Supervised Pre-Training of Graph Foundation Models: A Knowledge-Based Perspective. Here every pretext of each paper is listed and briefly explained. You can find all pretexts and their corresponding papers with detailed metadata below, including additional pretexts and literature not listed in our paper.
A kind reminder: to search for a certain paper, type the title or the abbreviation of the proposed method (recommended) into the browser search bar (Ctrl + F). Some papers fall under multiple sections.
- Relevant surveys and empirical studies
- Microscopic pretexts
- Macroscopic pretexts
- Task adaptation strategies
Note: ๐ธ๏ธ โgraph-related; ๐ค LLM-related; ๐ฌ empirical study
Node features
- Feature prediction: to predict the original node features by decoding low-dimensional representations
- Feature denoising: to add (generally continuous, e.g. isotropic Gaussian) noises to the original features and try to reconstruct them
- Masked feature prediction: a special, discrete case of feature denoising, which predicts the original features of masked nodes by representations of unmasked ones. It is "autoregressive" if the predicted nodes are generated one-by-one
- Feature recovery: to predict the original node features by the trivial synthetic features
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| MGAE: Marginalized Graph Autoencoder for Graph Clustering | CIKM'17 | Feature prediction | Graph partitioning | link |
| Symmetric Graph Convolutional Autoencoder for Unsupervised Graph Representation Learning (GALA) | ICCV'19 | Feature prediction | Node clustering; link prediction; image clustering | link |
| Strategies for Pre-training Graph Neural Networks (AttrMask) | ICLR'20 | Masked feature prediction | Graph classification; biological function prediction | link |
| Graph Representation Learning via Graphical Mutual Information Maximization (GMI) | WWW'20 | Feature prediction (JS) | Node classification; link prediction | link |
| When Does Self-Supervision Help Graph Convolutional Networks? (GraphComp) | ICML'20 | Masked feature prediction | Node classification | link |
| GPT-GNN: Generative Pre-Training of Graph Neural Networks | KDD'20 | Masked feature prediction (autoregressive) | Node classification; edge regression (recommendation score); meta-path prediction | link |
| Graph Attention Auto-Encoders (GATE) | ICTAI'20 | Feature prediction | Node classification | link |
| Graph-Bert: Only Attention is Needed for Learning Graph Representations | arXiv:2001 | Feature prediction | Node classification; node clustering | link |
| Self-supervised Learning on Graphs: Deep Insights and New Direction (AttributeMask) | arXiv:2006 | Masked feature prediction | Node classification | link |
| SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks | NeurIPS'21 | Masked feature prediction | Node classification; image classification | link |
| Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (MGSSL) | NeurIPS'21 | Masked feature prediction | Graph classification | link |
| Multi-Scale Variational Graph AutoEncoder for Link Prediction (MSVGAE) | WSDM'22 | Feature prediction | Link prediction | -- |
| Self-Supervised Representation Learning via Latent Graph Prediction (LaGraph) | ICML'22 | Masked feature prediction | Node classification; graph classification | link |
| GraphMAE: Self-Supervised Masked Graph Autoencoders | KDD'22 | Masked feature prediction | Node classification; graph classification | link |
| Graph Masked Autoencoders with Transformers (GMAE) | arXiv:2202 | Masked feature prediction | Node classification; graph classification | link |
| Wiener Graph Deconvolutional Network Improves Graph Self-Supervised Learning (WGDN) | AAAI'23 | Feature prediction | Node classification; graph classification | link |
| Heterogeneous Graph Masked Autoencoders (HGMAE) | AAAI'23 | Feature prediction; masked feature prediction | (Heterogeneous) node classification; node clustering | link |
| Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules | ICLR'23 | Masked feature prediction | Graph classification; graph regression | link |
| GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner | WWW'23 | Masked feature prediction | Node classification | link |
| Graph Neural Networks can Recover the Hidden Features Solely from the Graph Structure (GNNRecover) | ICML'23 | Feature recovery | Node classification | link |
| Directional Diffusion Models for Graph Representation Learning (DDM) | NeurIPS'23 | Feature denoising | Node classification; graph classification | link |
| DiP-GNN: Discriminative Pre-Training of Graph Neural Networks | NeurIPS Workshop (GLFrontiers)'23 | Masked feature prediction | Node classification; link prediction | -- |
| Towards Effective and Robust Graph Contrastive Learning With Graph Autoencoding (AEGCL) | TKDE'23 | Feature prediction | Node classification; node clustering; link prediction | link |
| RARE: Robust Masked Graph Autoencoder | TKDE'23 | Masked feature prediction | Node classification; graph classification; image classification | link |
| Incomplete Graph Learning via Attribute-Structure Decoupled Variational Auto-Encoder (ASD-VAE) | WSDM'24 | Feature prediction | Node classification; node attribute completion | link |
| Deep Contrastive Graph Learning with Clustering-Oriented Guidance (DCGL) | AAAI'24 | Feature prediction | Node clustering | link |
| Rethinking Graph Masked Autoencoders through Alignment and Uniformity (AUG-MAE) | AAAI'24 | Masked feature prediction | Node classification; graph classification | link |
| Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery (DGPM) | AAAI'24 | Masked feature prediction | Graph classification | link |
| Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning (GCMAE) | ICDE'24 | Masked feature prediction | Node classification; node clustering; graph classification; link prediction | link |
| Where to Mask: Structure-Guided Masking for Graph Masked Autoencoders (StructMAE) | IJCAI'24 | Masked feature prediction | Graph classification | link |
- Latent feature matching: to minimize the (Euclidean) distance between pairs of positive representation vectors
- Instance discrimination: to minimize/maximize the distance between pairs of positive/negative representation samples. Jenson-Shannon (JS), InfoNCE (incl. NT-Xent), Triplet margin, and Bootstrapping are all estimators of mutual information (MI) between nodes. Other contrastive losses:
- DP stands for the dot-product-based contrastive loss (with negative sampling), e.g. the population spectral contrastive loss
- BPR stands for Bayesian Personalized Ranking loss, mostly used in recommendation
- Dimension discrimination: to minimize/maximize the mutual information (MI) between pairs of positive/negative representation dimensions. Could be either intra-sample or inter-sample
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Deep Graph Contrastive Representation Learning (GRACE) | ICML Workshop (GRL+)'20 | Instance discrimination (InfoNCE) | Node classification | link |
| GraphTER: Unsupervised Learning of Graph Transformation Equivariant Representations via Auto-Encoding Node-wise Transformations | CVPR'20 | Latent feature matching | Graph (point cloud) classification; node classification (point cloud segmentation) | link |
| Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning (CG3) | AAAI'21 | Instance discrimination (InfoNCE) | Node classification | link |
| Graph Contrastive Learning with Adaptive Augmentation (GCA) | WWW'21 | Instance discrimination (InfoNCE) | Node classification | link |
| SelfGNN: Self-supervised Graph Neural Networks without Explicit Negative Sampling | WWW Workshop (SSL)'21 | Instance discrimination (Bootstrapping) | Node classification | link |
| Self-supervised Graph Learning for Recommendation (SGL) | SIGIR'21 | Instance discrimination (InfoNCE) | Recommendation | link |
| Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning (MERIT) | IJCAI'21 | Instance discrimination (InfoNCE) | Node classification | link |
| Pre-training on Large-Scale Heterogeneous Graph (PT-HGNN) | KDD'21 | Instance discrimination (InfoNCE) | (Heterogeneous) node classification; link prediction | link |
| Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning (HeCo); Hierarchical Contrastive Learning Enhanced Heterogeneous Graph Neural Network (HeCo++) | KDD'21; TKDE'23 | Instance discrimination (InfoNCE) | (Heterogeneous) node classification; node clustering | link |
| InfoGCL: Information-Aware Graph Contrastive Learning | NeurIPS'21 | Instance discrimination (Bootstrapping) | Node classification; graph classification | -- |
| From Canonical Correlation Analysis to Self-supervised Graph Neural Networks (CCA-SSG) | NeurIPS'21 | Dimension discrimination; latent feature matching | Node classification | link |
| Self-Supervised GNN that Jointly Learns to Augment (GraphSurgeon) | NeurIPS Workshop (SSL)'21 | Latent feature matching; dimension discrimination | Node classification | link |
| Simple Unsupervised Graph Representation Learning (SUGRL) | AAAI'22 | Instance discrimination (Triplet margin) | Node classification | link |
| Large-Scale Representation Learning on Graphs via Bootstrapping (BGRL) | ICLR'22 | Instance discrimination (Bootstrapping) | Node classification | link |
| VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning (VICReg) | ICLR'22 | Dimension discrimination; latent feature matching | Node classification | link |
| Adversarial Graph Contrastive Learning with Information Regularization (ARIEL) | WWW'22 | Instance discrimination (InfoNCE) | Node classification; graph classification | link |
| Self-Supervised Representation Learning via Latent Graph Prediction (LaGraph) | ICML'22 | Latent feature matching | Node classification; graph classification | link |
| ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learning | ICML'22 | Instance discrimination (InfoNCE) | Node classification | link |
| COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning | KDD'22 | Instance discrimination (InfoNCE) | Node classification | link |
| Relational Self-Supervised Learning on Graphs (RGRL) | CIKM'22 | Instance discrimination (Bootstrapping) | Node classification; link prediction | link |
| Revisiting Graph Contrastive Learning from the Perspective of Graph Spectrum (SpCo) | NeurIPS'22 | Instance discrimination (InfoNCE) | Node classification | link |
| Contrastive Graph Structure Learning via Information Bottleneck for Recommendation (CGI) | NeurIPS'22 | Instance discrimination (InfoNCE) | Recommendation | link |
| Uncovering the Structural Fairness in Graph Contrastive Learning (GRADE) | NeurIPS'22 | Instance discrimination (InfoNCE) | Node classification | link |
| Co-Modality Graph Contrastive Learning for Imbalanced Node Classification (CM-GCL) | NeurIPS'22 | Instance discrimination (InfoNCE) | Node classification (imbalanced) | link |
| Graph Barlow Twins: A Self-supervised Representation Learning Framework for Graphs (G-BT) | Knowledge-Based Systems'22 | Dimension discrimination | Node classification | link |
| Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming (G-Zoom) | TNNLS'22 | Instance discrimination (InfoNCE) | Node classification | -- |
| Neural Eigenfunctions Are Structured Representation Learners (NeuralEF) | arXiv:2210 | Dimension discrimination | Node classification; image retrieval; object detection; instance segmentation | link |
| MA-GCL: Model Augmentation Tricks for Graph Contrastive Learning | AAAI'23 | Instance discrimination (InfoNCE) | Node classification | link |
| ImGCL: Revisiting Graph Contrastive Learning on Imbalanced Node Classification | AAAI'23 | Instance discrimination (InfoNCE) | Node classification (imbalanced) | -- |
| Link Prediction with Non-Contrastive Learning (T-BGRL) | ICLR'23 | Instance discrimination (Bootstrapping) | Link prediction | link |
| LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation | ICLR'23 | Instance discrimination (InfoNCE) | Recommendation | link |
| GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner | WWW'23 | Latent feature matching | Node classification | link |
| Graph Self-supervised Learning with Augmentation-aware Contrastive Learning (ABGML) | WWW'23 | Instance discrimination (Bootstrapping) | Node classification; node clustering; similarity search | link |
| Randomized Schur Complement Views for Graph Contrastive Learning (rLap) | ICML'23 | Instance discrimination (InfoNCE, Bootstrapping) | Node classification | link |
| Graph Contrastive Learning with Generative Adversarial Network (GACN) | KDD'23 | Instance discrimination (InfoNCE, BPR) | Node classification; link prediction | -- |
| GiGaMAE: Generalizable Graph Masked Autoencoder via Collaborative Latent Space Reconstruction | CIKM'23 | Instance discrimination (InfoNCE) | Node classification; node clustering; link prediction | link |
| Provable Training for Graph Contrastive Learning (POT) | NeurIPS'23 | Instance discrimination (InfoNCE) | Node classification | link |
| Graph Contrastive Learning with Stable and Scalable Spectral Encoding (Sp2GCL) | NeurIPS'23 | Instance discrimination (InfoNCE) | Node classification; graph classification; graph regression | link |
| RARE: Robust Masked Graph Autoencoder | TKDE'23 | Latent feature matching | Node classification; graph classification; image classification | link |
| Hierarchically Contrastive Hard Sample Mining for Graph Self-Supervised Pretraining (HCHSM) | TNNLS'23 | Instance discrimination (JS) | Node classification; node clustering | link |
| Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph (SP-GCL) | TMLR'23 | Instance discrimination (DP) | Node classification | link |
| Oversmoothing: A Nightmare for Graph Contrastive Learning? (BlockGCL) | arXiv:2306 | Dimension discrimination | Node classification | link |
| Rethinking and Simplifying Bootstrapped Graph Latents (SGCL2) | WSDM'24 | Instance discrimination (Bootstrapping) | Node classification | link |
| Towards Alignment-Uniformity Aware Representation in Graph Contrastive Learning (AUAR) | WSDM'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | -- |
| ReGCL: Rethinking Message Passing in Graph Contrastive Learning | AAAI'24 | Instance discrimination (InfoNCE) | Node classification | link |
| A New Mechanism for Eliminating Implicit Conflict in Graph Contrastive Learning (PiGCL) | AAAI'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | link |
| ASWT-SGNN: Adaptive Spectral Wavelet Transform-Based Self-Supervised Graph Neural Network | AAAI'24 | Instance discrimination (InfoNCE) | Node classification; graph classification | -- |
| Graph Contrastive Invariant Learning from the Causal Perspective (GCIL) | AAAI'24 | Dimension discrimination | Node classification | link |
| HypeBoy: Generative Self-Supervised Representation Learning on Hypergraphs | ICLR'24 | Instance discrimination (InfoNCE) | Node classification; Hyperedge prediction | -- |
| A Graph is Worth 1-bit Spikes: When Graph Contrastive Learning Meets Spiking Neural Networks (SpikeGCL) | ICLR'24 | Instance discrimination (Triplet margin) | Node classification | link |
| Self-supervised Heterogeneous Graph Learning: a Homophily and Heterogeneity View (HERO) | ICLR'24 | Latent feature matching | (Heterogeneous) node classification; similarity search | link |
| Graph Augmentation for Recommendation (GraphAug) | ICDE'24 | Instance discrimination (InfoNCE, BPR) | Recommendation | link |
| Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning (GCMAE) | ICDE'24 | Instance discrimination (InfoNCE) | Node classification; node clustering; graph classification; link prediction | link |
| Graph Contrastive Learning with Cohesive Subgraph Awareness (CTAug) | WWW'24 | Instance discrimination (InfoNCE) | Node classification | link |
| Towards Expansive and Adaptive Hard Negative Mining: Graph Contrastive Learning via Subspace Preserving (GRAPE) | WWW'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | link (Unavailable) |
| MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning | WWW'24 | Instance discrimination (InfoNCE) | Node classification; graph classification | link |
| Graph Contrastive Learning via Interventional View Generation (GCL-IVG) | WWW'24 | Instance discrimination (InfoNCE) | Node classification; node clustering | -- |
| Graph Contrastive Learning with Kernel Dependence Maximization for Social Recommendation (CL-KDM) | WWW'24 | Instance discrimination (InfoNCE, BPR) | Recommendation | -- |
| High-Frequency-aware Hierarchical Contrastive Selective Coding for Representation Learning on Text-attributed Graphs (HASH-CODE) | WWW'24 | Instance discrimination (DP) | Node classification; link prediction | -- |
Node properties
- Property prediction: a regression task to predict the property of a node (e.g. degree)
- Centrality ranking: to estimate whether the centrality score of a node is greater/lower than that of another node
- Node order matching: to match the output node order with the input order
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Unsupervised Pre-training of Graph Convolutional Networks (ScoreRank) | ICLR Workshop (RLGM)'19 | Centrality ranking | Node classification | -- |
| Self-supervised Learning on Graphs: Deep Insights and New Direction (NodeProperty) | arXiv:2006 | Property prediction (degree, clustering coefficient, etc.) | Node classification | link |
| Permutation-Invariant Variational Autoencoder for Graph-Level Representation Learning (PIGAE) | NeurIPS'21 | Node order matching | Graph classification | link |
| Graph Auto-Encoder Via Neighborhood Wasserstein Reconstruction (NWR-GAE) | ICLR'22 | Property prediction (degree) | Node classification; structural role identification | link |
| What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders (MaskGAE) | KDD'23 | Property prediction (degree) | Node classification; link prediction | link |
Links
- Link prediction: a generally binary classification task that predicts if two nodes are connected by a link
- Link denoising: to add (generally continuous) noises to the original edge set and try to reconstruct it
- Masked link prediction: to predict the masked links by node representations propagated on the unmasked graph. It is "autoregressive" if the predicted links are generated one-by-one
- Meta-path prediction: link prediction on heterogeneous graphs, to predict if two nodes are connected by a meta-path
- (Masked) edge feature prediction: to predict the original (masked) edge features by node representations
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Variational Graph Auto-Encoders (GAE, VGAE) | NIPS Workshop (BDL)'16 | Link prediction | Link prediction | link |
| Adversarially Regularized Graph Autoencoder for Graph Embedding (ARGA, ARVGA) | IJCAI'18 | Link prediction | Link prediction; node clustering | link |
| Unsupervised Pre-training of Graph Convolutional Networks (DenoisingRecon) | ICLR Workshop (RLGM)'19 | Masked link prediction | Node classification | -- |
| Graphite: Iterative Generative Modeling of Graphs | ICML'19 | Link prediction | Node classification; link prediction | link |
| Semi-Implicit Graph Variational Auto-Encoders (SIG-VAE) | NeurIPS'19 | Link prediction | Node classification; link prediction; node clustering; graph generation | link |
| Strategies for Pre-training Graph Neural Networks (AttrMask) | ICLR'20 | Masked edge feature prediction | Graph classification; biological function prediction | link |
| GPT-GNN: Generative Pre-Training of Graph Neural Networks | KDD'20 | Masked link prediction (autoregressive) | Node classification; edge classification; meta-path prediction | link |
| Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs (SELAR) | NeurIPS'20 | Meta-path prediction | (Heterogeneous) node classification; link prediction | link |
| Self-supervised Learning on Graphs: Deep Insights and New Direction (EdgeMask) | arXiv:2006 | Masked link prediction | Node classification | link |
| Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning (CG3) | AAAI'21 | Link prediction | Node classification | link |
| How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision (SuperGAT) | ICLR'21 | Link prediction | Node classification; link prediction | link |
| Permutation-Invariant Variational Autoencoder for Graph-Level Representation Learning (PIGAE) | NeurIPS'21 | Link prediction; edge feature prediction | Graph classification | link |
| Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (MGSSL) | NeurIPS'21 | Masked edge feature prediction | Graph classification | link |
| Self-Supervised Graph Representation Learning via Topology Transformations (TopoTER) | TKDE'21 | Masked link prediction | Node classification; graph classification; link prediction | link |
| Directed Graph Auto-Encoders (DiGAE) | AAAI'22 | Link prediction | (Directed) link prediction | link |
| GPPT: Graph Pre-training and Prompt Tuning to Generalize Graph Neural Networks | KDD'22 | Masked link prediction | Node classification | link |
| Link Prediction with Contextualized Self-Supervision (CSSL2) | TKDE'22 | Link prediction | Link prediction | link |
| MGAE: Masked Autoencoders for Self-Supervised Learning on Graphs; S2GAE: Self-Supervised Graph Autoencoders are Generalizable Learners with Graph Masking | WSDM'23 | Masked link prediction | Node classification; graph classification; link prediction | link |
| Dual Low-Rank Graph Autoencoder for Semantic and Topological Networks (DLR-GAE) | AAAI'23 | Link prediction | Node classification | link |
| Heterogeneous Graph Masked Autoencoders (HGMAE) | AAAI'23 | Masked meta-path prediction | (Heterogeneous) node classification; node clustering | link |
| Deep Manifold Graph Auto-Encoder for Attributed Graph Embedding (DMGAE, DMVGAE) | ICASSP'23 | Link prediction | Node clustering; link prediction | -- |
| Multi-head Variational Graph Autoencoder Constrained by Sum-product Networks (SPN-MVGAE) | WWW'23 | Link prediction | Node classification; link prediction | link (Unavailable) |
| SeeGera: Self-supervised Semi-implicit Graph Variational Auto-encoders with Masking | WWW'23 | Masked link prediction | Node classification; link prediction; attribute prediction | link |
| DiP-GNN: Discriminative Pre-Training of Graph Neural Networks | NeurIPS Workshop (GLFrontiers)'23 | Masked link prediction | Node classification; link prediction | -- |
| Maximizing Mutual Information Across Feature and Topology Views for Representing Graphs (MVMI-FT) | TKDE'23 | Link prediction | Node classification; node clustering | link |
| Towards Effective and Robust Graph Contrastive Learning With Graph Autoencoding (AEGCL) | TKDE'23 | Link prediction | Node classification; node clustering; link prediction | link |
| ULTRA-DP: Unifying Graph Pre-training with Multi-task Graph Dual Prompt | arXiv:2310 | Link prediction | Node classification; link prediction | link |
| Incomplete Graph Learning via Attribute-Structure Decoupled Variational Auto-Encoder (ASD-VAE) | WSDM'24 | Edge feature prediction | Node classification; node attribute completion | link |
| Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning (GCMAE) | ICDE'24 | Link prediction | Node classification; node clustering; graph classification; link prediction | link |
| Masked Graph Autoencoder with Non-discrete Bandwidths (Bandana) | WWW'24 | Link denoising | Node classification; link prediction | link |
Context
- Context discrimination: to distinguish between contextual nodes and non-contextual nodes. LE stands for Laplacian Eigenmaps objective
- Factorized context discrimination: to maximize the log-likelihood of context representations given the corresponding central node conditioned on multiple disentangled latent factors
- Contextual subgraph discrimination: to distinguish between representations aggregated from different contextual subgraphs (maybe from different receptive fields). CE stands for cross-entropy
- Neighbor feature prediction: node feature prediction but to reconstruct the features of k-hop neighbors instead (BPR stands for Bayesian Personalized Ranking loss)
- Contextual property prediction: to predict the properties of contextual subgraphs (e.g. node / edge types contained, total node / edge counts, structural coefficient)
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Inductive Representation Learning on Large Graphs (GraphSAGE) | NIPS'17 | Context discrimination (JS) | Node classification | link |
| Strategies for Pre-training Graph Neural Networks (ContextPred) | ICLR'20 | Contextual subgraph discrimination (CE) | Graph classification; biological function prediction | link |
| GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding | ICLR'20 | Contextual subgraph discrimination (CE) | Node classification; link prediction | link |
| GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training | KDD'20 | Contextual subgraph discrimination (InfoNCE) | Node classification; graph classification; similarity search | link |
| Graph Attention Auto-Encoders (GATE) | ICTAI'20 | Context discrimination (JS) | Node classification | link |
| Sub-Graph Contrast for Scalable Self-Supervised Graph Representation Learning (Subg-Con) | ICDM'20 | Context discrimination (Triplet margin) | Node classification | link |
| Self-Supervised Graph Transformer on Large-Scale Molecular Data (GROVER) | NeurIPS'20 | Contextual property prediction | Graph classification; graph regression | link |
| Pre-training on Large-Scale Heterogeneous Graph (PT-HGNN) | KDD'21 | Context discrimination (InfoNCE) | (Heterogeneous) node classification; link prediction | link |
| Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization (EGI) | NeurIPS'21 | Context discrimination (JS) | Role identification; relation prediction | link |
| Contrastive Laplacian Eigenmaps (COLES) | NeurIPS'21 | Context discrimination (LE) | Node classification; node clustering | link |
| Graph-MLP: Node Classification without Message Passing in Graph | arXiv:2106 | Context discrimination (InfoNCE) | Node classification | link |
| Augmentation-Free Self-Supervised Learning on Graphs (AFGRL) | AAAI'22 | Context discrimination (Bootstrapping) | Node classification; node clustering; similarity search | link |
| Simple Unsupervised Graph Representation Learning (SUGRL) | AAAI'22 | Context discrimination (Triplet margin) | Node classification | link |
| SAIL: Self-Augmented Graph Contrastive Learning | AAAI'22 | Neighbor feature prediction (BPR) | Node classification; node clustering; link prediction | -- |
| Robust Self-Supervised Structural Graph Neural Network for Social Network Prediction | WWW'22 | Contextual subgraph discrimination (InfoNCE) | Node classification; graph classification; similarity search | -- |
| Node Representation Learning in Graph via Node-to-Neighbourhood Mutual Information Maximization (N2N) | CVPR'22 | Context discrimination (InfoNCE) | Node classification | link |
| RoSA: A Robust Self-Aligned Framework for Node-Node Graph Contrastive Learning | IJCAI'22 | Contextual subgraph discrimination (InfoNCE) | Node classification | link |
| Graph Auto-Encoder Via Neighborhood Wasserstein Reconstruction (NWR-GAE) | ICLR'22 | Neighbor feature prediction (Wasserstein distance) | Node classification; structural role identification | link |
| Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction (GIANT) | ICLR'22 | Neighbor matching1 | Node classification | link |
| Towards Self-supervised Learning on Graphs with Heterophily (HGRL) | CIKM'22 | Context discrimination (InfoNCE) | Node classification; node clustering | link |
| Unifying Graph Contrastive Learning with Flexible Contextual Scopes (UGCL) | ICDM'22 | Context discrimination (InfoNCE) | Node classification | link |
| Generalized Laplacian Eigenmaps (GLEN) | NeurIPS'22 | Context discrimination (LE) | Node classification; node clustering | link |
| Decoupled Self-supervised Learning for Graphs (DSSL) | NeurIPS'22 | Factorized context discrimination | Node classification | link |
| Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming (G-Zoom) | TNNLS'22 | Context discrimination (JS) | Node classification | -- |
| Link Prediction with Contextualized Self-Supervision (CSSL2) | TKDE'22 | Context discrimination (CE) | Link prediction | link |
| Localized Graph Contrastive Learning (Local-GCL) | arXiv:2212 | Context discrimination (InfoNCE) | Node classification | link |
| Deep Graph Structural Infomax (DGSI) | AAAI'23 | Context discrimination (JS) | Node classification | link |
| Neighbor Contrastive Learning on Learnable Graph Augmentation (NCLA) | AAAI'23 | Context discrimination (InfoNCE) | Node classification | link |
| Eliciting Structural and Semantic Global Knowledge in Unsupervised Graph Contrastive Learning (S3-CL) | AAAI'23 | Contextual subgraph discrimination (InfoNCE) | Node classification; node clustering | link |
| Contrastive Learning Meets Homophily: Two Birds with One Stone (NeCo) | ICML'23 | Context discrimination (InfoNCE) | Node classification | -- |
| Contrastive Cross-scale Graph Knowledge Synergy (CGKS) | KDD'23 | Context discrimination (LE); contextual subgraph discrimination (InfoNCE) | Node classification; graph classification | -- |
| Simple and Asymmetric Graph Contrastive Learning without Augmentations (GraphACL) | NeurIPS'23 | Context discrimination (InfoNCE) | Node classification | link |
| Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks (APT) | NeurIPS'23 | Context discrimination (InfoNCE) | Node classification; graph classification | link |
| Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning (HTML) | AAAI'24 | Contextual property prediction (structural coefficient) | Graph classification | link |
| Graph Contrastive Learning Reimagined: Exploring Universality (ROSEN) | WWW'24 | Context discrimination (InfoNCE) | Node classification; node clustering | -- |
| High-Frequency-aware Hierarchical Contrastive Selective Coding for Representation Learning on Text-attributed Graphs (HASH-CODE) | WWW'24 | Context discrimination (DP); contextual subgraph discrimination (DP) | Node classification; link prediction | -- |
| [Efficient Contrastive Learning for Fast and Accurate Inference on Graphs] (GraphECL) | ICML'24 | Context discrimination (InfoNCE) | Node classification | link |
1GIANT fine-tunes a language model to extract structural node features for GNN input
Long-range similarities
- Similarity prediction: to predict a similarity matrix between nodes. The pairwise similarity can be defined by shortest path distance, PageRank, Katz index, Jaccard coefficient,
$\ell_2$ distance & cosine similarity between output representations / input-output, etc - Masked path prediction: similar to masked link prediction, but the links are masked in paths
- Similarity-based discrimination: instance discrimination that is node similarity-aware
- Similarity graph alignment: to construct an additional similarity graph based on pairwise similarities of node features or graph topology, and minimize the distance of representation distributions between them (the original and similarity graph, or two different similarity graphs)
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Adaptive Graph Encoder for Attributed Graph Embedding (AGE) | KDD'20 | Similarity prediction (cosine similarity) | Node clustering; link prediction | link |
| AM-GCN: Adaptive Multi-channel Graph Convolutional Networks | KDD'20 | Similarity graph alignment (MSE) | Node classification | link |
| Graph-Bert: Only Attention is Needed for Learning Graph Representations | arXiv:2001 | Similarity prediction (PageRank, etc.) | Node classification; node clustering | link |
| Self-supervised Learning on Graphs: Deep Insights and New Direction (PairwiseDistance, PairwiseAttrSim) | arXiv:2006 | Similarity prediction (shortest path distance; cosine similarity) | Node classification | link |
| SAIL: Self-Augmented Graph Contrastive Learning | AAAI'22 | Similarity prediction (cosine similarity) | Node classification; node clustering; link prediction | -- |
| Self-Supervised Graph Representation Learning via Global Context Prediction; A New Self-supervised Task on Graphs: Geodesic Distance Prediction (S2GRL) | Information Sciences'22 | Similarity prediction (shortest path distance) | Node classification; node clustering; link prediction | -- |
| Dual Low-Rank Graph Autoencoder for Semantic and Topological Networks (DLR-GAE) | AAAI'23 | Similarity graph alignment (CE) | Node classification | link |
| Attribute and Structure Preserving Graph Contrastive Learning (ASP) | AAAI'23 | Similarity graph alignment (InfoNCE) | Node classification | link |
| Deep Manifold Graph Auto-Encoder for Attributed Graph Embedding (DMGAE, DMVGAE) | ICASSP'23 | Similarity prediction ( |
Node clustering; link prediction | -- |
| Self-Supervised Teaching and Learning of Representations on Graphs (GraphTL) | WWW'23 | Similarity-based discrimination (InfoNCE) | Node classification | -- |
| Graph Self-supervised Learning via Proximity Divergence Minimization (PDM) | UAI'23 | Similarity prediction (heat kernel, personalized PageRank, SimRank) | Node classification | link |
| What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders (MaskGAE) | KDD'23 | Masked path prediction | Node classification; link prediction | link |
| Maximizing Mutual Information Across Feature and Topology Views for Representing Graphs (MVMI-FT) | TKDE'23 | Similarity graph alignment (JS) | Node classification; node clustering | link |
| Towards Effective and Robust Graph Contrastive Learning With Graph Autoencoding (AEGCL) | TKDE'23 | Similarity graph alignment (InfoNCE) | Node classification; node clustering; link prediction | link |
| ULTRA-DP: Unifying Graph Pre-training with Multi-task Graph Dual Prompt | arXiv:2310 | Similarity prediction (cosine similarity) | Node classification; link prediction | link |
| Deep Contrastive Graph Learning with Clustering-Oriented Guidance (DCGL) | AAAI'24 | Similarity-based discrimination (InfoNCE) | Node clustering | link |
| Generative and Contrastive Paradigms Are Complementary for Graph Self-Supervised Learning (GCMAE) | ICDE'24 | Similarity-based discrimination (InfoNCE) | Node classification; node clustering; graph classification; link prediction | link |
Motifs
- Motif prediction: to assign each node (or supernode in the fragment graph) a motif pseudo-label given by unsupervised motif discovery algorithms (e.g. RDKit) and learn to predict them. It is "autoregressive" if the predicted supernodes are generated one-by-one
- Motif-based masked feature prediction: similar to masked feature prediction, but the features are masked in motifs
- Motif-based discrimination: to perform contrast between the original graph view and the fragment graph view
- Motif adversarial generation: to generate motifs with an adversarial min-max optimizer
| Papers | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Self-Supervised Graph Transformer on Large-Scale Molecular Data (GROVER) | NeurIPS'20 | Contextual property prediction | Graph classification; graph regression | link |
| Motif-Driven Contrastive Learning of Graph Representations (MICRO-Graph) | WWW Workshop (SSL)'21 | Motif-based discrimination (InfoNCE) | Graph classification | link |
| Motif-based Graph Self-Supervised Learning for Molecular Property Prediction (MGSSL) | NeurIPS'21 | Motif prediction (autoregressive) | Graph classification | link |
| Fragment-based Pretraining and Finetuning on Molecular Graphs (GraphFP) | NeurIPS'23 | Motif prediction; motif-based discrimination (InfoNCE) | Graph classification; graph regression | link |
| Motif-aware Attribute Masking for Molecular Graph Pre-training (MoAMa) | NeurIPS Workshop (GLFrontiers)'23 | Motif-based masked feature prediction | Graph classification | link |
| Motif-aware Riemannian Graph Neural Network with Generative-Contrastive Learning (MotifRGC) | AAAI'24 | Motif adversarial generation | Node classification; link prediction | link |
| Empowering Dual-Level Graph Self-Supervised Pretraining with Motif Discovery (DGPM) | AAAI'24 | Motif prediction | Graph classification | link |
Clusters
- Synthetic graph discrimination: binary classification between two synthetic graphs with different synthesizers (Erdลs-Rรฉnyi generator / SBM generator)
- Node clustering: to assign each node a cluster centroid (prototype) and - i) minimize the distance between nodes and their corresponding centroids in the latent space; or ii) minimize the distance between the learned centroids and the ground-truth centroids given by unsupervised feature clustering algorithms (e.g. K-means, DeepCluster)
- Graph partitioning: to assign each node a cluster centroid (prototype) and - i) predict the quality of the learned partitions evaluated by some metrics, e.g. maximizing modularity or minimizing the normalized edge weights of a graph cut (spectral clustering); or ii) predict the cluster membership of each node given by unsupervised graph partitioning algorithms (structure-based, e.g. METIS, Louvain)
- Cluster/partition-based instance discrimination: instance discrimination that is aware of graph clustering/partitioning memberships
- Cluster/partition-conditioned link prediction: to maximize the log-likelihood of existing links, but conditioned by the graph cluster/partition distributions
- Partition-conditioned masked link prediction: similar to masked link prediction, but the links are masked in clusters
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| SGR: Self-Supervised Spectral Graph Representation Learning | KDD Workshop (DLD)'18 | Synthetic graph discrimination | Graph classification | -- |
| Unsupervised Pre-training of Graph Convolutional Networks (ClusterDetect) | ICLR Workshop (RLGM)'19 | Graph partitioning | Node classification | -- |
| Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labeled Nodes (M3S) | AAAI'20 | Node clustering | Node classification | link |
| Collaborative Graph Convolutional Networks: Unsupervised Learning Meets Semi-Supervised Learning (CGCN) | AAAI'20 | Partition-conditioned link prediction | Node classification; node clustering | link (Deleted) |
| When Does Self-Supervision Help Graph Convolutional Networks? (NodeCluster, GraphPar) | ICML'20 | Node clustering; graph partitioning | Node classification | link |
| CommDGI: Community Detection Oriented Deep Graph Infomax | CIKM'20 | Cluster-based instance discrimination (JS); graph partitioning | Node clustering | link |
| Dirichlet Graph Variational Autoencoder (DGVAE) | NeurIPS'20 | Partition-conditioned link prediction | Graph generation; node clustering | link |
| Self-supervised Learning on Graphs: Deep Insights and New Direction (Distance2Clusters) | arXiv:2006 | Graph partitioning | Node classification | link |
| Mask-GVAE: Blind Denoising Graphs via Partition | WWW'21 | Graph partitioning; partition-conditioned masked link prediction | Node clustering; graph denoising | link |
| Motif-Driven Contrastive Learning of Graph Representations (MICRO-Graph) | WWW Workshop (SSL)'21 | Graph partitioning | Graph classification | link |
| Self-supervised Graph-level Representation Learning with Local and Global Structure (GraphLoG) | ICML'21 | Node clustering | Graph classification; biological function prediction | link |
| Graph Communal Contrastive Learning (gCooL) | WWW'22 | Partition-based instance discrimination (InfoNCE) | Node classification; node clustering | link |
| Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering (SHGP) | NeurIPS'22 | Graph partitioning | (Heterogeneous) node classification; node clustering | link |
| Eliciting Structural and Semantic Global Knowledge in Unsupervised Graph Contrastive Learning (S3-CL) | AAAI'23 | Cluster-based instance discrimination (InfoNCE) | Node classification; node clustering | link |
| CSGCL: Community-Strength-Enhanced Graph Contrastive Learning | IJCAI'23 | Partition-based instance discrimination (InfoNCE) | Node classification; node clustering; link prediction | link |
| HomoGCL: Rethinking Homophily in Graph Contrastive Learning | KDD'23 | Node clustering; cluster-based instance discrimination (InfoNCE) | Node classification; node clustering | link |
| CARL-G: Clustering-Accelerated Representation Learning on Graphs | KDD'23 | Node clustering | Node classification; node clustering; similarity search | link |
| Towards Alignment-Uniformity Aware Representation in Graph Contrastive Learning (AUAR) | WSDM'24 | Node clustering | Node classification; node clustering | -- |
| Deep Contrastive Graph Learning with Clustering-Oriented Guidance (DCGL) | AAAI'24 | Cluster-based instance discrimination (InfoNCE) | Node clustering | link |
| StructComp: Substituting propagation with Structural Compression in Training Graph Contrastive Learning | ICLR'24 | Partition-based instance discrimination (JS, InfoNCE, etc.) | Node classification | link |
| MARIO: Model Agnostic Recipe for Improving OOD Generalization of Graph Contrastive Learning | WWW'24 | Cluster-based instance discrimination | Node classification; graph classification | link |
| Graph Contrastive Learning with Kernel Dependence Maximization for Social Recommendation (CL-KDM) | WWW'24 | Partition-based instance discrimination (BPR) | Recommendation | -- |
| Community-Invariant Graph Contrastive Learning (CI-GCL) | ICML'24 | Partition-based instance discrimination (InfoNCE) | Graph classification; graph regression | link |
Global structure
- Global-local instance discrimination: instance discrimination between the representation of each node and a global representation vector, usually aggregated from the whole graph by a readout function
- Group discrimination: a simplified global-local instance discrimination that binarily classifies if a node belongs to the original or the perturbed graph
- Global instance discrimination: to discriminate between global representations of different graph views (small-scale only)
- Global dimension discrimination: dimension discrimination of different graph representations
- Factorized global instance discrimination: to maximize the log-likelihood of the target graph instance given the corresponding instance conditioned on multiple disentangled latent factors
- Graph similarity prediction: to predict various kinds of similarity functions between pairs of graphs, e.g. graph kernels (graphlet kernel, random walk kernel, graph edit distance kernel, etc)
- Half-graph matching: to divide each graph into two halves and predict if two halves are from the same original graph
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Pre-training Graph Neural Networks with Kernels (KernelPred) | arXiv:1811 | Graph similarity prediction | Graph classification | -- |
| Deep Graph InfoMax (DGI) | ICLR'19 | Global-local instance discrimination (JS) | Node classification | link |
| InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization | ICLR'20 | Global-local instance discrimination (JS) | Graph classification | link |
| Graph Contrastive Learning with Augmentations (GraphCL) | NeurIPS'20 | Global instance discrimination (InfoNCE) | Graph classification | link |
| Contrastive Multi-View Representation Learning on Graphs (MVGRL) | ICML'20 | Global-local instance discrimination (JS) | Node classification; graph classification | link |
| Contrastive Self-supervised Learning for Graph Classification (CSSL1) | AAAI'21 | Global instance discrimination (InfoNCE) | Graph classification | -- |
| Pairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks (PHD); An Effective Self-Supervised Framework for Learning Expressive Molecular Global Representations to Drug Discovery (MPG) | IJCAI'21; Briefings in Bioinformatics'21 | Half-graph matching | Graph classification | link |
| Graph Contrastive Learning Automated (JOAO) | ICML'21 | Global instance discrimination (InfoNCE) | Graph classification | link |
| Adversarial Graph Augmentation to Improve Graph Contrastive Learning (AD-GCL) | NeurIPS'21 | Global instance discrimination (InfoNCE) | Graph classification | link |
| InfoGCL: Information-Aware Graph Contrastive Learning | NeurIPS'21 | Global instance discrimination (Bootstrapping); global-local instance discrimination (Bootstrapping) | Node classification; graph classification | -- |
| Graph Adversarial Self-Supervised Learning (GASSL) | NeurIPS'21 | Global instance discrimination (Bootstrapping) | Graph classification | link (Unavailable) |
| Disentangled Contrastive Learning on Graphs (DGCL) | NeurIPS'21 | Factorized global instance discrimination | Graph classification | link |
| Bringing Your Own View: Graph Contrastive Learning without Prefabricated Data Augmentations (GraphCL-LP) | WSDM'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
| Self-Supervised Graph Neural Networks via Diverse and Interactive Message Passing (DIMP) | AAAI'22 | Global-local instance discrimination (JS) | Node classification; node clustering; graph classification | link |
| AutoGCL: Automated Graph Contrastive Learning via Learnable View Generators | AAAI'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
| Group Contrastive Self-Supervised Learning on Graphs (GroupCL; GroupIG) | TPAMI'22 | Global instance discrimination (JS; contrastive log-ratio upper bound (CLUB)) | Graph classification | -- |
| Towards Graph Self-Supervised Learning with Contrastive Adjusted Zooming (G-Zoom) | TNNLS'22 | Global-local instance discrimination (JS) | Node classification | -- |
| SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation | WWW'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
| Let Invariant Rationale Discovery Inspire Graph Contrastive Learning (RGCL) | ICML'22 | Global instance discrimination (InfoNCE) | Graph classification | link |
| M-Mix: Generating Hard Negatives via Multi-sample Mixing for Contrastive Learning | KDD'22 | Global instance discrimination (InfoNCE) | Node classification; node clustering; graph classification; graph edit distance prediction | link |
| AdaGCL: Adaptive Subgraph Contrastive Learning to Generalize Large-scale Graph Training | CIKM'22 | Global-local instance discrimination (JS) | Node classification | link |
| Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination (GGD) | NeurIPS'22 | Group discrimination | Node classification | link |
| Graph Self-supervised Learning with Accurate Discrepancy Learning (D-SLA) | NeurIPS'22 | Group discrimination; graph similarity prediction | Graph classification; link prediction | link |
| Deep Graph Structural Infomax (DGSI) | AAAI'23 | Global-local instance discrimination (JS) | Node classification | link |
| Spectral Augmentation for Self-Supervised Learning on Graphs (SPAN) | ICLR'23 | Global-local instance discrimination (InfoNCE) | Node classification; graph classification; graph regression | link |
| Mole-BERT: Rethinking Pre-training Graph Neural Networks for Molecules | ICLR'23 | Global instance discrimination (InfoNCE; Triplet margin) | Graph classification; graph regression | link |
| Spectral Augmentations for Graph Contrastive Learning (SGCL1) | AISTATS'23 | Global instance discrimination (InfoNCE) | Node classification; graph classification; similarity search | -- |
| Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning (CGC) | WWW'23 | Global instance discrimination (InfoNCE) | Graph classification | link (Unavailable) |
| Multi-Scale Subgraph Contrastive Learning (MSSGCL) | IJCAI'23 | Global-local instance discrimination (InfoNCE); global instance discrimination (InfoNCE) | Graph classification | link |
| Boosting Graph Contrastive Learning via Graph Contrastive Saliency (GCS) | ICML'23 | Global instance discrimination (InfoNCE) | Graph classification | link |
| SEGA: Structural Entropy Guided Anchor View for Graph Contrastive Learning | ICML'23 | Global instance discrimination (InfoNCE) | Graph classification | link |
| Randomized Schur Complement Views for Graph Contrastive Learning (rLap) | ICML'23 | Global-local instance discrimination (InfoNCE); global instance discrimination (InfoNCE) | Graph classification | link |
| Graph Self-Contrast Representation Learning (GraphSC) | ICDM'23 | Global instance discrimination (Triplet margin); global dimension discrimination | Graph classification | -- |
| Graph Contrastive Learning with Stable and Scalable Spectral Encoding (Sp2GCL) | NeurIPS'23 | Global instance discrimination (InfoNCE) | Node classification; graph classification; graph regression | link |
| Maximizing Mutual Information Across Feature and Topology Views for Representing Graphs (MVMI-FT) | TKDE'23 | Global-local instance discrimination (JS) | Node classification; node clustering | link |
| Hierarchically Contrastive Hard Sample Mining for Graph Self-Supervised Pretraining (HCHSM) | TNNLS'23 | Global-local instance discrimination (JS) | Node classification; node clustering | link |
| HeGCL: Advance Self-Supervised Learning in Heterogeneous Graph-Level Representation | TNNLS'23 | Global-local instance discrimination (JS) | (Heterogeneous) node classification; graph classification | link |
| Hierarchical Topology Isomorphism Expertise Embedded Graph Contrastive Learning (HTML) | AAAI'24 | Global instance discrimination (InfoNCE); graph similarity prediction (Jaccard coef-based isomorphic similarity) | Graph classification | link |
| TopoGCL: Topological Graph Contrastive Learning | AAAI'24 | Global instance discrimination (InfoNCE) | Graph classification | link (Unavailable) |
| Graph Contrastive Learning with Cohesive Subgraph Awareness (CTAug) | WWW'24 | Global instance discrimination (InfoNCE) | Graph classification | link |
| Graph Contrastive Learning with Personalized Augmentation (GPA) | TKDE'24 | Global instance discrimination (InfoNCE) | Graph classification | link |
Manifolds
- Cross-manifold discrimination: to perform instance discrimination between different manifolds (e.g. Euclidean vs. Hyperbolic)
- Ricci curvature prediction: to predict the aggregated Ricci curvature of each node's neighborhood
- Curvature-based node clustering: to assign each node a cluster centroid and maximize/minimize the curvature-based density within/across clusters
- Hyperbolic angle prediction: to pool representations to 2-dimensional angle vectors in a unit hyperbola. These vectors serve as pseudo-labels for regression
| Paper | Venue | Pretext | Downstream | Code |
|---|---|---|---|---|
| Enhancing Hyperbolic Graph Embeddings via Contrastive Learning (HGCL) | NeurIPS Workshop (SSL)'21 | Cross-manifold discrimination (InfoNCE) | Node classification | -- |
| A Self-supervised Mixed-curvature Graph Neural Network (SelfMGNN) | AAAI'22 | Cross-manifold discrimination (InfoNCE) | Node classification | -- |
| Dual Space Graph Contrastive Learning (DSGC) | WWW'22 | Cross-manifold discrimination (InfoNCE) | Graph classification | link (Unavailable) |
| CONGREGATE: Contrastive Graph Clustering in Curvature Spaces | IJCAI'23 | Ricci curvature prediction; cross-manifold discrimination (InfoNCE); curvature-based node clustering | Node clustering | link |
| Graph-level Representation Learning with Joint-Embedding Predictive Architectures (GraphJEPA) | arXiv:2309 | Hyperbolic angle prediction | Graph classification; graph regression | link |
| Motif-aware Riemannian Graph Neural Network with Generative-Contrastive Learning (MotifRGC) | AAAI'24 | Cross-manifold discrimination (InfoNCE) | Node classification; link prediction | link |
Task adaptation strategies
- Multi-task learning: to combine different pretexts and jointly learn them for task-generalizable performance
- Fine-tuning: to jointly learn downstream branches as well as the original pre-trained model. Parameter-efficient fine-tuning (PEFT) only updates part of the pre-trained model, e.g. adapter layers
- Probing: to freeze the parameters of the pre-trained model during the downstream task adaptation. The probe can be either parameterized or non-parameterized
- Prompting: to encode downstream data and the corresponding task information as tokens to instruct the behavior of pre-trained models for downstream adaptation
1Can be seen as a GNN probing method prompted by the pre-trained LLM
โค๏ธ Contributions by issues and pull requests to this source list are always welcome! Feel free to initiate a discussion with me, or give me a reminder if there are oversights of papers/hyperlinks or categorical mistakes.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent