arXiv subjects

Andreas Roth

Publications and source records attributed to Andreas Roth.

17 recordsLinked to original sources

arXiv2026

Towards Understanding and Avoiding Limitations of Convolutions on Graphs

While message-passing neural networks (MPNNs) have shown promising results, their real-world impact remains limited. Although various limitations have been identified, their theoretical foundations remain poorly understood, leading to fragmented research efforts. In this thesis, we provide an in-depth theoretical analysis and identify several key properties limiting the performance of MPNNs. Building on these findings, we propose several frameworks that address these shortcomings. We identify two properties exhibited by many MPNNs: shared component amplification (SCA), where each message-passing iteration amplifies the same components across all feature channels, and component dominance (CD), where a single component gets increasingly amplified as more message-passing steps are applied. These properties lead to the observable phenomenon of rank collapse of node representations, which generalizes the established over-smoothing phenomenon. By generalizing and decomposing over-smoothing, we enable a deeper understanding of MPNNs, more targeted solutions, and more precise communication within the field. To avoid SCA, we show that utilizing multiple computational graphs or edge relations is necessary. Our multi-relational split (MRS) framework transforms any existing MPNN into one that leverages multiple edge relations. Additionally, we introduce the spectral graph convolution for multiple feature channels (MIMO-GC), which naturally uses multiple computational graphs. A localized variant, LMGC, approximates the MIMO-GC while inheriting its beneficial properties. To address CD, we demonstrate a close connection between MPNNs and the PageRank algorithm. Based on personalized PageRank, we propose a variant of MPNNs that allows for infinitely many message-passing iterations, while preserving initial node features. Collectively, these results deepen the theoretical understanding of MPNNs.

cs.LG↗

arXiv2025

What Can We Learn From MIMO Graph Convolutions?

Most graph neural networks (GNNs) utilize approximations of the general graph convolution derived in the graph Fourier domain. While GNNs are typically applied in the multi-input multi-output (MIMO) case, the approximations are performed in the single-input single-output (SISO) case. In this work, we first derive the MIMO graph convolution through the convolution theorem and approximate it directly in the MIMO case. We find the key MIMO-specific property of the graph convolution to be operating on multiple computational graphs, or equivalently, applying distinct feature transformations for each pair of nodes. As a localized approximation, we introduce localized MIMO graph convolutions (LMGCs), which generalize many linear message-passing neural networks. For almost every choice of edge weights, we prove that LMGCs with a single computational graph are injective on multisets, and the resulting representations are linearly independent when more than one computational graph is used. Our experimental results confirm that an LMGC can combine the benefits of various methods.

cs.LG↗

arXiv2024

Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graph

The ability of message-passing neural networks (MPNNs) to fit complex functions over graphs is limited as most graph convolutions amplify the same signal across all feature channels, a phenomenon known as rank collapse, and over-smoothing as a special case. Most approaches to mitigate over-smoothing extend common message-passing schemes, e.g., the graph convolutional network, by utilizing residual connections, gating mechanisms, normalization, or regularization techniques. Our work contrarily proposes to directly tackle the cause of this issue by modifying the message-passing scheme and exchanging different types of messages using multi-relational graphs. We identify a sufficient condition to ensure linearly independent node representations. As one instantion, we show that operating on multiple directed acyclic graphs always satisfies our condition and propose to obtain these by defining a strict partial ordering of the nodes. We conduct comprehensive experiments that confirm the benefits of operating on multi-relational graphs to achieve more informative node representations.

cs.LG↗

arXiv2024

Simplifying the Theory on Over-Smoothing

Graph convolutions have gained popularity due to their ability to efficiently operate on data with an irregular geometric structure. However, graph convolutions cause over-smoothing, which refers to representations becoming more similar with increased depth. However, many different definitions and intuitions currently coexist, leading to research efforts focusing on incompatible directions. This paper attempts to align these directions by showing that over-smoothing is merely a special case of power iteration. This greatly simplifies the existing theory on over-smoothing, making it more accessible. Based on the theory, we provide a novel comprehensive definition of rank collapse as a generalized form of over-smoothing and introduce the rank-one distance as a corresponding metric. Our empirical evaluation of 14 commonly used methods shows that more models than were previously known suffer from this issue.

cs.LG↗

arXiv2023

Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks

Models with similar performances exhibit significant disagreement in the predictions of individual samples, referred to as prediction churn. Our work explores this phenomenon in graph neural networks by investigating differences between models differing only in their initializations in their utilized features for predictions. We propose a novel metric called Influence Difference (ID) to quantify the variation in reasons used by nodes across models by comparing their influence distribution. Additionally, we consider the differences between nodes with a stable and an unstable prediction, positing that both equally utilize different reasons and thus provide a meaningful gradient signal to closely match two models even when the predictions for nodes are similar. Based on our analysis, we propose to minimize this ID in Knowledge Distillation, a domain where a new model should closely match an established one. As an efficient approximation, we introduce DropDistillation (DD) that matches the output for a graph perturbed by edge deletions. Our empirical evaluation of six benchmark datasets for node classification validates the differences in utilized features. DD outperforms previous methods regarding prediction stability and overall performance in all considered Knowledge Distillation experiments.

cs.LG↗

arXiv2023

Curvature-based Pooling within Graph Neural Networks

Over-squashing and over-smoothing are two critical issues, that limit the capabilities of graph neural networks (GNNs). While over-smoothing eliminates the differences between nodes making them indistinguishable, over-squashing refers to the inability of GNNs to propagate information over long distances, as exponentially many node states are squashed into fixed-size representations. Both phenomena share similar causes, as both are largely induced by the graph topology. To mitigate these problems in graph classification tasks, we propose CurvPool, a novel pooling method. CurvPool exploits the notion of curvature of a graph to adaptively identify structures responsible for both over-smoothing and over-squashing. By clustering nodes based on the Balanced Forman curvature, CurvPool constructs a graph with a more suitable structure, allowing deeper models and the combination of distant information. We compare it to other state-of-the-art pooling approaches and establish its competitiveness in terms of classification accuracy, computational complexity, and flexibility. CurvPool outperforms several comparable methods across all considered tasks. The most consistent results are achieved by pooling densely connected clusters using the sum aggregation, as this allows additional information about the size of each pool.

cs.LG↗

arXiv2023

Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks

Our study reveals new theoretical insights into over-smoothing and feature over-correlation in graph neural networks. Specifically, we demonstrate that with increased depth, node representations become dominated by a low-dimensional subspace that depends on the aggregation function but not on the feature transformations. For all aggregation functions, the rank of the node representations collapses, resulting in over-smoothing for particular aggregation functions. Our study emphasizes the importance for future research to focus on rank collapse rather than over-smoothing. Guided by our theory, we propose a sum of Kronecker products as a beneficial property that provably prevents over-smoothing, over-correlation, and rank collapse. We empirically demonstrate the shortcomings of existing models in fitting target functions of node classification tasks.

cs.LG↗

arXiv2022

Forecasting Unobserved Node States with spatio-temporal Graph Neural Networks

Forecasting future states of sensors is key to solving tasks like weather prediction, route planning, and many others when dealing with networks of sensors. But complete spatial coverage of sensors is generally unavailable and would practically be infeasible due to limitations in budget and other resources during deployment and maintenance. Currently existing approaches using machine learning are limited to the spatial locations where data was observed, causing limitations to downstream tasks. Inspired by the recent surge of Graph Neural Networks for spatio-temporal data processing, we investigate whether these can also forecast the state of locations with no sensors available. For this purpose, we develop a framework, named Forecasting Unobserved Node States (FUNS), that allows forecasting the state at entirely unobserved locations based on spatio-temporal correlations and the graph inductive bias. FUNS serves as a blueprint for optimizing models only on observed data and demonstrates good generalization capabilities for predicting the state at entirely unobserved locations during the testing stage. Our framework can be combined with any spatio-temporal Graph Neural Network, that exploits spatio-temporal correlations with surrounding observed locations by using the network's graph structure. Our employed model builds on a previous model by also allowing us to exploit prior knowledge about locations of interest, e.g. the road type. Our empirical evaluation of both simulated and real-world datasets demonstrates that Graph Neural Networks are well-suited for this task.

cs.LG↗

arXiv2022

Transforming PageRank into an Infinite-Depth Graph Neural Network

Popular graph neural networks are shallow models, despite the success of very deep architectures in other application domains of deep learning. This reduces the modeling capacity and leaves models unable to capture long-range relationships. The primary reason for the shallow design results from over-smoothing, which leads node states to become more similar with increased depth. We build on the close connection between GNNs and PageRank, for which personalized PageRank introduces the consideration of a personalization vector. Adopting this idea, we propose the Personalized PageRank Graph Neural Network (PPRGNN), which extends the graph convolutional network to an infinite-depth model that has a chance to reset the neighbor aggregation back to the initial state in each iteration. We introduce a nicely interpretable tweak to the chance of resetting and prove the convergence of our approach to a unique solution without placing any constraints, even when taking infinitely many neighbor aggregations. As in personalized PageRank, our result does not suffer from over-smoothing. While doing so, time complexity remains linear while we keep memory complexity constant, independently of the depth of the network, making it scale well to large graphs. We empirically show the effectiveness of our approach for various node and graph classification tasks. PPRGNN outperforms comparable methods in almost all cases.

cs.LG↗

arXiv2020

Medley2K: A Dataset of Medley Transitions

The automatic generation of medleys, i.e., musical pieces formed by different songs concatenated via smooth transitions, is not well studied in the current literature. To facilitate research on this topic, we make available a dataset called Medley2K that consists of 2,000 medleys and 7,712 labeled transitions. Our dataset features a rich variety of song transitions across different music genres. We provide a detailed description of this dataset and validate it by training a state-of-the-art generative model in the task of generating transitions between songs.

cs.SD↗

arXiv2019

Instantaneous control of interacting particle systems in the mean-field limit

Controlling large particle systems in collective dynamics by a few agents is a subject of high practical importance, e.g., in evacuation dynamics. In this paper we study an instantaneous control approach to steer an interacting particle system into a certain spatial region by repulsive forces from a few external agents, which might be interpreted as shepherd dogs leading sheep to their home. We introduce an appropriate mathematical model and the corresponding optimization problem. In particular, we are interested in the interaction of numerous particles, which can be approximated by a mean-field equation. Due to the high-dimensional phase space this will require a tailored optimization strategy. The arising control problems are solved using adjoint information to compute the descent directions. Numerical results on the microscopic and the macroscopic level indicate the convergence of optimal controls and optimal states in the mean-field limit,i.e., for an increasing number of particles.

math.OC↗

arXiv2016

Controlling a self-organizing system of individuals guided by a few external agents -- particle description and mean-field limit

Optimal control of large particle systems with collective dynamics by few agents is a subject of high practical importance (e.g. in evacuation dynamics), but still limited mathematical basis. In particular the transition from discrete optimal control to a continuum setting as the number of particles tends to infinity is by far not fully understood. In this paper we contribute to this issue by studying a canonical model of controlling an interacting particle system into a certain spatial region by repulsive forces from few external agents, which might be interpreted as shepherd dogs leading sheep to their home. We discuss the appropriate modelling of such a problem and the associated optimality systems, providing some connections between the Lagrange multipliers in the discrete and continuum setting. As control strategies we investigate an Instantaneous Control and a global Optimal Control approach. The solutions of a family of control problems for the particle system with external agents are numerically compared to the mean-field controls as the number of particles tends to infinity. In both cases, this leads to a high dimensional phase space requiring tailored optimization strategies. All control problems arising are solved using adjoint information to compute the descent directions. The numerical results indicate the convergence of controls for both optimization strategies.

math.OC↗

arXiv2015

First-order quarter- and mixed-moment realizability theory and Kershaw closures for a Fokker-Planck equation in two space dimensions

Mixed-moment models, introduced before for one space dimension, are a modification of the method of moments applied to a (linear) kinetic equation, by choosing mixtures of different partial moments. They are well-suited to handle such equations where collisions of particles are modelled with a Laplace-Beltrami operator. We generalize the concept of mixed moments to two dimension. The resulting hyperbolic system of equations has desirable properties, removing some drawbacks of the well-known $\MN[1]$ model. We furthermore provide a realizability theory for a first-order system of mixed moments by linking it to the corresponding quarter-moment theory. Additionally, we derive a type of Kershaw closures for mixed- and quarter-moment models, giving an efficient closure (compared to minimum-entropy models). The derived closures are investigated for different benchmark problems.

math.AP↗

arXiv2015

A Retarded Mean-Field Approach for Interacting Fiber Structures

We consider an interacting system of one-dimensional structures modelling fibers with fiber-fiber interaction in a fiber lay-down process. The resulting microscopic system is investigated by looking at different asymptotic limits of the corresponding stochastic model. Equations arising from mean-field and diffusion limits are considered. Furthermore, numerical methods for the stochastic system and its mean-field counterpart are discussed. A numerical comparison of solutions corresponding to the different scales (microscopic, mesoscopic and macroscopic) is included.

math.DS↗

arXiv2011

Spin polarization of the quantum spin Hall edge states

While the helical character of the edge channels responsible for charge transport in the quantum spin Hall regime of a two-dimensional topological insulator is by now well established, an experimental confirmation that the transport in the edge channels is spin-polarized is still outstanding. We report experiments on nanostructures fabricated from HgTe quantum wells with an inverted band structure, in which a split gate technique allows us to combine both quantum spin Hall and metallic spin Hall transport in a single device. In these devices, the quantum spin Hall effect can be used as a spin current injector and detector for the metallic spin Hall effect, and vice versa, allowing for an all-electrical detection of spin polarization.

cond-mat.mes-hall↗

arXiv2009

Nonlocal edge state transport in the quantum spin Hall state

We present direct experimental evidence for nonlocal transport in HgTe quantum wells in the quantum spin Hall regime, in the absence of any external magnetic field. The data conclusively show that the non-dissipative quantum transport occurs through edge channels, while the contacts lead to equilibration between the counter-propagating spin states at the edge. We show that the experimental data agree quantitatively with the theory of the quantum spin Hall effect.

cond-mat.mes-hall↗

arXiv2007

Quantum Spin Hall Insulator State in HgTe Quantum Wells

Recent theory predicted that the Quantum Spin Hall Effect, a fundamentally novel quantum state of matter that exists at zero external magnetic field, may be realized in HgTe/(Hg,Cd)Te quantum wells. We have fabricated such sample structures with low density and high mobility in which we can tune, through an external gate voltage, the carrier conduction from n-type to the p-type, passing through an insulating regime. For thin quantum wells with well width d < 6.3 nm, the insulating regime shows the conventional behavior of vanishingly small conductance at low temperature. However, for thicker quantum wells (d > 6.3 nm), the nominally insulating regime shows a plateau of residual conductance close to 2e^2/h. The residual conductance is independent of the sample width, indicating that it is caused by edge states. Furthermore, the residual conductance is destroyed by a small external magnetic field. The quantum phase transition at the critical thickness, d = 6.3 nm, is also independently determined from the magnetic field induced insulator to metal transition. These observations provide experimental evidence of the quantum spin Hall effect.

cond-mat.mes-hall↗