arXiv Science⌕ Search

arXiv subjects

Hemanth Saratchandran

Publications and source records attributed to Hemanth Saratchandran.

At least 55 records · Page 3Linked to original sources
arXiv2024

A Sampling Theory Perspective on Activations for Implicit Neural Representations

Implicit Neural Representations (INRs) have gained popularity for encoding signals as compact, differentiable entities. While commonly using techniques like Fourier positional encodings or non-traditional activation functions (e.g., Gaussian, sinusoid, or wavelets) to capture high-frequency content, their properties lack exploration within a unified theoretical framework. Addressing this gap, we conduct a comprehensive analysis of these activations from a sampling theory perspective. Our investigation reveals that sinc activations, previously unused in conjunction with INRs, are theoretically optimal for signal encoding. Additionally, we establish a connection between dynamical systems and INRs, leveraging sampling theory to bridge these two paradigms.

cs.LG↗
arXiv2024

Analyzing the Neural Tangent Kernel of Periodically Activated Coordinate Networks

Recently, neural networks utilizing periodic activation functions have been proven to demonstrate superior performance in vision tasks compared to traditional ReLU-activated networks. However, there is still a limited understanding of the underlying reasons for this improved performance. In this paper, we aim to address this gap by providing a theoretical understanding of periodically activated networks through an analysis of their Neural Tangent Kernel (NTK). We derive bounds on the minimum eigenvalue of their NTK in the finite width setting, using a fairly general network architecture which requires only one wide layer that grows at least linearly with the number of data samples. Our findings indicate that periodically activated networks are \textit{notably more well-behaved}, from the NTK perspective, than ReLU activated networks. Additionally, we give an application to the memorization capacity of such networks and verify our theoretical predictions empirically. Our study offers a deeper understanding of the properties of periodically activated neural networks and their potential in the field of deep learning.

cs.LG↗
arXiv2024

Architectural Strategies for the optimization of Physics-Informed Neural Networks

Physics-informed neural networks (PINNs) offer a promising avenue for tackling both forward and inverse problems in partial differential equations (PDEs) by incorporating deep learning with fundamental physics principles. Despite their remarkable empirical success, PINNs have garnered a reputation for their notorious training challenges across a spectrum of PDEs. In this work, we delve into the intricacies of PINN optimization from a neural architecture perspective. Leveraging the Neural Tangent Kernel (NTK), our study reveals that Gaussian activations surpass several alternate activations when it comes to effectively training PINNs. Building on insights from numerical linear algebra, we introduce a preconditioned neural architecture, showcasing how such tailored architectures enhance the optimization process. Our theoretical findings are substantiated through rigorous validation against established PDEs within the scientific literature.

cs.LG↗
arXiv2023

On skip connections and normalisation layers in deep optimisation

We introduce a general theoretical framework, designed for the study of gradient optimisation of deep neural networks, that encompasses ubiquitous architecture choices including batch normalisation, weight normalisation and skip connections. Our framework determines the curvature and regularity properties of multilayer loss landscapes in terms of their constituent layers, thereby elucidating the roles played by normalisation layers and skip connections in globalising these properties. We then demonstrate the utility of this framework in two respects. First, we give the only proof of which we are aware that a class of deep neural networks can be trained using gradient descent to global optima even when such optima only exist at infinity, as is the case for the cross-entropy cost. Second, we identify a novel causal mechanism by which skip connections accelerate training, which we verify predictively with ResNets on MNIST, CIFAR10, CIFAR100 and ImageNet.

cs.LG↗
arXiv2023

Curvature-Aware Training for Coordinate Networks

Coordinate networks are widely used in computer vision due to their ability to represent signals as compressed, continuous entities. However, training these networks with first-order optimizers can be slow, hindering their use in real-time applications. Recent works have opted for shallow voxel-based representations to achieve faster training, but this sacrifices memory efficiency. This work proposes a solution that leverages second-order optimization methods to significantly reduce training times for coordinate networks while maintaining their compressibility. Experiments demonstrate the effectiveness of this approach on various signal modalities, such as audio, images, videos, shape reconstruction, and neural radiance fields.

cs.CV↗
arXiv2023

On the effectiveness of neural priors in modeling dynamical systems

Modelling dynamical systems is an integral component for understanding the natural world. To this end, neural networks are becoming an increasingly popular candidate owing to their ability to learn complex functions from large amounts of data. Despite this recent progress, there has not been an adequate discussion on the architectural regularization that neural networks offer when learning such systems, hindering their efficient usage. In this paper, we initiate a discussion in this direction using coordinate networks as a test bed. We interpret dynamical systems and coordinate networks from a signal processing lens, and show that simple coordinate networks with few layers can be used to solve multiple problems in modelling dynamical systems, without any explicit regularizers.

cs.LG↗
arXiv2022

How You Start Matters for Generalization

Characterizing the remarkable generalization properties of over-parameterized neural networks remains an open problem. In this paper, we promote a shift of focus towards initialization rather than neural architecture or (stochastic) gradient descent to explain this implicit regularization. Through a Fourier lens, we derive a general result for the spectral bias of neural networks and show that the generalization of neural networks is heavily tied to their initialization. Further, we empirically solidify the developed theoretical insights using practical, deep networks. Finally, we make a case against the controversial flat-minima conjecture and show that Fourier analysis grants a more reliable framework for understanding the generalization of neural networks.

cs.LG↗
arXiv2022

A new higher order Yang--Mills--Higgs flow in Riemannian $4$-manifold

Let $(M,g)$ be a closed Riemannian $4$-manifold and let $E$ be a vector bundle over $M$ with structure group $G$, where $G$ is a compact Lie group. In this paper, we consider a new higher order Yang--Mills--Higgs functional, in which the Higgs field is a section of $Ω^0(\textmd{ad}E)$. We show that, under suitable conditions, solutions to the gradient flow do not hit any finite time singularities. In the case that $E$ is a line bundle, we are able to use a different blow up procedure and obtain an improvement of the long time result in \cite{Z1}. The proof is rather relevant to the properties of the Green function, which is very different from the previous techniques in \cite{Ke,Sa,Z1}.

math.DG↗
arXiv2021

Realisations of elliptic operators on compact manifolds with boundary

This paper investigates realisations of elliptic differential operators of general order on manifolds with boundary following the approach of Bär-Ballmann to first order elliptic operators. The space of possible boundary values of elements in the maximal domain is described as a Hilbert space densely sandwiched between two mixed order Sobolev spaces. The description uses Calderón projectors which, in the first order case, is equivalent to results of Bär-Bandara using spectral projectors of an adapted boundary operator. Boundary conditions that induce Fredholm as well as regular realisations, and those that admit higher order regularity, are characterised. In addition, results concerning spectral theory, homotopy invariance of the Fredholm index, and well-posedness for higher order elliptic boundary value problems are proven.

math.AP↗
arXiv2021

Generalized Ornstein--Uhlenbeck Semigroups in weighted $L^p$-spaces on Riemannian Manifolds

Let $\mathcal{E}$ be a Hermitian vector bundle over a Riemannian manifold $M$ with metric $g$, let $\nabla$ be a metric covariant derivative on $\mathcal{E}$. We study the generalized Ornstein-Uhlenbeck differential expression $P^{\nabla}=\nabla^{\dagger}\nabla u+\nabla_{(dϕ)^{\sharp}}u-\nabla_{X}u+Vu$, where $\nabla^{\dagger}$ is the formal adjoint of $\nabla$, $(dϕ)^{\sharp}$ is the vector field corresponding to $dϕ$ via $g$, $X$ is a smooth real vector field on $M$, and $V$ is a self-adjoint locally integrable section of the bundle $\textrm{End }\mathcal{E}$. We show that (the negative of) the maximal realization $-H_{p,\max}$ of $P^{\nabla}$ generates an analytic quasi-contractive semigroup in $L^p_μ(\mathcal{E})$, $1<p<\infty$, where $dμ=e^{-ϕ}dν_{g}$, with $ν_{g}$ being the volume measure. Additionally, we describe a Feynman-Kac representation for the semigroup generated by $-H_{p,\max}$. For the Ornstein-Uhlenbeck differential expression acting on functions, that is, $P^{d}=Δu+(dϕ)^{\sharp}u-Xu+Vu$, where $Δ$ is the (non-negative) scalar Laplacian on $M$ and $V$ is a locally integrable real-valued function, we consider another way of realizing $P^{d}$ as an operator in $L^p_μ(M)$ and, by imposing certain geometric conditions on $M$, we prove another semigroup generation result.

math.AP↗
arXiv2021

On the Chern character in Higher Twisted K-theory and spherical T-duality

In this paper, we construct for higher twists that arise from cohomotopy classes, the Chern character in higher twisted K-theory, that maps into higher twisted cohomology. We show that it gives rise to an isomorphism between higher twisted K-theory and higher twisted cohomology over the reals. Finally we compute spherical T-duality in higher twisted K-theory and higher twisted cohomology in very general cases.

math.DG↗
arXiv2020

Essential self-adjointness of perturbed biharmonic operators via conformally transformed metrics

We give sufficient conditions for the essential self-adjointness of perturbed biharmonic operators acting on sections of a Hermitian vector bundle over a Riemannian manifold with additional assumptions, such as lower semi-bounded Ricci curvature or bounded sectional curvature. In the case of lower semi-bounded Ricci curvature, we formulate our results in terms of the completeness of the metric that is conformal to the original one, via a conformal factor that depends on a minorant of the perturbing potential $V$. In the bounded sectional curvature situation, we are able to relax the growth condition on the minorant of $V$ imposed in an earlier article. In this context, our growth condition on the minorant of $V$ is consistent with the literature on the self-adjointness of perturbed biharmonic operators on $\mathbb{R}^n$.

math.AP↗
arXiv2019

Essential self-adjointness of perturbed quadharmonic operators on Riemannian manifolds with an application to the separation problem

We consider perturbed quadharmonic operators, $Δ^4 + V$, acting on sections of a Hermitian vector bundle over a complete Riemannian manifold, with the potential $V$ satisfying a bound from below by a non-positive function depending on the distance from a point. Under a bounded geometry assumption on the Hermitian vector bundle and the underlying Riemannian manifold, we give a sufficient condition for the essential self-adjointness of such operators. We then apply this to prove the separation property in $L^2$ when the perturbed operator acts on functions.

math.DG↗
arXiv2018

Inequalities and separation for covariant Schrödinger operators

We consider a differential expression $L^{\nabla}_{V}=\nabla^{\dagger}\nabla+V$, where $\nabla$ is a metric covariant derivative on a Hermitian bundle $E$ over a geodesically complete Riemannian manifold $(M,g)$ with metric $g$, and $V$ is a linear self-adjoint bundle map on $E$. In the language of Everitt and Giertz, the differential expression $L^{\nabla}_{V}$ is said to be separated in $L^p(E)$ if for all $u\in L^p(E)$ such that $L^{\nabla}_{V}u\in L^p(E)$, we have $Vu\in L^p(E)$. We give sufficient conditions for $L^{\nabla}_{V}$ to be separated in $L^2(E)$. We then study the problem of separation of $L^{\nabla}_{V}$ in the more general $L^p$-spaces, and give sufficient conditions for $L^{\nabla}_{V}$ to be separated in $L^p(E)$, when $1<p<\infty$.

math.AP↗
arXiv2018

Higher order Seiberg-Witten functionals and their associated gradient flows

We define functionals generalising the Seiberg-Witten functional on closed $spin^c$ manifolds, involving higher order derivatives of the curvature form and spinor field. We then consider their associated gradient flows and, using a gauge fixing technique, are able to prove short time existence for the flows. We then prove energy estimates along the flow, and establish local $L^2$-derivative estimates. These are then used to show long time existence of the flow in sub-critical dimensions. In the critical dimension, we are able to show that long time existence is obstructed by an $L^{k+2}$ curvature concentration phenomenon.

math.DG↗
arXiv2017

Essential self-adjointness of powers of first-order differential operators on non-compact manifolds with low-regularity metrics

We consider first-order differential operators with locally bounded measurable coefficients on vector bundles with measurable coefficient metrics. Under a mild set of assumptions, we demonstrate the equivalence between the essential self-adjointness of such operators to a negligible boundary property. When the operator possesses higher regularity coefficients, we show that higher powers are essentially self-adjoint if and only if this condition is satisfied. In the case that the low-regularity Riemannian metric induces a complete length space, we demonstrate essential self-adjointness of the operator and its higher powers up to the regularity of its coefficients. We also present applications to Dirac operators on Dirac bundles when the metric is non-smooth.

math.FA↗
arXiv2015

Complements of tori in $\#_{2k}S^2 \times S^2$ that admit a hyperbolic structure

We construct examples of codimension two hyperbolic link complements in closed smooth 4-manifolds with homeomorphism type $\#_{2k}S^2 \times S^2$. All our examples are based on a construction of J. Ratcliffe and S. Tschantz, who constructed 1171 non-compact finite volume hyperbolic 4-manifolds of minimal volume. We then give necessary conditions for a closed smooth simply connected 4-manifold to contain a codimension two link complement that admits a hyperbolic structure.

math.GT↗