Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective

Abstract

In this work, we propose a novel layerwise adaptive construction method for neural network architectures. Our approach is based on a goal--oriented dual-weighted residual technique for the optimal control of neural differential equations. This leads to an ordinary differential equation constrained optimization problem with controls acting as coefficients and a specific loss function. We implement our approach on the basis of a DG(0) Galerkin discretization of the neural ODE, leading to an explicit Euler time marching scheme. The resulting optimization problem is solved using the Adam algorithm and a BFGS method adapted to the $H^1$ topology induced by the regularization term. Finally, we apply our method to the construction of neural networks for the classification of data sets, where we present results for a selection of well known examples from the literature.