arXiv subjects

Panos Stinis

Publications and source records attributed to Panos Stinis.

At least 55 records · Page 3Linked to original sources

arXiv2023

Exploring Learned Representations of Neural Networks with Principal Component Analysis

Understanding feature representation for deep neural networks (DNNs) remains an open question within the general field of explainable AI. We use principal component analysis (PCA) to study the performance of a k-nearest neighbors classifier (k-NN), nearest class-centers classifier (NCC), and support vector machines on the learned layer-wise representations of a ResNet-18 trained on CIFAR-10. We show that in certain layers, as little as 20% of the intermediate feature-space variance is necessary for high-accuracy classification and that across all layers, the first ~100 PCs completely determine the performance of the k-NN and NCC classifiers. We relate our findings to neural collapse and provide partial evidence for the related phenomenon of intermediate neural collapse. Our preliminary work provides three distinct yet interpretable surrogate models for feature representation with an affine linear model the best performing. We also show that leveraging several surrogate models affords us a clever method to estimate where neural collapse may initially occur within the DNN.

cs.LG↗

arXiv2023

Physics-informed machine learning of redox flow battery based on a two-dimensional unit cell model

In this paper, we present a physics-informed neural network (PINN) approach for predicting the performance of an all-vanadium redox flow battery, with its physics constraints enforced by a two-dimensional (2D) mathematical model. The 2D model, which includes 6 governing equations and 24 boundary conditions, provides a detailed representation of the electrochemical reactions, mass transport and hydrodynamics occurring inside the redox flow battery. To solve the 2D model with the PINN approach, a composite neural network is employed to approximate species concentration and potentials; the input and output are normalized according to prior knowledge of the battery system; the governing equations and boundary conditions are first scaled to an order of magnitude around 1, and then further balanced with a self-weighting method. Our numerical results show that the PINN is able to predict cell voltage correctly, but the prediction of potentials shows a constant-like shift. To fix the shift, the PINN is enhanced by further constrains derived from the current collector boundary. Finally, we show that the enhanced PINN can be even further improved if a small number of labeled data is available.

cs.LG↗

arXiv2023

Physics-informed machine learning of the correlation functions in bulk fluids

The Ornstein-Zernike (OZ) equation is the fundamental equation for pair correlation function computations in the modern integral equation theory for liquids. In this work, machine learning models, notably physics-informed neural networks and physics-informed neural operator networks, are explored to solve the OZ equation. The physics-informed machine learning models demonstrate great accuracy and high efficiency in solving the forward and inverse OZ problems of various bulk fluids. The results highlight the significant potential of physics-informed machine learning for applications in thermodynamic state theory.

physics.comp-ph↗

arXiv2023

A Multifidelity deep operator network approach to closure for multiscale systems

Projection-based reduced order models (PROMs) have shown promise in representing the behavior of multiscale systems using a small set of generalized (or latent) variables. Despite their success, PROMs can be susceptible to inaccuracies, even instabilities, due to the improper accounting of the interaction between the resolved and unresolved scales of the multiscale system (known as the closure problem). In the current work, we interpret closure as a multifidelity problem and use a multifidelity deep operator network (DeepONet) framework to address it. In addition, to enhance the stability and accuracy of the multifidelity-based closure, we employ the recently developed "in-the-loop" training approach from the literature on coupling physics and machine learning models. The resulting approach is tested on shock advection for the one-dimensional viscous Burgers equation and vortex merging using the two-dimensional Navier-Stokes equations. The numerical experiments show significant improvement of the predictive ability of the closure-corrected PROM over the un-corrected one both in the interpolative and the extrapolative regimes.

physics.comp-ph↗

arXiv2023

Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations

Physics-informed neural networks have emerged as an alternative method for solving partial differential equations. However, for complex problems, the training of such networks can still require high-fidelity data which can be expensive to generate. To reduce or even eliminate the dependency on high-fidelity data, we propose a novel multi-fidelity architecture which is based on a feature space shared by the low- and high-fidelity solutions. In the feature space, the projections of the low-fidelity and high-fidelity solutions are adjacent by constraining their relative distance. The feature space is represented with an encoder and its mapping to the original solution space is effected through a decoder. The proposed multi-fidelity approach is validated on forward and inverse problems for steady and unsteady problems described by partial differential equations.

cs.LG↗

arXiv2023

ViTO: Vision Transformer-Operator

We combine vision transformers with operator learning to solve diverse inverse problems described by partial differential equations (PDEs). Our approach, named ViTO, combines a U-Net based architecture with a vision transformer. We apply ViTO to solve inverse PDE problems of increasing complexity, namely for the wave equation, the Navier-Stokes equations and the Darcy equation. We focus on the more challenging case of super-resolution, where the input dataset for the inverse problem is at a significantly coarser resolution than the output. The results we obtain are comparable or exceed the leading operator network benchmarks in terms of accuracy. Furthermore, ViTO`s architecture has a small number of trainable parameters (less than 10% of the leading competitor), resulting in a performance speed-up of over 5x when averaged over the various test cases.

cs.CV↗

arXiv2023

SDYN-GANs: Adversarial Learning Methods for Multistep Generative Models for General Order Stochastic Dynamics

We introduce adversarial learning methods for data-driven generative modeling of the dynamics of $n^{th}$-order stochastic systems. Our approach builds on Generative Adversarial Networks (GANs) with generative model classes based on stable $m$-step stochastic numerical integrators. We introduce different formulations and training methods for learning models of stochastic dynamics based on observation of trajectory samples. We develop approaches using discriminators based on Maximum Mean Discrepancy (MMD), training protocols using conditional and marginal distributions, and methods for learning dynamic responses over different time-scales. We show how our approaches can be used for modeling physical systems to learn force-laws, damping coefficients, and noise-related parameters. The adversarial learning approaches provide methods for obtaining stable generative models for dynamic tasks including long-time prediction and developing simulations for stochastic systems.

cs.LG↗

arXiv2023

A Hybrid Deep Neural Operator/Finite Element Method for Ice-Sheet Modeling

One of the most challenging and consequential problems in climate modeling is to provide probabilistic projections of sea level rise. A large part of the uncertainty of sea level projections is due to uncertainty in ice sheet dynamics. At the moment, accurate quantification of the uncertainty is hindered by the cost of ice sheet computational models. In this work, we develop a hybrid approach to approximate existing ice sheet computational models at a fraction of their cost. Our approach consists of replacing the finite element model for the momentum equations for the ice velocity, the most expensive part of an ice sheet model, with a Deep Operator Network, while retaining a classic finite element discretization for the evolution of the ice thickness. We show that the resulting hybrid model is very accurate and it is an order of magnitude faster than the traditional finite element model. Further, a distinctive feature of the proposed model compared to other neural network approaches, is that it can handle high-dimensional parameter spaces (parameter fields) such as the basal friction at the bed of the glacier, and can therefore be used for generating samples for uncertainty quantification. We study the impact of hyper-parameters, number of unknowns and correlation length of the parameter distribution on the training and accuracy of the Deep Operator Network on a synthetic ice sheet model. We then target the evolution of the Humboldt glacier in Greenland and show that our hybrid model can provide accurate statistics of the glacier mass loss and can be effectively used to accelerate the quantification of uncertainty.

physics.comp-ph↗

arXiv2022

SMS: Spiking Marching Scheme for Efficient Long Time Integration of Differential Equations

We propose a Spiking Neural Network (SNN)-based explicit numerical scheme for long time integration of time-dependent Ordinary and Partial Differential Equations (ODEs, PDEs). The core element of the method is a SNN, trained to use spike-encoded information about the solution at previous timesteps to predict spike-encoded information at the next timestep. After the network has been trained, it operates as an explicit numerical scheme that can be used to compute the solution at future timesteps, given a spike-encoded initial condition. A decoder is used to transform the evolved spiking-encoded solution back to function values. We present results from numerical experiments of using the proposed method for ODEs and PDEs of varying complexity.

math.NA↗

arXiv2022

Vibrational Levels of a Generalized Morse Potential

A Generalized Morse Potential (GMP) is an extension of the Morse Potential (MP) with an additional exponential term and an additional parameter that compensate for MP's erroneous behavior in the long range part of the interaction potential. Because of the additional term and parameter, the vibrational levels of the GMP cannot be solved analytically, unlike the case for the MP. We present several numerical approaches for solving the vibrational problem of the GMP based on Galerkin methods, namely the Laguerre Polynomial Method (LPM), the Symmetrized Laguerre Polynomial Method (SLPM) and the Polynomial Expansion method (PEM) and apply them to the vibrational levels of the homonuclear diatomic molecules B$_2$, O$_2$ and F$_2$, for which high level theoretical Full CI potential energy surfaces and experimentally measured vibrational levels have been reported. Overall the LPM produces vibrational states for the GMP that are converged to within spectroscopic accuracy of 0.01 cm$^{-1}$ in between 1 and 2 orders of magnitude faster and with much fewer basis functions / grid points than the Colbert-Miller Discrete Variable Representation (CM-DVR) method for the three homonuclear diatomic molecules examined in this study.

physics.chem-ph↗

arXiv2022

Physics-constrained deep neural network method for estimating parameters in a redox flow battery

In this paper, we present a physics-constrained deep neural network (PCDNN) method for parameter estimation in the zero-dimensional (0D) model of the vanadium redox flow battery (VRFB). In this approach, we use deep neural networks (DNNs) to approximate the model parameters as functions of the operating conditions. This method allows the integration of the VRFB computational models as the physical constraints in the parameter learning process, leading to enhanced accuracy of parameter estimation and cell voltage prediction. Using an experimental dataset, we demonstrate that the PCDNN method can estimate model parameters for a range of operating conditions and improve the 0D model prediction of voltage compared to the 0D model prediction with constant operation-condition-independent parameters estimated with traditional inverse methods. We also demonstrate that the PCDNN approach has an improved generalization ability for estimating parameter values for operating conditions not used in the DNN training.

physics.chem-ph↗

arXiv2022

Enhanced physics-constrained deep neural networks for modeling vanadium redox flow battery

Numerical modeling and simulation have become indispensable tools for advancing a comprehensive understanding of the underlying mechanisms and cost-effective process optimization and control of flow batteries. In this study, we propose an enhanced version of the physics-constrained deep neural network (PCDNN) approach [1] to provide high-accuracy voltage predictions in the vanadium redox flow batteries (VRFBs). The purpose of the PCDNN approach is to enforce the physics-based zero-dimensional (0D) VRFB model in a neural network to assure model generalization for various battery operation conditions. Limited by the simplifications of the 0D model, the PCDNN cannot capture sharp voltage changes in the extreme SOC regions. To improve the accuracy of voltage prediction at extreme ranges, we introduce a second (enhanced) DNN to mitigate the prediction errors carried from the 0D model itself and call the resulting approach enhanced PCDNN (ePCDNN). By comparing the model prediction with experimental data, we demonstrate that the ePCDNN approach can accurately capture the voltage response throughout the charge--discharge cycle, including the tail region of the voltage discharge curve. Compared to the standard PCDNN, the prediction accuracy of the ePCDNN is significantly improved. The loss function for training the ePCDNN is designed to be flexible by adjusting the weights of the physics-constrained DNN and the enhanced DNN. This allows the ePCDNN framework to be transferable to battery systems with variable physical model fidelity.

physics.chem-ph↗

arXiv2021

Machine-learning custom-made basis functions for partial differential equations

Spectral methods are an important part of scientific computing's arsenal for solving partial differential equations (PDEs). However, their applicability and effectiveness depend crucially on the choice of basis functions used to expand the solution of a PDE. The last decade has seen the emergence of deep learning as a strong contender in providing efficient representations of complex functions. In the current work, we present an approach for combining deep neural networks with spectral methods to solve PDEs. In particular, we use a deep learning technique known as the Deep Operator Network (DeepONet), to identify candidate functions on which to expand the solution of PDEs. We have devised an approach which uses the candidate functions provided by the DeepONet as a starting point to construct a set of functions which have the following properties: i) they constitute a basis, 2) they are orthonormal, and 3) they are hierarchical i.e., akin to Fourier series or orthogonal polynomials. We have exploited the favorable properties of our custom-made basis functions to both study their approximation capability and use them to expand the solution of linear and nonlinear time-dependent PDEs.

math.NA↗

arXiv2021

Structure-preserving Sparse Identification of Nonlinear Dynamics for Data-driven Modeling

Discovery of dynamical systems from data forms the foundation for data-driven modeling and recently, structure-preserving geometric perspectives have been shown to provide improved forecasting, stability, and physical realizability guarantees. We present here a unification of the Sparse Identification of Nonlinear Dynamics (SINDy) formalism with neural ordinary differential equations. The resulting framework allows learning of both "black-box" dynamics and learning of structure preserving bracket formalisms for both reversible and irreversible dynamics. We present a suite of benchmarks demonstrating effectiveness and structure preservation, including for chaotic systems.

cs.LG↗

arXiv2021

Machine learning structure preserving brackets for forecasting irreversible processes

Forecasting of time-series data requires imposition of inductive biases to obtain predictive extrapolation, and recent works have imposed Hamiltonian/Lagrangian form to preserve structure for systems with reversible dynamics. In this work we present a novel parameterization of dissipative brackets from metriplectic dynamical systems appropriate for learning irreversible dynamics with unknown a priori model form. The process learns generalized Casimirs for energy and entropy guaranteed to be conserved and nondecreasing, respectively. Furthermore, for the case of added thermal noise, we guarantee exact preservation of a fluctuation-dissipation theorem, ensuring thermodynamic consistency. We provide benchmarks for dissipative systems demonstrating learned dynamics are more robust and generalize better than either "black-box" or penalty-based approaches.

physics.comp-ph↗

arXiv2021

Time-dependent stochastic basis adaptation for uncertainty quantification

We extend stochastic basis adaptation and spatial domain decomposition methods to solve time varying stochastic partial differential equations (SPDEs) with a large number of input random parameters. Stochastic basis adaptation allows the determination of a low dimensional stochastic basis representation of a quantity of interest (QoI). Extending basis adaptation to time-dependent problems is challenging because small errors introduced in the previous time steps of the low dimensional approximate solution accumulate over time and cause divergence from the true solution. To address this issue we have introduced an approach where the basis adaptation varies at every time step so that the low dimensional basis is adapted to the QoI at that time step. We have coupled the time-dependent basis adaptation with domain decomposition to further increase the accuracy in the representation of the QoI. To illustrate the construction, we present numerical results for one-dimensional time varying linear and nonlinear diffusion equations with random space-dependent diffusion coefficients. Stochastic dimension reduction techniques proposed in the literature have mainly focused on quantifying the uncertainty in time independent and scalar QoI. To the best of our knowledge, this is the first attempt to extend dimensional reduction techniques to time varying and spatially dependent quantities such as the solution of SPDEs.

math.NA↗

arXiv2021

Optimal renormalization of multi-scale systems

While model order reduction is a promising approach in dealing with multi-scale time-dependent systems that are too large or too expensive to simulate for long times, the resulting reduced order models can suffer from instabilities. We have recently developed a time-dependent renormalization approach to stabilize such reduced models. In the current work, we extend this framework by introducing a parameter that controls the time-decay of the memory of such models and optimally selecting this parameter based on limited fully resolved simulations. First, we demonstrate our framework on the inviscid Burgers equation whose solution develops a finite-time singularity. Our renormalized reduced order models are stable and accurate for long times while using for their calibration only data from a full order simulation before the occurrence of the singularity. Furthermore, we apply this framework to the 3D Euler equations of incompressible fluid flow, where the problem of finite-time singularity formation is still open and where brute force simulation is only feasible for short times. Our approach allows us to obtain for the first time a perturbatively renormalizable model which is stable for long times and includes all the complex effects present in the 3D Euler dynamics. We find that, in each application, the renormalization coefficients display algebraic decay with increasing resolution, and that the parameter which controls the time-decay of the memory is problem-dependent.

physics.flu-dyn↗

arXiv2019

Model reduction for a power grid model

We apply model reduction techniques to the DeMarco power grid model. The DeMarco model, when augmented by an appropriate line failure mechanism, can be used to study cascade failures. Here we examine the DeMarco model without the line failure mechanism and we investigate how to construct reduced order models for subsets of the state variables. We show that due to the oscillating nature of the solutions and the absence of timescale separation between resolved and unresolved variables, the construction of accurate reduced models becomes highly non-trivial since one has to account for long memory effects. In addition, we show that a reduced model which includes even a short memory is drastically better than a memoryless model.

eess.SP↗