arXiv Science⌕ Search

arXiv subjects

Zhiyuan Li

Publications and source records attributed to Zhiyuan Li.

At least 271 records · Page 15Linked to original sources
arXiv2022

Understanding Gradient Descent on Edge of Stability in Deep Learning

Deep learning experiments by Cohen et al. [2021] using deterministic Gradient Descent (GD) revealed an Edge of Stability (EoS) phase when learning rate (LR) and sharpness (i.e., the largest eigenvalue of Hessian) no longer behave as in traditional optimization. Sharpness stabilizes around $2/$LR and loss goes up and down across iterations, yet still with an overall downward trend. The current paper mathematically analyzes a new mechanism of implicit regularization in the EoS phase, whereby GD updates due to non-smooth loss landscape turn out to evolve along some deterministic flow on the manifold of minimum loss. This is in contrast to many previous results about implicit bias either relying on infinitesimal updates or noise in gradient. Formally, for any smooth function $L$ with certain regularity condition, this effect is demonstrated for (1) Normalized GD, i.e., GD with a varying LR $η_t =\fracη{\| \nabla L(x(t)) \|}$ and loss $L$; (2) GD with constant LR and loss $\sqrt{L- \min_x L(x)}$. Both provably enter the Edge of Stability, with the associated flow on the manifold minimizing $λ_{1}(\nabla^2 L)$. The above theoretical results have been corroborated by an experimental study.

cs.LG↗
arXiv2022

Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models

Language modeling on large-scale datasets leads to impressive performance gains on various downstream language tasks. The validation pre-training loss (or perplexity in autoregressive language modeling) is often used as the evaluation metric when developing language models since the pre-training loss tends to be well-correlated with downstream performance (which is itself difficult to evaluate comprehensively). Contrary to this conventional wisdom, this paper shows that 1) pre-training loss cannot fully explain downstream performance and 2) flatness of the model is well-correlated with downstream performance where pre-training loss is not. On simplified datasets, we identify three ways to produce models with the same (statistically optimal) pre-training loss but different downstream performance: continue pre-training after convergence, increasing the model size, and changing the training algorithm. These experiments demonstrate the existence of implicit bias of pre-training algorithms/optimizers -- among models with the same minimal pre-training loss, they implicitly prefer more transferable ones. Toward understanding this implicit bias, we prove that SGD with standard mini-batch noise implicitly prefers flatter minima in language models, and empirically observe a strong correlation between flatness and downstream performance among models with the same minimal pre-training loss. We also prove in a synthetic language setting that among the models with the minimal pre-training loss, the flattest model transfers to downstream tasks.

cs.LG↗
arXiv2022

A note on Fourier-Mukai partners of abelian varieties over positive characteristic fields

Over complex numbers, the Fourier-Mukai partners of abelian varieties are well-understood. A celebrated result is Orlov's derived Torelli theorem. In this note, we study the FM-partners of abelian varieties in positive characteristic. We notice that, in odd characteristics, two abelian varieties of odd dimension are derived equivalent if their associated Kummer stacks are derived equivalent, which is Krug and Sosna's result over complex numbers. For abelian surfaces in odd characteristic, we show that two abelian surfaces are derived equivalent if and only if their associated Kummer surfaces are isomorphic. This extends the result [doi:10.1215/s0012-7094-03-12036-0] to odd characteristic fields, which solved a classical problem originally from Shioda. Furthermore, we establish the derived Torelli theorem for supersingular abelian varieties and apply it to characterize the quasi-liftable birational models of supersingular generalized Kummer varieties.

math.AG↗
arXiv2022

Few-shot Generation of Personalized Neural Surrogates for Cardiac Simulation via Bayesian Meta-Learning

Clinical adoption of personalized virtual heart simulations faces challenges in model personalization and expensive computation. While an ideal solution is an efficient neural surrogate that at the same time is personalized to an individual subject, the state-of-the-art is either concerned with personalizing an expensive simulation model, or learning an efficient yet generic surrogate. This paper presents a completely new concept to achieve personalized neural surrogates in a single coherent framework of meta-learning (metaPNS). Instead of learning a single neural surrogate, we pursue the process of learning a personalized neural surrogate using a small amount of context data from a subject, in a novel formulation of few-shot generative modeling underpinned by: 1) a set-conditioned neural surrogate for cardiac simulation that, conditioned on subject-specific context data, learns to generate query simulations not included in the context set, and 2) a meta-model of amortized variational inference that learns to condition the neural surrogate via simple feed-forward embedding of context data. As test time, metaPNS delivers a personalized neural surrogate by fast feed-forward embedding of a small and flexible number of data available from an individual, achieving -- for the first time -- personalization and surrogate construction for expensive simulations in one end-to-end learning framework. Synthetic and real-data experiments demonstrated that metaPNS was able to improve personalization and predictive accuracy in comparison to conventionally-optimized cardiac simulation models, at a fraction of computation.

cs.LG↗
arXiv2022

Implicit Regularization and Convergence for Weight Normalization

Normalization methods such as batch [Ioffe and Szegedy, 2015], weight [Salimansand Kingma, 2016], instance [Ulyanov et al., 2016], and layer normalization [Baet al., 2016] have been widely used in modern machine learning. Here, we study the weight normalization (WN) method [Salimans and Kingma, 2016] and a variant called reparametrized projected gradient descent (rPGD) for overparametrized least-squares regression. WN and rPGD reparametrize the weights with a scale g and a unit vector w and thus the objective function becomes non-convex. We show that this non-convex formulation has beneficial regularization effects compared to gradient descent on the original objective. These methods adaptively regularize the weights and converge close to the minimum l2 norm solution, even for initializations far from zero. For certain stepsizes of g and w , we show that they can converge close to the minimum norm solution. This is different from the behavior of gradient descent, which converges to the minimum norm solution only when started at a point in the range space of the feature matrix, and is thus more sensitive to initialization.

cs.LG↗
arXiv2022

A Chandra Survey of Milky Way Globular Clusters. III. Searching for X-ray Signature of Intermediate-mass Black Holes

Globular clusters (GCs) are thought to harbor the long-sought population of intermediate-mass black holes (IMBHs). We present a systematic search for a putative IMBH in 81 Milky Way GCs, based on archival Chandra X-ray observations. We find in only six GCs a significant X-ray source positionally coincident with the cluster center, which have 0.5-8 keV luminosities between $\sim1\times 10^{30}~{\rm erg~s^{-1}}$ to $\sim 4\times10^{33}~{\rm erg~s^{-1}}$. However, the spectral and temporal properties of these six sources can also be explained in terms of binary stars. The remaining 75 GCs do not have a detectable central source, most with $3σ$ upper limits ranging between $10^{29-32}~{\rm erg~s^{-1}}$ over 0.5-8 keV, which are significantly lower than predicted for canonical Bondi accretion. To help understand the feeble X-ray signature, we perform hydrodynamic simulations of stellar wind accretion onto a $1000~{\rm M_\odot}$ IMBH from the most-bound orbiting star, for stellar wind properties consistent with either a main-sequence (MS) star or an asymptotic giant branch (AGB) star. We find that the synthetic X-ray luminosity for the MS case ($\sim 10^{19}\rm~erg~s^{-1}$) is far below the current X-ray limits. The predicted X-ray luminosity for the AGB case ($\sim 10^{34}\rm~erg~s^{-1}$), on the other hand, is compatible with the detected central X-ray sources, in particular the ones in Terzan 5 and NGC 6652. However, the probability of having an AGB star as the most-bound star around the putative IMBH is very low. Our study strongly suggests that it is very challenging to detect the accretion-induced X-ray emission from IMBHs, even if they were prevalent in present-day GCs.

astro-ph.HE↗
arXiv2022

Mukai's program for non-primitive curves on K3 surfaces

Mukai's program seeks to recover a K3 surface $X$ from any curve $C$ on it by exhibiting it as a Fourier-Mukai partner to a Brill-Noether locus of vector bundles on the curve. In the case $X$ has Picard number one and the curve $C\in |H|$ is primitive, this was confirmed by Feyzbakhsh for $g\geq 11$ and $g\neq 12$. More recently, Feyzbakhsh has shown that certain moduli spaces of stable bundles on $X$ are isomorphic to the Brill-Noether locus of curves in $|H|$ if $g$ is sufficiently large. In this paper, we work with irreducible curves in a non-primitive ample linear system $|mH|$ and prove that Mukai's program is valid for any irreducible curve when $g\neq 2$, $mg\geq 11$ and $mg\neq 12$. Furthermore, we introduce the destabilising regions to improve Feyzbakhsh's analysis. We show that there are hyper-Kähler varieties as Brill-Noether loci of curves in every dimension.

math.AG↗
arXiv2022

A Novel Ontology-guided Attribute Partitioning Ensemble Learning Model for Early Prediction of Cognitive Deficits using Quantitative Structural MRI in Very Preterm Infants

Structural magnetic resonance imaging studies have shown that brain anatomical abnormalities are associated with cognitive deficits in preterm infants. Brain maturation and geometric features can be used with machine learning models for predicting later neurodevelopmental deficits. However, traditional machine learning models would suffer from a large feature-to-instance ratio (i.e., a large number of features but a small number of instances/samples). Ensemble learning is a paradigm that strategically generates and integrates a library of machine learning classifiers and has been successfully used on a wide variety of predictive modeling problems to boost model performance. Attribute (i.e., feature) bagging method is the most commonly used feature partitioning scheme, which randomly and repeatedly draws feature subsets from the entire feature set. Although attribute bagging method can effectively reduce feature dimensionality to handle the large feature-to-instance ratio, it lacks consideration of domain knowledge and latent relationship among features. In this study, we proposed a novel Ontology-guided Attribute Partitioning (OAP) method to better draw feature subsets by considering the domain-specific relationship among features. With the better partitioned feature subsets, we developed an ensemble learning framework, which is referred to as OAP-Ensemble Learning (OAP-EL). We applied the OAP-EL to predict cognitive deficits at 2 years of age using quantitative brain maturation and geometric features obtained at term equivalent age in very preterm infants. We demonstrated that the proposed OAP-EL approach significantly outperformed the peer ensemble learning and traditional machine learning approaches.

cs.LG↗
arXiv2022

What Happens after SGD Reaches Zero Loss? --A Mathematical Framework

Understanding the implicit bias of Stochastic Gradient Descent (SGD) is one of the key challenges in deep learning, especially for overparametrized models, where the local minimizers of the loss function $L$ can form a manifold. Intuitively, with a sufficiently small learning rate $η$, SGD tracks Gradient Descent (GD) until it gets close to such manifold, where the gradient noise prevents further convergence. In such a regime, Blanc et al. (2020) proved that SGD with label noise locally decreases a regularizer-like term, the sharpness of loss, $\mathrm{tr}[\nabla^2 L]$. The current paper gives a general framework for such analysis by adapting ideas from Katzenberger (1991). It allows in principle a complete characterization for the regularization effect of SGD around such manifold -- i.e., the "implicit bias" -- using a stochastic differential equation (SDE) describing the limiting dynamics of the parameters, which is determined jointly by the loss function and the noise covariance. This yields some new results: (1) a global analysis of the implicit bias valid for $η^{-2}$ steps, in contrast to the local analysis of Blanc et al. (2020) that is only valid for $η^{-1.6}$ steps and (2) allowing arbitrary noise covariance. As an application, we show with arbitrary large initialization, label noise SGD can always escape the kernel regime and only requires $O(κ\ln d)$ samples for learning an $κ$-sparse overparametrized linear model in $\mathbb{R}^d$ (Woodworth et al., 2020), while GD initialized in the kernel regime requires $Ω(d)$ samples. This upper bound is minimax optimal and improves the previous $\tilde{O}(κ^2)$ upper bound (HaoChen et al., 2020).

cs.LG↗
arXiv2022

Robust Training of Neural Networks Using Scale Invariant Architectures

In contrast to SGD, adaptive gradient methods like Adam allow robust training of modern deep networks, especially large language models. However, the use of adaptivity not only comes at the cost of extra memory but also raises the fundamental question: can non-adaptive methods like SGD enjoy similar benefits? In this paper, we provide an affirmative answer to this question by proposing to achieve both robust and memory-efficient training via the following general recipe: (1) modify the architecture and make it scale invariant, i.e. the scale of parameter doesn't affect the output of the network, (2) train with SGD and weight decay, and optionally (3) clip the global gradient norm proportional to weight norm multiplied by $\sqrt{\tfrac{2λ}η}$, where $η$ is learning rate and $λ$ is weight decay. We show that this general approach is robust to rescaling of parameter and loss by proving that its convergence only depends logarithmically on the scale of initialization and loss, whereas the standard SGD might not even converge for many initializations. Following our recipe, we design a scale invariant version of BERT, called SIBERT, which when trained simply by vanilla SGD achieves performance comparable to BERT trained by adaptive methods like Adam on downstream tasks.

cs.LG↗
arXiv2022

Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent

As part of the effort to understand implicit bias of gradient descent in overparametrized models, several results have shown how the training trajectory on the overparametrized model can be understood as mirror descent on a different objective. The main result here is a characterization of this phenomenon under a notion termed commuting parametrization, which encompasses all the previous results in this setting. It is shown that gradient flow with any commuting parametrization is equivalent to continuous mirror descent with a related Legendre function. Conversely, continuous mirror descent with any Legendre function can be viewed as gradient flow with a related commuting parametrization. The latter result relies upon Nash's embedding theorem.

cs.LG↗
arXiv2022

Very Large Array Multi-band Radio Imaging of the Triple AGN Candidate SDSS J0849+1114

Kpc-scale triple active galactic nuclei (AGNs), potential precursors of gravitationally-bound triple massive black holes (MBHs), are rarely seen objects and believed to play an important role in the evolution of MBHs and their host galaxies. In this work we present a multi-band (3.0, 6.0 10.0, and 15.0 GHz), high-resolution radio imaging of the triple AGN candidate, SDSS J0849+1114, using the Very Large Array. Two of the three nuclei (A and C) are detected at 3.0, 6.0, and 15 GHz for the first time, both exhibiting a steep spectrum over 3--15 GHz (with a spectral index $-0.90 \pm 0.05$ and $-1.03 \pm 0.04$) consistent with a synchrotron origin. Nucleus A, the strongest nucleus among the three, shows a double-sided jet, with the jet orientation changing by $\sim20^{\circ}$ between its inner 1" and the outer 5.5" (8.1 kpc) components, which may be explained as the MBH's angular momentum having been altered by merger-enhanced accretion. Nucleus C also shows a two-sided jet, with the western jet inflating into a radio lobe with an extent of 1.5" (2.2 kpc). The internal energy of the radio lobe is estimated to be $\rm 5.0 \times 10^{55}$ erg, for an equipartition magnetic field strength of $\rm \sim 160\ μG$. No significant radio emission is detected at all four frequencies for nucleus B, yielding an upper limit of 15, 15, 15, and 18 $\rm μJy\ beam^{-1}$ at 3.0, 6.0, 10.0, and 15.0 GHz, based on which we constrain the star formation rate in nucleus B to be $\lesssim 0.4~\rm M_{\odot}~yr^{-1}$.

astro-ph.GA↗
arXiv2022

Interpretability of Neural Network With Physiological Mechanisms

Deep learning continues to play as a powerful state-of-art technique that has achieved extraordinary accuracy levels in various domains of regression and classification tasks, including images, video, signal, and natural language data. The original goal of proposing the neural network model is to improve the understanding of complex human brains using a mathematical expression approach. However, recent deep learning techniques continue to lose the interpretations of its functional process by being treated mostly as a black-box approximator. To address this issue, such an AI model needs to be biological and physiological realistic to incorporate a better understanding of human-machine evolutionary intelligence. In this study, we compare neural networks and biological circuits to discover the similarities and differences from various perspective views. We further discuss the insights into how neural networks learn from data by investigating human biological behaviors and understandable justifications.

cs.NE↗
arXiv2022

Peridynamic modeling for impact failure of wet concrete considering the influence of saturation

In this paper, a modified intermediately homogenized peridynamic (IH-PD) model for analyzing impact failure of wet concrete has been presented under the configuration of ordinary state-based peridynamic theory. The meso-structural properties of concrete are linked to the macroscopic mechanical behavior in the IH-PD model, where the heterogeneity of concrete is taken into account, and the calculation cost does not increase. Simultaneously, the porosity of concrete is considered, which is implemented by deleting the bond between two material points, as well as the influence of porosity on the mechanical properties of concrete. Moreover, the effective bulk and shear modulus of cement mortar in wet concrete (saturated and unsaturated concrete) are calculated respectively. The dynamic model for wet concrete is described from three aspects: strength, dynamic increase factor, and equation of state. Validation of the proposed model is established through analyzing some benchmark tests and comparing with the corresponding experiment and other available numerical results.

math.NA↗
arXiv2022

Nonlinear optical radiation of a lithium niobate microcavity

The nonlinear optical radiation of an integrated lithium niobate microcavity is demonstrated, which has been neglected in previous studies of nonlinear photonic devices. We find that the nonlinear coupling between confined optical modes on the chip and continuum modes in free space can be greatly enhanced on the platform of integrated microcavity, with feasible relaxation of the phase-matching condition. With an infrared pump laser, we observe the vertical radiation of second-harmonic wave at the visible band, which indicates a robust phase-matching-free chip-to-free-space frequency converter and also unveils an extra energy dissipation channel for integrated devices. Such an unexpected coherent nonlinear interaction between the free-space beam and the confined mode is also validated by the different frequency generation. Furthermore, based on the phase-matching-free nature of the nonlinear radiation, we build an integrated atomic gas sensor to characterize Rb isotopes with a single telecom laser. The unveiled mechanism of nonlinear optical radiation is universal for all dielectric photonic integrated devices, and provides a simple and robust chip-to-free-space as well as visible-to-telecom interface.

physics.optics↗
arXiv2022

Unpolarized Shafarevich conjectures for hyper-Kähler varieties

Shafarevich conjecture/problem is about the finiteness of isomorphism classes of a family of varieties defined over a number field with good reduction outside a finite collection of places. For K3 surfaces, such a finiteness result was proved by Y. She. For hyper-Kähler varieties, which are higher-dimensional analogs of K3 surfaces, Y. André has verified the Shafarevich conjecture for hyper-Kähler varieties of a given dimension and admitting a very ample polarization of bounded degree. In this paper, we provide a unification of both results by proving the (unpolarized) Shafarevich conjecture for hyper-Kähler varieties in a given deformation type. In a similar fashion, generalizing a result of Orr and Skorobogatov on K3 surfaces, we prove the finiteness of geometric isomorphism classes of hyper-Kähler varieties of CM type in a given deformation type defined over a number field with bounded degree. A key to our approach is a uniform Kuga--Satake map, inspired by She's work, and we study its arithmetic properties, which are of independent interest.

math.AG↗
arXiv2022

CAHA/PPAK Integral-field Spectroscopic Observations of M81 -- I. Circumnuclear ionized gas

Galactic circumnuclear environments of nearby galaxies provide unique opportunities for our understanding of the co-evolution between super-massive black holes and their host galaxies. Here we present a detailed study of ionized gas in the central kiloparsec region of M81, which hosts the closest prototype low-luminosity active galactic nucleus, based on optical integral-field spectroscopic observations taken with the CAHA 3.5m telescope. It is found that much of the circumnuclear ionized gas is concentraed within a bright core of $\sim$200 pc in extent and a surrounding spiral-like structure known as the nuclear spiral. The total mass of the ionized gas is estimated to be $\sim2\times10^5\rm~M_\odot$, which corresponds to a few percent of the cold gas mass in this region, as traced by co-spatial dust extinction features. Plausible signature of a bi-conical outflow along the disk plane is suggested by a pair of blueshifted/redshifted low-velocity features, symmetrically located at $\sim$ 120 -- 250 pc from the nucleus. The spatially-resolved line ratios of [N\,{\sc ii}]/H$α$ and [O\,{\sc iii}]/H$β$ demonstrate that much of the circumnuclear region can be classified as LINER (low-ionization nuclear emission-line region). However, substantial spatial variations in the line intensities and line ratios strongly suggest that different ionization/excitation mechanisms, rather than just a central dominant source of photoionization, are simultaneously at work to produce the observed line signatures.

astro-ph.GA↗