arXiv ScienceSearch

subject

math.OC

math.OC: explore 248 source-linked works published from 2017 to 2026, with original documents and citations.

This collection is a preview while coverage and quality are evaluated.

Search within this collection

Coverage and selection

Includes records with this source-supplied label or an explicit phrase match in their metadata. Matches indicate a mention, not proof that a paper uses a method or tests a material. Source versions are consolidated by DOI.

Sources: arxiv. Collection updated 2026-09-15. Counts describe this index, not the complete source archives.

A Complete Resolution of Forsythe's Conjecture for Restarted Conjugate Gradients

Forsythe's conjecture, published in 1968, asserts that for each restart length $s$, every exact-arithmetic restarted conjugate-gradient iteration on a real symmetric positive definite problem either terminates or has normalised residuals that converge separately along the even and odd restart subsequences. Apart from the classical steepest-descent case, this asymptotic question remained unresolved in full generality for nearly six decades. We give a complete classification by restart length in the original finite-dimensional setting and identify a sharp threshold. For $s=2$ and $s=3$, every problem either terminates or has convergent even and odd residual directions. For every $s\ge4$, there is a diagonal positive definite counterexample of dimension $s+4$ which never terminates and whose even residual directions do not converge. Together with Akaike's theorem for $s=1$, this shows that the conjectured universal conclusion is true precisely for $s\in\{1,2,3\}$ and false for every $s\ge4$. The positive results follow from a degree-independent double-orthogonality identity and an analysis of the low-degree fixed-point sets. At restart length four, rational interval arithmetic and Sturm sequences certify a transverse Hopf point of the leading vector field of the rescaled squared-weight map. Analytic periodic-orbit and shadowing arguments yield the counterexample at restart length four, and degree elevation extends the construction to every larger restart length. The classification for all $s\ge2$ is also formally verified in Lean.

math.NA

Input-to-State Stability Framework for Fully Distributed Primal-Dual Dynamics for Quadratic GNEPs Without Multiplier Consensus

Generalized Nash Equilibrium Problems (GNEPs) often arise in multi-agent engineering applications that require distributed algorithms. Unlike traditional approaches that enforce consensus on multipliers, our method removes the need to share multipliers, reducing communication and improving privacy. As a result, different initializations can lead to different GNEs, including non-variational ones. We establish convergence under sufficient conditions using an input-to-state stability (ISS) framework.

math.OC

Equivalence Between Continuous-Time Risk-Sensitive Control and R\'enyi Divergence Minimization

In this study, we show that a continuous-time risk-sensitive control problem is equivalent to a R\'enyi divergence minimization problem over trajectory path measures. Reformulating stochastic optimal control as probabilistic inference via Kullback-Leibler (KL) divergence minimization avoids the computational intractability of the Hamilton-Jacobi-Bellman equation. However, standard KL control is inherently risk-neutral, and recent minimax extensions remain restricted to risk-averse settings. Our equivalence result resolves this limitation by offering a unified probabilistic framework for arbitrary risk attitudes in continuous-time nonlinear systems. Based on Girsanov theorem, we explicitly map the risk sensitivity to the R\'enyi divergence order, deriving a noise-dependent control penalty scaled by risk preference. This formulation seamlessly modulates tail-weighting behaviors, interpolating between zero-forcing for risk-averse policies and mass-covering for risk-seeking policies. These findings bridge stochastic control and information-theoretic inference, providing a foundation for sampling-based control algorithms.

math.OC

Structure-Preserving Data-Driven Identification of Port-Hamiltonian Differential-Algebraic Systems

We present a data-driven approach to identifying linear index-1 differential-algebraic pH systems (pH-DAEs) based on input-output measurements. In comparison to the identification of port-Hamiltonian (pH) systems, the algebraic constraint and the index condition pose additional challenges. First, we establish a structure-preserving formulation of the considered pH-DAE class and derive an implicit midpoint discretization that preserves the algebraic constraints and discrete dissipation inequality. We formulate the identification problem as a regularized least-squares minimization problem subject to the pH-DAE dynamics. Exploiting the index-1 structure, we reduce the constrained problem to an unconstrained optimization problem over the system parameters while preserving the port-Hamiltonian structure. Next, we derive an adjoint-based formulation to efficiently evaluate the gradient of the resulting reduced cost functional. This enables us to use gradient-based optimization methods for parameter estimation. Under suitable assumptions on the admissible parameter set, the existence of a minimizer is established. Numerical experiments demonstrate that the proposed approach can identify surrogate pH-DAE systems that accurately reproduce the input-output behavior of reference systems. Further investigations show the approach's potential for identifying reduced-order surrogate models. Cross-validation with independent input signals confirms the predictive capability of the identified models.

math.NA

Mathematical Programming in Machine Learning and Artificial Intelligence: A Unified Taxonomy of Models and Applications

Mathematical programming provides a common language for many decisions embedded in modern machine-learning (ML) and artificial-intelligence (AI) systems: selecting retrieval context, routing tokens, allocating inference compute, fitting structured predictors, protecting against distribution shift, and balancing competing objectives. However, the relevant literature is fragmented across optimization, information retrieval, recommendation, natural-language processing, computer vision, and learning theory. This paper organizes various applications under common mathematical programming paradigms: linear, quadratic, binary and mixed-integer, conic, bilevel, multi-objective, inverse, distributionally robust, submodular, and min--max optimization. We normalize the models with a mostly unified notation and, for every application, identify inputs, decision variables, a principal formulation, structural properties, solution strategies, and limitations. Across paradigms, we compare tractability, relaxation quality, decomposition, approximation guarantees, and scalability bottlenecks. The paper shows that mathematical programming is most useful not as a claim that all learning is LP or MIP, but as a disciplined interface between predictions and constrained decisions.

math.OC

MpSub: A Momentum $p$-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models

Full-parameter fine-tuning of large language models has substantial memory costs because backpropagation stores activations and gradients. Zeroth-order optimization avoids this by estimating update directions from loss evaluations, but existing methods require tuning a sensitive learning rate for each model and task. We propose the momentum $p$-dimensional subspace trust-region method (MpSub). At each iteration, MpSub searches within a $p$-dimensional subspace: one direction preserves historical momentum from the most recent accepted step, while the remaining directions explore via fresh random sampling. The subspace gradient is estimated by central differences, a trial step is computed from a linear trust-region model, and the trust-region radius adapts according to the agreement between predicted and observed loss reduction, eliminating the learning rate. For LLM fine-tuning, evaluations within an iteration share a minibatch, and directions are regenerated in place from seeds, using forward passes alone. For smooth deterministic objectives under unorthogonalized Gaussian directions, we bound the finite-difference error, quantify gradient energy captured by the subspace, and prove that $\lim_{k\to\infty} \|\nabla f(x_k)\|_2 = 0$ almost surely under a safeguarded radius update. Under a matched budget of 8,400 training-objective forward passes, we fine-tune OPT-125M and OPT-350M on CommitmentBank. With the same preset parameters at both model sizes, MpSub attains mean test accuracies of 0.673 and 0.690 over three seeds, matching tuned MeZO (0.685) without any learning-rate search.

cs.LG

A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay

Understanding generalization remains a central challenge in machine learning because it requires jointly considering data, architecture, and training dynamics. In this paper, we develop a theoretical framework that characterizes how these factors jointly shape generalization performance throughout training. More precisely, we study a broad class of neural networks trained under the $\ell^2$ loss by gradient descent (GD) with weight decay, and prove the convergence of GD to a neighbourhood of the global minimizers of the empirical loss. By partitioning the space based on the input data, we then decompose the population error into data error, optimization error, and prediction variation error, and bound them separately. In particular, for the prediction variation error, which measures the oscillations of the learned function, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds for its evolution along the training trajectory. These bounds yield two important implications: a necessary condition of improved generalization explains differences in layerwise generalization behavior; a sufficient condition describes delayed generalization and provides a theoretical characterization of grokking.

cs.LG

Configuration-Constrained Tube MPC for Difference-of-Convex Nonlinear Systems

This paper develops a convex tube model predictive control formulation for constrained nonlinear systems. We consider dynamics described by a discrete-time state-space model with parametric and additive uncertainty that admits a difference-of-convex decomposition. Convex directional bounds of the nonlinear dynamics are combined with configuration-constrained polytopic tubes, whose predefined combinatorial structure yields an affine parameterization of their vertices. The resulting finite-dimensional sufficient conditions certify robust one-step tube propagation under parametric uncertainty and additive disturbances, while allowing the tube geometry and an associated vertex control law to be optimized jointly in a single convex program. An implicit terminal condition guarantees recursive feasibility and convergence of the predicted tube to a target robust control invariant set. Numerical results illustrate the closed-loop properties and the trade-off between geometric flexibility and computational complexity.

math.OC

Accelerating Fourier--Motzkin elimination: redundancy removal and the choice of variable elimination order

Fourier-Motzkin elimination computes an inequality description of the projection of a polyhedron onto a subset of its coordinates by eliminating one variable at a time. It is used in several areas of optimisation and computer science, and it is a standard way of obtaining the entropic constraints of a causal structure, where the marginalisation over the latent variables produces such a projection. Its limitation is the growth of the intermediate systems of inequalities, which can be doubly exponential in the number of eliminated variables even though the projection itself grows only as a single exponential. In practice the computational overload of the method therefore depends on two choices: how the redundant inequalities are removed after each step, and the order in which the variables are eliminated. We consider both. We first show, by an explicit example, that Imbert's redundancy test cannot be interleaved with redundancy removal by linear programming. We show that the two methods, however, can be combined soundly if the derivation records used by Imbert's test are re-initialised after every step at which linear programming is used. We then propose a rule for choosing the elimination order of the variables that gives a significant computational advantage, however, at the cost of increased resource usage. We demonstrate this advantage on some random polytopes, where the rule reduces the running time by factors of between 6 and 25 compared with the same elimination under a fixed order. For entropic descriptions of causal structures, with more than 250 inequalities and more than 100 variables to eliminate, our rule keeps the number of inequalities handled at each step one to two orders of magnitude lower than a fixed order.

cs.CC

Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study

We study the control of Markov decision processes in which the quality of a policy is evaluated by a dynamic, time-consistent Markov risk measure rather than by an expected discounted cost. The main obstacle to combining such measures with reinforcement learning is that a transition risk mapping depends on the transition kernel in a nonlinear way, and therefore cannot be estimated from a single observed transition. We remove this obstacle by employing mini-batch transition risk mappings: the mapping is applied to the empirical measure of $N$ independent next-state samples, and the result is averaged. The resulting mapping is again coherent. However, as an expected value of a function of $N$ next-state values, it admits an unbiased one-sample estimator. We embed this mapping into a double deep Q-network, analyze the two sources of estimation bias that arise, and obtain a risk-averse Q-learning method applicable to state spaces far beyond the reach of tabular schemes. The method is applied to an underwater robot navigation problem, in which a vehicle must visit collection points, gather stochastic information payloads, and deliver them at transmission points, while exposed at each step to the risk of destruction. A hierarchical decomposition delegates path execution to an exact graph search and confines learning to the high-level ``collect or transmit'' decision. A low-dimensional feature map, invariant under the symmetries of the problem, replaces the raw state--configuration encoding. In experiments on $300$ held-out environments, the resulting policies transfer to instance sizes never seen in training, and already $N=2$ reduces the upper semideviation of the outcome distribution while simultaneously improving its mean whenever the simulator is misspecified---an empirical counterpart of the duality between coherent risk measures and distributional robustness.

cs.AI

Bi-ZOL: Bilevel Zeroth-Order Learning with Nonsmooth Responses

This paper studies lower-level-constrained bilevel optimization in a response-oracle setting, where lower-level model information is unavailable and the induced response mapping is locally Lipschitz but potentially nonsmooth. In this setting, the classical response Jacobian and reduced hypergradient may fail to exist. We propose Bilevel Zeroth-Order Learning (Bi-ZOL), a structure-guided zeroth-order method for finding stationary points of the nonsmooth reduced problem. Instead of estimating the gradient of a fully smoothed reduced hyperobjective, Bi-ZOL separates the bilevel chain-rule structure: it keeps the exact upper-level partial gradients at the queried response and uses zeroth-order sampling only to estimate the response Jacobian. This construction yields an approximate hypergradient that is more directly aligned with the Clarke chain-rule subdifferential. We show that the Bi-ZOL direction admits a partial-smoothing interpretation, quantify its pointwise structural bias, and prove finite-time convergence to a $(\delta,\epsilon)$-Bi-ZOL Frank--Wolfe stationary point. The bias is $O(\delta)$ for piecewise $C^{1,1}$ responses under local regularity and vanishes for piecewise affine responses on active-cell neighborhoods. Experiments on incentive-based tracking problems show that Bi-ZOL achieves smaller stationarity gaps and lower hyperobjective values than vanilla zeroth-order smoothing under comparable response-oracle budgets.

math.OC

Collision-Free Velocity Scheduling for Multi-Agent Systems on Predefined Routes via Inexact-Projection ADMM

In structured multi-agent transportation systems, agents often must follow predefined routes, making spatial rerouting undesirable or impossible. This paper addresses route-constrained multi-agent coordination by optimizing waypoint passage times while preserving each agent's assigned waypoint order and nominal route assignment. A differentiable surrogate trajectory model maps waypoint timings to smooth position profiles and captures first-order tracking lag, enabling pairwise safety to be encoded through distance-based penalties evaluated on a dense temporal grid spanning the mission horizon. The resulting nonlinear and nonconvex velocity-scheduling problem is solved using an inexact-projection Alternating Direction Method of Multipliers (ADMM) algorithm that combines structured timing updates with gradient-based collision-correction steps and avoids explicit integer sequencing variables. Numerical experiments on random-crossing, bottleneck, and graph-based network scenarios show that the proposed method computes feasible and time-efficient schedules across a range of congestion levels and yields shorter mission completion times than a representative hierarchical baseline in the tested bottleneck cases.

cs.RO

Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima

An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a \emph{single scalar output}, providing a normal form for large-step GD in a neighbourhood of an \emph{isolated} flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with \emph{vector-valued outputs} (including regression with arbitrarily many observations), and (2) to a neighbourhood of a \emph{manifold} of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of \cite{macdonaldeos} to this broader setting, overcoming several technical challenges. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.

cs.LG

From Relaxed Indexability to Exact Indexability: A $t$-Step Approach for Partially Observable Restless Bandits

Whittle index policies offer a scalable method for restless multi-armed bandits, but under partial observability even determining the indifference subsidy at a single belief requires solving an infinite-horizon belief-state problem with no closed-form value function. Liu [10] addresses this difficulty by linearizing the unknown decision boundary, leading to a linear system and a closed-form approximate Whittle index. However, the resulting threshold uses only a one-step active--passive comparison and does not account for longer-horizon continuation values. We extend this framework to a \emph{$t$-step lookahead threshold policy}. For each subsidy $m$, the threshold is defined by the active-minus-passive advantage under $t$-step finite-horizon value iteration. At $t=1$, the threshold is $m$-independent and recovers the linear threshold of Liu [10]; for $t>1$, it becomes subsidy-dependent through the induced first-crossing structure and tracks the exact decision boundary more closely. The proposed algorithmic framework does not require indexability as an input and includes an indexability verification. Under the original Whittle indexability, we prove that the $t$-step approximate Whittle index converges geometrically to the exact Whittle index, \[ |\widehat W_t(\omega)-W(\omega)|=O(\beta^t). \] Numerically, all 2,715 tested three-state instances are verified with computable priority index functions according to the proposed criterion. The P95 index error decreases from $2.18\times10^{-2}$ at $t=1$ to $8.93\times10^{-4}$ at $t=8$. In an exact-comparable instance with $\beta=0.9999$, $t=2$ already recovers the exact Whittle-index ordering. Moderate-depth threshold policies also outperform the one-step baseline and remain close to the optimal dynamic-programming benchmark, while runtime grows mildly with $t$.

cs.LG

1.73-Optimal Echelon Stock (R,nQ) Policies in Two-Stage Stochastic Serial Systems

We consider a classical two-stage continuous-review serial inventory system with unit-sized Poisson demand of rate lambda, backlog cost rate p, lead times L_1, L_2, echelon holding-cost rates h_1, h_2, and fixed shipment costs K_1, K_2. The classical echelon stock (R,nQ) policy offers a simple operating rule: Stage 1 orders a fixed Q_1-lot, Stage 2 orders n >= 1 such lots at a time, and a Stage-1 request waits until a complete lot is available. Existing uniform guarantees restrict lead times or induced lot sizes, and the known all-instance guarantee is primitive-dependent, leaving open whether this restrictive integer-ratio class admits any uniform guarantee over the full primitive space. We develop a setup-preserving cost-allocation lower bound and show that the infimum cost within the classical class is at most 1.73 times the optimal cost over a rate-balanced admissible comparison class, for every lambda, p, L_1, L_2, h_1, h_2, K_1, K_2 >= 0. The result holds for exact integer lot sizes and under both setup conventions, one charging each positive dispatch and the other every complete lot. On the boundaries the guarantee is one, as an equality of infima, when lambda=0, p=0, or h_2=0, and improves to 5/3 when K_2=0. Numerically, across three broad parameter grids, the best-found (R,nQ) policy costs at most 1.098 times the evaluated lower bound, with a median ratio of 1.007; a deliberately adversarial stress test reports 1.6004 under the conservative lot-cost convention. At that point, finite searches under shipment-cost accounting find no benefit from allowing Stage 2 to ship an incomplete Q_1-lot, which is consistent with looseness in the evaluated lower bound. The simple (R,nQ) policy, therefore, combines a straightforward fixed-lot implementation with a uniform guarantee over the full nonnegative primitive space.

math.OC

Computation and Applications of Euclidean and Normed Representations of Massive Data

This thesis investigates Euclidean-space and $\ell_p$-norm representations of different forms of data, with a focus on efficient algorithms for computing these representations in settings where the amount of data is very large. The applications of such representations are also discussed: they may be used to summarize the data in a more compact form for downstream tasks while preserving its most salient properties; and further, they may also be used to extract insights about the data that may not be apparent in its original form. This thesis presents novel algorithms for computing Euclidean representations in the case where the data comes with linear structure, as well as in the case where the data comes only with metric, or distance structure. In addition, we give algorithms for computing $\ell_p$-norm representations in linear structured cases. The analyses of these algorithms use tools from geometry, probability, and optimization, and these tools are used to illuminate other algorithms for similar problems.

cs.DS

Rough variational principles and applications to adjoint systems

We consider a Type-II variational principle driven by geometric rough path with split boundary conditions naturally suited to adjoint systems. From this rough variational principle we derive rough Hamilton's equations, establish their pathwise conservation laws and associated Hamilton--Jacobi equation. We then specialise the framework to rough adjoint systems, obtaining pathwise conservation and quasi-conservation laws that underpin adjoint sensitivity analysis with respect to initial conditions and parameters. On the discrete side, we construct a rough Galerkin discretisation of the rough Type-II variational principle and show that it generates a symplectic flow with discrete analogues of the continuous conservation laws. We establish it's equivalence to a class of Rough Symplectic Partitioned Runge--Kutta (RSPRK) methods and analyse its convergence and naturality properties. Lastly, we perform numerical experiments to validate the predicted convergence rates and demonstrate that RSPRK methods preserve the adjoint conservation laws to machine precision, yielding more accurate and stable gradients in optimisation problems than non-symplectic alternatives.

math.NA
Compare source metadata on this page
WorkPublishedSource identifierSource
A Complete Resolution of Forsythe's Conjecture for Restarted Conjugate Gradients2026-09-042609.04659arxiv
Input-to-State Stability Framework for Fully Distributed Primal-Dual Dynamics for Quadratic GNEPs Without Multiplier Consensus2026-09-072609.06983arxiv
Equivalence Between Continuous-Time Risk-Sensitive Control and R\'enyi Divergence Minimization2026-09-072609.07045arxiv
Structure-Preserving Data-Driven Identification of Port-Hamiltonian Differential-Algebraic Systems2026-09-072609.07187arxiv
Mathematical Programming in Machine Learning and Artificial Intelligence: A Unified Taxonomy of Models and Applications2026-09-072609.07254arxiv
MpSub: A Momentum $p$-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models2026-09-072609.07666arxiv
A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay2026-09-072609.07755arxiv
Configuration-Constrained Tube MPC for Difference-of-Convex Nonlinear Systems2026-09-072609.07809arxiv
Accelerating Fourier--Motzkin elimination: redundancy removal and the choice of variable elimination order2026-09-072609.07960arxiv
Mini-Batch Risk-Averse Deep Q-Learning: A Robot Navigation Case Study2026-09-072609.07998arxiv
Bi-ZOL: Bilevel Zeroth-Order Learning with Nonsmooth Responses2026-09-072609.08021arxiv
Nonsmooth Convex Optimization using the Specular Gradient Method with Root-Linear Convergence2024-12-302412.20747arxiv
Collision-Free Velocity Scheduling for Multi-Agent Systems on Predefined Routes via Inexact-Projection ADMM2026-03-23IEEE Robotics and Automation Letters, 2026arxiv
Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima2026-07-092607.08380arxiv
From Relaxed Indexability to Exact Indexability: A $t$-Step Approach for Partially Observable Restless Bandits2026-08-252608.24167arxiv
1.73-Optimal Echelon Stock (R,nQ) Policies in Two-Stage Stochastic Serial Systems2026-09-062609.06318arxiv
Computation and Applications of Euclidean and Normed Representations of Massive Data2026-09-062609.06329arxiv
Rough variational principles and applications to adjoint systems2026-09-062609.06337arxiv

These are bibliographic comparisons, not experimental rankings. Follow the original document for methods and conditions.