arXiv Science⌕ Search

arXiv subjects

Naum Dimitrieski

Publications and source records attributed to Naum Dimitrieski.

4 recordsLinked to original sources

Adaptive Constraint Sampling for Stochastic Gradient Methods in Constrained Optimization

This paper proposes an adaptive constraint-sampling technique for stochastic gradient-based methods to efficiently solve constrained optimization problems with a large number of inequality constraints. The proposed constraint-sampling technique maintains a set of recently violated constraints and draws the constraint used in each iteration uniformly either from this set or from the set of all constraints. This keeps its per-iteration arithmetic overhead independent of the number of constraints. In simulations on synthetic convex optimization problems and in the synthesis of a control barrier function for the Dubins Car, the constraint-sampling technique is shown to significantly outperform uniform constraint sampling.

math.OC↗

Global Stability and Step Size Robustness of RMSProp

In this paper, an input-to-state Lyapunov function for the RMSProp optimization algorithm is introduced. Global asymptotic stability of the RMSProp algorithm for constant step sizes and robustness properties with respect to arbitrary bounded time-varying step size rules are established.

math.OC↗

Stochastic Gradient Descent for Constrained Optimization based on Adaptive Relaxed Barrier Functions

This paper presents a novel stochastic gradient descent algorithm for constrained optimization. The proposed algorithm randomly samples constraints and components of the finite sum objective function and relies on a relaxed logarithmic barrier function that is appropriately adapted in each optimization iteration. For a strongly convex objective function and affine inequality constraints, step-size rules and barrier adaptation rules are established that guarantee asymptotic convergence with probability one. The theoretical results in the paper are complemented by numerical studies which highlight potential advantages of the proposed algorithm for optimization problems with a large number of constraints.

math.OC↗

Time-delay Induced Stochastic Optimization and Extremum Seeking

In this paper a novel stochastic optimization and extremum seeking algorithm is presented, one which is based on time-delayed random perturbations and step size adaptation. For the case of a one-dimensional quadratic unconstrained optimization problem, global exponential convergence in expectation and global exponential practical convergence of the variance of the trajectories are proven. The theoretical results are complemented by numerical simulations for one- and multi-dimensional quadratic and non-quadratic objective functions.

math.OC↗