arXiv subjects

Pontus Giselsson

Publications and source records attributed to Pontus Giselsson.

At least 19 recordsLinked to original sources

arXiv2026

HeatKV: Head-tuned KV-cache Compression for Visual Autoregressive Modeling

Visual Autoregressive (VAR) models have recently demonstrated impressive image generation quality while maintaining low latency. However, they suffer from severe KV-cache memory constraints, often requiring gigabytes of memory per generated image. We introduce HeatKV, a novel compression method that adapts cache allocation in each head based on its attention to previously generated scales. Using a small offline calibration set, the attention heads are ranked according to their attention scores over prior scales. Based on this ranking, we construct a static pruning schedule tailored to a given memory budget. Applied to the Infinity-2B model, HeatKV achieves $2 \times$ higher compression ratio in memory allocation for KV cache compared to existing methods, while maintaining similar or better image fidelity, prompt alignment and human perception score. Our method achieves a new state-of-the-art (SOTA) for VAR model KV-cache compression, showcasing the effectiveness of fine-grained, head-specific cache allocation. Code and calibration script available at https://github.com/arm-research/heatkv.

cs.CV↗

arXiv2025

Extending Douglas-Rachford Splitting for Convex Optimization

The Douglas-Rachford splitting method is a classical and widely used algorithm for solving monotone inclusions involving the sum of two maximally monotone operators. It was recently shown to be the unique frugal, no-lifting resolvent-splitting method that is unconditionally convergent in the general two-operator setting. In this work, we show that this uniqueness does not hold in the convex optimization case: when the operators are subdifferentials of proper, closed, convex functions, a strictly larger class of frugal, no-lifting resolvent-splitting methods is unconditionally convergent. We provide a complete characterization of all such methods in the convex optimization setting and prove that this characterization is sharp: unconditional convergence holds exactly on the identified parameter regions. These results immediately yield new families of convergent ADMM-type and Chambolle-Pock-type methods obtained through their Douglas-Rachford reformulations.

math.OC↗

arXiv2025

The AutoLyap software suite for computer-assisted Lyapunov analyses of first-order methods

We introduce AutoLyap, a software suite that assists with Lyapunov analyses of a wide class of first-order methods for structured optimization and inclusion problems. Lyapunov analyses are structured proof patterns, with historical roots in the study of dynamical systems, commonly used to establish convergence results for first-order methods. Building on previous work, the core idea behind AutoLyap is to recast the verification of the existence of a Lyapunov analysis as a semidefinite program (SDP), which can then be solved numerically using standard SDP solvers. Users of the package specify (i) the class of optimization or inclusion problems, (ii) the first-order method in question, and (iii) the type of Lyapunov analysis they wish to test. Once these inputs are provided, AutoLyap handles the SDP modeling and proceeds to solve the SDP numerically. We use the package to numerically verify and extend several convergence results. AutoLyap is currently available in Python and Julia.

math.OC↗

arXiv2025

The Symmetry Coefficient of Positively Homogeneous Functions

The Bregman distance is a central tool in convex optimization, particularly in first-order gradient descent and proximal-based algorithms. Such methods enable optimization of functions without Lipschitz continuous gradients by leveraging the concept of relative smoothness, with respect to a reference function $h$. A key factor in determining the full range of allowed step sizes in Bregman schemes is the symmetry coefficient, $\alpha(h)$, of the reference function $h$. While some explicit values of $\alpha(h)$ have been determined for specific functions $h$, a general characterization has remained elusive. This paper explores two problems: ($\textit{i}$) deriving calculus rules for the symmetry coefficient and ($\textit{ii}$) computing $\alpha(\lVert\cdot\rVert_2^p)$ for general $p$. We establish upper and lower bounds for the symmetry coefficient of sums of positively homogeneous Legendre functions and, under certain conditions, provide exact formulas for these sums. Furthermore, we demonstrate that $\alpha(\lVert\cdot\rVert_2^p)$ is independent of dimension and propose an efficient algorithm for its computation. Additionally, we prove that $\alpha(\lVert\cdot\rVert_2^p)$ asymptotically equals, and is lower bounded by, the function $1/(2p)$, offering a simpler upper bound for step sizes in Bregman schemes. Finally, we present closed-form computations for specific cases such as $p \in \{6,8,10\}$.

math.OC↗

arXiv2025

Splitting the Forward-Backward Algorithm: A Full Characterization

We study frugal splitting algorithms with minimal lifting for solving monotone inclusion problems involving sums of maximal monotone and cocoercive operators. Building on a foundational result by Ryu, we fully characterize all methods that use only individual resolvent evaluations, direct evaluations of cocoercive operators, and minimal memory resources while ensuring convergence via averaged fixed-point iterations. We show that all such methods are captured by a unified framework, which includes known schemes and enables new ones with promising features. Systematic numerical experiments lead us to propose three design heuristics to achieve excellent performances in practice, yielding significant gains over existing methods.

math.OC↗

arXiv2025

A Lyapunov analysis of Korpelevich's extragradient method with fast and flexible extensions

We develop a Lyapunov-based analysis of Korpelevich's extragradient method and show that it achieves an $o(1/k)$ last-iterate convergence rate of the constructed Lyapunov function. This Lyapunov function simultaneously upper bounds several standard measures of optimality, which allows our analysis to sharpen existing last-iterate convergence guarantees for these measures. Moreover, the same analysis enables the design of a class of flexible extensions of the extragradient method in which extragradient steps are adaptively blended with user-specified directions via a Lyapunov-guided line-search procedure. These extensions retain global convergence under practical assumptions and can attain superlinear rates when the directions are chosen appropriately. Numerical experiments confirm the simplicity and efficiency of the proposed framework.

math.OC↗

arXiv2023

The Chambolle--Pock method converges weakly with $\theta>1/2$ and $\tau \sigma \|L\|^2<4/(1+2\theta)$

The Chambolle--Pock method is a versatile three-parameter algorithm designed to solve a broad class of composite convex optimization problems, which encompass two proper, lower semicontinuous, and convex functions, along with a linear operator $L$. The functions are accessed via their proximal operators, while the linear operator is evaluated in a forward manner. Among the three algorithm parameters $\tau $, $\sigma $, and $\theta$; $\tau,\sigma >0$ serve as step sizes for the proximal operators, and $\theta$ is an extrapolation step parameter. Previous convergence results have been based on the assumption that $\theta=1$. We demonstrate that weak convergence is achievable whenever $\theta> 1/2$ and $\tau \sigma \|L\|^2<4/(1+2\theta)$. Moreover, we establish tightness of the step size bound by providing an example that is nonconvergent whenever the second bound is violated.

math.OC↗

arXiv2023

Automated tight Lyapunov analysis for first-order methods

We present a methodology for establishing the existence of quadratic Lyapunov inequalities for a wide range of first-order methods used to solve convex optimization problems. In particular, we consider i) classes of optimization problems of finite-sum form with (possibly strongly) convex and possibly smooth functional components, ii) first-order methods that can be written as a linear system in state-space form in feedback interconnection with the subdifferentials of the functional components of the objective function, and iii) quadratic Lyapunov inequalities that can be used to draw convergence conclusions. We present a necessary and sufficient condition for the existence of a quadratic Lyapunov inequality within a predefined class of Lyapunov inequalities, which amounts to solving a small-sized semidefinite program. We showcase our methodology on several first-order methods that fit the framework. Most notably, our methodology allows us to significantly extend the region of parameter choices that allow for duality-gap convergence in the Chambolle-Pock method when the linear operator is the identity mapping.

math.OC↗

arXiv2022

Circuit Analysis using Monotone+Skew Splitting

It is shown that the behavior of an $m$-port circuit of maximal monotone elements can be expressed as a zero of the sum of a maximal monotone operator containing the circuit elements, and a structured skew-symmetric linear operator representing the interconnection structure, together with a linear output transformation. The Condat-V\~u algorithm solves inclusion problems of this form, and may be used to solve for the periodic steady-state behavior, given a periodic excitation at each port, using an iteration in the space of periodic trajectories.

math.OC↗

arXiv2022

Incorporating History and Deviations in Forward--Backward Splitting

We propose a variation of the forward--backward splitting method for solving structured monotone inclusions. Our method integrates past iterates and two deviation vectors into the update equations. These deviation vectors bring flexibility to the algorithm and can be chosen arbitrarily as long as they together satisfy a norm condition. We present special cases where the deviation vectors, selected as predetermined linear combinations of previous iterates, always meet the norm condition. Notably, we introduce an algorithm employing a scalar parameter to interpolate between the conventional forward--backward splitting scheme and an accelerated O(1/n^2)-convergent forward--backward method that encompasses both the accelerated proximal point method and the Halpern iteration as special cases. The existing methods correspond to the two extremes of the allowed scalar parameter range. By choosing the interpolation scalar near the midpoint of the permissible range, our algorithm significantly outperforms these previously known methods when addressing a basic monotone inclusion problem stemming from minimax optimization.

math.OC↗

arXiv2022

Frugal Splitting Operators: Representation, Minimal Lifting, and Convergence

We investigate frugal splitting operators for finite sum monotone inclusion problems. These operators utilize exactly one direct or resolvent evaluation of each operator of the sum, and the splitting operator's output is dictated by linear combinations of these evaluations' inputs and outputs. To facilitate analysis, we introduce a novel representation of frugal splitting operators via a generalized primal-dual resolvent. The representation is characterized by an index and four matrices, and we provide conditions on these that ensure equivalence between the classes of frugal splitting operators and generalized primal-dual resolvents. Our representation paves the way for new results regarding lifting numbers and the development of a unified convergence analysis for frugal splitting operator methods, contingent on the directly evaluated operators being cocoercive. The minimal lifting number is $n-1-f$ where $n$ is the number of monotone operators and $f$ is the number of direct evaluations in the splitting. Notably, this lifting number is achievable only if the first and last operator evaluations are resolvent evaluations. These results generalize the minimal lifting results by Ryu and Malitsky--Tam that consider frugal resolvent splittings. Building on our representation, we delineate a constructive method to design frugal splitting operators, exemplified in the design of a novel, convergent, and parallelizable frugal splitting operator with minimal lifting.

math.OC↗

arXiv2022

DWIFOB: A Dynamically Weighted Inertial Forward-Backward Algorithm for Monotone Inclusions

We propose a novel dynamically weighted inertial forward-backward algorithm (DWIFOB) for solving structured monotone inclusion problems. The scheme exploits the globally convergent forward-backward algorithm with deviations in [26] as the basis and combines it with the extrapolation technique used in Anderson acceleration to improve local convergence. We also present a globally convergent primal-dual variant of DWIFOB and numerically compare its performance to the primal-dual method of Chambolle-Pock and a Tikhonov regularized version of Anderson acceleration applied to the same problem. In all our numerical evaluations, the primal-dual variant of DWIFOB outperforms the Chambolle-Pock algorithm. Moreover, our numerical experiments suggest that our proposed method is much more robust than the regularized Anderson acceleration, which can fail to converge and be sensitive to algorithm parameters. These numerical experiments highlight that our method performs very well while still being robust and reliable.

math.OC↗

arXiv2021

Nonlinear Forward-Backward Splitting with Momentum Correction

The nonlinear, or warped, resolvent recently explored by Giselsson and B\`ui-Combettes has been used to model a large set of existing and new monotone inclusion algorithms. To establish convergent algorithms based on these resolvents, corrective projection steps are utilized in both works. We present a different way of ensuring convergence by means of a nonlinear momentum term, which in many cases leads to cheaper per-iteration cost. The expressiveness of our method is demonstrated by deriving a wide range of special cases. These cases cover and expand on the forward-reflected-backward method of Malitsky-Tam, the primal-dual methods of V\~u-Condat and Chambolle-Pock, and the forward-reflected-Douglas-Rachford method of Ryu-V\~u. A new primal-dual method that uses an extra resolvent step is also presented as well as a general approach for adding momentum to any special case of our nonlinear forward-backward method, in particular all the algorithms listed above.

math.OC↗

arXiv2021

Forward--Backward Splitting with Deviations for Monotone Inclusions

We propose and study a weakly convergent variant of the forward--backward algorithm for solving structured monotone inclusion problems. Our algorithm features a per-iteration deviation vector which provides additional degrees of freedom. The only requirement on the deviation vector to guarantee convergence is that its norm is bounded by a quantity that can be computed online. This approach provides great flexibility and opens up for the design of new and improved forward--backward-based algorithms, while retaining global convergence guarantees. These guarantees include linear convergence of our method under a metric subregularity assumption without the need to adapt the algorithm parameters. Choosing suitable monotone operators allows for incorporating deviations into other algorithms, such as Chambolle--Pock and Krasnoselsky--Mann iterations. We propose a novel inertial primal--dual algorithm by selecting the deviations along a momentum direction and deciding their size using the norm condition. Numerical experiments demonstrate our convergence claims and show that even this simple choice of deviation vector can improve the performance, compared, e.g., to the standard Chambolle--Pock algorithm.

math.OC↗

arXiv2021

Hybrid Acceleration Scheme for Variance Reduced Stochastic Optimization Algorithms

Stochastic variance reduced optimization methods are known to be globally convergent while they suffer from slow local convergence, especially when moderate or high accuracy is needed. To alleviate this problem, we propose an optimization algorithm -- which we refer to as a hybrid acceleration scheme -- for a class of proximal variance reduced stochastic optimization algorithms. The proposed optimization scheme combines a fast locally convergent algorithm, such as a quasi--Newton method, with a globally convergent variance reduced stochastic algorithm, for instance SAGA or L--SVRG. Our global convergence result of the hybrid acceleration method is based on specific safeguard conditions that need to be satisfied for a step of the locally fast convergent method to be accepted. We prove that the sequence of the iterates generated by the hybrid acceleration scheme converges almost surely to a solution of the underlying optimization problem. We also provide numerical experiments that show significantly improved convergence of the hybrid acceleration scheme compared to the basic stochastic variance reduced optimization algorithm.

math.OC↗

arXiv2021

Joint Analog Beam Selection and Digital Beamforming in Millimeter Wave Cell-Free Massive MIMO Systems

Cell-free massive MIMO systems consist of many distributed access points with simple components that jointly serve the users. In millimeter wave bands, only a limited set of predetermined beams can be supported. In a network that consolidates these technologies, downlink analog beam selection stands as a challenging task for the network sum-rate maximization. Low-cost digital filters can improve the network sum-rate further. In this work, we propose low-cost joint designs of analog beam selection and digital filters. The proposed joint designs achieve significantly higher sum-rates than the disjoint design benchmark. Supervised machine learning (ML) algorithms can efficiently approximate the input-output mapping functions of the beam selection decisions of the joint designs with low computational complexities. Since the training of ML algorithms is performed off-line, we propose a well-constructed joint design that combines multiple initializations, iterations, and selection features, as well as beam conflict control, i.e., the same beam cannot be used for multiple users. The numerical results indicate that ML algorithms can retain 99-100% of the original sum-rate results achieved by the proposed well-constructed designs.

cs.IT↗

arXiv2021

Generalized Alternating Projections on Manifolds and Convex Sets

In this paper, we extend the previous convergence results for the generalized alternating projection method applied to subspaces in [arXiv:1703.10547] to hold also for smooth manifolds. We show that the algorithm locally behaves similarly in the subspace and manifold settings and that the same rates are obtained. We also present convergence rate results for when the algorithm is applied to non-empty, closed, and convex sets. The results are based on a finite identification property that implies that the algorithm after an initial identification phase solves a smooth manifold feasibility problem. Therefore, the rates in this paper hold asymptotically for problems in which this identification property is satisfied. We present a few examples where this is the case and also a counter example for when this is not.

math.OC↗

arXiv2020

Sampling and Update Frequencies in Proximal Variance-Reduced Stochastic Gradient Methods

Variance-reduced stochastic gradient methods have gained popularity in recent times. Several variants exist with different strategies for the storing and sampling of gradients and this work concerns the interactions between these two aspects. We present a general proximal variance-reduced gradient method and analyze it under strong convexity assumptions. Special cases of the algorithm include SAGA, L-SVRG and their proximal variants. Our analysis sheds light on epoch-length selection and the need to balance the convergence of the iterates with how often gradients are stored. The analysis improves on other convergence rates found in the literature and produces a new and faster converging sampling strategy for SAGA. Problem instances for which the predicted rates are the same as the practical rates are presented together with problems based on real world data.

math.OC↗