Restart-Free Oracle-Efficient Methods for Strongly Convex Composite Optimization
We consider strongly convex composite optimization problems whose objective is the sum of a general smooth component, a general nonsmooth component, and a relatively simple nonsmooth term. Although existing first-order methods can achieve optimal convergence guarantees under strong convexity, they typically rely on explicit restart schemes, leading to multiphase algorithms with increased structural and implementation complexity. To address this issue, we develop a restart-free stochastic gradient sliding method that incorporates strong convexity directly through a novel parameter selection strategy. We establish that the proposed method computes an $ε$-solution with $\mathcal{O}\bigl(\log(1/ε)\bigr)$ gradient evaluations of the smooth component and $\mathcal{O}(1/ε)$ stochastic subgradient evaluations of the general nonsmooth component. These bounds match the corresponding complexity guarantees of existing multiphase, restart-based methods, while eliminating the need for explicit restarts. Furthermore, when subgradients of the general nonsmooth component are difficult to compute, but a compatible smooth approximation and its first-order oracle are available, we develop a restart-free accelerated stochastic smoothed gradient sliding (RF-ASSGS) method. RF-ASSGS requires $\mathcal{O}\bigl(\log(1/ε)\bigr)$ gradient evaluations of the smooth component and $\mathcal{O}\bigl(1/\sqrtε+σ^2/ε\bigr)$ stochastic-oracle evaluations. Overall, our results demonstrate that restart-free gradient sliding methods can preserve the optimal convergence guarantees of their restart-based counterparts for strongly convex objectives, while substantially simplifying the algorithmic structure and implementation.