Forgetting matters: when subspace optimization accelerates without the strong convexity parameter

Abstract

Sequential subspace optimization (SESOP) minimizes the objective exactly over a few directions built from past iterates. It needs no problem constants, is accelerated on smooth convex functions and behaves like an accelerated method in experiments. This led to the conjecture that it is accelerated in the strongly convex case without knowing the strong convexity parameter. We show that the conjecture fails in the worst case. For every subspace method whose memory forgets slowly, namely a window of recent gradients and steps together with cumulative sums with subexponential weights, we construct a counterexample: acceleration requires a window of length of order $\sqrtκ$, where $κ$ is the condition number. SESOP and the plane form of conjugate gradients are therefore not accelerated. Memories that forget fast avoid this obstruction. Pairs of geometric filters give accelerated methods with logarithmic memory or only two stored vectors. They need neither the strong convexity parameter nor the smoothness constant, never restart the iterate and restart their memory at most $O(\logκ)$ times. Finally, the method with fixed activation times remains asymptotically accelerated on functions with Lipschitz Hessian even when each subproblem is solved by a single Newton step.