Decentralized Inexact Cubic Newton Method with Consensus Procedure

Abstract

Distributed optimization is widely used in large-scale and privacy-preserving machine learning, where each agent stores a local objective and communicates only with its neighbors in a connected network. We study decentralized second-order optimization and focus on consensus procedures that approximately average local iterates, gradients, and Hessians through neighbor-to-neighbor communications. We propose a general Decentralized Cubic Newton method for convex optimization under $L_1$-smoothness of gradients and $L_2$-Lipschitz continuity of Hessians, and develop a theory that accurately tracks the inaccuracies caused by consensus and by disagreement between local iterates. Under these assumptions, the method matches the iteration complexity of the exact Cubic Newton method and requires only additional polylogarithmic communication-round overhead to reach the necessary consensus accuracy. We further propose an Accelerated Decentralized Cubic Newton method for strongly convex objectives and show that it matches the iteration complexity of the exact Accelerated Cubic Newton method, again with only additional polylogarithmic communication-round overhead. Finally, although the general method requires exchanging full $d \times d$ Hessian matrices, we show how it can be implemented for generalized linear models by transmitting only vectors, making the approach substantially more practical in high dimensions.