Error Bounds and a Smoothing Penalty Method for Binary Orthogonal Optimization

Abstract

Binary orthogonal optimization arises in semantic hashing, orthogonal binary projection, and Hadamard matrix design. Existing methods often handle binarity and orthogonality separately, so their outputs need not satisfy both constraints simultaneously and generally come without a residual-based guarantee of exact feasibility. Using an exact box-constrained reformulation, we establish local and global Lipschitz error bounds and prove that the Lipschitz exponent is optimal. In particular, when the binary orthogonal feasible set is nonempty, any box-constrained matrix whose orthogonality residual is below $1$ admits a feasible sign rounding, and the rounding distance is bounded by this residual. These bounds also yield a globally exact nonsmooth orthogonality-residual penalty over the box and a uniformly sharp local penalty threshold. We further develop a smoothing penalty method for binary orthogonal optimization (SPBO). We prove that every accumulation point is stationary for the limiting nonsmooth penalty problem. Numerical experiments on orthogonal binary projection, supervised hashing, and Hadamard completion show that SPBO substantially reduces orthogonality violations while retaining competitive objective values or retrieval accuracy.