Maximum vanishing subspace problem, CAT(0)-space relaxation, and block-triangularization of partitioned matrix

Abstract

In this paper, we address the following algebraic generalization of the bipartite stable set problem. We are given a block-structured matrix (partitioned matrix) $A = (A_{αβ})$, where $A_{αβ}$ is an $m_α$ by $n_β$ matrix over field ${\bf F}$ for $α=1,2,\ldots,μ$ and $β= 1,2,\ldots,ν$. The maximum vanishing subspace problem (MVSP) is to maximize $\sum_α \dim X_α + \sum_β \dim Y_β$ over vector subspaces $X_α \subseteq {\bf F}^{m_α}$ for $α=1,2,\ldots,μ$ and $Y_β \subseteq {\bf F}^{n_β}$ for $β= 1,2,\ldots,ν$ such that each $A_{αβ}$ vanishes on $X_α \times Y_β$ when $A_{αβ}$ is viewed as a bilinear form ${\bf F}^{m_α} \times {\bf F}^{n_β} \to {\bf F}$. This problem arises from a study of a canonical block-triangular form of $A$ by Ito, Iwata, and Murota~(1994), and is closely related to the noncommutative rank of a matrix with indeterminates. We prove that a weighted version (WMVP) of MVSP can be solved in psuedo polynomial time, provided arithmetic operations on ${\bf F}$ can be done in constant time. Our proof is a novel combination of submodular optimization on modular lattice and convex optimization on CAT(0)-space. We present implications of this result on block-triangularization of partitioned matrix.