On triangular similarity of nilpotent triangular matrices

Abstract

Let $B_n$ (resp. $U_n$, $N_n$) be the set of $n\times n$ nonsingular (resp. unit, nilpotent) upper triangular matrices. We use a novel approach to explore the $B_n$-similarity orbits in $N_n$. The Belitski\u{\i}'s canonical form of $A\in N_n$ under $B_n$-similarity is in $QU_n$ where $Q$ is the subpermutation such that $A\in B_n QB_n$. Using graph representations and $U_n$-similarity actions stablizing $QU_n$, we obtain new properties of the Belitski\u{\i}'s canonical forms and present an efficient algorithm to find the Belitski\u{\i}'s canonical forms in $N_n$. As consequences, we construct new Belitski\u{\i}'s canonical forms in all $N_n$'s, list all Belitski\u{\i}'s canonical forms for $n=7, 8$, and show examples of 3-nilpotent Belitski\u{\i}'s canonical forms in $N_n$ with arbitrary numbers of parameters up to $\operatorname{O}(n^2)$.