Matrix spherical analysis on nilmanifolds

Abstract

Given a nilpotent Lie group $N$, a compact subgroup $K$ of automorphisms of $N$ and an irreducible unitary representation $(\tau,W_\tau)$ of $K$, we study conditions on $\tau$ for the commutativity of the algebra of $\mathrm{End}(W_\tau)$-valued integrable functions on $N$, with an additional property that generalizes the notion of $K$-invariance. A necessary condition, proved by F. Ricci and A. Samanta, is that $(K\ltimes N,K)$ must be a Gelfand pair. In this article we determine all the commutative algebras from a particular class of Gelfand pairs constructed by J. Lauret.