On distinguished orbits of reductive representations

Abstract

Let $G$ be a real reductive Lie group and ${\tau}:G \longrightarrow GL(V)$ be a real reductive representation of $G$ with (restricted) moment map $m_{\ggo}: V-{0} \longrightarrow \ggo$. In this work, we introduce the notion of "nice space" of a real reductive representation to study the problem of how to determine if a $G$-orbit is "distinguished" (i.e. it contains a critical point of the norm squared of $m_{\ggo}$). We give an elementary proof of the well-known convexity theorem of Atiyah-Guillemin-Sternberg in our particular case and we use it to give an easy-to-check sufficient condition for a $G$-orbit of a element in a nice space to be distinguished. In the case where $G$ is algebraic and $\tau$ is a rational representation, the above condition is also necessary (making heavy use of recent results of M. Jablonski), obtaining a generalization of Nikolayevsky's nice basis criterium. We also provide useful characterizations of nice spaces in terms of the weights of $\tau$. Finally, some applications to ternary forms and minimal metrics on nilmanifolds are presented.