On distinguished orbits of reductive representations

Let G be a real reductive Lie group and let τ:G→GL(V)τ:G→GL(V) be a real reductive representation of G with (restricted) moment map mg:V∖{0}→gmg:V∖{0}→g. In this work, we introduce the notion of nice space of a real reductive representation to study the problem of how to determine if a G-orbit is distinguished   (i.e. it contains a critical point of the norm squared of mgmg). We give an elementary proof of the well-known convexity theorem of Atiyah–Guillemin–Sternberg in our particular case and we use it to give an easy-to-check sufficient condition for a G-orbit of an element in a nice space to be distinguished. In the case where G is algebraic and τ is a rational representation, the above condition is also necessary (making heavy use of recent results of Michael Jablonski), obtaining a generalization of Nikolayevskyʼs nice basis criterion. We also provide useful characterizations of nice spaces in terms of the weights of τ. Finally, some applications to ternary forms are presented.