Sonine formulas and intertwining operators in Dunkl theory

Abstract

Let $V_k$ denote Dunkl's intertwining operator associated with some root system $R$ and multiplicity function $k$. For two multiplicities $k, k^\prime$ on $R$, we study the operator $V_{k^\prime,k} = V_{k^\prime}\circ V_k^{-1}$, which intertwines the Dunkl operators for multiplicity $k$ with those for multiplicity $k^\prime.$ While it is well-known that the operator $V_k$ is positive for nonnegative $k$, it has been a long-standing conjecture that its generalizations $V_{k^\prime,k}$ are also positive if $k^\prime \geq k \geq 0,$ which is known to be true in rank one. In this paper, we disprove this conjecture by constructing examples for root system $B_n$ with multiplicites $k^\prime \geq k \geq 0$ for which $V_{k^\prime, k}$ is not positive. This matter is closely related to the existence of integral representations of Sonine type between the Dunkl kernels and Bessel functions associated with the relevant multiplicities. In our examples, such Sonine formulas do not exist. As a consequence, we obtain necessary conditions on Sonine-type integral formulas for Heckman-Opdam hypergeometric functions of type $BC_n$ as well as conditions on the existence of positive branching coefficients between systems of multivariable Jacobi polynomials.