Fractional Laplacians on domains, a development of Hörmander's theory of mu-transmission pseudodifferential operators

Abstract

Let $P$ be a classical pseudodifferential operator of complex order $m$ on an $n$-dimensional smooth manifold $Ω_1$. For the truncation $P_Ω$ to a smooth subset $Ω$ there is a well-known theory of boundary value problems when $P_Ω$ has the transmission property (preserves $C^\infty (\barΩ)$) and is of integer order; the calculus of Boutet de Monvel. Many interesting operators, such as for example complex powers of the Laplacian $(-Δ)^μ$ with noninteger mu, are not covered. They have instead the mu-transmission property defined in Hörmander's books, mapping $x_n^μC^\infty (\barΩ)$ into $C^\infty (\barΩ)$. In an unpublished lecture note from 1965, Hörmander described an $L_2$-solvability theory for mu-transmission operators, departing from Vishik and Eskin's results. We here develop the theory in $L_p$ Sobolev spaces ($1<p<\infty$) in a modern setting. It leads to not only Fredholm solvability statements but also regularity results in full scales of Sobolev spaces (for $s\to \infty$). The solution spaces have a singularity at the boundary that we describe in detail. We moreover obtain results in Hölder spaces, which radically improve recent regularity results for fractional Laplacians.