Asymptotic behaviour of sampling and transition probabilities in coalescent models under selection and parent dependent mutations

Abstract

The results in this paper provide new information on asymptotic properties of classical models: the neutral Kingman coalescent under a general finite-alleles, parent-dependent mutation mechanism, and its generalisation, the ancestral selection graph. Several relevant quantities related to these fundamental models are not explicitly known when mutations are parent dependent. Examples include the probability that a sample taken from a population has a certain type configuration, and the transition probabilities of their block counting jump chains. In this paper, asymptotic results are derived for these quantities, as the sample size goes to infinity. It is shown that the sampling probabilities decay polynomially in the sample size with multiplying constant depending on the stationary density of the Wright-Fisher diffusion and that the transition probabilities converge to the limit of frequencies of types in the sample.