The star-shaped Lambda-coalescent and Fleming-Viot process

Abstract

The star-shaped $Λ$-coalescent and corresponding $Λ$-Fleming-Viot process where the $Λ$ measure has a single atom at unity are studied in this paper. The transition functions and stationary distribution of the $Λ$-Fleming-Viot process are derived in a two-type model with mutation. The distribution of the number of non-mutant lines back in time in the star-shaped $Λ$-coalescent is found. Extensions are made to a model with $d$ types, either with parent independent mutation or general Markov mutation, and an infinitely-many-types model when $d\to \infty$. An eigenfunction expansion for the transition functions is found which has polynomial right eigenfunctions and left eigenfunctions described by hyperfunctions. A further star-shaped model with general frequency dependent change is considered and the stationary distribution in the Fleming-Viot process derived. This model includes a star-shaped $Λ$-Fleming-Viot process with mutation and selection. In a general $Λ$-coalescent explicit formulae for the transition functions and stationary distribution when there is mutation are unknown, however in this paper explicit formulae are derived in the star-shaped coalescent.