Enumeration of coalescent histories for caterpillar species trees and $p$-pseudocaterpillar gene trees

Abstract

For a fixed set $X$ containing $n$ taxon labels, an ordered pair consisting of a gene tree topology $G$ and a species tree $S$ bijectively labeled with the labels of $X$ possesses a set of coalescent histories -- mappings from the set of internal nodes of $G$ to the set of edges of $S$ describing possible lists of edges in $S$ on which the coalescences in $G$ take place. Enumerations of coalescent histories for gene trees and species trees have produced suggestive results regarding the pairs $(G,S)$ that, for a fixed $n$, have the largest number of coalescent histories. We define a class of 2-cherry binary tree topologies that we term $p$-pseudocaterpillars, examining coalescent histories for non-matching pairs $(G,S)$, in the case in which $S$ has a caterpillar shape and $G$ has a $p$-pseudocaterpillar shape. Using a construction that associates coalescent histories for $(G,S)$ with a class of "roadblocked" monotonic paths, we identify the $p$-pseudocaterpillar labeled gene tree topology that, for a fixed caterpillar labeled species tree topology, gives rise to the largest number of coalescent histories. The shape that maximizes the number of coalescent histories places the "second" cherry of the $p$-pseudocaterpillar equidistantly from the root of the "first" cherry and from the tree root. A symmetry in the numbers of coalescent histories for $p$-pseudocaterpillar gene trees and caterpillar species trees is seen to exist around the maximizing value of the parameter $p$. The results provide insight into the factors that influence the number of coalescent histories possible for a given gene tree and species tree.