Estimating Neandertal contribution to the Upper Paleolithic H. sapiens gene pool using genetic drift effect in branching process model

摘要

Abstract: - In the paper, there is considered the effect of the genetic drift, which could eliminate the hypothetical Neandertal mtDNA admixture from the Upper Paleolithic gene pool of anatomically modern humans. To model the demography, the slightly supercritical Markov's branching process (BP) based on the O'Connell model has been proposed. Relying on relatively fast convergence to the O'Connell's limiting properties it is possible to estimate the time of extinction of the Neandertals relatively to the time of the root of the mtDNA polymorphism of modern humans. The results of the study indicate that the maximum hypothetical contribution of Neandertal mtDNA which could be eliminated by the genetic drift at 0.05 significance level is about 12% for studying mtDNA record only. However, the expected value of the admixture, estimated to be about 3.9 % based on mtDNA record only, is reduced to around 2.3 % if additional nuclear genome data is utilized for Bayesian inference. Relevance of the paper lies in treating mtDNA-based studies as complementary approaches to those based on nuclear DNA sequenced by the Neandertal genome project.