Deconvolution estimation of a mixture distribution with boundary effects motivated by mutation effect distribution.

摘要

Density estimation in measurement error models has been widely studied. However, most existing methods consider only continuous target variables, hence they cannot be applied directly to many real problems. Motivated by an evolutionary biology study, we consider more general cases: the target distribution is a mixture of a continuous component and finite numbers of pointmasses, which can cover most of practical problems. In this dissertation, we approach the estimation of the distribution in three different ways under the framework of measurement error models.;Our first proposal is of the Fourier type, which is obtained by generalizing Liu and Taylor (1989). The proposed estimator has a closed form, and gives continuous and smooth density estimators for the continuous mixture component. In addition, its convergence rate is comparably fast. However, when the target distribution has non-smooth boundaries, it suffers from a strong boundary effect. This motivates us to to propose two other methods of the sieve type; one is based on maximum likelihood (ML), and the other uses least squares (LS). By easily reflecting the known boundary information, they remarkably reduce the boundary problems, which is another major contribution of this dissertation. Moreover, the use of penalization improves the smoothness of the resulting estimator, especially the ML based estimator, and reduces the estimation variance.;For each estimator, some asymptotic properties are explored by mathematical computation, and finite sample performances are illustrated via simulation studies. In addition, the proposed estimators are applied to the virus lineage data in Burch et al. (2007), which originally motivates this study. In this application, we not only estimate the mutation effect distribution, but also visually validate the classical exponential assumption on the mutation effect distribution, using density envelope plots.;Keywords: Boundary effect, Deconvolution, Fourier transformation, Mixture distribution, Measurement error, Penalization.