Multiple testing is often carried out in practice using approximated p-valuesobtained, for instance, via bootstrap or permutation tests. We are interestedin allocating a pre-specified total number of samples (that is draws from abootstrap distribution or permutations) to all hypotheses in order toapproximate their p-values in an optimal way, in the sense that the allocationminimizes the total expected number of misclassified hypotheses. By amisclassified hypothesis we refer to a decision on single hypotheses whichdiffers from the one obtained if all p-values were known analytically. Neitherusing a constant number of samples per p-value estimate nor more sophisticatedapproaches available in the literature guarantee the computation of an optimalallocation in the above sense. This article derives the optimal allocation of afinite total number of samples to a finite number of hypotheses tested usingthe Bonferroni correction. Simulation studies show that a simple samplingalgorithm based on Thompson Sampling asympotically mimics this optimalallocation.
展开▼
