On Differentially Private Stochastic Convex Optimization with Heavy-tailed Data

摘要

In this paper, we consider the problem of designing Differentially Private (DP) algorithms for Stochastic Convex Optimization (SCO) on heavy-tailed data. The irregularity of such data violates some key assumptions used in almost all existing DP-SCO and DP-ERM methods, resulting in failure to provide the DP guarantees. To better understand this type of challenges, we provide in this paper a comprehensive study of DP-SCO under various settings. First, we consider the case where the loss function is strongly convex and smooth. For this case, we propose a method based on the sample-and-aggregate framework, which has an excess population risk of O(_d~3/nε~4) (after omitting other factors), where n is the sample size and d is the dimensionality of the data. Then, we show that with some additional assumptions on the loss functions, it is possible to reduce the expected excess population risk to O (d~2/nε~2). To lift these additional conditions, we also provide a gradient smoothing and trimming based scheme to achieve excess population risks of O(d~2/nε~2) and O(d~(2/3)/(nε~2)~(1/3)) for strongly convex and general convex loss functions, respectively, with high probability. Experiments suggest that our algorithms can effectively deal with the challenges caused by data irregularity.