A tasks reordering model to reduce transfers overhead on GPUs

摘要

Abstract The compute capabilities of current GPUs allow exploiting concurrency when several independent tasks are simultaneously launched. These tasks are typically composed by data transfer commands and kernel computation commands. In this paper we develop a run-time approach to optimize the concurrency between data transfers and kernel computation operations in a multithreaded scenario where each CPU thread is sending tasks to the GPU. Our solution is based on a temporal execution model for concurrent tasks that is able to establish the tasks execution order that minimizes the total execution time, including data transfers. Moreover, a heuristic to select the best order has been developed, which is able to improve the execution time achieved by the hardware scheduler of current NVIDIA cards. Our approach obtains performance improvements, under real workloads, of up to 19% with respect to the execution using multiple hardware queues managed by Hyper-Q. Highlights • Execution of sets of independent tasks using CUDA streams is studied. • A temporal model for simulating the execution of independent tasks is introduced. • A dynamic scheduling method to dispatch tasks in a specific order is presented. • A heuristic for obtaining a near minimum execution time is developed. • An increase in concurrency, compared with Hyper-Q, is demonstrated.