Real-space finite difference PAW method for large-scale applications on massively parallel computers

摘要

Simulations of materials from first-principles have improved drastically over the last decades, benefitting from newly developed methods and access to increasingly larger computing resources. Nevertheless, a quantum mechanical description of a solid without approximations is not feasible. In the wide field of methods for ab-initio calculations of the electronic structure, density functional theory and, in particular, the local density approximation have turned out to make also simulations of large systems accessible. Density functional calculations provide insight into the processes happening in a vast range of materials by their access to an understandable electronic structure in the framework of the Kohn-Sham single particle wave functions. A lot of functionalities in the fields of electronic devices, catalytic surfaces, molecular synthesis and magnetic materials can be explained analyzing the resulting total energies, ground state structures and Kohn-Sham spectra. However, challenging physical problems are often accompanied with calculations including a huge number of atoms in the simulation volume, mostly due to very low symmetry. The total workload of wave function based DFT scales roughly quadratically with the number of atoms. This leads to the necessity of supercomputer usage. In the present work, an implementation of DFT on real-space grids has been developed, suited for making use of the massively parallel computing resources of moder supercomputers. Massively parallel machines are based on distributed memory and huge numbers of compute nodes, easily exceeding 100,000 parallel processes. An efficient parallelization of density functional calculations is only possible when the data can be stored process-local and the amount of inter-node communication is kept low. Our real-space grid approach with a three-dimensional domain decomposition provides an intrinsic data locality and solves both, the Poisson equation for the electrostatic problem and the Kohn-Sham eigenvalue problem, on a uniform real-space grid. The derivative operators are approximated by finite-differences leading to localized operators which require only communication with the nearest neighbor processes. This causes an excellent parallel performance at large system sizes. Treating only valence electrons, we apply the projector augmented wave method for an accurate modelling of energy contributions and scattering properties of the atomic cores. In addition to the real-space grid parallelization, we apply a distribution of the workload of different Kohn-Sham states onto parallel processes. This second parallelization level avoids the memory bottleneck at large system sizes and introduces even more parallel speedup. Calculations of systems with up to 3584 atoms of Ge, Sb and Te have been performed on (up to) all 294,912 cores of JUGENE, the massively parallel supercomputer installed at the Forschungszentrum Jülich.