A NUMA Aware Spark on Many-cores and Large Memory Servers

摘要

Within the scope of the CloudDBAppliance project, we investigate how Apache Spark can leverage a many cores and large memory platform, with a scale up approach as opposed to the commonly used scale out one: that is, the approach is to deploy a spark cluster to few large servers with many cores (up to several hundreds) and large memory (up to several tera-byte), rather than spreading it on many vanilla servers, and to stack several Spark executor processes per cluster node when running a job. It requires to cope with the non-uniform memory access within such servers, so we inculcate NUMA awareness to Spark, that provides a smart and application transparent placement of executor processes. We experiment it on a BullSequana S series platform with the Intel HiBench suite benchmark and compare performance where NUMA awareness is off or on.