Efficient Stream Processing of Scientific Data

摘要

Modern particle physics produces volumes of experimental data that challenge any data processing system. To illustrate, the trigger system of the LHCb experiment at CERN must sustain a data rate of 4 TB/s, yet maintain real-time characteristics. In this work, we report on ELPACO, a distributed event processing platform for scientific data. Its key characteristics are excellent scalability and high resource efficiency. ELPACO inherits its favorable scalability from Apache Storm, which we used as a basis for our platform. For resource efficiency, we tailored ELPACO to Eriador, a parallel, ARM-based hardware substrate with excellent energy/performance characteristics. With experiments on realistic data, we confirm a linear scalability (throughput vs. core count) and a 2.5 × improvement in energy efficiency compared to existing solutions.