Alchemy: Distributed financial quantitative analysis system with high-level programming model

摘要

Nowadays, the financial securities investment and transaction are processed digitally. The securities companies have accumulated a large amount of financial data. Given that, quantitative analysis has become a common method for securities investors. However, as the scale of data continues to grow, data storing and processing have increasingly become a big challenge to financial quantitative analysts. First, most existing stand-alone quantitative analysis systems are hard to accelerate data processing in a distributed way. Second, existing distributed computing frameworks such as Apache Spark demands financial quantitative analysts with expertise knowledge on distributed computing. Third, transition from traditional stand-alone financial quantitative analysis (FQA) system to such distributed computing system often introduces learning efforts and development overheads. To solve these problems, we propose Alchemy, a distributed processing platform tailored for FQA with high-level programming interface. Alchemy offers distributed computing service and provides a pipeline-based programming model which shares many similarities with the traditional stand-alone systems, reducing learning efforts for financial quantitative analysts. The performance of Alchemy is evaluated in both experimental environment and a real-world production environment in Huatai Securites which is a leading financial company in China. The results show that Alchemy is able to achieve up to 300 times speedup compared against existing financial quantitative analysis systems.