Distributed computing has got extensive application with the coming of the big data era .Open source distrib‐uted computing frameworks headed by Hadoop and Spark lead the relevant industry standards .However ,there are difficul‐ties in using and second‐round developing Hadoop and Spark ,while the former is programmed with Java and the latter is pro‐grammed with Scala .But Dpark ,a distributed computing framework programmed with Python ,extremely improves work ef‐ficiency of data analysis ,because it not only inherits the mechanism of memory calculation and lazy evaluation from Spark , but also combines with the concise syntax of Python .What's more ,it is able to cooperate with MooseFS ,which is a distribu‐ted file system ,Beansdb ,which is a distributed database ,and Mesos ,which is a distributed resources scheduling frame‐work ,naturally .The work efficiency of traditional Python program and the Dpark‐based program in data preprocessing are compared ,while the performance and scalability of the latter is better than the former .%随着大数据时代的来临，以 Hadoop 和 Spark 为首的开源分布式计算框架主导着相关行业的事实标准。然而，无论是使用 Java 编写的 Hadoop ，还是使用 Scala 编写的 Spark ，使用及对其进行二次开发的难度都比较大，而使用 Py‐thon 编写的分布式计算框架 Dpark ，具有继承自 Spark 的内存计算和惰性求值机制，结合 Python 的简洁语法，同时又配合分布式文件系统 MooseFS 、分布式数据库 Beansdb 和分布式资源调度框架 Mesos ，可以极大提高数据分析的工作效率。文章主要对比了传统 Python 程序和基于 Dpark 的 Python 程序在完成数据预处理工作上的运行效率，得出后者的性能和可扩展性至少优于前者数十倍的结论。
展开▼
