基于Hadoop Streaming的Last比对软件并行化的研究与实现

摘要

With the arrival of the next generation sequencing technology, stand-alone version of the Last alignment soft-ware is not good enough for massive data processing while Hadoop Streaming technology can solve the problem by rapidly deploying Last software to the distributed cluster. It will be ensured to a balanced and effective data segmentation as well as control of the parallel granularity by the NFS-based customizing segmentation of data set and Partitioner based distribu-tion of tasks. Experimental results show that this method can guarantee the same results with the stand-alone operation, effectively reduce the running time with a relatively high speed.%随着下一代测序技术的到来，单机版Last比对软件已经不能满足海量数据的处理需求。使用Hadoop Streaming技术将Last比对软件快速部署到云计算环境中，解决当前单机版Last比对软件处理大数据能力差的问题。通过自定义的基于NFS 文件系统的数据集切分方法和基于Partitioner的任务分配方式能够实现均衡高效的数据切分，并保证并行化粒度可控。实验结果表明，在保证与单机运行结果一致的情况下，这种方法能有效缩减软件运行时间，具有较高的加速比。