Parallel Acceleration of IBM Alignment Model based on Lock-Free Hash Table

摘要

Word alignment is still the most time-consuming task in statistical machine translation. This paper deeply analyzes the performance bottleneck of MGIZA++, which is the most commonly used word alignment tool, and proposes a lock-free programming method to speed up the modern word alignment parallel algorithm. Firstly, a word alignment oriented linear lock-free hash table is introduced. Secondly, three sub-tasks of IBM alignment model, i.e. model initialization, expectation calculation and normalization, are parallel designed and implemented. Experimental results show that the method proposed in this paper can significantly reduce the execution time compared with MGIZA++.