Dynamic Server Pool Data Segmentation Using Dynamic Ordinal Partition Key Without Locks

摘要

Aspects described herein may relate to a data processing engine that executes on a computing device in order to store data from one or more feed files, which may be heterogeneous, to a destination data structure on a designated computing device. Because the files may be huge in size, it is important that the files be stored in a manner in order to reduce the time to move the data and to support am efficient mechanism for recovering from errors. A feed file may be dynamically partitioned into groups of contiguous rows based on a dynamic partitioning key, where data chunks are loaded into a plurality of clone tables and subsequently moved into a destination data structure. The data processing engine may determine a row size for the clone files and request for resources from a computing cloud to obtain those resources.