Systems and methods are described for conducting data processing against a target data set using multiple, duration-limited task executions in an on-demand code execution system. The on-demand code execution system can enable execution of user-specified code as a task, but may establish duration limits for each task execution. Such duration limits can sometimes cause a task execution to fail, such as when processing a large data set. To overcome this problem, this disclosure enables data processing to occur over multiple task executions. Each task execution can process a portion of a data set selected such that data processing of the portion can complete prior to expiration of the duration limit. If an additional portion remains, the task execution can generate state information to enable an additional execution to conduct data processing on an additional portion, and pass that state information within a call to start the additional execution.
展开▼
