Hybrid local/remote infrastructure for data processing with lightweight setup, powerful debuggability, controllability, integration, and productivity features

摘要

The technology provides a hybrid local/remote hosted MapReduce framework and infrastructure comprising systems and methods for improving setup, configuration, controllability, debuggability, and integration of a compute job and systems and methods for increasing programmer productivity. The system applies an interpreted programming language for the programmer's custom Map and Reduce algorithms, such that those algorithms can execute identically on both the hosted service as well as locally (e.g., on the programmer's local computing system or device) for development and debugging purposes. Furthermore, the system delivers this service—a hosted MapReduce infrastructure—in a simple and transparent web service.