System and method for scalable processing of collected knowledge by creating knowledge generation nodes

摘要

A knowledge generation machine (KGM) that can scale to the high volumes of software processing that may be required, as well as manage the unpredictable nature of the size of each processing element, is provided. A KGM master dynamically creates and destroys KGM nodes based on availability, performance, and resource allocation. Unlike the conventional systems, the present KGM receives performance feedback from KGM nodes and can use that performance information in determining which KGM nodes should be assigned information, reused, or destroyed. Further, efficient garbage collection ensures that resources can be de-allocated when needed. Thus, the KGM can scale to handle large spikes of incoming information.