Shareable, Persistent, In-Memory, Read-Only Data

摘要

Ensuring that multiple processes can access required data in a timely fashion is a critical aspect of successful parallel processing. The larger the data set, the more profoundly the overall run time is affected by reading the data and integrating it into the appropriate data structures and/or communicating the data between processes. Either each process must read and configure the data, or they must waste time in requesting and receiving data from another process. What is described in this paper is a Python-specific technique for having persistent, in-memory data that multiple processes can access without the overhead of loading or communicating the data.