Management of very large streaming data sets for efficient writes and reads to and from persistent storage

摘要

A method to produce a reverse skip list data structure in a computer readable medium, comprising: inputting streamed data to packets created in a temporary memory so as to create a sequence of packets; upon completion of creation of a packet in the stream, transferring the completed packet from the temporary memory to persistent memory; providing each of a plurality of respective packets with a respective pointer that skips over at least one other packet in the packet sequence and that indicates a location in persistent memory of a different respective packet in the packet sequence that was transferred to persistent memory prior to such providing of the respective pointer.