ELF: A Constraint-Aware XQuery Engine for Processing XML Streams with Minimized Memory Footprint

摘要

XML and XQuery [1] have been widely accepted as the standard data representation and query language for web applications. When the input consists of a large amount of XML tokens, the main memory buffer requirement in XML stream processing can be significant, which might also lead to a significant CPU consumption due to the manipulation cost on the buffered data. To provide real-time responses, serious challenges in memory utilization are faced by the XQuery evaluation over XML streams. In many practical applications, XML streams are generated following a pre-defined semantic constraint such as the Document Type Definition (DTD) and XML schema [1], shown by the following two scenarios: Network Traffic Monitoring. For monitoring network traffic, anomalies of network traffic flow may need to be detected from the statistical data sent in XML streams. In such a case, the XML stream, which would be generated by a work-flow engine or simply a customized program, will follow a pre-defined schema. News Publishing. In such scenario, the news server retrieves news from a large number of sources, such as different reporter devices, different broadcast agencies and government sources and disseminates messages as an XML stream to subscribers. The sources may all agree with a pre-defined schema. Utilizing such semantic knowledge of the stream input enables us to on-the-fly predict the non-occurrence of a given pattern within a bound context. This can help to avoid data buffering and to release buffered data at an earlier moment, thus achieving a minimized memory footprint. Let us consider a query as below: FOR $a IN $ROOT/news WHERE $a/location = "Boston" RETURN $a/entry, $a/comment Without semantic knowledge, for each bound news element, the earliest we can perform the predicate checking on locations, output entries and comments then release the corresponding buffer is after the news has been completely received. Assume we are given the semantics of the news element type as the DTD below: . By such schema knowledge of the input stream, if two consecutive advertisements subelements are met within an news element, we can guarantee that no more location can occur under this news. If none of the received locations within the news is equal to "Boston", buffered entries of the binding can be purged from memory and future coming comments can be directly dropped without any buffering because this news is guaranteed to be failing. More examples on query optimization by utilizing such semantic knowledge can be found in [5]. State-of-the-Art. Reducing memory consumption is very important for stream applications. Only a limited number of XML stream processing engines [3] [6] [4] have looked at the schema-based query optimization opportunity. Among them, FluXQuery [4] does not support filtering-related optimizations. The focus of [6] and [3] is not on buffer minimization. They only statically capture limited constraints from the given schema. ELF Solution. We propose ELF (Constraint-aware XQuery Engine for Processing XML Streams with Minimized Memory Footprint) in this demonstration. Given the DTD of the input stream, ELF on the fly detects the Pattern Non-Occurrence (PNO) constraints [5] and then utilize such runtime constraint knowledge to adjust the buffering strategy dynamically, thus effectively decreasing the memory consumption.