Research of Distributed Query and Optimization Method Based onMetadata

摘要

A method of distributed query based on metadata, which uses metadata to define and manage the virtual tablecontaining key information of the data source, has been studied in this paper. Then, in view of the different data level, designedtwo different data solutions based on query and optimization, for applying to common data and huge data respectively.In common data query, using the virtual table, the syntax analysis tree and memory database was realized by; copying,moving, and dividing the branch from virtual SQL query syntax tree to make the query optimized. In terms of hugeamounts of data query, Pig, Hadoop, Python is used to implement data query; by optimizing the Pig code, using multipleprocesses, processing file merging and file uploading or downloading in HDFS, making index on high frequency businessand so on to achieve optimization of big data.