Continuous Cross Identification in Large-Scale Dynamic Astronomical Data Flow

摘要

In modern astronomy, Short-Timescale and Large Field-of-view (STLF) sky survey produce large volume data and face a great challenge in cross identification. Furthermore, transient survey projects are required to select the candidates fast from large volume data. However, traditional cross identification methods didn't satisfy the observation of transient survey. We present a fast and efficient cross identification system for large-scale astronomical data streams. By receiving a high-frequency star catalog and maintaining a local star catalog, the system partitions the star catalog and cross identification with the object catalog. A coding strategy is used to manage the unique ID of the all-sky star. After processing data, all the results are stored in Redis and generate the light curve. Our experiment shows that the method could meet the strict performance requirements and good recognition accuracy on fast real-time sky survey project. Additionally, our system shows good performance in low latency large volume astronomical data processing and our system has been successfully applied in the Ground-based Wide Angle Camera (GWAC) online data processing pipeline.