W36 - Beyond mAP: Reassessing the Evaluation of Object Detection

摘要

The BMREOD workshop was conducted to address the future of how the computer vision community compares and analyses object detectors, and promote the new field of probabilistic object detection (PrOD). It had been identified that, in recent years, through a focus on evaluation using challenges, the object detection community had been able to quickly identify the effectiveness of different methods by standardized testing and examining performance metrics. However, it was seen as important to reassess these standardized procedures to ensure they provided enough insight into the specific aspects of object detection that can be analyzed, and that performance adequately aligned with how they might be used in practical applications. A key concept that drove this aspect of the workshop was that quantitative results should be easily reconciled with a detector's performance in applied tasks. This workshop gave the computer vision community a platform to assess the current evaluation standards, find any weaknesses therein and propose new directions forward to improve object detection evaluation. The workshop also hosted the 3rd PrOD challenge, aimed at promoting object detection which provides both spatial and semantic uncertainty estimates accurately. Participants that achieved high scores were provided with the opportunity to supply a paper to the workshop and receive monetary prizes. The final workshop comprised of 12 presentations, including keynote presentations, workshop paper presentations, and PrOD challenge presentations. Three keynote presentations were provided by invited speakers Assoc. Prof. Emre Akbas, Assoc. Prof. Walter Scheirer, and Dr Larry Zitnick, providing their thoughts on the future of object detection evaluation research. Two presentations were given by workshop organizer Dr David Hall, one adding to the general object detection evaluation discussion, and one summarizing the PrOD challenge. Including all workshop papers and PrOD challenge papers, nine papers were submitted to the workshop. After review by the workshop organizers, seven were accepted for the workshop, comprizing of four workshop papers and three PrOD challenge papers. We would like to thank all our fellow researchers for their contributions to the BMREOD workshop, and both the ECCV general and workshop chairs for providing the material and support for this workshop. This workshop was supported by the Australian Research Council Centre of Excellence for Robotic Vision (project number CE140100016), and the QUT Centre for Robotics, Australia.