Performance Evaluation of Sensors on Mobile Vehicles Using a Large Data Repository and Ground Truth

摘要

Progress in algorithm development and transfer of results to practical applications such as military robotics requires standard qualitative and quantitative measurements for performance evaluation and validation. Although the evaluation and validation of algorithms have been discussed for well over a decade, the research community still faces a lack of well-defined and standardized methodology. In this research, we describe three methods for creating ground truth databases and benchmarks using multiple sensors, for use in mobile robotics. The databases and benchmarks provide researchers with high quality data from suites of sensors operating in complex environments representing real-world problems. At NIST, we have equipped a High Mobility Multi-purpose Wheeled Vehicle (HMMWV) with a suite of sensors including a Riegl ladar, GDRS ladar, stereo CCD, several color cameras, Global Positioning System (GPS), Inertial Navigation System (INS), pan/tilt encoders, and odometry. All sensors are calibrated and registered with each other in space and time. This allows a database of features and terrain elevation to be built. Ground truth information is collected through aerial surveys, from maps, by human annotation, and by previous traverses of the terrain by the vehicle. Ground truth may include terrain elevation information, feature information (roads, road signs, trees, ponds, fences, etc.) and constraint information (e.g., one-way streets). We have implemented our a priori database using One Semi- Automated Forces (OneSAF), a military simulation environment. Using the Inertial Navigation System and Global Positioning System (GPS) on the HMMWV to provide indexing into the database, we extract all the elevation and feature information for a region surrounding the vehicle as it moves about the NIST campus. Ground truth for each sensor can be obtained by projecting this information into the sensors' coordinate systems.