Scoring, Truthing, And Registration Toolkit (START)
The Situation
Every year, large volumes of imagery are collected for the sole purpose of evaluating Automatic Target Recognition (ATR) algorithms. However, ATR evaluation requires image truth for each target in every image of the video sequence. Truth information includes attributes of the target’s position, as well as other metadata about the operating scenario, sensor type and other factors. Specifying truth information for a large number of images is tedious, time consuming, and error prone. The best custom-designed truthing software in current use requires the user to draw a bounding region around each entity in an image frame. This amounts at least one hundred thousand clicks per hour of video, requiring days or weeks due to the precision required.
The Charles River Analytics Solution
We developed a complete truthing system we call the Scoring, Truthing, And Registration Toolkit (START) (figure 1). The truthing component of START only requires the user to truth a small number of key-frames and to specify some target motion parameters for each video sequence. START computes the truth information intelligently between key frames using a correlation-based template-matching algorithm, which allows for target rotation, translation, and scale change. A set of diagnostic measures is used to predict when the automatically generated truth values may be erroneous, allowing for operator intervention and correction. The scoring component of START computes and reports a set of standard scoring metrics on the output from the candidate ATR algorithm, using the truthed image data (figure 2). All truth data and scoring reports are stored in XML for ease of integration with large-scale simulation systems.
The Benefit
All current and recent DoD ATR efforts require toolkits such as START for assessing algorithm performance regardless of the type of sensor and platform used. Lack of a standard toolkit causes redundant developments costs every time a new contract or DoD program is developed. START offers a consistent, cost-effective approach to many fundamental elements of ATR evaluation.
Related Papers
Irvine, J., Ralph, S. K., Stevens, M. R., Kenyon, S., Anderson, D., Snorrason, M., and Gwilt, D. (2004) "A Scoring, Truthing, and Registration Toolkit for Evaluation of Target Detection and Tracking," Proceedings SPIE Defense & Security, vol. 5426. Orlando, FL (April). Abstract
Irvine, J. M., Ralph, S., Stevens, M. R., Marvel, J., Snorrason, M., and Gwilt, D. (2005) "START for Evaluation of Target Detection and Tracking," Proceedings SPIE Defense & Security, vol 5807. Orlando, FL (April). Abstract
Ralph, S. K., Irvine, J., Stevens, M. R., Snorrason, M., and Gwilt, D. (2004) "Assessing the Performance of an Automated Video Ground Truthing Application,'' Proceedings of Applied Imagery Pattern Recognition, Washington DC (October). Abstract
Ralph, S. K., Irvine, J., Stevens, M. R., and Snorrason, M. (2005) "START: A Tool for Rapidly Generating Image-Truth and Evaluating ATR Performance," Proceedings of WACV. Breckenridge, CO (January). Abstract
Stevens, M. R., Ralph, S. K., and Snorrason, M. (2004) "Interactive Truthing Tools for Moving Platforms and Moving Targets,'' Automatic Target Recognition Working Group, Eglin, FL (February).
