Evaluation strategies for automatic linguistic indexing of pictures

With the rapid technological advances in machine learning and data mining, it is now possible to train computers with hundreds of semantic concepts for the purpose of annotating images automatically using keywords and textual descriptions. We have developed a system, the Automatic Linguistic Indexing of Pictures (ALIP) system, using a 2-D multiresolution hidden Markov model. The evaluation of such approaches opens up challenges and interesting research questions. The goals of linguistic indexing are often different from those of other fields including image retrieval, image classification, and computer vision. In many application domains, computer programs that can provide semantically relevant keyword annotations are desired, even if the predicted annotations are different from those of the gold standard. In this paper, we discuss evaluation strategies for automatic linguistic indexing of pictures. We provide both objective and subjective evaluation methods. Finally, we report experimental results using our ALIP system.