This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via Google Scholar Google Scholar Articles by Uitdenbogerd, A. L. Articles by Zobel, J. Search for Related Content

Methodologies for Evaluation of Note-Based Music-Retrieval Systems

School of Computer Science and IT, RMIT University, GPO Box 2476V, Melbourne 3001, Australia
School of Computer Science and IT, RMIT University, GPO Box 2476V, Melbourne 3001, Australia
School of Computer Science and IT, RMIT University, GPO Box 2476V, Melbourne 3001, Australia
alu{at}cs.rmit.edu.au
abhijit{at}cs.rmit.edu.au
jz{at}cs.rmit.edu.au

There have been many proposed music-retrieval systems, based on a variety of principles. How the effectiveness of these systems compares is not clear. The evaluation of some systems has been informal, without the rigor applied in other areas of information retrieval, and comparison of systems is difficult because of the lack of a common data set, queries, or relevance judgments. In this paper we explain how we collected artificial and expert music queries and name-based relevance judgments, and describe software we developed for collection of manual relevance judgments. Together with a collection of downloaded musical instrument digital interface (MIDI) files, these sets of queries and relevance judgments provide valuable tools for measuring music-retrieval systems. As an example of the value of these tools, we use them to compare the effect of using the expert queries and manual judgments to that of the artificial queries and manual judgments used in our earlier experiments.

Key words: relevance assessment; music-retrieval-system evaluation; queries; relevance judgments
History: received February 2004; revised January 2005; accepted March 2005.