SCIENCE@home

How Well do Offline and Online Evaluation Metrics Measure User Satisfaction in Web Image Search?

Fan Zhang,K. Zhou,Yunqiu Shao,Cheng Luo,Min Zhang,Shaoping Ma

Published 2018 in Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

ABSTRACT

Comparing to general Web search engines, image search engines present search results differently, with two-dimensional visual image panel for users to scroll and browse quickly. These differences in result presentation can significantly impact the way that users interact with search engines, and therefore affect existing methods of search evaluation. Although different evaluation metrics have been thoroughly studied in the general Web search environment, how those offline and online metrics reflect user satisfaction in the context of image search is an open question. To shed light on this, we conduct a laboratory user study that collects both explicit user satisfaction feedbacks as well as user behavior signals such as clicks. Based on the combination of both externally assessed topical relevance and image quality judgments, offline image search metrics can be better correlated with user satisfaction than merely using topical relevance. We also demonstrate that existing offline Web search metrics can be adapted to evaluate on a two-dimensional presentation for image search. With respect to online metrics, we find that those based on image click information significantly outperform offline metrics. To our knowledge, our work is the first to thoroughly establish the relationship between different measures and user satisfaction in image search.

PUBLICATION RECORD

  • Publication year

    2018

  • Venue

    Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

  • Publication date

    2018-06-27

  • Fields of study

    Computer Science

  • Identifiers
    DOI 10.1145/3209978.3210059
  • External record

    Open on Semantic Scholar

  • Source metadata

    Semantic Scholar

CITATION MAP

EXTRACTION MAP

CLAIMS

  • No claims are published for this paper.

CONCEPTS

  • No concepts are published for this paper.

REFERENCES

Showing 1-58 of 58 references · Page 1 of 1

CITED BY

Showing 1-22 of 22 citing papers · Page 1 of 1