Okapi at TREC-3
đź“ś Abstract
The emphasis in TREC-3 has been on further refinement of term-weighting functions; an investigation of run-time passage determination and searching; expansion of ad hoc queries by terms extracted from the top documents retrieved by a trial search; new methods for choosing query expansion terms after relevance feedback, now split into: methods of ranking terms prior to selection; subsequent selection procedures; and the development of a user interface and search procedure within the new TREC interactive search framework. The two successes have been in query expansion and in routing term selection. The modified term-weighting functions and passage retrieval have had small beneficial effects. For TREC-3 there were to be topics without the CONCEPTS fields, which had proved to be by far the most useful source of query terms. Query expansion, passage retrieval and the modified weighting functions, used together, have gone a long way towards compensating for this loss.
✨ Summary
Paper summary
- The paper reports the City University Okapi system’s TREC-3 experiments in probabilistic information retrieval.
- It develops BM25, a family of weighting functions that extends the Robertson–Sparck Jones probabilistic model with within-document term frequency, query term frequency, and document-length normalization. The experiments generally favored a document-length parameter near 0.75, while the more elaborate nonlinear parameterization provided little additional benefit.
- Runtime passage retrieval was feasible but computationally expensive. Its standalone gains over whole-document retrieval were modest, although its combination with query expansion produced more noticeable improvements.
- Query expansion using terms extracted from top-ranked documents in an initial search—without explicit relevance judgments—was unexpectedly effective for ad hoc retrieval, particularly when combined with passage retrieval.
- For routing, ranking candidate expansion terms with RSV, RSV2, or ARSV produced broadly similar results. A computationally intensive stepwise procedure that accepted or rejected terms according to training-set performance produced substantially better retrospective and predictive results, but raised practical concerns about efficiency and generalization.
- The interactive experiments introduced phrase handling, user-controlled term sets, relevance feedback, and removal of noisy automatically extracted terms. The authors caution that interactive and automatic results were not directly comparable because the automatic routing experiments used much more relevance information.
Subsequent influence
The paper is a primary reference for Okapi BM25. Apache Lucene’s documentation explicitly identifies this paper as the work in which BM25 was introduced and provides a production implementation of the model. (lucene.apache.org) BM25 later became the default text-similarity model in Elasticsearch and is also the default similarity for applicable fields in modern Solr configurations, demonstrating direct adoption in widely used search infrastructure. (elastic.co) The paper is also catalogued as a heavily cited TREC contribution in bibliographic and research-impact records. (impact.ref.ac.uk)