Web-Scale Responsive Visual Search at Bing

✨ Summary

Summary

The paper describes a production visual-search system deployed in Microsoft Bing for searching across a general web-scale image index containing tens of billions of images. Its design addresses three competing requirements: relevance, response latency, and storage scalability.

The system uses a three-stage retrieval and ranking pipeline. First, Level-0 matching applies a sharded inverted index based on Bag-of-Visual-Words representations to reduce the search space by roughly four orders of magnitude. Level-1 ranking uses a relatively inexpensive deep neural network feature and Product Quantization (PQ) to reduce the candidate set from millions to thousands. Level-2 ranking retrieves richer visual features and applies a LambdaMART model that combines multiple signals, including DNN embeddings, category features, face features, color, object-detection outputs, text-matching signals, and deduplication features.

The visual representations are produced by several CNN architectures, including AlexNet, ZFSPPNet, GoogleNet, GoogleNetBN, and ResNet-50. The authors train models using Bing-specific data, human relevance judgments, web-scraped data, classification objectives, pairwise ranking loss, and triplet loss. Object detection, using Faster R-CNN and SSD, supports interactive hotspots and helps isolate objects such as fashion and furniture items. The system also detects shopping intent and can return related products with merchant and price information.

To reduce memory and computation costs, the system applies PCA followed by PQ. For example, a 2048-dimensional ResNet representation is compressed from approximately 8 KB to 25 bytes per image. The visual-word representation uses approximately 64 bytes per image. Features and indexes are distributed across hundreds of machines, with SSD-based storage used for faster random access. CPU inference is optimized through vectorized low-level implementations, multithreading, caching, and a graph-based feature-extraction runtime; GPU deployment is reported to accelerate feature extraction by roughly 10–20 times for applicable models.

On the reported evaluation set, the proposed multi-feature system achieves NDCG@5 of 74.20, compared with 70.81 for the strongest listed single-feature PQ baseline. PCA and PQ cause modest relevance reductions for the tested ResNet feature, lowering NDCG@5 from 71.84 for the raw feature to 70.81 after quantization. Reported median-style 50% latencies are 174 ms for an image already present in the index and 1083 ms for a user-uploaded image; feature extraction accounts for much of the latter cost, while cascaded ranking is reported at 49 ms.

The evaluation has important limitations: relevance is measured on a constrained set of candidate images using human pairwise judgments, and the reported NDCG primarily evaluates final reranking rather than the complete billions-scale retrieval process. The paper therefore presents an engineering account of how approximate indexing, cascaded ranking, compressed representations, distributed retrieval, and heterogeneous hardware can jointly make general visual search practical.

Influence and subsequent use

The paper became a documented reference point for later commercial visual-search work. Alibaba’s contemporaneous KDD system described a separate large-scale visual-search architecture for e-commerce, addressing similar production concerns such as deep feature learning, object localization, indexing, reranking, and high-throughput serving. (kdd.org) Later Microsoft work on web-scale generic object detection explicitly cited the Bing visual-search system as an example of using object detection to improve visual-search engagement and relevance, extending the object-understanding component toward a broader production detection platform. (researchgate.net) The paper is also cited in subsequent visual-search, e-commerce retrieval, and content-based image-retrieval literature as a production-scale example of combining deep representations with cascaded retrieval and compressed indexing. (doi.org)