Practice with Graph-based ANN Algorithms on Sparse Data: Chi-square Two-tower model, HNSW, Sign Cauchy Projections

by   Ping Li, et al.

Sparse data are common. The traditional “handcrafted” features are often sparse. Embedding vectors from trained models can also be very sparse, for example, embeddings trained via the “ReLuactivation function. In this paper, we report our exploration of efficient search in sparse data with graph-based ANN algorithms (e.g., HNSW, or SONG which is the GPU version of HNSW), which are popular in industrial practice, e.g., search and ads (advertising). We experiment with the proprietary ads targeting application, as well as benchmark public datasets. For ads targeting, we train embeddings with the standard “cosine two-tower” model and we also develop the “chi-square two-tower” model. Both models produce (highly) sparse embeddings when they are integrated with the “ReLu” activation function. In EBR (embedding-based retrieval) applications, after we the embeddings are trained, the next crucial task is the approximate near neighbor (ANN) search for serving. While there are many ANN algorithms we can choose from, in this study, we focus on the graph-based ANN algorithm (e.g., HNSW-type). Sparse embeddings should help improve the efficiency of EBR. One benefit is the reduced memory cost for the embeddings. The other obvious benefit is the reduced computational time for evaluating similarities, because, for graph-based ANN algorithms such as HNSW, computing similarities is often the dominating cost. In addition to the effort on leveraging data sparsity for storage and computation, we also integrate “sign cauchy random projections” (SignCRP) to hash vectors to bits, to further reduce the memory cost and speed up the ANN search. In NIPS'13, SignCRP was proposed to hash the chi-square similarity, which is a well-adopted nonlinear kernel in NLP and computer vision. Therefore, the chi-square two-tower model, SignCRP, and HNSW are now tightly integrated.


Understanding and Generalizing Monotonic Proximity Graphs for Approximate Nearest Neighbor Search

Graph-based algorithms have shown great empirical potential for the appr...

Constrained Approximate Similarity Search on Proximity Graph

Search engines and recommendation systems are built to efficiently displ...

2-Bit Random Projections, NonLinear Estimators, and Approximate Near Neighbor Search

The method of random projections has become a standard tool for machine ...

Composite Code Sparse Autoencoders for first stage retrieval

We propose a Composite Code Sparse Autoencoder (CCSA) approach for Appro...

Sign-Full Random Projections

The method of 1-bit ("sign-sign") random projections has been a popular ...

Graph Reordering for Cache-Efficient Near Neighbor Search

Graph search is one of the most successful algorithmic trends in near ne...

Analysis of SparseHash: an efficient embedding of set-similarity via sparse projections

Embeddings provide compact representations of signals in order to perfor...

Please sign up or login with your details

Forgot password? Click here to reset