Dynamic Query Modification for Binary Locality Sensitive Hashing

Abstract

Our context of interest is how binary locality sensitive hash (LSH) functions can be used to solve the approximate near neighbour (ANN) problem in hyperspherical spaces, which seeks to find the $k$ closest elements of some dataset $X \subset \mathbb{R}^d$ to some further point $\mathbf{q} \in \mathbb{R}^d$ presented as a query. Binary locality sensitive function families $\mathcal{H}$ are sets of functions each with signature $h: \mathbf{x} \in \mathbb{R}^d \rightarrow \{ 0, 1 \}$. A function is locality sensitive if the output of the function is more likely to be equal (a `hash collision') if two close vectors are used as input than if two far vectors are used. In this paper we introduce dynamic query modification. This procedure changes $\mathbf{q}$ at query time using information acquired during search to approximate the point $\langle \mathbf{c} \rangle$, which is the $\ell_2$-normalised centroid of the $k$-nearest neighbour set of $\mathbf{q}$. By theoretical and experimental analysis we demonstrate $\langle \mathbf{c} \rangle$ has two significant advantages when compared to $\mathbf{q}$. Firstly, we show that the hash function output of $\langle \mathbf{c} \rangle$ is likely to have a higher average collision probability with the $k$-nearest neighbours than the hash output of $\mathbf{q}$. Secondly, in our experiments, we observe no instances of $\langle \mathbf{c} \rangle$ failing to collide with any member of the near-neighbour set; a property which we demonstrate is not true for $\mathbf{q}$. To demonstrate the efficacy of the technique, we define a novel structure MQ-Forest, a modified version of RP-Forest. Both are binary LSH-based ANN mechanisms, but MQ-Forest dynamically estimates $\langle\mathbf{c}\rangle$ during the query process. We show that MQ-Forest reduces both build and query times by up to $40\%$ when measured over several large, high-dimensional benchmark datasets.