TapTell: Interactive visual search for mobile task recommendation

•We present a system-driven TapTell, demonstrates efficiency and usability.•It tackles a real mobile based visual search and task recommendation problem.•This paper provides a natural user interaction for better visual search.•We propose a context-embedded vocabulary tree (CVT) method.•CVT can leverage object context of the visual query for better search results.

Mobile devices are becoming ubiquitous. People use them as personal concierge to search information and make decisions. Therefore, understanding user intent and subsequently provide meaningful and personalized suggestions is important. While existing efforts have predominantly focused on understanding the intent expressed by a textual or a voice query, this paper presents a new and alternative perspective which understands user intent visually, i.e., via visual signal captured by the built-in camera. We call this kind of intent “visual intent” as it can be naturally expressed through a visual form. To accomplish the discovery of visual intent on the phone, we develop TapTell, an exemplary real application on Windows Phone seven, by taking advantages of user interaction and rich context to enable interactive visual searches and contextual recommendations. Through the TapTell system, a mobile user can take a photo and indicate an object-of-interest within the photo via different drawing patterns. Then, the system performs a search-based recognition using a proposed large-scale context-embedded vocabulary tree. Finally, contextually relevant entities (i.e., local businesses) are recommended to the user for completing mobile tasks (those tasks which are natural to be raised and subsequently executed when the user utilizes mobile devices). We evaluated TapTell in a variety of scenarios with millions of images, and compared our results to state-of-the-art algorithms for image retrieval.