Efficient processing of substring match queries with inverted variable-length gram indexes

•We study substring matching with inverted indexes using variable-length grams.•We develop several pruning rules to reduce the large search space significantly.•We present both optimal and approximate algorithms to reduce search space.•We did performance study to show the effectiveness of our algorithms.•Our algorithms are much more effective than traditional algorithms in experiments.

With the widespread use of the internet, text-based data sources have become ubiquitous and the demand for effective support of string matching queries continues to increase. The relational query language SQL supports LIKE clauses for string data to handle substring matching queries. Due to the popularity of such substring matching queries, there have been many studies on designing efficient indexes to support the LIKE clause in SQL. Among them, the indexes based on q-grams with fixed or variable sizes have been studied extensively.In this paper, we show that the optimal processing of substring matching queries with inverted variable-length gram indexes should be decided judiciously. However, the search space of finding the optimal variable-length gram set among those available is exponential with the number of grams. To reduce the search space, we present effective pruning strategies which do not sacrifice optimality. Based on the cost estimation, we propose the optimal algorithms to find the best plan with the minimum cost for substring matching queries using these pruning strategies. We also provide the approximate algorithms to overcome the exponential nature of search space to find an optimal query plan. Our performance study confirms that the proposed techniques improve query execution time for substring matching significantly compared to the running time of the traditional algorithms.