CN103365998B - A kind of similar character string search method - Google Patents

Abstract

The invention discloses a similar character string retrieval method, comprising: using the overlapping information between inverted linked lists in an inverted index to obtain a gram identification degree index; integrating the gram identification degree index with the length ratio of the inverted linked list to form a comprehensive Index; According to the comprehensive index, the candidate set is obtained through the generalized prefix filter, and the real edit distance of the strings in the candidate set is calculated to obtain the retrieval result. The similar character retrieval method of the present invention can improve the retrieval efficiency, especially when the editing distance threshold increases, the retrieval efficiency can be greatly improved.

Description

A method for searching similar character strings

technical field

The invention belongs to the technical field of databases, and in particular relates to a method for retrieving similar character string data.

Background technique

In recent years, a lot of research has been done on string similarity queries in the database field. String similarity query can be widely used in data fusion, data cleaning, duplication detection, natural language processing and bioinformatics. For example, non-homologous database records representing the same entity in the real world may have some errors, and data fusion of these same entities is required when building a data warehouse; in protein detection, it is necessary to quickly obtain relevant information about new protein sequences, At this time, the protein sequence most similar to the new protein can be used to describe the new protein sequence; in the existing search engine, since human input often contains errors, the search engine needs to provide some error correction functions at this time. Prompt and help users identify errors in their input.

The latest existing methods for solving string similarity queries include VChunk, asymmetric-Search and Adaptive-Search. Among them, VChunk uses a structure based on the statistical Chunk boundary dictionary to split the string to generate a signature for retrieval. Asymmetric-Search uses different signatures for queries and indexes (one uses gram, the other uses chunk) to reduce matching items, thereby improving performance. Adaptive-Search is mainly based on an observation: Through experimental observations, it is found that when using a prefix filter, appropriately increasing the number of grams used to access the posting list can improve performance. The basic filter used by the first two methods is a prefix filter. The traditional prefix filter will generate a large number of candidate sets; the third method intends to improve the traditional prefix filter by increasing the number of grams actually used to access the inverted index. But because it is only a heuristic rule based on experimental observations, it does not achieve the best performance.

Contents of the invention

The invention overcomes the defects in the prior art that the candidate set is too large due to the increase of the edit distance threshold, various filters cannot be adaptively selected, and the quantitative index of the existing query optimization is inaccurate, etc., and proposes a similar string retrieval method .

The present invention proposes a kind of similar character string retrieval method, comprises the following steps:

Step 1: Using the overlapping information between the inverted linked lists in the inverted index to obtain the gram recognition index;

Step 2: Integrating the gram recognition index and the length ratio of the inverted linked list to form a comprehensive index;

Step 3: According to the comprehensive index, the candidate set is obtained through the generalized prefix filter, and the search result is obtained by calculating the real edit distance of the character strings in the candidate set.

The present invention proposes a method for retrieving similar character strings, wherein, before step 1, an inverted index is first established for a set of character strings to be retrieved, and a MinHash signature corresponding to the inverted index is generated.

The present invention proposes a similar character string retrieval method, wherein, by numbering the character string and splitting the q-gram, an inverted index is established by taking the split gram as an entry and mapping to the character string containing the gram .

The present invention proposes a similar character string retrieval method, wherein a hash function is selected according to the length of the character string, and a MinHash signature is generated in combination with the inverted index.

The present invention proposes a similar character string retrieval method, wherein, in step 1, the gram recognition degree index is represented by the sum of the number of grams having the same MinHash signature as the target gram in the gram collection of the character string to be queried, and the sum is not Contains the target gram.

The present invention proposes a similar character string retrieval method, wherein, in step 2, the length of the inverted linked list is calculated based on the number of elements of the inverted linked list actually used, and the gram is integrated by adding the weights based on the ratio of the length of the linked list The recognition index and the length index of the posting list form a comprehensive index.

The present invention proposes a similar character string retrieval method, wherein said step 3 obtaining retrieval results through said generalized prefix filter comprises the following steps:

Step A1: After splitting the query string by gram, sort the split query string in ascending order according to the comprehensive index;

Step A2: Select the first τ+1 grams of the non-overlapping part of the string as the basic gram query set, where τ is the distance threshold;

Step A3: Add new grams in a loop and calculate the increased gain until the newly added gram has no gain or no new gram;

Step A4: Merge all the grams, and take out the character strings whose occurrences are at least LB times and add them to the candidate set, and perform real edit distance calculation on the character strings in the candidate set, so as to obtain the retrieval results; where LB refers to the The lower limit of the number of identical grams used by the generalized prefix filter described above.

The present invention proposes a similar character string retrieval method, wherein, the gain in the step A3 is represented by the following formula:

In the formula, G represents the gain, C _i represents the size of the current candidate set, |I(g _n )| represents the length of the inverted list of g _n , J _s represents the Jaccard similarity, and cost(Q) represents a real Average time spent on edit distance calculation.

The invention proposes a generalized prefix filter, and proposes a new index for measuring gram quality by utilizing the coincidence information between inverted linked lists. The present invention can improve the retrieval efficiency through the optimized generalized prefix filter, especially when the edit distance threshold becomes higher, the performance can be greatly improved. The present invention simultaneously overcomes the shortcomings of the traditional method that cannot select the optimal filter self-adaptively.

Description of drawings

Fig. 1 represents the flow chart of similar character string retrieval method of the present invention;

Fig. 2 shows the inverted index in the embodiment.

Fig. 3 shows the MinHash signature in the embodiment.

detailed description

The present invention will be further described in detail in conjunction with the following specific embodiments and accompanying drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the content specifically mentioned below, are common knowledge and common knowledge in this field, and the present invention has no special limitation content.

The similar character string retrieval method of the present invention introduces a new generalized prefix filter to reduce the candidate set to the greatest extent. The similar character string retrieval method of the present invention makes full use of the information of the inverted index, proposes to use the overlapping information between the inverted linked lists as an index, and increases high-quality grams to reduce candidate sets without reducing performance. Although it increases the estimation overhead compared to the state-of-the-art, it greatly improves the retrieval performance. As shown in Figure 1, similar character string retrieval method of the present invention comprises the following steps:

Step 1: Use the overlapping information between the inverted linked lists in the inverted index to obtain the gram identification index; where the gram identification index in step 1 is from the gram collection of the query string, which has the same MinHash signature as the target gram The sum of the number of grams, which does not include the target gram.

Further, before step 1, an inverted index is established for the set of character strings to be retrieved, and a MinHash signature corresponding to the inverted index is generated. In the present invention, the character string is numbered and q-gram split, and the split gram is used as an entry to map to a character string containing the gram to establish an inverted index. Select the hash function according to the length of the string, and combine the inverted index to generate the MinHash signature.

Step 2: Integrate the gram recognition index with the length ratio of the posting list to form a comprehensive index. Wherein, in step 2, the length of the posting list is calculated based on the number of posting list elements actually used, and the gram recognition index and posting list length index are integrated by adding the weight based on the ratio of the linked list length to form a comprehensive index.

Step 3: According to the comprehensive index, the candidate set is obtained through the generalized prefix filter, and the real edit distance of the strings in the candidate set is calculated to obtain the retrieval result. Wherein, the step 3 to obtain the retrieval result through the generalized prefix filter includes the following steps:

Step A1: After splitting the query string by gram, sort the split query string in ascending order according to the comprehensive index;

Step A2: Select the first τ+1 grams of the non-overlapping part of the string as the basic gram query set, where τ is the distance threshold;

Step A3: Add new grams in a loop and calculate the increased gain until the newly added gram has no gain or no new gram; wherein, the gain in step A3 is expressed by the following formula:

In the formula, G represents the gain, C _i represents the size of the current candidate set, |I(g _n )| represents the length of the inverted list of g _n , J _s represents the Jaccard similarity, and cost(Q) represents a real Average time spent on edit distance calculation.

Step A4: Merge all the grams, and take out strings with occurrences of at least LB times and add them to the candidate set, and perform real edit distance calculation on the strings in the candidate set to obtain the retrieval results, where LB refers to generalized prefix filtering The lower limit of the number of same grams used by the processor.

In the present invention, the generalized prefix filter refers to the filtering of only the first few grams of the sequence to filter irrelevant character strings in the gram collection sorted in ascending order according to the comprehensive index when the strings are searched for similarity Way.

Example:

In the present invention, two parts need to be completed in the preprocessing stage: establishing an inverted index and generating a MinHash signature.

The inverted index needs to record the mapping relationship from gram to string. Table 1 is a set of character strings, including 5 character strings, which are numbered from 0 to 4.

Table 1 The character string of this embodiment

id Strings 0 rich 1 stick 2 stich 3 stuck 4 stud

First split all 5 character strings into 2-grams, and then use all the appearing grams as entries to create an inverted index to obtain an inverted index as shown in Figure 2. Wherein, all strings in the string collection containing the current gram are saved on the inverted list.

Since the generation process of the MinHash signature requires a replacement operation, a hash function is usually used to simulate the replacement operation. Table 2 is two hash functions. Use the same set of hash functions for strings of the same length. In this embodiment, each set of hash functions includes only one hash function.

Table 2 Hash functions used by different groups

string length hash function 4 (x+1)%2

5 (2x+3)%5

The first hash function is used for string number 0 because its length is 4, and the second hash function is used for string numbers 1-4 because they all have length 5. Taking ch as an example, in the interval of length 4, because there is only one character string, its MinHash signature value is (0+1)%2=1. In the interval of length 5, there are 4 strings, so calculate the first non-zero value of each string after being shuffled by the hash function. Because the numbers in the inverted linked list represent the bits that are 1 after vectorization, directly substitute the numbers in the linked list into the hash function in turn, and select the smallest bit. Use the second hash function on the two non-zero bits to get (2*2+3)%5=2 and (2*4+3)%5=1, because the latter one is the smallest, so choose the latter one The hash value is signed as a MinHash of length interval 5. The final MinHash signature of ch is 1-1. Figure 3 shows the MinHash signature generated based on the inverted index in Figure 2.

For the quality of the gram, the present invention proposes a new comprehensive index: based on the length of the gram inverted list and the recognition degree of the gram. The recognition index of a gram is used to measure how many candidate results a gram generates. The lower the recognition index, the less likely it is for a gram to generate candidate strings together with other grams. Therefore, the quality of a gram is higher, and it should be preferred over other grams. . Since the selection of any D+LB gram in the generalized gram filter can guarantee the correct result, but the performance of the filter produced by different gram combinations is very different. The quality of the gram. Gram recognition is a relative index, which must be based on a specific q-gram set. Its specific definition is as follows: For the q-gram set Gq(S) of a given query string S, for any gram g _i ∈ Gq(S), I(g _i ) represents the collection of all character strings in the inverted linked list with gram g _i as the entry in the inverted index. For any e _j ∈ I(g _i ), first count Gq(S) The number of grams of e _j is included in the inverted list of each gram in , and the value is subtracted by 1, which is recorded as the recognition degree of each element, and the identification of each element in the I(g _i ) inverted list is Degrees are summed to obtain the recognition degree of gram g _i . However, since the inverted list is usually very long, it is very time-consuming to actually calculate the index in the query. Therefore, the present invention estimates the gram recognition degree based on the MinHash signature. The estimation method is similar to the calculation of the real recognition degree. The degree method can be changed from the original reverse to MinHash signature. For example, if the current query is strick, and τ=1, the 2-gram set of the query is {st, tr, ri, ic, ck}. Taking st as an example, since τ=1, the string length L of strick is 6. According to the nature of string retrieval, only the part in the interval [L-τ, L+τ] in the inverted list needs to be considered, that is, only The MinHash signature of interval 5 needs to be considered. In the length interval 5, the MinHash signature of st is 0. Among the 2-grams split from all queries, there are 3 ck, ic and st whose signatures are also 0 under the same hash function, so the estimated recognition degree of st is 3-1=2. The total weight of the gram generated by combining the length of the inverted list and the degree of recognition can be directly added to the weight of 1/2 after regularization. Table 3 indicates that the combined weight represents the gram quality of each gram in each interval.

Table 3 The weight of gram recognition degree and the length ratio of inverted list

gram Recognition 4 Recognition 5 length 5 Merge Weight 5 total weight st 0 2 4 0.42 0.42 tr null null null null null the ri 1 0 0 0 0 ic 1 2 2 0.29 0.29 ck 0 2 2 0.29 0.29

When merging the weights of different intervals, the sum is performed in proportion to the number of character strings contained in different intervals. Therefore, for st, the total recognition is 0*0.2+0.42*0.8=0.336.

The search framework based on the generalized prefix filter of the present invention is divided into two steps: generating base query gram sets and adding new grams. Before execution, the gram generated by the query must be sorted in ascending order according to the comprehensive quality index of the gram. As shown by the final weights in Table 3, the final order is ri, ic, ck, st. Since the grams containing the same position will be affected by the editing operation at the common position at the same time, if the newly added gram contains the selected position, it will not bring about a performance increase. Therefore, when selecting the gram, If the previous gram is selected, the newly added gram cannot share the position with the selected gram. For example, the string abcdef, if gram ab is selected, then bc cannot be used, because ab shares b with bc. After sorting, select the top minimum τ+1 grams that satisfy the no-shared position as the base query gram set. In this embodiment, ri and ck are selected (because ri and ic have shared parts, so ic is not selected), and the inverted linked lists of these two grams are merged, and all elements 1 and 3 appearing on the linked list are used as candidate results. The step of adding new grams is a cyclic process, and the loop stops when there is no performance gain. Using Indicates gain. Among them, C _i represents the size of the current candidate set, |I(g _n )| represents the length of the inverted list of g _n , J _s represents the Jaccard similarity, and cost(Q) represents the average value of a real edit distance calculation for the query Time consuming. If the gain is greater than 1, it means that the newly added gram can improve performance and can be added to the candidate set. Search for newly added grams in order according to the previous sorting, and then perform gain evaluation on st. Through calculation, increasing st has no performance gain, so it cannot be added to the candidate set, and the loop stops. In this embodiment, the candidate set is {1, 3}, that is, {stick, stuck}. After calculating the real edit distance between these two candidate results and strick, only the real edit distance of string 1 satisfies the query strick distance within 1, so the string 1 is obtained as the result that satisfies the condition.

The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the inventive concept, changes and advantages conceivable by those skilled in the art are all included in the present invention, and the appended claims are the protection scope.

Claims (4)

1. a similar character string search method, it is characterised in that comprise the following steps:

Step one: utilize the coincidence information between the table of falling row chain in inverted index, obtains gram identification index；Wherein, described Gram identification index is by the gram set of character string to be checked, has identical MinHash signature with target gram The summation of gram number represents, does not comprise target gram in this summation described；

Step 2: described gram identification index integrated with the table of falling row chain length ratio, forms aggregative indicator；Wherein, The table of falling row chain length is calculated with the number of the actually used table element of falling row chain, and by ratio based on chained list length by weight Add and integrate described gram identification index and the table of falling row chain length index, forming aggregative indicator；

Step 3: obtain Candidate Set by extensive prefix filter according to described aggregative indicator, by calculating in described Candidate Set The real editing distance of character string obtain retrieval result；Wherein, retrieval result bag is obtained by described extensive prefix filter Include following step:

Step A1: after splitting inquiry string by gram, the inquiry string after splitting by described aggregative indicator enters Row ascending sort；

Step A2: choose+1 gram of front τ of the non-overlapping part of character string as base gram query set, wherein τ is distance threshold values；

Step A3: circulate the gain after adding new gram and calculating increase, until newly-increased gram does not has gain or does not has Till new gram；Wherein, described gain represents with following formula:

$G=\left(\frac{\frac{\left|C_i\right|}{\left|I\left(g_n\right)\right|}+1}{J_s(C_i,I(g_n\left)\right)+1}-1\right)*\cos t\left(Q\right);$

In formula, G represents gain, C_iRepresent the size of current Candidate Set, | I (g_n) | represent g_nThe table of falling row chain length, gn represents Gram after the fractionation of sequence number n, J_sRepresent that Jaccard similarity, cost (Q) expression carry out once true editing distance to inquiry The average time calculated is time-consuming；

Step A4: all gram are merged, and take out occurrence number be at least LB time character string add Candidate Set, to institute State the character string in Candidate Set and carry out real editing distance calculating, thus obtain retrieval result；Wherein, LB refers to described extensive The lower limit of the identical gram number that prefix filter uses.

Similar character string search method the most according to claim 1, it is characterised in that before step one, first to needs The string assemble being retrieved sets up inverted index, and generates the MinHash signature that inverted index is corresponding.

Similar character string search method the most according to claim 2, it is characterised in that by character string is numbered with Q-gram splits, and the gram after splitting sets up inverted index for entry maps to the relation of the character string comprising described gram.

Similar character string search method the most according to claim 2, it is characterised in that select according to the length of described character string Take hash function, generate MinHash signature in conjunction with described inverted index.