CN111444325B - Method for measuring document similarity by position coding single random replacement hash - Google Patents

Abstract

Position coding single random replacement hash measurement document similarityThe method belongs to the field of searching similar texts in information retrieval, and comprises the following steps: s1, preliminarily extracting text features to generate a single random replacement hash set O _x The method comprises the steps of carrying out a first treatment on the surface of the S2, further extracting text features to generate a single random replacement position coding hash set P _x : traversing set O in S1 _x The non-empty area in the data processing system takes the serial number of the non-empty area as key and the hash value as value, and generates key value pairs with the structure of < k and v > by mixed coding to form a set P _x The method comprises the steps of carrying out a first treatment on the surface of the S3: similarity measure: traversal P _a 、P _b All key value pairs in (1) according to the similarity

The similarity of the two documents a, b is compared. The invention has high calculation precision and keeps consistent with OPH; as the number of empty areas increases, the POPH method for measuring the document similarity saves the calculation time and the storage space.

Description

A Method for Measuring Document Similarity by Position-encoded Single Random Permutation Hash

technical field

The invention belongs to the field of searching similar texts in information retrieval, and more specifically relates to a method for measuring document similarity by position coding single random permutation hash.

Background technique

The WEB is experiencing explosive growth, and more and more documents are published on the Internet. This trend makes the document resources on the Internet grow exponentially, which provides unprecedented convenience for human beings to share knowledge and create wealth. Modernization has a positive role in promoting. However, while these digital resources provide help to people, the easy availability of resources also makes the illegal copying, plagiarism, plagiarism and other behaviors of documents more and more rampant, so that there may be comparisons in documents such as various papers and project applications serious plagiarism. At the same time, with the country's large investment in education and scientific research, it has provided funding for various educational and technological projects, such as: National Natural Science Foundation projects, doctoral programs of the Ministry of Education, fund projects of various provinces and cities, and various scientific and technological plans. Since these projects belong to different functional departments and units, there are multiple declarations and multiple declarations in the project application. The phenomenon of plagiarism, multiple declarations and multiple declarations of applications seriously affects the objectivity and fairness of project approval, and has a negative impact on the rational allocation of national scientific research funds, resulting in the possibility that scientific research funds may not be used efficiently. In order to prevent plagiarism and correct the academic atmosphere, it is very meaningful to conduct research on document similarity detection technology. As a result, search engines, libraries, foundations, dissertation databases, and intellectual property departments all over the world have invested huge manpower, material resources, and financial resources, and are working hard to explore and explore document similarity detection in order to break through the similarity as soon as possible. The key scientific issues of detection provide a good solution for papers, project applications, award declarations, patent duplication checks or search engine webpage deduplication, etc.

Similarity detection data is characterized by massive amounts. Taking the applications for the National Natural Science Foundation of China as an example, the current number of applications in 2019 has reached more than 200,000, and will continue to grow at a relatively rapid rate every year. As another example, in recent years, the number of college graduates in China is about 7 million each year, and most of the graduation thesis need similarity testing. Not only does it need to be checked against the data of the year, but it also needs to be checked against historical data. For such a large amount of documents, it is impossible to rely on conventional detection methods. Therefore, it is urgent to use hash similarity estimation technology to establish a set of accuracy The detection mechanism with excellent efficiency and high efficiency realizes the similarity comparison technology for massive documents. The concept of text similarity measurement and related technologies also came into being. A good text similarity measurement method is of great significance in research fields such as similarity detection, automatic question answering system, intelligent retrieval, web page deduplication, and natural language processing.

Text similarity refers to a measurement parameter of the degree of matching between two or more texts. The higher the similarity, the greater the similarity between the two texts, and vice versa. The traditional text similarity measurement method is the Vector Space Model (VSM), which calculates the frequency vector inner product with weights between the document to be searched and a certain document in the data set to obtain the similarity between the two documents. The algorithm needs to store a large number of feature words, has the disadvantages of slow comparison speed and low accuracy, and cannot be applied to similarity measurement in massive data. Based on the Minwise similarity measurement algorithm as the most mainstream and mature similarity detection method, by converting the similarity problem into an event probability problem, the text vocabulary set is mapped to the hash value set, and the string comparison problem is transformed into The feature fingerprint comparison problem is suitable for the similarity measurement of massive data.

The similarity measurement algorithm based on Minwise and its variants have high estimation accuracy, but various research institutions are still pursuing higher accuracy. This is due to the diversity and randomness of the actual detection data, and it is often easy for a class of large texts to contain small texts (f ₁ >>f ₂ ≈a). Among them, f ₁ and f ₂ are the word set sizes of document 1 and document 2, and a is the intersection size. Because f ₁ >>f ₂ , the similarity is very small, and because f ₂ ≈a, the inclusion rate of document 2 relative to document 1 is close to 1, such a high inclusion rate also shows that document 2 is completely plagiarized from document 1 . For such cases of low similarity rate and high inclusion rate, the variance of the similarity measurement algorithm based on Minwise is large and the accuracy is not enough. Although this is a relatively special type of data, it is not uncommon in practice. Sometimes the similarity deviation can be as high as more than 20%, and there is no better processing method at present.

However, the disadvantage of the Minwise-based similarity measurement algorithm is that k random independent permutations are required to generate k hash values, and then one-to-one comparison of k hash values is performed to calculate the similarity value of the two document pairs, and k times Replacement is time-consuming, accounting for 80% of the total time. One Permutation Hashing (OPH) proposes that only one permutation can achieve the effect of k permutations and generate k hash values, thereby improving the calculation efficiency.

The invention patent published on 2018.08.17 with the publication number CN108415889A and the name of a text similarity detection method based on a weighted one-time permutation hash algorithm proposes a method for non-uniform division of regions, which can be achieved by setting a threshold Reduce the comparison of hash values, which can effectively improve the calculation efficiency.

However, when there are too many empty hash values in the region, both the above-mentioned single random permutation hashing method and the weighted single permutation hashing method have the problem of excessive performance consumption.

Contents of the invention

In order to solve the above-mentioned technical problems, the present invention proposes a method for measuring document similarity by Position One Permutation Hashing (POPH), which is used to solve the performance consumption problem of comparing hash values when OPH generates excessive empty areas. Improving computing performance has important scientific significance and practical application value.

The present invention takes following technical scheme:

A method for measuring document similarity by position coding single random permutation hash, comprising the following steps:

S1, initially extracting text features and generating a single random permutation hash set O _x ;

S2, further extract text features, and generate a single random permutation position encoding hash set P _x : traverse the non-empty area in the set O _x in S1, use the serial number of the non-empty area as the key, and the hash value as the value, and mix encoding Generate key-value pairs with the structure <k,v> to form a set P _x ;

比较两文档a、b的相似度；S3: Similarity measurement: traverse all key-value pairs in P _a and P _b , according to the similarity

Compare the similarity of two documents a and b;

Among them, the subscript x represents any document, P _a and P _b are the key-value pairs <k, v> sets generated by the method of S2 for documents a and b respectively, _Nemp is the empty area in the sets O _a and O _b at the same time N _mat represents the number of non-empty and equal hash values in the sets O _a and O _b , and k is the number of set areas in the sets O _a and O _b that have a larger total area except the end bit.

Further, the generation steps of the hash set O _x described in S1 are:

S1.1: Perform word segmentation and noise filtering on the document x to obtain the word segmentation set S _x ;

S1.2: Use the Rabin function to map S _x to obtain a new set S _xD , perform a random permutation on the set S _xD to generate a set π(S _xD );

S1.3: The hash value generated by π(S _xD ) on the complete set Ω is S _xR ;

S1.4: Perform compression coding on S _xR to obtain O _x .

其中，数据集Y∈Ω且y∈Y，π为一个随机minwise排列。Further, the random permutation of S _{x D} in S1.2 satisfies: any element y in the data set Y has the same probability of being the minimum value of this data set after the permutation under random permutation π, namely

Among them, the data set Y∈Ω and y∈Y, π is a random minwise arrangement.

Further, the complete set Ω in S1.3 is evenly divided into k regions, all regions are equal in size and m, and each region is numbered from 1 to k.

Further, each area in the complete set Ω generates a hash value: if there is no non-zero element in a certain area, the area is an empty area, and its hash value is "*"; if there is a non-zero element in a certain area, the The area is a non-empty area, and the minimum non-zero element in the area is used as the hash value of the area; the hash value set of the empty area and the non-empty area forms S _xR .

Further, the compression encoding process in the S1.4 adopts the encoding compression function f(hash)=hashmodm, wherein, mod is a modulus function, m is the area size of the complete set Ω, for each hash value in S _xR The set O _x is generated after applying the compression coding function.

Further, the steps of the similarity measurement method in the S3 are:

S3.1: Let N _mat =0, _Nemp =0, i=1, read the key-value pairs of the non-empty areas in P _a and P _b respectively from the beginning, and the minindex is the current smaller one in the sets P _a and P _b key value;

S3.2: When the serial number of the non-empty area of P _a read is not equal to the serial number of the non-empty area of P _b , N _emp =N _emp +minindex-i, N _mat remains unchanged, and the larger area of the current P _a and P _b The key-value pair of the serial number continues to be compared with the next non-empty zone key-value pair in the set with the smaller zone serial number;

S3.2.1: i=minindex+1, minindex becomes the current smaller key value in the sets P _a and P _b , when the serial number of the non-empty area of P _a read is not equal to the serial number of the non-empty area of P _b , N _emp = N _emp +minindex-i, N _mat unchanged, otherwise, go to step S3.3;

S3.3: When the serial number of the read P _a non-empty area is equal to the serial number of the P _b non-empty area, i=minindex+1, and the minindex becomes the current smaller key value in the set P _a and P _b , if the two key values The values in the pair are equal, N _mat = N _mat +1, N _emp remains unchanged, otherwise, enter step S3.3.1;

S3.3.1: If the values in the two key-value pairs are not equal, N _emp =N _emp +minindex-i, N _mat remains unchanged, continue to read the key-value pair of the next non-empty area in P _a and P _b , i= minindex+1, minindex becomes the current smaller key value in the sets P _a and P _b , and enters step S3.4;

S3.4: If all the key-value pairs in P _a and P _b are traversed to the end bit, then stop the comparison, otherwise go to step S3.2.

The beneficial effects of the present invention are:

(1) The calculation accuracy is high, because the similarity R calculation formula is consistent with OPH, and the accuracy is also consistent;

(2) With the increase of the number of empty areas, the time consumed by POPH will be shorter and shorter than that of OPH. The method of POPH to measure the similarity of documents only compares the hash values of non-empty areas, and then calculates N _mat through position coding , which not only saves computing time but also saves storage space.

Description of drawings

Fig. 1 is a document generation single random permutation hash set OPH(S _x ) process diagram of an embodiment of the present invention;

Fig. 2 is a region map corresponding to the single random permutation hash value compression encoding in Fig. 1;

Fig. 3 is the position code hash set POPH (S _x ) process diagram that generates a single random permutation for the compression coding in Fig. 2;

Fig. 4 is a comparison chart of the time used for the comparison of different documents using OPH and POPH to complete the comparison of hash values when there is no empty space;

Fig. 5 is a time comparison diagram for the first pair of documents in Fig. 4 to use OPH and POPH to complete the comparison of hash values when the empty area exists;

Fig. 6 is a time comparison diagram for the second pair of documents in Fig. 4 to use OPH and POPH to complete the comparison of hash values when the empty area exists;

Fig. 7 is a time comparison diagram for the third pair of documents in Fig. 4 to use OPH and POPH to complete the comparison of hash values when the empty area exists;

Fig. 8 is a time comparison diagram for the fourth pair of documents in Fig. 4 to use OPH and POPH to complete the comparison of hash values when the empty area exists;

FIG. 9 is a comparison of sets P _a and P _b when the counter i=1 in one embodiment;

Fig. 10 is the comparison situation of sets P _a and P _b when the counter i=3 of the embodiment of Fig. 9;

Fig. 11 is the comparison situation of sets P _a and P _b when the counter i=4 of the embodiment of Fig. 9;

Fig. 12 is the comparison situation of sets P _a and P _b when the counter i=6 of the embodiment of Fig. 9;

FIG. 13 is a comparison between sets P _a and P _b when the counter i=8 in the embodiment of FIG. 9 .

Detailed ways

The present invention will be further described below in conjunction with specific examples. Unless otherwise specified, the methods used in the examples of the present invention are conventionally used methods in the art.

Example 1

A method for measuring document similarity by position coding single random permutation hash, comprising the following steps:

S1, initially extracting text features and generating a single random permutation hash set O _x ;

S2, further extract text features, and generate a single random permutation position encoding hash set P _x : traverse the non-empty area in the set O _x in S1, use the serial number of the non-empty area as the key, and the hash value as the value, and mix encoding Generate key-value pairs with the structure <k,v> to form a set P _x ;

比较两文档a、b的相似度；S3: Similarity measurement: traverse all key-value pairs in P _a and P _b , according to the similarity

Compare the similarity of two documents a and b;

Among them, the subscript x represents any document, P _a and P _b are the key-value pairs <k, v> sets generated by the method of S2 for documents a and b respectively, _Nemp is the empty area in the sets O _a and O _b at the same time N _mat represents the number of non-empty and equal hash values in the sets O _a and O _b , and k is the number of set areas in the sets O _a and O _b that have a larger total area except the end bit.

The details of S1 are as follows: First, scan and analyze the text information, use the Chinese word segmentation algorithm to segment the document, use the stop word list to filter out the text noise data, and the word segmentation set is the word set S _x of the document x. Noise refers to meaningless words in the text, generally high-frequency and low-meaning auxiliary words, function words, etc.;

For word set S _x (the subscript x represents any document), the Rabin function (which can map the string set into a 32-bit or 64-bit natural data set) is used to map the 32-bit integer, and the set is named S _xD after mapping. Assume that the complete set Ω={0,1,...,D-1}, a ₀ a ₁ ...... a _D-1 HSI is a permutation on Ω, the vector (a ₀ ,a ₁ ,·· , a _D-1 ) represents a permutation of Ω:

If for a data set Y∈Ω and y∈Y, there exists a permutation π such that

Then π is a random minwise arrangement. In other words, any element y in the data set Y has the same probability of being the minimum value of the data set after the replacement of π. The set generated by a random permutation of the set S _xD is named π(S _xD ).

The complete set Ω is evenly divided into k regions (referred to as Bins), so the size of all regions is equal. Let the size of the region be m, and each region (Bin) is numbered from 1 to k. The number is called BinId (abbreviated as Bid). Each value in π(S _xD ) can be found in the Bin of the complete set Ω, and then a hash value is generated in each Bin. The specific process of hash value generation: if there is no non-zero element in the Bin, it will be "*" as the hash value of this area, and this Bin is also named as an empty area (all zeros are empty). If there is a non-zero element in the i-th Bin, select a minimum non-zero element value in the Bin as the hash value of the area. Because the Bin has non-empty elements, it is named a non-empty area, and the empty area and non-empty The hash value of the empty area is collectively referred to as Binhash, and the hash value set generated by π(S _{x D} ) on the complete set Ω is S _{x R} ; finally, S _{x R} is compressed and encoded, and the compression encoding process is: set the encoding compression function: f(hash)= hashmod m, where mod is a modulo function, and the set generated by applying the compression coding function to each hash value in S _xR is OPH(S _x ), referred to as O _x .

As shown in Figure 1 and Figure 2, assuming that the complete set Ω={0,1,2,...35} (D=36), the complete set Ω is evenly divided into 9 regions, that is, k=9; the word set of a certain document x is S _x , and then, use the Rabin function on the word set S _x to map it into a 32-bit integer. After the mapping, the set is named S _xD , and then the set formed by random replacement of S _xD is π(S _xD )={6, 19,25,32}; Correspond each value in the set π(S _xD ) to the complete set Ω to generate the hash value, the generated hash value set is S _xR , then S _xR ={*,6,*, *,19,*,25,*,32}; Finally, the hash value in S _xR is compressed and coded, and the compressed coded set is generated as O _x , then O _x ={*,2,*,*,3, *,1,*,0}.

S2 specifically includes: as shown in Figure 3, the Bid of a certain area is used as a location feature and the hash value of the Bin is mixed and encoded to generate a key-value pair with a structure of <k, v>. Traverse all non-empty areas in the set O _x , use the Bin's Bid as the key, and the Bin's Binhash as the value, and store them in the <k, v> structure to form the set P _x .

The steps of the similarity measurement method in S3 are:

S3.1: Let N _mat =0, _Nemp =0, i=1, read the key-value pairs of the non-empty areas in P _a and P _b respectively from the beginning, and the minindex is the current smaller one in the sets P _a and P _b key value;

S3.2: When the serial number of the non-empty area of P _a read is not equal to the serial number of the non-empty area of P _b , N _emp =N _emp +minindex-i, N _mat remains unchanged, and the larger area of the current P _a and P _b The key-value pair of the serial number continues to be compared with the next non-empty zone key-value pair in the set with the smaller zone serial number;

S3.2.1: i=minindex+1, minindex becomes the current smaller key value in the sets P _a and P _b , when the serial number of the non-empty area of P _a read is not equal to the serial number of the non-empty area of P _b , N _emp = N _emp +minindex-i, N _mat unchanged, otherwise, go to step S3.3;

S3.3: When the serial number of the read P _a non-empty area is equal to the serial number of the P _b non-empty area, i=minindex+1, and the minindex becomes the current smaller key value in the set P _a and P _b , if the two key values The values in the pair are equal, N _mat = N _mat +1, N _emp remains unchanged, otherwise, enter step S3.3.1;

S3.3.1: If the values in the two key-value pairs are not equal, N _emp =N _emp +minindex-i, N _mat remains unchanged, continue to read the key-value pair of the next non-empty area in P _a and P _b , i= minindex+1, minindex becomes the current smaller key value in the sets P _a and P _b , and enters step S3.4;

S3.4: If all the key-value pairs in P _a and P _b are traversed to the end bit, then stop the comparison, otherwise go to step S3.2.

计算出相似度R。Finally apply the formula

Calculate the similarity R.

Example 2

Select 4 pairs of documents in the experimental data set to form the data set, divide the document pairs into 4 groups according to the similarity from high to low, and randomly select a pair of words in each document pair to represent the document pair. The experimental data is shown in Table 1 below ( f ₁ and f ₂ are the word set sizes of document 1 and document 2, and a is the intersection size), if there is no empty area in the randomly permuted set π(S _xD ), when calculating _Nemp and N _mat , POPH needs to To compare Bid, Binhash must be compared at the same time, while OPH only needs to compare Binhash, so the calculation speed is better than POPH, as shown in Figure 4.

其中f₁＝|S₁|,f₂＝|S₂|,a＝|S₁∩S₂|，当在全集Ω中均匀划分出的总区数不变时，通过减少a的值，就可以形成不同数量的空区，因此通过减少表1中每个文档对中a的数量来增加空区的数量。Count the time it takes for OPH and POPH to complete the comparison of hash values when empty areas appear in different proportions: this data set is constructed on the basis of the first data set above, according to the similarity between the measurement sets S1 and S2 The basic formula used is:

Where f ₁ =|S ₁ |, f ₂ =|S ₂ |, a=|S ₁ ∩S ₂ |, when the total number of regions evenly divided in the complete set Ω remains unchanged, by reducing the value of a, the A different number of voids can be formed, so increase the number of voids by reducing the number of a in each document pair in Table 1.

Table 1

As shown in Table 2 below, each document pair has 5 different empty area percentages. For example, in the document pair "RIGHTS-RESERVED", a=a*0.8 means that the number of empty areas in the document accounts for 20% of the total area , in the same way, a=a*0.7, a=a*0.5, a=a*0.3 and a=a*0.2 respectively represent "RIGHTS-RESERVED" document pairs, containing 30%, 50%, 70% and 80 respectively % empty area, under different percentages of empty area, the comparison of the time consumed by OPH and POPH to complete the hash value comparison is shown in Figure 5-8 below.

Table 2

Therefore, as the percentage of empty areas increases, the calculation time of POPH and OPH decreases; as the number of empty areas increases, the time consumed by POPH will become shorter and shorter, because POPH calculates _Nemp and N _mat , and Unlike OPH, which traverses all areas, it only compares the hash values of non-empty areas, and then calculates N _mat through position coding, which not only saves calculation time, but also saves storage space.

Example 3

As shown in FIG. 9 to FIG. 13 , this embodiment enumerates two sets of key-value pairs P _a and P _b , and the number of regions is 10. The illustrated process shows the calculation process of the similarity R between P _a and P _b .

Among them, i is a counter, indicating the 1-10th area, minindex refers to the current smaller k value in the set P _a and P _b , and the counter i increases with the change of minindex (i=minindex+1), not OPH The calculation method uses a loop to traverse k areas, so POPH saves time.

Specifically, P _a ={*,2,*,*,3,*,1,*,0,end}, P _b ={*,*,3,*,*,*,1,*,1, end}, N _emp is the number of empty areas in the sets P _a and P _b at the same time, N _mat is the number of non-empty and equal hash values in the sets P _a and P _b , and k is the number of areas in the sets P _a and P _b The largest number of collection areas obtained except the end bit, P _a and P _b are respectively the key-value pair <k, v> collection generated by the position encoding single random permutation hash method (POPH) of documents a and b .

S1: Data initialization, at this time N _mat =0, N _emp =0;

S2: let i=1, at this time the first non-empty areas in P _a and P _b are <2,2> and <3,3> respectively, then minindex=2, N _emp =minindex-i=1, N _mat = 0;

S3: i=minindex+1=3, at this time, the non-empty areas in P _a and P _b are <5,3> and <3,3> respectively, then minindex=3, N _emp =N _emp +minindex -i=1 (no newly added quantity that is empty area at the same time, maintain the quantity in S2), N _mat =0;

S4: i=minindex+1=4, at this time the non-empty areas in P _a and P _b are <5,3> and <7,1> respectively, then minindex=5, N _emp =N _emp +minindex - i=2, _Nmat =0;

S5: i=minindex+1=6, at this time the non-empty areas in P _a and P _b are <7,1> and <7,1> respectively, then minindex=7, N _emp =N _emp +minindex -i=3, _Nmat =1;

S6: i=minindex+1=8, at this time, the non-empty areas in P _a and P _b are <9,0> and <9,1> respectively, then minindex=9, N _emp =N _emp +minindex -i=4, _Nmat =1;

S7: i=minindex+1=10, the traversal of the non-empty areas of P _a and P _b has been completed at this time, the counter reaches the end bit, and the comparison is stopped, then k=9, N _emp =4, N _mat =1, so similar Spend

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included within the protection scope of the present invention.

Claims (4)

1. A method for measuring document similarity by position coding single random permutation hash, which is characterized by comprising the following steps:

s1, preliminarily extracting text features to generate a single random replacement hash set O _x ；

S1.1: word segmentation and noise filtering are carried out on the document x to obtain a word segmentation set S _x ；

S1.2: using Rabin function pair S _x Mapping to obtain a new set S _xD For the set S _xD Performs random permutation once to generate a set pi (S _xD )；

S1.3：π(S _xD ) The hash value generated on the corpus Ω is S _xR ；

S1.4: for S _xR Compression encoding to obtain O _x ；

S2, further extracting textCharacterization, generating a single random permutation position-coding hash set P _x : traversing set O in S1 _x The non-empty area in the data processing system takes the serial number of the non-empty area as key and the hash value as value, and generates key value pairs with the structure of < k and v > by mixed coding to form a set P _x ；

S3, similarity measurement: traversal P _a 、P _b All key value pairs in (1) according to the similarity

Comparing the similarity of the two documents a and b;

wherein, the whole set Ω is uniformly divided into k regions, all regions have the same size and m, each region is numbered from 1 to k, and each region in the whole set Ω generates a hash value: if a certain area does not have non-zero elements, the area is a dead area, and the hash value of the area is "/x"; if a certain area has non-zero elements, the area is a non-empty area, and the minimum non-zero element in the area is used as a hash value of the area; hash value set of empty area and non-empty area forms S _xR Subscript x denotes any document, P _a 、P _b Key value pairs < k, v > set and N generated by S2 method for documents a and b respectively _emp For set O _a 、O _b The number of the middle and empty areas N _mat Representing set O _a 、O _b Is not null and the hash value is equal, k is set O _a 、O _b The number of the integrated areas of the end dividing bits, which is larger in the total area number.

2. The method of claim 1, wherein S in S1.2 is _xD The random permutation performed satisfies: the probability that any one element Y in the data set Y has the same probability under random permutation pi is the minimum value after the data set is permuted, namely

Wherein, the data set Y ε Ω and Y ε Y, pi is a random minwise permutation.

3. The method of claim 1, wherein the compression encoding process in S1.4 uses an encoding compression function f (hash) =hashmom, where mod is a modulo function, m is the region size of the corpus Ω, for S _xR Each hash value in the set is used for generating a set O after a compression coding function is applied _x 。

4. The method for measuring similarity of documents according to claim 1, wherein the step of the similarity measuring method in S3 is as follows:

s3.1: let N _mat ＝0，N _emp Respectively read P from the beginning =0, i=1 _a 、P _b Key value pairs of the non-empty area, and mini is set P _a And P _b A current smaller key value of (a);

s3.2: when P is read _a Sequence number and P of non-hollow area _b When the sequence numbers of the non-empty areas are not equal, N _emp ＝N _emp +minindex-i，N _mat Unchanged, current P _a And P _b The key value pair with the larger area sequence number in the middle is continuously compared with the key value pair with the next non-empty area in the set with the smaller area sequence number;

s3.2.1: i=mini+1, mini becomes set P _a And P _b When the current smaller key value of P is read _a Sequence number and P of non-hollow area _b When the sequence numbers of the non-empty areas are not equal, N _emp ＝N _emp +minindex-i，N _mat If not, entering a step S3.3;

s3.3: when P is read _a Sequence number and P of non-hollow area _b When the sequence numbers of the non-empty areas are equal, i=mini+1, and mini becomes a set P _a And P _b If the value of the two key value pairs is equal to the current smaller key value, N _mat ＝N _mat +1，N _emp Unchanged, otherwise, go to step S3.3.1;

s3.3.1: if the two key value pairs are not equal, N _emp ＝N _emp +minindex-i，N _mat Unchanged, continue reading P _a 、P _b Key value of next non-empty region in the listFor i=mini+1, mini becomes set P _a And P _b The step S3.4 is entered;

s3.4: if go through P _a 、P _b If the key value pairs reach the end bit, the comparison is stopped, otherwise, the step S3.2 is started.

Images