CN105718430A - Grouping minimum value-based method for calculating fingerprint similarity - Google Patents

Abstract

The invention discloses a grouping minimum value-based method for calculating the fingerprint similarity. According to the method, the limitation of random arrangement is broken through, fingerprints are not generated through the random arrangement, and the fingerprints can still be used for estimating the similarity, so that the problem that the random arrangement is low in efficiency and complicated is solved; and the method is capable of simplifying the Hash process stages of a Minwise algorithm and variant algorithms of the Minwise algorithm, and is an optimization method for Hash functions in a detection algorithm.

Description

A kind of method calculating similarity as fingerprint based on packet minima

Technical field

The present invention relates to a kind of method calculating similarity as fingerprint based on packet minima.

Background technology

WEB is just experiencing explosive increase, increasing documents and materials begin at online announcement, this trend makes document resources on network become geometric growth, shares knowledge for the mankind and provides unprecedented facility with creating the wealth, also the modernization construction of China is had positive impetus.But, while these digitalization resources are offered help to people, easily the obtaining property of resource also makes the behaviors such as the bootlegging of document, plagiarism, plagiarization more and more rampant so that in various papers and project application book etc., it is understood that there may be than more serious plagiarism phenomenon.Meanwhile, along with the country's a large amount of inputs to education and scientific research, it is provided that the subsidy of various education and science and technology item, such as center for doctors's project of: project of national nature science fund project, the Ministry of Education, the fund project in each province and city, various plans of science and technology etc..Being in charge of owing to these projects belong to different functional institutions's unit, this allows for project application book and also exists and repeatedly declare the phenomenon declared with bull.The plagiarism of application, repeatedly declare and declare phenomenon with bull and had a strong impact on objectivity and the fairness of project examination, the reasonable distribution of country's research funding is exerted an adverse impact, causes scientific research funds to be likely to can not get efficient utilization.For preventing plagiarism, rectifying academic atmosphere, the research carrying out document similarity detection technique is very meaningful.Thus, search engine all over the world, library, foundation, paper storehouse, Intellectual Property Department etc. all put into huge human and material resources and financial resources, grope on document similarity detects just hardy and probe into, to breaking through the key scientific problems of similarity detection as early as possible, provide good solution for paper, project application book, award return, the duplicate checking of patent or the removing duplicate webpages etc. of search engine.

Similarity detection data have the feature of magnanimity, and for state natural sciences fund application, at present with regard to applications in 2013, application quantity reaches more than 170,000 parts, every year also will with speed increment faster.And for example, the annual graduates' number of China was about 7,000,000 in recent years, the thesis of wherein most is required for carrying out similarity detection, annual May, paper detection limit peaked, average daily more than several ten thousand parts, similarity detection not only to carry out duplicate checking with data then, also need to detect with historical data, and the document of such magnanimity, it is at all impracticable for depending conventional sense mode alone, therefore in the urgent need to setting up a set of precision and all excellent testing mechanism of efficiency, it is achieved the similarity comparison technology to magnanimity document.

The structure basis estimating son of Minwise Hash and mutation thereof is in that random alignment, the ultimate principle of Minwise Hash and mutation thereof are:

Make complete or collected works Ω=0,1 ..., D-1}, obtain relevant shingles by shingling document d and gather S_d.Document S₁And S₂Similarity be defined as:Wherein f₁=| S₁|, f₂=| S₂|, a=| S₁∩S₂|.Assuming that the random independent arrangement on complete or collected works Ω a: π. Ω → Ω, Ω=0,1 ..., D-1}, by the arrangement π of k independent random₁,π₂,...,π_k, just the shingles of any one document d is gathered and is converted to: $\stackrel{\‾}{S_d}=\left(\min \right\{{\mathrm{\π}}_1\left(S_d\right)\},\min \{{\mathrm{\π}}_2\left(S_d\right)\},...,\min \{{\mathrm{\π}}_k\left(S_d\right)\left\}\right).$

The similarity of Minwise Hash estimates that sub-R is:

$R=\mathrm{Pr}\left(\min \right\{\mathrm{\π}\left(S_1\right)\}=min\{\mathrm{\π}\left(S_2\right)\left\}\right)=\frac{\left|S_1\∩S_2\right|}{\left|S_1\∪S_2\right|}---\left(1\right)$

In formula (1), function min{ π (X) } it is exactly Minwise hash function.

And the measurement formula of the unbiased esti-mator of R is:

$\hat{R}=\frac{1}{k}\underset{j=1}{\overset{k}{\Σ}}1\{min\left({\mathrm{\π}}_j\right(S_1\left)\right)=min\left({\mathrm{\π}}_j\right(S_2\left)\right)\}---\left(2\right)$

The variance estimated is:

$Var\left({\hat{R}}_M\right)=\frac{1}{k}R(1-R)---\left(3\right)$

Wherein, k is sample size (or experiment number).

As shown in formula (2), k is the number of times of experiment, and k π can obtain k fingerprint, thus the equal ratio of comparison fingerprint carrys out approximate solution similarity.Wherein, k is for estimating that calculation has great meaning, and k is more big, and the variance estimated is more little, and the accuracy rate of estimation and recall rate are more high；K is more little, and the variance of estimation is more big, and the accuracy rate of estimation and recall rate are more low, therefore often requires that k in real system > 1000, variance generally just can be reduced to user's acceptable scope.

No matter it is generate random alignment or use random alignment, is required for calculating in a large number the time, this is because random alignment scope is to complete or collected works [0,2³²) arrange.So big arrangement is general all unavailable in systems in practice.It is generally adopted approximate alignment, to [0,2³²) carrying out less codomain delivery, raising efficiency is limited, also reduces precision, remains a need for still more generating k > 1000 random alignment, calculates the used time still very long.

Above-mentioned Hash fingerprint technique is a kind of conventional technology, but due to the restriction of its random alignment, causes the fingerprint poor efficiency of its generation, and then cause that the accuracy that the property compared detects is not high.

Summary of the invention

In order to overcome above-mentioned problem of the prior art, the present invention proposes a kind of fingerprint similarity based method based on packet minima, adopts the method can break through the restriction of random alignment, is namely a kind of fingerprint generation method without random alignment.

In order to solve above-mentioned technical problem, the technical scheme is that

A kind of fingerprint similarity based method based on packet minima is provided, comprises the following steps:

S1. Text character extraction: be used for extracting text feature set S_d；

Being scanned text message analyzing, text message carries out participle, and filters the noise data in text message, the participle set obtained is the word set S of text_shgs；To word set S_shgsAdopt Rabin function, map the integer of 32, after mapping, gather called after S_d；

S2. packet: given 2 document S₁And S₂, to S₁And S₂In element be grouped by fixed size m, group number be set to k, complete or collected works be Ω=0,1,2 ..., 2³²-1}, | Ω |=2³², the element having in set is designated as 1, the element of nothing is designated as 0；

S3. select in group and represent, form fingerprint:

Often group chooses a representative, and this is represented as maximum, minima or the neutral element often organized；If element-free in group, it is called sky group, is then set to e, S_dIt is the formation of k to be represented as the set of element in organizing, i.e. fingerprint set；

S4. calculating similarity R, its calculation is:

R=N_mat/(k-N_emp)(4)

Wherein N_matRepresent that often group represents equal number of times, N_empRepresent the number of times of sky group；I_mat,iRepresent i-th group of equal counting when comparing；I_emp,iExpression is empty counting simultaneously.

Specific as follows:

$N_{mat}=\underset{i=1}{\overset{k}{\Σ}}{I}_{mat,i}---\left(5\right)$

$N_{emp}=\underset{i=1}{\overset{k}{\Σ}}{I}_{emp,i}---\left(6\right)$

$I_{mat,i}=\left\{\begin{array}{cc}1,& \begin{array}{cc}if& \min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_1=\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_2\≠e\end{array}\\ 0,& otherwise\end{array}\right.---\left(7\right)$

Formula (7) is specifically: when comparing two elements not for e, when namely be sky, and two elements are equal, then I_mat,i=1, otherwise I_mat,i=0；

$I_{emp,i}=\left\{\begin{array}{cc}1,& \begin{array}{cc}if& min{\left(\begin{array}{cc}i-th& group\end{array}\right)}_1=min{\left(\begin{array}{cc}i-th& group\end{array}\right)}_2=e\end{array}\\ 0,& otherwise\end{array}\right.---\left(8\right)$

Formula (8) is specifically: be all e when comparing two elements, when being namely all empty, then and I_emp,i=1, otherwise I_emp,i=0.

Compared with prior art, advantages of the present invention is:

1) the method eliminates random alignment π₁,π₂,...,π_kGeneration and storage, it is possible to more save storage overhead；2) the method need not map fingerprint by random alignment, decreases the calculating time generating fingerprint.

Accompanying drawing explanation

Fig. 1 is the flow chart of the present invention.

Detailed description of the invention

Below in conjunction with accompanying drawing, the present invention will be further described, but embodiments of the present invention are not limited to this.

A kind of fingerprint similarity based method based on packet minima, particularly as follows:

Step one, Text character extraction step: this step is used for extracting text feature set S_d；

First, being scanned text message analyzing, utilize Chinese Word Automatic Segmentation that text message carries out participle, utilization disables the participle set after vocabulary filters out text noise data and is the word set S of text message_shgs.Noise is in text insignificant word, is usually the auxiliary word of the low justice of high frequency, function word etc.；To word set S_shgsAdopt Rabin function, map the integer of 32, after mapping, gather called after S_d。

Step 2, packet

Given 2 document S₁And S₂, to S₁And S₂In element be grouped by fixed size m, group number be set to k, complete or collected works be Ω=0,1,2 ..., 2³²-1}, | Ω |=2³².The element having in set is designated as 1, and the element of nothing is designated as 0.

In the present embodiment, if Ω=0,1,2 ... 15}, | Ω |=16, m=4, k=4, m*k=| Ω |,

S₁={ 0,3,4,6,713,16}

S₂={ 0,1,3,6,7,13,14,16}

To S₁And S₂In element be grouped by fixed size m after:

Therefore, now

S₁=1001 | 1011 | 0000 | 0100}

S₂=1101 | 0011 | 0000 | 0110}

Select representative in step 3, group, form fingerprint.

Often group chooses a representative, can choose maximum, minima or the neutral element often organized, as the representative of a group.If element-free in group, it is called sky group, is then set to e, S_dIt is the formation of k to be represented as the set of element in organizing, is defined herein as fingerprint set.

Lower example adopts minima to be representative:

Printfinger(S₁)=[0,4, e, 13]

Printfinger(S₂)=[0,6, e, 13]

Because one group is up to 4 elements, therefore the representative often organized to 4 deliverys, can save memory space.

Printfinger(S₁)=[0,0,0,1]

Printfinger(S₂)=[0,2, e, 1]

Step 4, calculating similarity R.

Computing formula is:

R=N_mat/(k-N_emp)(4)

Wherein N_matRepresent that often group represents equal number of times.N_empRepresent the number of times of sky group.Specific formula for calculation is:

$N_{mat}=\underset{i=1}{\overset{k}{\Σ}}{I}_{mat,i}---\left(5\right)$

$N_{emp}=\underset{i=1}{\overset{k}{\Σ}}{I}_{emp,i}---\left(6\right)$

$I_{mat,i}=\left\{\begin{array}{cc}1,& \begin{array}{cc}if& \min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_1=\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_2\≠e\end{array}\\ 0,& otherwise\end{array}\right.---\left(7\right)$

Formula (7) is specifically: when comparing two elements not for e, when namely be sky, and two elements are equal, then I_mat,i=1, otherwise I_mat,i=0；

$I_{emp,i}=\left\{\begin{array}{cc}1,& \begin{array}{cc}if& min{\left(\begin{array}{cc}i-th& group\end{array}\right)}_1=min{\left(\begin{array}{cc}i-th& group\end{array}\right)}_2=e\end{array}\\ 0,& otherwise\end{array}\right.---\left(8\right)$

Formula (8) is specifically: be all e when comparing two elements, when being namely all empty, then and I_emp,i=1, otherwise I_emp,i=0.

The present embodiment particularly as follows:

Initially, N_mat=0, N_emp=0

Group 1,0=0, then N_mat=Nmat+1,

Now N_mat=1, N_emp=0.

Group 2,0 ≠ 2, then N_matIt is constant,

Now N_mat=1, N_emp=0.

Group 3, all there is empty group e, then N in both sides_emp=N_emp+ 1,

Now N_mat=1, N_emp=1.

Group 4,1=1, then N_mat=N_mat+ 1,

Now N_mat=2, N_emp=1.

Calculating similarity is R=N_mat/(k-N_emp)=2/ (4-1)=2/3=0.667=66.7%

Below, it was demonstrated that this formula (4) R=N_mat/(k-N_emp) correctness.

Have only to identity mathematics expectation E [N_mat/(k-N_emp)]=R

(1) k is group number, is naturally larger than the quantity N of empty group_emp, then k-N_emp>0。

k-N_emp> 0=> Pr (k-N_emp> 0)=1

(2) according to definition it can be seen that

I_emp,i=1=> I_mat,i=0 (9)

$\begin{array}{c}E\left(N_{mat}\right|\left(k-N_{emp}=m\right))=\underset{i=1}{\overset{k}{\Σ}}E\left(I_{mat,i}\right|\left(k-N_{emp}=m\right))\\ =\underset{i=1}{\overset{k}{\Σ}}\frac{m}{k}*R=k*\frac{m}{k}*R=mR\end{array}---\left(11\right)$

$E\left(\frac{N_{mat}}{k-N_{emp}}\right|\left(k-N_{emp}=m\right))=R==>E\left(\frac{N_{mat}}{k-N_{emp}}\right)=R_{mat}---\left(12\right)$

Compared with prior art, advantages of the present invention is:

1) the method eliminates random alignment π₁,π₂,...,π_kGeneration and storage；2) the method need not map fingerprint by random alignment.

Step 5, can also further optimize: by fingerprint compression to 1.

Computing formula is:

R_mat,b=2E_mat,b-1(13)

Hereinafter prove the correctness of formula (13):

Making B is the special family of functions being independently distributed of a class, and wherein from definition territory [0, | Ω |-1], element can be mapped to that { 0,1}, namely the probability of 50% is mapped as 0, and 50% probability is mapped as 1 with non-uniform probability by function b ∈ B, b.Therefore, if u, v be ∈ [0, | Ω |-1], as u ≠ v, Pr_b∈B{ b (u)=b (v) }=1/2, as u=v, Pr_b∈B{ b (u)=b (v) }=1.In conjunction with B and formula (4), use symbol E_mat,bRepresent minimum value function b be mapped to probability equal after 0,1}, its formula is:

$\begin{array}{c}{E}_{mat,b}={\mathrm{Pr}}_{b\∈B}\{b\left(\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_1\right)=b\left(\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_2\right)\≠e\}\\ =R+\frac{1-R}{2}=\frac{1+R}{2}\end{array}---\left(14\right)$

Prove said method: the generation event of R is min (i-thgroup)₁=min (i-thgroup)₂≠ e, at this moment has the probability b (min (i-thgroup) of 100%₁)=b (min (i-thgroup)₂) ≠ e；1-R probability min (i-thgroup)₁≠min(i-thgroup)₂≠ e, at this moment, has the probability b (min (i-thgroup) of 1/2₁)=b (min (i-thgroup)₂) ≠ e,

Therefore, by formula (14), R is used_bThe calculating formula of similarity representing the fingerprint after compression is: R_b=2E_mat,b-1。

The embodiment of invention described above, is not intended that limiting the scope of the present invention.Any amendment done within the spiritual principles of the present invention, equivalent replacement and improvement etc., should be included within the claims of the present invention.

Claims (3)

1. the fingerprint similarity based method based on packet minima, it is characterised in that comprise the following steps:

S1. Text character extraction: be used for extracting text feature set S_d；

S2. packet: given 2 document S₁And S₂, to S₁And S₂In element be grouped by fixed size m, group number be set to k, complete or collected works be Ω=0,1,2 ..., 2³²-1}, | Ω |=2³², the element having in set is designated as 1, the element of nothing is designated as 0；

S3. select in group and represent, form fingerprint:

Often group chooses a representative, and this is represented as maximum, minima or the neutral element often organized, selects minima here；If element-free in group, it is called sky group, is set to e, S_dIt is the formation of k to be represented as the set of element in organizing, i.e. fingerprint set；

S4. calculating similarity R, its calculation is:

R=N_mat/(k-N_emp)(4)

Wherein, N_matRepresent S₁And S₂Fingerprint set often group represent equal number of times, N_empRepresent S₁And S₂Fingerprint set be the number of times of empty group simultaneously；I_mat,iRepresent i-th group of equal counting when comparing；I_emp,iExpression is empty counting simultaneously.

Specific as follows:

$N_{mat}=\underset{i=1}{\overset{k}{\Σ}}{I}_{mat,i}---\left(5\right)$

$N_{emp}=\underset{i=1}{\overset{k}{\Σ}}{I}_{emp,i}---\left(6\right)$

$I_{mat,i}=\left\{\begin{array}{cc}1,& if\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_1=\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_2\≠e\\ 0,& otherwise\end{array}\right.---\left(7\right)$

Formula (7) is specifically: when comparing two elements not for e, when namely be sky, and two elements are equal, then I_mat,i=1, otherwise I_mat,i=0；

$I_{emp,i}=\left\{\begin{array}{cc}1,& if\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_1=\min {\left(\begin{array}{cc}i-th& group\end{array}\right)}_2=e\\ 0,& otherwise\end{array}\right.---\left(8\right)$

Formula (8) is specifically: be all e when comparing two elements, when being namely all empty, then and I_emp,i=1, otherwise I_emp,i=0.

2. the fingerprint similarity based method based on packet minima according to claim 1, it is characterised in that the packet of described step S2, group number is set to k, and the element having in set is designated as 1, and the element of nothing is designated as 0；Select the minima in group representatively, form fingerprint.

3. the fingerprint similarity based method based on packet minima according to claim 2, it is characterised in that described step S4 also includes, by fingerprint compression to 1, using R after calculating similarity R_bRepresenting that the similarity after compression estimates son, formula is:

R_b=2E_mat,b-1

Wherein, E_mat,bRepresent that minimum value function b is mapped to { probability equal after 0,1}.