CN105718430A - Grouping minimum value-based method for calculating fingerprint similarity - Google Patents
Grouping minimum value-based method for calculating fingerprint similarity Download PDFInfo
- Publication number
- CN105718430A CN105718430A CN201610019243.9A CN201610019243A CN105718430A CN 105718430 A CN105718430 A CN 105718430A CN 201610019243 A CN201610019243 A CN 201610019243A CN 105718430 A CN105718430 A CN 105718430A
- Authority
- CN
- China
- Prior art keywords
- group
- fingerprint
- mat
- emp
- similarity
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/194—Calculation of difference between files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Collating Specific Patterns (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a grouping minimum value-based method for calculating the fingerprint similarity. According to the method, the limitation of random arrangement is broken through, fingerprints are not generated through the random arrangement, and the fingerprints can still be used for estimating the similarity, so that the problem that the random arrangement is low in efficiency and complicated is solved; and the method is capable of simplifying the Hash process stages of a Minwise algorithm and variant algorithms of the Minwise algorithm, and is an optimization method for Hash functions in a detection algorithm.
Description
Technical field
The present invention relates to a kind of method calculating similarity as fingerprint based on packet minima.
Background technology
WEB is just experiencing explosive increase, increasing documents and materials begin at online announcement, this trend makes document resources on network become geometric growth, shares knowledge for the mankind and provides unprecedented facility with creating the wealth, also the modernization construction of China is had positive impetus.But, while these digitalization resources are offered help to people, easily the obtaining property of resource also makes the behaviors such as the bootlegging of document, plagiarism, plagiarization more and more rampant so that in various papers and project application book etc., it is understood that there may be than more serious plagiarism phenomenon.Meanwhile, along with the country's a large amount of inputs to education and scientific research, it is provided that the subsidy of various education and science and technology item, such as center for doctors's project of: project of national nature science fund project, the Ministry of Education, the fund project in each province and city, various plans of science and technology etc..Being in charge of owing to these projects belong to different functional institutions's unit, this allows for project application book and also exists and repeatedly declare the phenomenon declared with bull.The plagiarism of application, repeatedly declare and declare phenomenon with bull and had a strong impact on objectivity and the fairness of project examination, the reasonable distribution of country's research funding is exerted an adverse impact, causes scientific research funds to be likely to can not get efficient utilization.For preventing plagiarism, rectifying academic atmosphere, the research carrying out document similarity detection technique is very meaningful.Thus, search engine all over the world, library, foundation, paper storehouse, Intellectual Property Department etc. all put into huge human and material resources and financial resources, grope on document similarity detects just hardy and probe into, to breaking through the key scientific problems of similarity detection as early as possible, provide good solution for paper, project application book, award return, the duplicate checking of patent or the removing duplicate webpages etc. of search engine.
Similarity detection data have the feature of magnanimity, and for state natural sciences fund application, at present with regard to applications in 2013, application quantity reaches more than 170,000 parts, every year also will with speed increment faster.And for example, the annual graduates' number of China was about 7,000,000 in recent years, the thesis of wherein most is required for carrying out similarity detection, annual May, paper detection limit peaked, average daily more than several ten thousand parts, similarity detection not only to carry out duplicate checking with data then, also need to detect with historical data, and the document of such magnanimity, it is at all impracticable for depending conventional sense mode alone, therefore in the urgent need to setting up a set of precision and all excellent testing mechanism of efficiency, it is achieved the similarity comparison technology to magnanimity document.
The structure basis estimating son of Minwise Hash and mutation thereof is in that random alignment, the ultimate principle of Minwise Hash and mutation thereof are:
Make complete or collected works Ω=0,1 ..., D-1}, obtain relevant shingles by shingling document d and gather Sd.Document S1And S2Similarity be defined as:Wherein f1=| S1|, f2=| S2|, a=| S1∩S2|.Assuming that the random independent arrangement on complete or collected works Ω a: π. Ω → Ω, Ω=0,1 ..., D-1}, by the arrangement π of k independent random1,π2,...,πk, just the shingles of any one document d is gathered and is converted to:
The similarity of Minwise Hash estimates that sub-R is:
In formula (1), function min{ π (X) } it is exactly Minwise hash function.
And the measurement formula of the unbiased esti-mator of R is:
The variance estimated is:
Wherein, k is sample size (or experiment number).
As shown in formula (2), k is the number of times of experiment, and k π can obtain k fingerprint, thus the equal ratio of comparison fingerprint carrys out approximate solution similarity.Wherein, k is for estimating that calculation has great meaning, and k is more big, and the variance estimated is more little, and the accuracy rate of estimation and recall rate are more high;K is more little, and the variance of estimation is more big, and the accuracy rate of estimation and recall rate are more low, therefore often requires that k in real system > 1000, variance generally just can be reduced to user's acceptable scope.
No matter it is generate random alignment or use random alignment, is required for calculating in a large number the time, this is because random alignment scope is to complete or collected works [0,232) arrange.So big arrangement is general all unavailable in systems in practice.It is generally adopted approximate alignment, to [0,232) carrying out less codomain delivery, raising efficiency is limited, also reduces precision, remains a need for still more generating k > 1000 random alignment, calculates the used time still very long.
Above-mentioned Hash fingerprint technique is a kind of conventional technology, but due to the restriction of its random alignment, causes the fingerprint poor efficiency of its generation, and then cause that the accuracy that the property compared detects is not high.
Summary of the invention
In order to overcome above-mentioned problem of the prior art, the present invention proposes a kind of fingerprint similarity based method based on packet minima, adopts the method can break through the restriction of random alignment, is namely a kind of fingerprint generation method without random alignment.
In order to solve above-mentioned technical problem, the technical scheme is that
A kind of fingerprint similarity based method based on packet minima is provided, comprises the following steps:
S1. Text character extraction: be used for extracting text feature set Sd;
Being scanned text message analyzing, text message carries out participle, and filters the noise data in text message, the participle set obtained is the word set S of textshgs;To word set SshgsAdopt Rabin function, map the integer of 32, after mapping, gather called after Sd;
S2. packet: given 2 document S1And S2, to S1And S2In element be grouped by fixed size m, group number be set to k, complete or collected works be Ω=0,1,2 ..., 232-1}, | Ω |=232, the element having in set is designated as 1, the element of nothing is designated as 0;
S3. select in group and represent, form fingerprint:
Often group chooses a representative, and this is represented as maximum, minima or the neutral element often organized;If element-free in group, it is called sky group, is then set to e, SdIt is the formation of k to be represented as the set of element in organizing, i.e. fingerprint set;
S4. calculating similarity R, its calculation is:
R=Nmat/(k-Nemp)(4)
Wherein NmatRepresent that often group represents equal number of times, NempRepresent the number of times of sky group;Imat,iRepresent i-th group of equal counting when comparing;Iemp,iExpression is empty counting simultaneously.
Specific as follows:
Formula (7) is specifically: when comparing two elements not for e, when namely be sky, and two elements are equal, then Imat,i=1, otherwise Imat,i=0;
Formula (8) is specifically: be all e when comparing two elements, when being namely all empty, then and Iemp,i=1, otherwise Iemp,i=0.
Compared with prior art, advantages of the present invention is:
1) the method eliminates random alignment π1,π2,...,πkGeneration and storage, it is possible to more save storage overhead;2) the method need not map fingerprint by random alignment, decreases the calculating time generating fingerprint.
Accompanying drawing explanation
Fig. 1 is the flow chart of the present invention.
Detailed description of the invention
Below in conjunction with accompanying drawing, the present invention will be further described, but embodiments of the present invention are not limited to this.
A kind of fingerprint similarity based method based on packet minima, particularly as follows:
Step one, Text character extraction step: this step is used for extracting text feature set Sd;
First, being scanned text message analyzing, utilize Chinese Word Automatic Segmentation that text message carries out participle, utilization disables the participle set after vocabulary filters out text noise data and is the word set S of text messageshgs.Noise is in text insignificant word, is usually the auxiliary word of the low justice of high frequency, function word etc.;To word set SshgsAdopt Rabin function, map the integer of 32, after mapping, gather called after Sd。
Step 2, packet
Given 2 document S1And S2, to S1And S2In element be grouped by fixed size m, group number be set to k, complete or collected works be Ω=0,1,2 ..., 232-1}, | Ω |=232.The element having in set is designated as 1, and the element of nothing is designated as 0.
In the present embodiment, if Ω=0,1,2 ... 15}, | Ω |=16, m=4, k=4, m*k=| Ω |,
S1={ 0,3,4,6,713,16}
S2={ 0,1,3,6,7,13,14,16}
To S1And S2In element be grouped by fixed size m after:
Therefore, now
S1=1001 | 1011 | 0000 | 0100}
S2=1101 | 0011 | 0000 | 0110}
Select representative in step 3, group, form fingerprint.
Often group chooses a representative, can choose maximum, minima or the neutral element often organized, as the representative of a group.If element-free in group, it is called sky group, is then set to e, SdIt is the formation of k to be represented as the set of element in organizing, is defined herein as fingerprint set.
Lower example adopts minima to be representative:
Printfinger(S1)=[0,4, e, 13]
Printfinger(S2)=[0,6, e, 13]
Because one group is up to 4 elements, therefore the representative often organized to 4 deliverys, can save memory space.
Printfinger(S1)=[0,0,0,1]
Printfinger(S2)=[0,2, e, 1]
Step 4, calculating similarity R.
Computing formula is:
R=Nmat/(k-Nemp)(4)
Wherein NmatRepresent that often group represents equal number of times.NempRepresent the number of times of sky group.Specific formula for calculation is:
Formula (7) is specifically: when comparing two elements not for e, when namely be sky, and two elements are equal, then Imat,i=1, otherwise Imat,i=0;
Formula (8) is specifically: be all e when comparing two elements, when being namely all empty, then and Iemp,i=1, otherwise Iemp,i=0.
The present embodiment particularly as follows:
Initially, Nmat=0, Nemp=0
Group 1,0=0, then Nmat=Nmat+1,
Now Nmat=1, Nemp=0.
Group 2,0 ≠ 2, then NmatIt is constant,
Now Nmat=1, Nemp=0.
Group 3, all there is empty group e, then N in both sidesemp=Nemp+ 1,
Now Nmat=1, Nemp=1.
Group 4,1=1, then Nmat=Nmat+ 1,
Now Nmat=2, Nemp=1.
Calculating similarity is R=Nmat/(k-Nemp)=2/ (4-1)=2/3=0.667=66.7%
Below, it was demonstrated that this formula (4) R=Nmat/(k-Nemp) correctness.
Have only to identity mathematics expectation E [Nmat/(k-Nemp)]=R
(1) k is group number, is naturally larger than the quantity N of empty groupemp, then k-Nemp>0。
k-Nemp> 0=> Pr (k-Nemp> 0)=1
(2) according to definition it can be seen that
Iemp,i=1=> Imat,i=0 (9)
Compared with prior art, advantages of the present invention is:
1) the method eliminates random alignment π1,π2,...,πkGeneration and storage;2) the method need not map fingerprint by random alignment.
Step 5, can also further optimize: by fingerprint compression to 1.
Computing formula is:
Rmat,b=2Emat,b-1(13)
Hereinafter prove the correctness of formula (13):
Making B is the special family of functions being independently distributed of a class, and wherein from definition territory [0, | Ω |-1], element can be mapped to that { 0,1}, namely the probability of 50% is mapped as 0, and 50% probability is mapped as 1 with non-uniform probability by function b ∈ B, b.Therefore, if u, v be ∈ [0, | Ω |-1], as u ≠ v, Prb∈B{ b (u)=b (v) }=1/2, as u=v, Prb∈B{ b (u)=b (v) }=1.In conjunction with B and formula (4), use symbol Emat,bRepresent minimum value function b be mapped to probability equal after 0,1}, its formula is:
Prove said method: the generation event of R is min (i-thgroup)1=min (i-thgroup)2≠ e, at this moment has the probability b (min (i-thgroup) of 100%1)=b (min (i-thgroup)2) ≠ e;1-R probability min (i-thgroup)1≠min(i-thgroup)2≠ e, at this moment, has the probability b (min (i-thgroup) of 1/21)=b (min (i-thgroup)2) ≠ e,
Therefore, by formula (14), R is usedbThe calculating formula of similarity representing the fingerprint after compression is: Rb=2Emat,b-1。
The embodiment of invention described above, is not intended that limiting the scope of the present invention.Any amendment done within the spiritual principles of the present invention, equivalent replacement and improvement etc., should be included within the claims of the present invention.
Claims (3)
1. the fingerprint similarity based method based on packet minima, it is characterised in that comprise the following steps:
S1. Text character extraction: be used for extracting text feature set Sd;
S2. packet: given 2 document S1And S2, to S1And S2In element be grouped by fixed size m, group number be set to k, complete or collected works be Ω=0,1,2 ..., 232-1}, | Ω |=232, the element having in set is designated as 1, the element of nothing is designated as 0;
S3. select in group and represent, form fingerprint:
Often group chooses a representative, and this is represented as maximum, minima or the neutral element often organized, selects minima here;If element-free in group, it is called sky group, is set to e, SdIt is the formation of k to be represented as the set of element in organizing, i.e. fingerprint set;
S4. calculating similarity R, its calculation is:
R=Nmat/(k-Nemp)(4)
Wherein, NmatRepresent S1And S2Fingerprint set often group represent equal number of times, NempRepresent S1And S2Fingerprint set be the number of times of empty group simultaneously;Imat,iRepresent i-th group of equal counting when comparing;Iemp,iExpression is empty counting simultaneously.
Specific as follows:
Formula (7) is specifically: when comparing two elements not for e, when namely be sky, and two elements are equal, then Imat,i=1, otherwise Imat,i=0;
Formula (8) is specifically: be all e when comparing two elements, when being namely all empty, then and Iemp,i=1, otherwise Iemp,i=0.
2. the fingerprint similarity based method based on packet minima according to claim 1, it is characterised in that the packet of described step S2, group number is set to k, and the element having in set is designated as 1, and the element of nothing is designated as 0;Select the minima in group representatively, form fingerprint.
3. the fingerprint similarity based method based on packet minima according to claim 2, it is characterised in that described step S4 also includes, by fingerprint compression to 1, using R after calculating similarity RbRepresenting that the similarity after compression estimates son, formula is:
Rb=2Emat,b-1
Wherein, Emat,bRepresent that minimum value function b is mapped to { probability equal after 0,1}.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610019243.9A CN105718430B (en) | 2016-01-13 | 2016-01-13 | A kind of method for calculating similarity as fingerprint based on packet minimum value |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610019243.9A CN105718430B (en) | 2016-01-13 | 2016-01-13 | A kind of method for calculating similarity as fingerprint based on packet minimum value |
Publications (2)
Publication Number | Publication Date |
---|---|
CN105718430A true CN105718430A (en) | 2016-06-29 |
CN105718430B CN105718430B (en) | 2018-05-04 |
Family
ID=56147793
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610019243.9A Active CN105718430B (en) | 2016-01-13 | 2016-01-13 | A kind of method for calculating similarity as fingerprint based on packet minimum value |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN105718430B (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111104484A (en) * | 2019-12-19 | 2020-05-05 | 南京中孚信息技术有限公司 | Text similarity detection method and device and electronic equipment |
CN111444325A (en) * | 2020-03-30 | 2020-07-24 | 湖南工业大学 | Method for measuring document similarity by position coding single random permutation hash |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100070509A1 (en) * | 2008-08-15 | 2010-03-18 | Kai Li | System And Method For High-Dimensional Similarity Search |
CN102222085A (en) * | 2011-05-17 | 2011-10-19 | 华中科技大学 | Data de-duplication method based on combination of similarity and locality |
CN102682104A (en) * | 2012-05-04 | 2012-09-19 | 中南大学 | Method for searching similar texts and link bit similarity measuring algorithm |
CN103020174A (en) * | 2012-11-28 | 2013-04-03 | 华为技术有限公司 | Similarity analysis method, device and system |
CN104636325A (en) * | 2015-02-06 | 2015-05-20 | 中南大学 | Document similarity determining method based on maximum likelihood estimation |
-
2016
- 2016-01-13 CN CN201610019243.9A patent/CN105718430B/en active Active
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100070509A1 (en) * | 2008-08-15 | 2010-03-18 | Kai Li | System And Method For High-Dimensional Similarity Search |
CN102222085A (en) * | 2011-05-17 | 2011-10-19 | 华中科技大学 | Data de-duplication method based on combination of similarity and locality |
CN102682104A (en) * | 2012-05-04 | 2012-09-19 | 中南大学 | Method for searching similar texts and link bit similarity measuring algorithm |
CN103020174A (en) * | 2012-11-28 | 2013-04-03 | 华为技术有限公司 | Similarity analysis method, device and system |
CN104636325A (en) * | 2015-02-06 | 2015-05-20 | 中南大学 | Document similarity determining method based on maximum likelihood estimation |
Non-Patent Citations (2)
Title |
---|
YUAN XIN-PAN 等: "Near-duplicate document detection with improved similarity measurement", 《SPRINGER》 * |
盛鑫海 等: "基于分组指纹的细粒度相似性检测系统", 《湖南工业大学学报》 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111104484A (en) * | 2019-12-19 | 2020-05-05 | 南京中孚信息技术有限公司 | Text similarity detection method and device and electronic equipment |
CN111104484B (en) * | 2019-12-19 | 2021-09-03 | 南京中孚信息技术有限公司 | Text similarity detection method and device and electronic equipment |
CN111444325A (en) * | 2020-03-30 | 2020-07-24 | 湖南工业大学 | Method for measuring document similarity by position coding single random permutation hash |
Also Published As
Publication number | Publication date |
---|---|
CN105718430B (en) | 2018-05-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
AU2018247340B2 (en) | Dvqa: understanding data visualizations through question answering | |
CN102722709B (en) | Method and device for identifying garbage pictures | |
CN104636325B (en) | A kind of method based on Maximum-likelihood estimation determination Documents Similarity | |
CN106294350B (en) | A kind of text polymerization and device | |
CN109710701A (en) | A kind of automated construction method for public safety field big data knowledge mapping | |
Wei et al. | On the spatial distribution of buildings for map generalization | |
RU2760471C1 (en) | Methods and systems for identifying fields in a document | |
CN103279478B (en) | A kind of based on distributed mutual information file characteristics extracting method | |
JP2012118977A (en) | Method and system for machine-learning based optimization and customization of document similarity calculation | |
CN103617280A (en) | Method and system for mining Chinese event information | |
CN109739997A (en) | Address control methods, apparatus and system | |
CN108090178B (en) | Text data analysis method, text data analysis device, server and storage medium | |
RU2768233C1 (en) | Fuzzy search using word forms for working with big data | |
JP2018537760A (en) | Method and apparatus for account mapping based on address information | |
Boubaker et al. | Online Arabic databases and applications | |
CN109918648B (en) | Rumor depth detection method based on dynamic sliding window feature score | |
AU2019200371A1 (en) | Utilizing artificial intelligence to integrate data from multiple diverse sources into a data structure | |
CN110287329A (en) | A kind of electric business classification attribute excavation method based on commodity text classification | |
CN113924582A (en) | Machine learning processing pipeline optimization | |
CN106127222A (en) | The similarity of character string computational methods of a kind of view-based access control model and similarity determination methods | |
CN110377690A (en) | A kind of information acquisition method and system based on long-range Relation extraction | |
CN106339481A (en) | Chinese compound new-word discovery method based on maximum confidence coefficient | |
CN112328657A (en) | Feature derivation method, feature derivation device, computer equipment and medium | |
CN114092948A (en) | Bill identification method, device, equipment and storage medium | |
CN105718430A (en) | Grouping minimum value-based method for calculating fingerprint similarity |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
TR01 | Transfer of patent right |
Effective date of registration: 20200529 Address after: Room g0044, headquarters building, Changsha Zhongdian Software Park Co., Ltd., No. 39, Jianshan Road, Changsha hi tech Development Zone, Changsha City, Hunan Province Patentee after: HUNAN YUN ZHI IOT NETWORKTECHNOLOGY Co.,Ltd. Address before: 412000 Taishan Road, Tianyuan District, Hunan, No. 88, No. Patentee before: HUNAN University OF TECHNOLOGY |
|
TR01 | Transfer of patent right |