CN104169917B - A kind of method based on whois lookup data flow point cutpoint and server - Google Patents

Abstract

Embodiments provide a kind of method based on whois lookup data flow point cutpoint.By judging in M window, in some window, whether at least part of data meet predetermined condition in the embodiment of the present invention, search data flow point cutpoint, when data at least part of in some window are unsatisfactory for predetermined condition, then skip N × U length, obtain next potential cut-point, improve data flow point cutpoint search efficiency.

Description

A kind of method based on whois lookup data flow point cutpoint and server

Technical field

The present invention relates to areas of information technology, particularly relate to a kind of based on whois lookup data stream The method of cut-point and server.

Background technology

The continuous growth of data volume so that the data storage providing sufficient becomes currently stored field The severe challenge faced.A kind of mode tackling this challenge at present is the number utilizing and needing storage According to redundancy properties, use data de-duplication technology, thus reduce the data volume of storage.

In prior art, repetition based on content piecemeal (Content Defined Chunk, CDC) Data deletion algorithm, first has to data stream to be stored is divided into a lot of data block, and by data Stream is divided into data block and is accomplished by searching suitable cut-point in a stream, two adjacent data streams Data between cut-point constitute a data block.Calculate the eigenvalue of data block, thus search Whether there is the data block of same characteristic features value, if finding the data block that same characteristic features refers to, then Think that existence repeats data.Concrete, data de-duplication technology based on content piecemeal is should Search by sliding window technique (Sliding Window Technique) content based on file The cut-point of piecemeal, i.e. determines data flow point by the Rabin fingerprint of data in calculation window Cutpoint.Assume to search cut-point from the left side of data stream to the right, calculate in sliding window every time The fingerprint of data, and by fingerprint value based on given integer K delivery after, with given remainder R compares；If equal, the right-hand member of window is data flow point cutpoints, is otherwise continued by window Turn right slip one byte, the most cyclically carry out calculating and comparison, until arrive data stream end Tail.During data de-duplication based on content piecemeal, search data flow point cutpoint, need Consume substantial amounts of calculating resource, thus become the bottleneck promoting data de-duplication performance.

Summary of the invention

First aspect, embodiments provides a kind of based on the segmentation of whois lookup data stream The method of point, is preset with rule on described server, and described rule is: for potential cut-point

K determines M some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x- A_x,p_x+B_x] corresponding predetermined condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；Described method includes:

A) it is current potential cut-point k according to described rule_iDetermine a p_izAnd described some p_izCorresponding Window W_iz[p_iz-A_z,p_iz+B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet pre- Fixed condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described some p_izAlong described data flow point cutpoint search direction jump N number of data stream segmentation Point minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), Obtain new potential cut-point, perform step a)；

C) as described current potential cut-point k_iM window in each window W_ix[p_ix- A_x,p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor data flow point cutpoint.

In conjunction with in first aspect, the first possible implementation, described rule also includes: extremely Few two some p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f。

In conjunction with the first possible implementation of first aspect, the implementation that the second is possible In, described rule also includes: described at least two point p_eAnd p_f, relative to described potential segmentation Point k, searches in the reverse direction at described data flow point cutpoint.

In conjunction with the realization side that the first possible implementation of first aspect or the second are possible Formula, in the third possible implementation, described rule also includes: described at least two point p_e And p_fBetween distance be 1 U.

In conjunction with first aspect, or first aspect first is to the third arbitrary possible implementation, In 4th kind of possible implementation, it is judged that described window W_iz[p_iz-A_z,p_iz+B_zAt least portion in] Whether divided data meets described predetermined condition C_z, specifically include:

Random function is used to judge described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are No meet described predetermined condition C_z。

In conjunction with the 4th kind of possible implementation of first aspect, the 5th kind of possible implementation In, described use random function judges described window W_iz[p_iz-A_z,p_iz+B_zAt least partly count in] According to whether meeting described predetermined condition C_z, it is specially and uses hash function to judge described window W_iz [p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z。

In conjunction with first aspect, or first aspect first is to the 5th kind of arbitrary possible implementation, In 6th kind of possible implementation, as described window W_iz[p_iz-A_z,p_iz+B_zIn] at least partly Data are unsatisfactory for described predetermined condition C_z, from described some p_izSearch along described data flow point cutpoint The direction N number of data flow point cutpoint minimum of jump searches unit U, it is thus achieved that described new potential segmentation Point, according to described rule, the some p determined for described new potential cut-point_icCorresponding window W_ic[p_ic-A_c,p_ic+B_c] left margin and described window W_iz[p_iz-A_z,p_iz+B_z] right margin weight The described some p closed or determine for described new potential cut-point_icCorresponding described window W_ic [p_ic-A_c,p_ic+B_c] left margin be positioned at described window W_iz[p_iz-A_z,p_iz+B_zWithin the scope of]；Its In, the described some p determined for described new potential cut-point_icIt is according to described rule, for institute State in the sequence that M point that new potential cut-point determines obtains according to data stream search direction The point of sequence first.

In conjunction with the 4th kind of possible implementation of first aspect, the 7th kind of possible implementation In, use random function to judge described window W_iz[p_iz-A_z,p_iz+B_zAt least part of data in] Whether meet described predetermined condition C_z, specifically include:

At described window W_iz[p_iz-A_z,p_iz+B_zF byte is selected, by described F byte in] Recycling H time, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[p_iz-A_z, p_iz+B_zIn], at least part of data meet described predetermined condition C_z。

Second aspect, embodiments provides a kind of based on the segmentation of whois lookup data stream The method of point, is preset with rule on described server, and described rule is: for potential cut-point K determines M window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined bar Part C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；

Described method includes:

A) it is current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz[k_i-A_z, k_i+B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet predetermined Condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined Condition C_z, from described current potential cut-point k_iSearch along described data flow point cutpoint The direction N number of data flow point cutpoint minimum of jump searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖), it is thus achieved that new potential cut-point, step a) is performed；

C) as described current potential cut-point k_iM window in each window W_ix[k_i-A_x, k_i+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

In conjunction with in second aspect, the first possible implementation, described rule also includes: extremely Few two window W_ie[k_i-A_e,k_i+B_e] and W_if[k_i-A_f,k_i+B_f], meet condition: | A_e+B_e |=| A_f+B_f|, C_e=C_f。

In conjunction with the first possible implementation of second aspect, the implementation that the second is possible In, described rule also includes: A_eAnd A_fFor positive integer.

In conjunction with the realization side that the first possible implementation of second aspect or the second are possible Formula, in the implementation that the third is possible, described rule also includes: A_e-1=A_f, B_e+ 1= B_f。

In conjunction with second aspect, or second aspect first is to the 3rd arbitrary possible implementation, the In four kinds of possible implementations, it is judged that described window W_iz[k_i-A_z,k_i+B_zAt least partly count in] Predetermined condition C is met according to the most described_z, specifically include:

Random function is used to judge described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are No meet described predetermined condition C_z。

In conjunction with the 4th kind of possible implementation of second aspect, the 5th kind of possible implementation In, described use random function judges described window W_iz[k_i-A_z,k_i+B_zAt least part of data in] Whether meet described predetermined condition C_z, it is specially and uses hash function to judge described window W_iz[k_i- A_z,k_i+B_zIn], whether at least part of data meet described predetermined condition C_z。

In conjunction with second aspect, or second aspect first is to the 5th arbitrary possible implementation, the In six kinds of possible implementations, as described window W_iz[k_i-A_z,k_i+B_zAt least partly count in] According to being unsatisfactory for described predetermined condition C_z, from described current potential cut-point k_iAlong described data flow point The cutpoint search direction N number of data flow point cutpoint minimum of jump searches unit U, it is thus achieved that described new diving At cut-point, according to described rule, the window W determined for described new potential cut-point_ic[k_i -A_c,k_i+B_c] left margin and described window W_iz[k_i-A_z,k_i+B_z] right margin overlap or The described window W determined for described new potential cut-point_ic[k_i-A_c,k_i+B_c] left margin position In described window W_iz[k_i-A_z,k_i+B_zWithin the scope of]；Wherein, for described new potential segmentation The described window W that point determines_ic[k_i-A_c,k_i+B_c] it is according to described rule, for described new diving Sequence the in the sequence that M the window determined at cut-point obtains according to data stream search direction The window of one.

In conjunction with the 4th kind of possible implementation of second aspect, the 7th kind of possible implementation In, use random function to judge described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are No meet described predetermined condition C_z, specifically include:

At described window W_iz[k_i-A_z,k_i+B_zF byte is selected, by anti-for described F byte in] Utilizing H time again, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[k_i-A_z, k_i+B_zIn], at least part of data meet described predetermined condition C_z。

The third aspect, embodiments provides a kind of clothes for searching data flow point cutpoint Business device, described server includes CPU and main storage, described CPU Communicating with described main storage, be preset with rule on described server, described rule is: for Potential cut-point k determines M some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window Mouth W_x[p_x-A_x,p_x+B_x] corresponding predetermined condition C_x, wherein, x be 1 to M continuous print from So number, M >=2, A_xAnd B_xFor integer；

Described main storage is used for storing executable instruction, and described CPU performs described Executable instruction, to perform following steps:

A) it is current potential cut-point k according to described rule_iDetermine a p_izAnd described some p_izCorresponding Window W_iz[p_iz-A_z,p_iz+B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet pre- Fixed condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described some p_izAlong described data flow point cutpoint search direction jump N number of data stream segmentation Point minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), Obtain new potential cut-point, perform step a)；

C) as described current potential cut-point k_iM window in each window W_ix[p_ix- A_x,p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor data flow point cutpoint.

In conjunction with in the third aspect, the first possible implementation, described rule also includes: extremely Few two some p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f。

In conjunction with the first possible implementation of the third aspect, the implementation that the second is possible In, described rule also includes: described at least two point p_eAnd p_f, relative to described potential point Cutpoint k, searches in the reverse direction at described data flow point cutpoint.

In conjunction with the realization side that the first possible implementation of the third aspect or the second are possible Formula, in the third possible implementation, described rule also includes: described at least two point p_e And p_fBetween distance be 1 U.

In conjunction with the third aspect, or first to the 3rd arbitrary possible implementation, the 4th kind may Implementation in, described CPU specifically for

Random function is used to judge described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are No meet described predetermined condition C_z。

In conjunction with the 4th kind of possible implementation of the third aspect, the 5th kind of possible implementation In, described CPU judges described window W specifically for using hash function_iz[p_iz-A_z, p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z。

In conjunction with the third aspect, or first to the 5th arbitrary possible implementation, the 6th kind may Implementation in, as described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for Described predetermined condition C_z, from described some p_izJump N number of along described data flow point cutpoint search direction Data flow point cutpoint minimum searches unit U, it is thus achieved that described new potential cut-point, according to described Rule, the some p determined for described new potential cut-point_icCorresponding window W_ic[p_ic-A_c,p_ic+ B_c] left margin and described window W_iz[p_iz-A_z,p_iz+B_z] right margin overlap or be described The described some p that new potential cut-point determines_icCorresponding described window W_ic[p_ic-A_c,p_ic+B_c] Left margin be positioned at described window W_iz[p_iz-A_z,p_iz+B_zWithin the scope of]；Wherein, for described newly Described some p determining of potential cut-point_icIt is according to described rule, for described new potential point M the point that cutpoint determines is according to the point of sequence first in the sequence of data stream search direction acquisition.

In conjunction with the 4th kind of possible implementation of the third aspect, the 7th kind of possible implementation In, described CPU uses random function to judge described window W_iz[p_iz-A_z,p_iz+B_z] In at least partly data whether meet described predetermined condition C_z, specifically include:

At described window W_iz[p_iz-A_z,p_iz+B_zF byte is selected, by described F byte in] Recycling H time, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[p_iz-A_z, p_iz+B_zIn], at least part of data meet described predetermined condition C_z.Fourth aspect, the present invention is real Execute example and provide a kind of server for searching data flow point cutpoint, during described server includes Central Processing Unit and main storage, described CPU and described main storage communication, Being preset with rule on described server, described rule is: determine M window for potential cut-point k Mouth W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, its In, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；

Described main storage is used for storing executable instruction, and described CPU performs described Executable instruction, to perform following steps:

A) it is current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz[k_i-A_z,k_i +B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet predetermined Condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined condition C_z, from described current potential cut-point k_iJump N number of along described data flow point cutpoint search direction Data flow point cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖), obtain Obtain potential cut-point newly, perform step a)；

C) as described current potential cut-point k_iM window in each window W_ix[k_i-A_x, k_i+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

In conjunction with in fourth aspect, the first possible implementation, described rule also includes: extremely Few two window W_ie[k_i-A_e,k_i+B_e] and W_if[k_i-A_f,k_i+B_f], meet condition: | A_e+B_e |=| A_f+B_f|, C_e=C_f。

In conjunction with the first possible implementation of fourth aspect, the implementation that the second is possible In, described rule also includes: A_eAnd A_fFor positive integer.

In conjunction with the realization side that the first possible implementation of fourth aspect or the second are possible Formula, in the implementation that the third is possible, described rule also includes: A_e-1=A_f, B_e+ 1= B_f。

In conjunction with fourth aspect, or first to the 3rd arbitrary possible implementation, the 4th kind may Implementation in, described CPU specifically for

Random function is used to judge described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are No meet described predetermined condition C_z。

In conjunction with the 4th kind of possible implementation of fourth aspect, the 5th kind of possible implementation In, described CPU judges described window W specifically for using hash function_iz[k_i-A_z, k_i+B_zIn], whether at least part of data meet described predetermined condition C_z。

In conjunction with fourth aspect, or first to the 5th arbitrary possible implementation, the 6th kind may Implementation in, as described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for Described predetermined condition C_z, from described current potential cut-point k_iSearch along described data flow point cutpoint The direction N number of data flow point cutpoint minimum of jump searches unit U, it is thus achieved that described new potential cut-point, According to described rule, the window W determined for described new potential cut-point_ic[k_i-A_c,k_i+B_c] Left margin and described window W_iz[k_i-A_z,k_i+B_z] right margin overlap or be described newly The described window W that determines of potential cut-point_ic[k_i-A_c,k_i+B_c] left margin be positioned at described window Mouth W_iz[k_i-A_z,k_i+B_zWithin the scope of]；Wherein, determine for described new potential cut-point Described window W_ic[k_i-A_c,k_i+B_c] it is according to described rule, for described new potential cut-point The sequence that M the window determined obtains according to data stream search direction sorts first window.

In conjunction with the 4th kind of possible implementation of fourth aspect, the 7th kind of possible implementation In, described CPU uses random function to judge described window W_iz[k_i-A_z,k_i+B_z] In at least partly data whether meet described predetermined condition C_z, specifically include:

At described window W_iz[k_i-A_z,k_i+B_zF byte is selected, by anti-for described F byte in] Utilizing H time again, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[k_i-A_z, k_i+B_zIn], at least part of data meet described predetermined condition C_z。

5th aspect, embodiments provides a kind of clothes for searching data flow point cutpoint Business device, is preset with rule on described server, and described rule is: true for potential cut-point k Determine M some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+ B_x] corresponding predetermined condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xWith B_xFor integer；

Described server includes: processing unit, is used for performing step a):

A) it is current potential cut-point k according to described rule_iDetermine a p_izAnd described some p_izCorresponding Window W_iz[p_iz-A_z,p_iz+B_z], i and z is integer, and 1≤z≤M；

Judge processing unit, be used for judging described window W_iz[p_iz-A_z,p_iz+B_zIn] at least partly Whether data meet predetermined condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described some p_izAlong described data flow point cutpoint search direction jump N number of data stream segmentation Point minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), Obtain new potential cut-point, the most described determine that unit is that described new potential cut-point performs step A)；

As described current potential cut-point k_iM window in each window W_ix[p_ix-A_x, p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

In conjunction with in the 5th aspect, the first possible implementation, described rule also includes: extremely Few two some p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f。

In conjunction with the first possible implementation of the 5th aspect, the implementation that the second is possible In, described rule also includes: described at least two point p_eAnd p_f, relative to described potential point Cutpoint k, searches in the reverse direction at described data flow point cutpoint.

In conjunction with the implementation that the first possible implementation of the 5th aspect or the second are possible, In the implementation that the third is possible, described rule also includes: described at least two point p_eWith p_fSpacing be 1 U.

In conjunction with the 5th aspect, or first to the 3rd arbitrary possible implementation, the 4th kind may Implementation in, the described specifically used random function of judgement processing unit judges described window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z。

In conjunction with the 4th kind of possible implementation of the 5th aspect, the 5th kind of possible implementation In, described decision process unit judges described window W specifically for using hash function_iz[p_iz-A_z, p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z。

In conjunction with the 5th aspect, or first to the 5th arbitrary possible implementation, the 6th kind may Implementation in, described judgement processing unit is for as described window W_iz[p_iz-A_z,p_iz+B_z] In at least partly data be unsatisfactory for described predetermined condition C_z, from described some p_izAlong described data stream Cut-point search direction N number of data flow point cutpoint minimum of jumping searches unit U, it is thus achieved that described new Potential cut-point, described determines that unit is that described new potential cut-point performs step a), root According to described rule, the some p determined for described new potential cut-point_icCorresponding window W_ic[p_ic- A_c,p_ic+B_c] left margin and described window W_iz[p_iz-A_z,p_iz+B_z] right margin overlap or The described window W determined for described new potential cut-point_ic[p_ic-A_c,p_ic+B_c] left margin It is positioned at described window W_iz[p_iz-A_z,p_iz+B_zWithin the scope of]；Wherein, for described new potential point The described window W that cutpoint determines_ic[p_ic-A_c,p_ic+B_c] it is according to described rule, for described new Sequence the in the sequence that potential cut-point determine M point obtains according to data stream search direction The point of one.

In conjunction with the 4th kind of possible implementation of the 5th aspect, the 7th kind of possible implementation In, described judgement processing unit judges described window W specifically for using random function_iz[p_iz- A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z, specifically include:

At described window W_iz[p_iz-A_z,p_iz+B_zF byte is selected, by described F byte in] Recycling H time, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[p_iz-A_z, p_iz+B_zIn], at least part of data meet described predetermined condition C_z。

6th aspect, embodiments provides a kind of clothes for searching data flow point cutpoint Business device, is preset with rule on described server, and described rule is: true for potential cut-point k Determine M window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, Wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；

Described server comprises determining that unit, is used for performing step a:

A) it is current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz[k_i-A_z,k_i +B_z], i and z is integer, and 1≤z≤M；

Judge processing unit, be used for judging described window W_iz[k_i-A_z,k_i+B_zAt least partly count in] According to whether meeting predetermined condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined condition C_z, from described current potential cut-point k_iJump N number of along described data flow point cutpoint search direction Data flow point cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖), obtain Obtain potential cut-point newly, perform step a)；

C works as described current potential cut-point k_iM window in each window W_ix[k_i-A_x, k_i+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

In conjunction with in the 6th aspect, the first possible implementation, described rule also includes: extremely Few two window W_ie[k_i-A_e,k_i+B_e] and W_if[k_i-A_f,k_i+B_f], meet condition: | A_e+B_e |=| A_f+B_f|, C_e=C_f。

In conjunction with the first possible implementation of the 6th aspect, the implementation that the second is possible In, described rule also includes: A_eAnd A_fFor positive integer.

In conjunction with the realization side that the first possible implementation of the 6th aspect or the second are possible Formula, in the implementation that the third is possible, described rule also includes: A_e-1=A_f, B_e+ 1= B_f。

In conjunction with the 6th aspect, or first to the 3rd arbitrary possible implementation, the 4th kind may Implementation in, described judgement processing unit specifically for

Random function is used to judge described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are No meet described predetermined condition C_z。

In conjunction with the 4th kind of possible implementation of the 6th aspect, the 5th kind of possible implementation In, described judgement processing unit judges described window W specifically for using hash function_iz[k_i-A_z, k_i+B_zIn], whether at least part of data meet described predetermined condition C_z。

In conjunction with the 6th aspect, or first to the 5th arbitrary possible implementation, the 6th kind may Implementation in, described judgement processing unit is for as described window W_iz[k_i-A_z,k_i+B_z] In at least partly data be unsatisfactory for described predetermined condition C_z, from described current potential cut-point k_iEdge The described data flow point cutpoint search direction N number of data flow point cutpoint minimum of jump searches unit U, obtains Described new potential cut-point, described determine that unit is that described new potential cut-point performs step A), according to described rule, the window W determined for described new potential cut-point_ic[k_i-A_c, k_i+B_c] left margin and described window W_iz[k_i-A_z,k_i+B_z] right margin overlap or be The described window W that described new potential cut-point determines_ic[k_i-A_c,k_i+B_c] left margin be positioned at Described window W_iz[k_i-A_z,k_i+B_zWithin the scope of]；Wherein, for described new potential cut-point The described window W determined_ic[k_i-A_c,k_i+B_c] it is according to described rule, for described new potential The sequence that M the window that cut-point determines obtains according to data stream search direction sorts first Window.

In conjunction with the 4th kind of possible implementation of the 6th aspect, the 7th kind of possible implementation In, described judgement processing unit uses random function to judge described window W_iz[k_i-A_z,k_i+B_z] In at least partly data whether meet described predetermined condition C_z, specifically include:

At described window W_iz[k_i-A_z,k_i+B_zF byte is selected, by anti-for described F byte in] Utilizing H time again, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[k_i-A_z, k_i+B_zIn], at least part of data meet described predetermined condition C_z。

The embodiment of the present invention at least partly counts in some window in M window by judging According to whether meeting predetermined condition, search data flow point cutpoint, when at least portion in some window Divided data is unsatisfactory for predetermined condition, then skip N*U length, it is thus achieved that next potential cut-point, Improve data flow point cutpoint search efficiency.

Accompanying drawing explanation

Fig. 1 is embodiment of the present invention one application scenarios schematic diagram；

Fig. 2 is data flow point cutpoint schematic diagrams；

Fig. 3 is for searching data flow point cutpoint schematic diagram；

Fig. 4 is embodiment of the present invention method schematic diagram；

Fig. 5 and Fig. 6 is for searching data flow point cutpoint embodiment schematic diagram；

Fig. 7 and Fig. 8 is for searching data flow point cutpoint embodiment schematic diagram；

Fig. 9 and Figure 10 is for looking for data flow point cutpoint embodiment schematic diagram；

Figure 11 and Figure 12 and Figure 13 is for looking for data flow point cutpoint embodiment schematic diagram；

Figure 14 and Figure 15 is for looking for data flow point cutpoint embodiment schematic diagram；

Figure 16 and Figure 17 is for judge in window, whether at least part of data meet predetermined condition schematic diagram；

Figure 18 is duplicate removal server architecture figure；

Figure 19 is duplicate removal server architecture figure；

Figure 20 is embodiment of the present invention method schematic diagram；

Figure 21 and Figure 22 is for searching data flow point cutpoint embodiment schematic diagram；

Figure 23 and Figure 24 is for searching data flow point cutpoint embodiment schematic diagram；

Figure 25 and Figure 26 is for looking for data flow point cutpoint embodiment schematic diagram；

Figure 27 and Figure 28 and Figure 29 is for looking for data flow point cutpoint embodiment schematic diagram；

Figure 30 and Figure 31 is for looking for data flow point cutpoint embodiment schematic diagram；

Figure 32 and Figure 33 is for judge in window, whether at least part of data meet predetermined condition schematic diagram.

Specific embodiment

Along with the continuous progress of memory technology, data generation amount is also being continuously increased, substantial amounts of number Highest demand is proposed according to memory capacity.While memory capacity increases, too increase IT Equipment purchase cost, in order to alleviate the demand contradictory between data volume and memory capacity, saves IT Equipment purchase cost, introduces data de-duplication technology in field of data storage.

Embodiment of the present invention one uses scene to be data backup scene.Data backup is for preventing The loss of data that a variety of causes causes, backs up data to other storages by backup server and is situated between The process of matter.Data backup system framework as shown in Figure 1.Data backup system includes client End (101a, 101b ... 101n), the backup server 102, (letter of data de-duplication server Claim duplicate removal server or heavily delete server) 103 and storage device (104a, 104b ... 104n). Wherein client (101a, 101b ... 101n) can be application server, work station etc.；Standby The data that part server 102 generates for backup client；Duplicate removal server 103 is used for performing standby The data de-duplication task of number evidence；Storage device (104a, 104b ... 104n) is as depositing The storage medium of the data after storage data de-duplication, can be that disk array, tape library etc. are deposited Storage media.Client (101a, 101b ... 101n), backup server 102, repetition data are deleted Except server 103 and storage device (104a, 104b ... 104n) can pass through switch, office The modes such as territory net, the Internet, optical fiber connect, and the said equipment may be located at same place, it is possible to To be positioned at different location.Backup server 102, heavily delete server 103, storage device (104a, 104b ... 104n) can be independent physical equipment, or be physically integrated in implementing It is integrated, or backup server 102 becomes one with heavily deleting server 103, or heavily delete Server 103 and storage device (104a, 104b ... 104n) become one.

The duplicate removal server 103 data stream execution data de-duplication operations to Backup Data, one As comprise the following steps:

1) data flow point cutpoint is searched: search data flow point in a stream according to special algorithm Cutpoint；

2) data block is divided according to the data flow point cutpoint found；

3) eigenvalue of data block is calculated: calculate the eigenvalue of data block as identifying these data The feature of block；Calculated eigenvalue is added to the data block of file corresponding to this data stream Feature list in；SHA-1 or MD5 algorithm is typically utilized to calculate the eigenvalue of data block；

4) identical block detection: the eigenvalue of calculated data block is special with data block Levy already present eigenvalue in list to compare to determine whether there is identical block；

5) deleting duplicated data block: detected by identical block, if it find that data block is special Levy the eigenvalue that in list, existence is identical with this data block, then need not to store again this data block or The repetition data block stored number that person determines according to backup policy decides whether to store this data block.

By duplicate removal server 103, the data stream of Backup Data is performed data de-duplication operations Step understand, data flow point cutpoint is searched as the committed step of data de-duplication operations, Directly determine the performance of data de-duplication.

In the embodiment of the present invention, duplicate removal server 103 receives the backup that backup server 102 sends File, performs data de-duplication to this document and processes.Usual pending backup file is in duplicate removal Presenting with data-stream form in server 103, duplicate removal server 103 searches the segmentation in data stream During point, data flow point cutpoint minimum to be determined searches unit, concrete as in figure 2 it is shown, such as Potential cut-point k₁Continuous two the data flow point cutpoint minimums being positioned at sequence number respectively 1 and 2 are looked into Looking between unit, potential cut-point refers to that needs carry out judging whether to split as data stream The point of point；As a k₁It is a data flow point cutpoint, data flow point cutpoint search direction such as Fig. 2 Shown in middle arrow, searching next potential cut-point is k₇, i.e. it is positioned at sequence number and is respectively 7 and 8 Continuous two data flow point cutpoint minimums search between unit, as potential cut-point k₇For data Flow point cutpoint, two the most adjacent data flow point cutpoint k₁、k₇Between data be 1 data Block.Data flow point cutpoint minimum search unit specifically can determine according to actual needs, here with As a example by 1 byte (Byte), i.e. the data flow point cutpoint minimum of serial number 1,2,7 and 8 is looked into Unit-sized is looked for be 1 byte.The data flow point usual table of cutpoint search direction as shown in Figure 2 Show and searched to end-of-file direction by file header, or by file Caudad file header direction, this enforcement In example as a example by searching to end-of-file direction from file header.

In data de-duplication scene, usual data block is the least, and data de-duplication rate is the highest, The most easily find repetition data block, but the metadata quantity thus generated is the biggest, and number According to block little to a certain extent after, data de-duplication rate would not add, but metadata Quantity but can sharply increase.Therefore, it is necessary to control data block size, in actual application, generally The minima of meeting setting data block, such as 4KB (4096 bytes), simultaneously take account of repetition Data deletion rate, also can the maximum of setting data block, i.e. data block size not can exceed that maximum Value, such as 12KB (12288 bytes).A kind of specific implementation is as it is shown on figure 3, go Weight server 103 is searching data flow point cutpoint, k along direction shown in arrow_aFor current lookup The data flow point cutpoint arrived, from k_aNext potential point is searched to data flow point cutpoint search direction Cutpoint, for meeting minimum data block requirement, it will usually start along data from data flow point cutpoint Flow point cutpoint search direction skips minimum data block size, from the beginning of minimum data block end position Search, namely using minimum data block end position as next potential cut-point k_i.At this In inventive embodiments, can be first from k_aPoint is along data flow point cutpoint search direction jump minimum data Block 4KB, i.e. 4*1024=4096 byte.From k_aPoint jumps along data flow point cutpoint search direction 4096 bytes, the end position the 4096th byte obtains some k_i, as potential cut-point, Such as k_iContinuous two the data flow point cutpoint minimums being positioned at sequence number respectively 4096 and 4097 are looked into Look between unit.Still as a example by Fig. 3, k_aThe data flow point cutpoint arrived for current lookup, edge Next data flow point cutpoint is searched in direction as shown in Figure 3, if it exceeds data block maximum is still So do not find next data flow point cutpoint, then from k_aPoint starts to look into data flow point cutpoint Direction is looked for reach the some k of data block maximum_zAs next data flow point cutpoint, force Segmentation.

The embodiment of the present invention provides a kind of side based on duplicate removal whois lookup data flow point cutpoint Method, as shown in Figure 4, including:

Being preset with rule on duplicate removal server 103, described rule is: true for potential cut-point k Determine M some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+ B_x] corresponding predetermined condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xWith B_xFor integer；Wherein, p_xSpacing d with potential cut-point k_xIndividual data flow point cutpoint is minimum Searching unit, data flow point cutpoint minimum is searched unit and is represented with U, and in the present embodiment, U=1 is individual Byte,.In the implementation shown in Fig. 3, about the value of M, one of which realization side Formula, M*U value is not more than the ultimate range between two the adjacent data flow point cutpoints preset, The data block greatest length i.e. preset.Judge some p_zCorresponding window W_z[p_z-A_z, p_z+B_z] In at least partly data whether meet predetermined condition C_z, wherein, z is integer, 1≤z≤M, (p_z -A_z) and (p_z+B_z) represent window W respectively_zTwo borders.When judging that any one puts p_z Window W_z[p_z-A_z, p_z+B_zIn], at least part of data are unsatisfactory for predetermined condition C_z, then from It is unsatisfactory for the window W of predetermined condition_z[p_z-A_z, p_z+B_z] corresponding some p_zSplit along data stream The point search direction N number of byte of jump, N≤‖ B_z‖+max_x(‖A_x‖+‖(k-p_x)‖).Its In, ‖ (k-p_x) ‖ represent M some p_xIn any one point with potential cut-point k between distance, max_x(‖A_x‖+‖(k-p_x) ‖) represent M some p_xIn any one point with potential cut-point k Between distance and A corresponding to this point_xThe maximum of absolute value sum；‖B_z‖ represents W_z [p_z-A_z, p_z+B_zB in]_zAbsolute value, embodiment below will specifically be introduced N value Principle.As each window W judged in M window_x[p_x-A_x, p_x+B_xIn] at least partly Data meet predetermined condition C_x, the most potential cut-point k is data flow point cutpoints.

Concrete, to current potential cut-point k_i, according to described rule, perform following steps:

Step 401: be current potential cut-point k according to described rule_iDetermine a p_izAnd described point p_izCorresponding window W_iz[p_iz-A_z,p_iz+B_z], i and z is integer, and 1≤z≤M；

Step 402: judge described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are the fullest Foot predetermined condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described some p_izAlong described data flow point cutpoint search direction jump N number of data stream segmentation Point minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), Obtain new potential cut-point, perform step 401；

As described current potential cut-point k_iM window in each window W_ix[p_ix-A_x, p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

Further, described rule also includes: at least two point p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f；

Described rule also includes: described at least two point p_eAnd p_f, relative to described potential point Cutpoint k, searches in the reverse direction at described data flow point cutpoint.

Described rule also includes: described at least two point p_eAnd p_fBetween distance be 1 U.

Judge described window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described pre- Fixed condition C_z, specifically include:

Random function is used to judge described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are No meet described predetermined condition C_z。

Described use random function judges described window W_iz[p_iz-A_z,p_iz+B_zAt least partly count in] According to whether meeting described predetermined condition C_z, it is specially and uses hash function to judge described window W_iz [p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z。

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described some p_izAlong described data flow point cutpoint search direction jump N number of data stream segmentation Point minimum searches unit U, it is thus achieved that described new potential cut-point, according to described rule, for institute State the some p that new potential cut-point determines_icCorresponding window W_ic[p_ic-A_c,p_ic+B_c] the left side Boundary and described window W_iz[p_iz-A_z,p_iz+B_z] right margin overlap or be described newly potential point The described some p that cutpoint determines_icCorresponding described window W_ic[p_ic-A_c,p_ic+B_c] left margin position In described window W_iz[p_iz-A_z,p_iz+B_zWithin the scope of]；Wherein, for described new potential segmentation The described some p that point determines_icIt is according to described rule, the M determined for described new potential cut-point The sequence that individual point obtains according to data stream search direction sorts first point.

The embodiment of the present invention at least partly counts in some window in M window by judging According to whether meeting predetermined condition, search data flow point cutpoint, when at least portion in some window Divided data is unsatisfactory for predetermined condition, then skip N*U length, and wherein, N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), it is thus achieved that next potential cut-point, improve Data flow point cutpoint search efficiency.

During data de-duplication, for ensureing that data block size is uniform, average can be considered According to block (also referred to as average piecemeal) size, i.e. meeting minimum data block size and maximum data While block size limits, can determine whether average data block size, big to ensure the data block obtained Little uniformly.Point p_xNumber M and some p_xCorresponding window W_x[p_x-A_x, p_x+B_xAt least portion in] Divided data meets predetermined condition C_xProbability, the two factor determines and finds data flow point cutpoint Probability (representing with P (n)).The former affects the length of jump, and the latter affects the probability of jump, The two joint effect average mark block size.It is said that in general, when average mark block size is fixed, point p_xNumber M increases, then a single point p_xCorresponding window W_x[p_x-A_x, p_x+B_xAt least portion in] Divided data meets predetermined condition C_xProbability also increase, such as on duplicate removal server 103 preset Rule be: determine 11 some p for potential cut-point k_x, it is natural that x is respectively 1 to 11 continuous print Number, any one some p in 11 points_xCorresponding window W_x[p_x-A_x, p_x+B_xIn] at least partly Data meet predetermined condition C_xProbability be 1/2.And another preset on duplicate removal server 103 Group rule is: 24 the some p selected for potential cut-point k_x, x is respectively 1 to 24 continuous print certainly So number, any one some p in 24 points_xCorresponding window W_x[p_x-A_x, p_x+B_xAt least portion in] Divided data meets predetermined condition C_xProbability 3/4.Concrete window W_x[p_x-A_x, p_x+B_xIn] extremely Small part data meet predetermined condition C_xProbability set and can be found in and judge window W_x[p_x-A_x, p_x +B_xIn], whether at least part of data meet predetermined condition C_xThe description of part.Point p_xNumber M with Point p_xCorresponding window W_x[p_x-A_x, p_x+B_xIn], at least part of data meet predetermined condition C_x's Probability the two factor determine P (n), P (n) represent: from data stream original position/data Flow point cutpoint is searched after n data flow point cutpoint minimum searches unit and is not found data flow point cutpoint Probability.The calculating process of P (n), actually one multistep is determined about the two factor Long Fibonacci ordered series of numbers, after will be described in detail.After obtaining P (n), 1-P (n) is data The distribution function of flow point cutpoint, (1-P (n))-(1-P (n-1))=P (n-1)-P (n), it is n-th Point finds the probability of data flow point cutpoint, the namely density function of data flow point cutpoint, according to The density function of data flow point cutpoint just can be with integrationThus try to achieve The desired length of data flow point cutpoint, i.e. average mark block size, wherein, 4*1024 (byte) Representing minimum data block length, 12*1024 (byte) represents maximum data block length.

On the basis of data flow point cutpoint as shown in Figure 3 is searched, at the embodiment shown in Fig. 5 In, duplicate removal server 103 is preset with rule, described rule is: true for potential cut-point k Fixed 11 some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] (it is called for short window W_x) and window Mouth W_x[p_x-A_x,p_x+B_x] corresponding predetermined condition C_x, wherein, A₁=A₂=A₃=A₄=A₅=A₆=A₇ =A₈=A₉=A₁₀=A₁₁=169, B₁=B₂=B₃=B₄=B₅=B₆=B₇=B₈=B₉=B₁₀=B₁₁=0, and C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁.Wherein, some p_xWith potential cut-point Spacing d of k_xIndividual byte, concrete, put p₁With 0 byte of spacing of potential cut-point k, Point p₂With 1 byte of spacing of potential cut-point k, put p₃Spacing with potential cut-point k 2 bytes, put p₄With 3 bytes of spacing of potential cut-point k, put p₅With potential cut-point 4 bytes of the spacing of k, put p₆With 5 bytes of spacing of potential cut-point k, put p₇With 6 bytes of the spacing of potential cut-point k, put p₈7 words of spacing with potential cut-point k Joint, puts p₉With 8 bytes of spacing of potential cut-point k, put p₁₀And between potential cut-point k 9 bytes of distance, put p₁₁With 10 bytes of spacing of potential cut-point k, and put p₂、p₃、 p₄、p₅、p₆、p₇、p₈、p₉、p₁₀And p₁₁It is respectively positioned on data relative to potential cut-point k Flow point cutpoint searches opposite direction.k_aFor data flow point cutpoint, the cutpoint of data flow point shown in Fig. 5 Search direction is from left to right, from data flow point cutpoint k_aAfter skipping minimum data block 4KB, Small data block 4KB end position is as next potential cut-point k_i, for potential cut-point k_iReally Fixed point p_ix, in the present embodiment, according to the rule preset on duplicate removal server 103, x is respectively It is 1 to 11 continuous print natural numbers.In the embodiment shown in Fig. 5, for potential cut-point k_iReally Fixed point is 11, respectively p_i1、p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、 p_i10And p_i11, put p_i1、p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11Right The window answered is respectively W_i1[p_i1-169,p_i1]、W_i2[p_i2-169,p_i2]、W_i3[p_i3-169,p_i3]、W_i4 [p_i4-169,p_i4]、W_i5[p_i5-169,p_i5]、W_i6[p_i6-169,p_i6]、W_i7[p_i7-169,p_i7]、W_i8 [p_i8-169,p_i8]、W_i9[p_i9-169,p_i9]、W_i10[p_i10-169,p_i10] and W_i11[p_i11-169,p_i11]。 Above-mentioned window is briefly referred to as W_i1、W_i2、W_i3、W_i4、W_i5、W_i6、W_i7、W_i8、W_i9、 W_i10And W_i11.Wherein, some p_ixWith potential cut-point k_iSpacing d_xIndividual byte, concrete, p_i1With k_iSpacing 0 byte, p_i2With k_iSpacing 1 byte, p_i3With k_iSpacing 2 bytes, p_i4 With k_iSpacing 3 bytes, p_i5With k_iSpacing 4 bytes, p_i6With k_iSpacing 5 bytes, p_i7With k_iSpacing 6 bytes, p_i8With k_iSpacing 7 bytes, p_i9With k_iSpacing 8 bytes, p_i10With k_i 9 bytes of spacing, p_i11With k_i10 bytes of spacing, and p_i2、p_i3、p_i4、p_i5、p_i6、 p_i7、p_i8、p_i9、p_i10And p_i11Relative to potential cut-point k_iIt is respectively positioned on data flow point cutpoint to look into Look for opposite direction.Judge W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁、 Judge W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂, judge W_i3 [p_i3-169,p_i3In], whether at least part of data meet predetermined condition C₃, judge W_i4[p_i4-169, p_i4In], whether at least part of data meet predetermined condition C₄, judge W_i5[p_i5-169,p_i5In] at least Whether part data meet predetermined condition C₅, judge W_i6[p_i6-169,p_i6At least part of data in] Whether meet predetermined condition C₆, judge W_i7[p_i7-169,p_i7In], whether at least part of data meet Predetermined condition C₇, judge W_i8[p_i8-169,p_i8In], whether at least part of data meet predetermined condition C₈, judge W_i9[p_i9-169,p_i9In], whether at least part of data meet predetermined condition C₉, judge W_i10[p_i10-169,p_i10In], whether at least part of data meet predetermined condition C₁₀With judge W_i11[p_i11 -169,p_i11In], whether at least part of data meet predetermined condition C₁₁.When judging window W_i1In extremely Small part data meet predetermined condition C₁, window W_i2In at least partly data meet predetermined condition C₂, window W_i3In at least partly data meet predetermined condition C₃, window W_i4In at least partly count According to meeting predetermined condition C₄, window W_i5In at least partly data meet predetermined condition C₅, window W_i6 In at least partly data meet predetermined condition C₆, window W_i7In at least partly data meet predetermined Condition C₇, window W_i8In at least partly data meet predetermined condition C₈, window W_i9In at least portion Divided data meets predetermined condition C₉, window W_i10In at least partly data meet predetermined condition C₁₀With Window W_i11In at least partly data meet predetermined condition C₁₁Time, the most current potential cut-point k_iFor Data flow point cutpoint.When data at least part of in any one window in 11 windows are unsatisfactory for correspondence Predetermined condition time, as shown in Figure 6, W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for Corresponding predetermined condition C₅, then from a p_i5Along the data flow point cutpoint search direction N number of word of jump Joint, the most N number of byte is not more than ‖ B₅‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), at Fig. 6 In shown embodiment, N number of byte of jumping is not more than 179 bytes, in the present embodiment, N=11, obtains next potential cut-point, for potential cut-point k_iDifference, here by new Potential cut-point is expressed as k_j.According in the embodiment shown in Fig. 5 at duplicate removal server 103 The upper rule preset, for potential cut-point k_jThe point determined is 11, respectively p_j1、p_j2、p_j3、 p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、p_j10And p_j11, determine a p_j1、p_j2、p_j3、p_j4、p_j5、 p_j6、p_j7、p_j8、p_j9、p_j10And p_j11Corresponding window is respectively W_j1[p_j1-169,p_j1]、W_j2[p_j2 -169,p_j2]、W_j3[p_j3-169,p_j3]、W_j4[p_j4-169,p_j4]、W_j5[p_j5-169,p_j5]、W_j6[p_j6 -169,p_j6]、W_j7[p_j7-169,p_j7]、W_j8[p_j8-169,p_j8]、W_j9[p_j9-169,p_j9]、 W_j10[p_j10-169,p_j10] and W_j11[p_j11-169,p_j11].Wherein, p_jxWith potential cut-point k_jIt Spacing d_xIndividual byte, concrete, p_j1With k_jSpacing 0 byte, p_j2With k_j1 word of spacing Joint, p_j3With k_jSpacing 2 bytes, p_j4With k_jSpacing 3 bytes, p_j5With k_j4 words of spacing Joint, p_j6With k_jSpacing 5 bytes, p_j7With k_jSpacing 6 bytes, p_j8With k_j7 words of spacing Joint, p_j9With k_jSpacing 8 bytes, p_j10With k_j9 bytes of spacing, p_j11With k_jSpacing 10 Byte, and p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、p_j10And p_j11Relatively In potential cut-point k_jIt is respectively positioned on data flow point cutpoint and searches opposite direction.Embodiment party as shown in Figure 6 In formula, when for potential cut-point k_jThe 11st the window W determined_j11[p_j11-169,p_j11], protecting Demonstrate,prove potential cut-point k_iWith potential cut-point k_jBetween scope all within determination range, then exist In present embodiment, it is necessary to assure window W_j11[p_j11-169,p_j11] left margin and W_i5[p_i5 -169,p_i5] right margin p_i5Overlap or be positioned at W_i5[p_i5-169,p_i5Within the scope of], wherein, institute State potential cut-point k_jThe point p determined_j11It is according to described rule, for described potential cut-point k_j M the point determined is according to the point of sequence first in the sequence of data stream search direction acquisition.Therefore, In this restriction, work as W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for predetermined condition C₅, From p_i5The distance jumped along data flow point cutpoint search direction is no more than ‖ B₅‖+max_x (‖A_x‖+‖(k_i-p_ix) ‖), wherein, M=11,11*U are not more than max_x(‖A_x‖+‖(k_i -p_ix) ‖), therefore, from p_i5The distance jumped along data flow point cutpoint search direction is little In 179.Judge W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁、 Judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂, judge W_j3 [p_j3-169,p_j3In], whether at least part of data meet predetermined condition C₃, judge W_j4[p_j4-169, p_j4In], whether at least part of data meet predetermined condition C₄, judge W_j5[p_j5-169,p_j5In] extremely Whether small part data meet predetermined condition C₅, judge W_j6[p_j6-169,p_j6In] at least partly Whether data meet predetermined condition C₆, judge W_j7[p_j7-169,p_j7In], at least part of data are No meet predetermined condition C₇, judge W_j8[p_j8-169,p_j8In], whether at least part of data meet Predetermined condition C₈, judge W_j9[p_j9-169,p_j9In], whether at least part of data meet predetermined bar Part C₉, judge W_j10[p_j10-169,p_j10In], whether at least part of data meet predetermined condition C₁₀ With judge W_j11[p_j11-169,p_j11In], whether at least part of data meet predetermined condition C₁₁.When The most in embodiments of the present invention, it is judged that potential cut-point k_aWhether it is also to abide by during data flow point cutpoint Follow this rule, implement and no longer describe, be referred to judge potential cut-point k_iDescription. When judging window W_j1In at least partly data meet predetermined condition C₁, window W_j2In at least partly Data meet predetermined condition C₂, window W_j3In at least partly data meet predetermined condition C₃, window Mouth W_j4In at least partly data meet predetermined condition C₄, window W_j5In at least partly data meet Predetermined condition C₅, window W_j6In at least partly data meet predetermined condition C₆, window W_j7In extremely Small part data meet predetermined condition C₇, window W_j8In at least partly data meet predetermined condition C₈, window W_j9In at least partly data meet predetermined condition C₉, window W_j10In at least partly count According to meeting predetermined condition C₁₀With window W_j11In at least partly data meet predetermined condition C₁₁Time, then Current potential cut-point k_jFor data flow point cutpoint, k_jWith k_aBetween data constitute 1 data Block, simultaneously according to k_aIdentical mode skips minimum piecemeal size 4KB, it is thus achieved that next latent At cut-point, and according to the rule preset on duplicate removal server 103, it is judged that next potential Whether cut-point is data flow point cutpoints.When judging potential cut-point k_jIt not data flow point cutpoints Time, according to k_iJump 11 bytes of identical mode obtain next potential cut-points, and press Impinge upon the rule preset on duplicate removal server 103 and said method judges next potential cut-point Whether it is data flow point cutpoints.When the maximum data block exceeding setting does not the most find data stream During cut-point, then from the end position of maximum data block as force-splitting point.

In the embodiment shown in Fig. 5, according to the rule preset on duplicate removal server 103, From judging W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁Start, when Judge W_i1[p_i1-169,p_i1In], at least part of data meet predetermined condition C₁, judge W_i2[p_i2 -169,p_i2In], at least part of data meet predetermined condition C₂, judge W_i3[p_i3-169,p_i3In] At least partly data meet predetermined condition C₃With judge W_i4[p_i4-169,p_i4At least partly count in] According to meeting predetermined condition C₄, it is judged that W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for making a reservation for Condition C₅Time, from a p_i5Jump 10 bytes along data flow point cutpoint search direction, the The end position of 10 bytes obtains new potential cut-point, for distinguishing with other potential cut-points, Here shown as k_g, according to the rule preset on duplicate removal server 103, for potential cut-point k_g Determine 11 some p_gx, x is respectively 1 to 11 continuous print natural number, respectively p_g1、p_g2、p_g3、 p_g4、p_g5、p_g6、p_g7、p_g8、p_g9、p_g10And p_g11, determine a p_g1、p_g2、p_g3、p_g4、 p_g5、p_g6、p_g7、p_g8、p_g9、p_g10And p_g11Corresponding window is respectively W_g1[p_g1-169,p_g1]、 W_g2[p_g2-169,p_g2]、W_g3[p_g3-169,p_g3]、W_g4[p_g4-169,p_g4]、W_g5[p_g5-169, p_g5]、W_g6[p_g6-169,p_g6]、W_g7[p_g7-169,p_g7]、W_g8[p_g8-169,p_g8]、W_g9[p_g9 -169,p_g9]、W_g10[p_g10-169,p_g10] and W_g11[p_g11-169,p_g11].Wherein, p_gxWith latent At cut-point k_gSpacing d_xIndividual byte, concrete, p_g1With k_gSpacing 0 byte, p_g2With k_gSpacing 1 byte, p_g3With k_gSpacing 2 bytes, p_g4With k_gSpacing 3 bytes, p_g5With k_gSpacing 4 bytes, p_g6With k_gSpacing 5 bytes, p_g7With k_gSpacing 6 bytes, p_g8With k_gSpacing 7 bytes, p_g9With k_gSpacing 8 bytes, p_g10With k_g9 bytes of spacing, p_g11With k_g10 bytes of spacing, and p_g2、p_g3、p_g4、p_g5、p_g6、p_g7、p_g8、p_g9、p_g10 And p_g11Relative to potential cut-point k_gIt is respectively positioned on data flow point cutpoint and searches opposite direction.Judge W_g1 [p_g1-169,p_g1In], whether at least part of data meet predetermined condition C₁, judge W_g2[p_g2-169, p_g2In], whether at least part of data meet predetermined condition C₂, judge W_g3[p_g3-169,p_g3In] extremely Whether small part data meet predetermined condition C₃, judge W_g4[p_g4-169,p_g4In] at least partly Whether data meet predetermined condition C₄, judge W_g5[p_g5-169,p_g5In], at least part of data are No meet predetermined condition C₅, judge W_g6[p_g6-169,p_g6In], whether at least part of data meet Predetermined condition C₆, judge W_g7[p_g7-169,p_g7In], whether at least part of data meet predetermined bar Part C₇, judge W_g8[p_g8-169,p_g8In], whether at least part of data meet predetermined condition C₈、 Judge W_g9[p_g9-169,p_g9In], whether at least part of data meet predetermined condition C₉, judge W_g10[p_g10-169,p_g10In], whether at least part of data meet predetermined condition C₁₀With judge W_g11 [p_g11-169,p_g11In], whether at least part of data meet predetermined condition C₁₁.Therefore, potential point Cutpoint k_gCorresponding some p_g11With potential cut-point k_iCorresponding some p_i5Overlap, and put p_g11Right The window W answered_g11[p_g11-169,p_g11] and some p_i5Corresponding window W_i5[p_i5-169,p_i5] overlap, And C₅=C₁₁, therefore, to as potential cut-point k_i, when judging W_i5[p_i5-169,p_i5In] at least Part data are unsatisfactory for predetermined condition C₅Time, from a p_i5Along data flow point cutpoint search direction Jump 10 bytes, it is thus achieved that potential cut-point k_gStill do not meet as data flow point cutpoint Condition.Therefore, if from a p_i5Along 10 the byte meetings of jump of data flow point cutpoint search direction There is double counting, from a p_i5Permissible along 11 bytes of data flow point cutpoint search direction jump Reduce double counting, in hgher efficiency.Therefore improve the speed searching data flow point cutpoint.When Preset rules midpoint p_xCorresponding window W_x[p_x-A_x,p_x+B_xIn], at least part of data meet pre- Fixed condition C_xProbability when being 1/2, i other words perform jump with the probability of 1/2, the most at most may be used With 179 bytes of jumping.

In the present embodiment, pre-defined rule is: determine 11 some p for potential cut-point k_x, point p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] corresponding predetermined bar Part C_x, x is respectively 1 to 11 continuous print natural numbers, wherein, puts p_xCorresponding window W_x[p_x-A_x, p_x+B_xThe probability that in], at least part of data meet predetermined condition is 1/2, by the two factor P (n) can be calculated.And A₁=A₂=A₃=A₄=A₅=A₆=A₇=A₈=A₉=A₁₀=A₁₁=169, B₁=B₂ =B₃=B₄=B₅=B₆=B₇=B₈=B₉=B₁₀=B₁₁=0, and C₁=C₂=C₃=C₄=C₅=C₆=C₇= C₈=C₉=C₁₀=C₁₁, wherein, p_xSpacing d with potential cut-point k_xIndividual byte, specifically , p₁With 0 byte of spacing of potential cut-point k, p₂With 1 byte of spacing of k, p₃ With 2 bytes of spacing of k, p₄With 3 bytes of spacing of k, p₅Spacing 4 with k Byte, p₆With 5 bytes of spacing of k, p₇With 6 bytes of spacing of k, p₈And between k 7 bytes of distance, p₉With 8 bytes of spacing of k, p₁₀With 9 bytes of spacing of k, p₁₁ With 10 bytes of spacing of k, and p₂、p₃、p₄、p₅、p₆、p₇、p₈、p₉、p₁₀ And p₁₁It is respectively positioned on data flow point cutpoint relative to potential cut-point k and searches opposite direction.The most whether There are at least part of data in each window in continuous 11 some correspondence windows and be satisfied by pre- Fixed condition C_xJust determine whether potential cut-point k is data flow point cutpoints.From data stream start bit Put/a upper data flow point cutpoint jumps after minimum 4096 bytes of piecemeal length, to data flow point Cutpoint searches 10 bytes of opposite direction rollback, finds the 4086th point, the most there is not number According to flow point cutpoint, so P (4086)=1, the like, P (4087)=1 ... P (4095) =1.At the 4096th point, i.e. at minimum piecemeal size, with the probability of (1/2) ^11 this In the window that 11 points are corresponding, in each window, at least partly data meet predetermined condition C_x, because of There is data flow point cutpoint with the probability of (1/2) ^11 in this, with the probability of 1-(1/2) ^11 not There is data flow point cutpoint, so P (11)=1-(1/2) ^11.

At n-th, 12 kinds of situations can be divided into carry out recursion P (n).

In 1: the n-th corresponding window of situation, at least part of data are unsatisfactory for the probability of 1/2 Predetermined condition, now n-1 point before n-th does not exist with the probability of P (n-1) continuously Window corresponding to 11 points in each window at least partly data meet predetermined bar respectively Part, therefore P (n) comprises 1/2*P (n-1).In n-th corresponding window at least partly There are 11 points of continuous print in n-1 the point that data are unsatisfactory for before predetermined condition, and at n-th In corresponding window, in each window, at least partly data meet the situation of predetermined condition respectively Unrelated with P (n).

In 2: the n-th corresponding windows of situation, at least part of data meet pre-with the probability of 1/2 Fixed condition, in the window that (n-1)th point is corresponding, at least partly data are unsatisfactory for pre-with the probability of 1/2 Fixed condition, now (n-1)th some n-2 point above does not exist with the probability of P (n-2) continuously Window corresponding to 11 points in each window at least partly data meet predetermined bar respectively Part, therefore P (n) comprises 1/2*1/2*P (n-2).At least portion in n-th corresponding window Divided data meets predetermined condition, and in the window that (n-1)th point is corresponding, at least partly data are unsatisfactory for Predetermined condition, and there is the window that 11 points of continuous print are corresponding in n-2 the point that (n-1)th point is above In Kou, in each window, at least part of data meet situation and P (n) nothing of predetermined condition respectively Close.

According to foregoing description, the window that situation 11: the n-th to n-9 point is corresponding at least partly counts The probability of (1/2) ^10 meets predetermined condition according to this, in the (n-1)th 0 windows that point is corresponding at least Part data are unsatisfactory for predetermined condition with the probability of 1/2, now the (n-1)th 0 some n-11 above Each window in the window that 11 points of continuous print are corresponding is there is not in individual point with the probability of P (n-11) In Kou, at least part of data meet predetermined condition respectively, and therefore P (n) comprises (1/2) ^10*1/2*P (n-11).In the window of the n-th to n-9 some correspondence, at least partly data are satisfied by predetermined condition, In the (n-1)th 0 windows that point is corresponding, at least partly data are unsatisfactory for predetermined condition, and the (n-1)th 0 Individual point above n-11 point exists in the window that 11 points of continuous print are corresponding in each window The situation that at least part of data meet predetermined condition respectively is unrelated with P (n).

In the window that 12: the n-th to n-10 point of situation is corresponding, at least part of data are with (1/2) ^11 Probability meet predetermined condition, this situation is unrelated with P (n).

Therefore, P (n)=1/2*P (n-1)+(1/2) ^2*P (n-2)+...+(1/2) ^11*P(n-11).Another kind of preset rules: determine 24 some p for potential cut-point k_x, point p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] corresponding predetermined bar Part C_x, x is respectively 1 to 24 continuous print natural numbers, wherein, puts p_xCorresponding window W_x[p_x-A_x, p_x+B_xIn], at least part of data meet predetermined condition C_xProbability be 3/4, by the two because of Element can calculate P (n).And A₁=A₂=A₃=A₄=A₅=A₆=A₇=A₈=A₉=A₁₀=A₁₁=169, B₁ =B₂=B₃=B₄=B₅=B₆=B₇=B₈=B₉=B₁₀=B₁₁=0, and C₁=C₂=C₃=C₄=C₅=C₆= C₇=C₈=C₉=...=C₂₂=C₂₃=C₂₄, wherein, p_xSpacing d with potential cut-point k_xIndividual Byte, concrete, p₁With 0 byte of spacing of potential cut-point k, p₂Spacing with k 1 byte, p₃With 2 bytes of spacing of k, p₄With 3 bytes of spacing of k, p₅With k it 4 bytes of spacing, p₆With 5 bytes of spacing of k, p₇With 6 bytes of spacing of k, p₈With 7 bytes of spacing of k, p₉With 8 bytes of spacing of k ... p₂₂Spacing with k 21 bytes, p₂₃With 22 bytes of spacing of k, p₂₄With 23 bytes of spacing of k, and p₂、p₃、p₄、p₅、p₆、p₇、p₈、p₉…p₂₂、p₂₃And p₂₄Relative to potential segmentation Point k is respectively positioned on data flow point cutpoint and searches opposite direction.The most whether there are continuous 24 some correspondences In each window in window, at least part of data are satisfied by predetermined condition C_xJust determine potential Whether cut-point k is data flow point cutpoints, can be calculated by equation below:

P (4073)=1, P (4074)=1 ... P (, 4095)=1, P (4096)=1- (3/4) ^24,

P (n)=1/4*P (n-1)+1/4* (3/4) * P (n-2)+...+1/4* (3/4) ^23*P(n-24)。

Through calculating, P (5*1024)=0.78, P (11*1024)=0.17, P (12*1024)=0.13, I.e. from data stream original position/a data flow point cutpoint find after 12KB the probability with 13% Do not find data flow point cutpoint yet, force to split.By this probability, try to achieve data stream The density function of cut-point, through integration try to achieve about averagely from data stream original position/on One data flow point cutpoint finds data flow point cutpoint when searching 7.6KB, i.e. average mark block length is big It is about 7.6KB.In the window corresponding with 11 points of continuous print at least part of data with 1/2 probability Meeting predetermined condition different, tradition CDC algorithm uses a window to meet with the probability of 1/2^12 During condition, the effect of average mark block length 7.6KB can be reached.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, at the embodiment shown in Fig. 7 In, duplicate removal server 103 is preset with rule, described rule is: true for potential cut-point k Fixed 11 some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] Corresponding predetermined condition C_x, x is respectively 1 to 11 continuous print natural numbers, wherein, puts p_xCorresponding Window W_x[p_x-A_x,p_x+B_xIn], at least part of data meet predetermined condition C_xProbability be 1/2, and A₁=A₂=A₃=A₄=A₅=A₆=A₇=A₈=A₉=A₁₀=A₁₁=169, B₁=B₂=B₃=B₄=B₅ =B₆=B₇=B₈=B₉=B₁₀=B₁₁=0, and C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀= C₁₁, wherein, p_xSpacing d with potential cut-point k_xIndividual byte, concrete, p₁With potential 2 bytes of the spacing of cut-point k, p₂With 3 bytes of spacing of k, p₃Spacing with k 4 bytes, p₄With 5 bytes of spacing of k, p₅With 6 bytes of spacing of k, p₆With k it 7 bytes of spacing, p₇With 8 bytes of spacing of k, p₈With 9 bytes of spacing of k, p₉With 10 bytes of spacing of k, p₁₀With 1 byte of spacing of k, p₁₁Spacing with k 0 byte, and p₁、p₂、p₃、p₄、p₅、p₆、p₇、p₈、p₉And p₁₀Relative to latent It is respectively positioned on data flow point cutpoint at cut-point k and searches opposite direction.k_aFor data flow point cutpoint, figure The cutpoint search direction of data flow point shown in 7 is from left to right, from data flow point cutpoint k_aSkip After minimum data block 4KB, at minimum data block 4KB end position as next potential segmentation Point k_i, for potential cut-point k_iDetermine a p_ix, in the present embodiment, according at duplicate removal server The rule preset on 103, x is respectively 1 to 11 continuous print natural numbers.The embodiment party shown in Fig. 7 In formula, according to pre-defined rule, for potential cut-point k_iThe point determined is 11, respectively p_i1、 p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11, put p_i1、p_i2、p_i3、 p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11Corresponding window is respectively W_i1[p_i1-169, p_i1]、W_i2[p_i2-169,p_i2]、W_i3[p_i3-169,p_i3]、W_i4[p_i4-169,p_i4]、W_i5[p_i5-169, p_i5]、W_i6[p_i6-169,p_i6]、W_i7[p_i7-169,p_i7]、W_i8[p_i8-169,p_i8]、W_i9[p_i9-169, p_i9]、W_i10[p_i10-169,p_i10] and W_i11[p_i11-169,p_i11].Wherein, some p_ixWith potential cut-point k_iSpacing d_ixIndividual byte, concrete, p_i1With k_iSpacing 2 bytes, p_i2With k_iSpacing 3 Individual byte, p_i3With k_iSpacing 4 bytes, p_i4With k_iSpacing 5 bytes, p_i5With k_iSpacing 6 Byte, p_i6With k_iSpacing 7 bytes, p_i7With k_iSpacing 8 bytes, p_i8With k_i9 words of spacing Joint, p_i9With k_iSpacing 10 bytes, p_i10With k_i1 byte of spacing, p_i11With k_i0 word of spacing Joint, and p_i1、p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、p_i9And p_i10Relative to latent At cut-point k_iIt is respectively positioned on data flow point cutpoint and searches opposite direction.Judge W_i1[p_i1-169,p_i1In] At least partly whether data meet predetermined condition C₁, judge W_i2[p_i2-169,p_i2In] at least partly Whether data meet predetermined condition C₂, judge W_i3[p_i3-169,p_i3In], whether at least part of data Meet predetermined condition C₃, judge W_i4[p_i4-169,p_i4In], whether at least part of data meet predetermined Condition C₄, judge W_i5[p_i5-169,p_i5In], whether at least part of data meet predetermined condition C₅、 Judge W_i6[p_i6-169,p_i6In], whether at least part of data meet predetermined condition C₆, judge W_i7 [p_i7-169,p_i7In], whether at least part of data meet predetermined condition C₇, judge W_i8[p_i8-169, p_i8In], whether at least part of data meet predetermined condition C₈, judge W_i9[p_i9-169,p_i9In] extremely Whether small part data meet predetermined condition C₉, judge W_i10[p_i10-169,p_i10At least partly count in] According to whether meeting predetermined condition C₁₀With judge W_i11[p_i11-169,p_i11In], whether at least part of data Meet predetermined condition C₁₁.When judging window W_i1In at least partly data meet predetermined condition C₁、 Window W_i2In at least partly data meet predetermined condition C₂, window W_i3In at least partly data full Foot predetermined condition C₃, window W_i4In at least partly data meet predetermined condition C₄, window W_i5In At least partly data meet predetermined condition C₅, window W_i6In at least partly data meet predetermined bar Part C₆, window W_i7In at least partly data meet predetermined condition C₇, window W_i8In at least partly Data meet predetermined condition C₈, window W_i9In at least partly data meet predetermined condition C₉, window Mouth W_i10In at least partly data meet predetermined condition C₁₀With window W_i11In at least partly data full Foot predetermined condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint.When 11 windows In time at least partly data are unsatisfactory for the predetermined condition of correspondence in any one window, as shown in Figure 8, W_i3[p_i3-169,p_i3In], at least part of data are unsatisfactory for predetermined condition C₃, put p_i3Along data stream It is described as a example by cut-point search direction 11 bytes of jump.As shown in Figure 8, when judging W₃No When meeting predetermined condition, with p₃For starting point, along data flow point cutpoint search direction jump N Individual byte, the most N number of byte is not more than ‖ B₃‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), In embodiment shown in Fig. 6, N number of byte of jumping, it is specially and is not more than 179 bytes, at this In embodiment, N=11, at the end position of the 11st byte, it is thus achieved that next potential segmentation Point, for potential cut-point k_iDifference, is expressed as k by new potential cut-point here_j, according to The rule preset on duplicate removal server 103, for potential cut-point k_jThe point determined is 11, It is respectively p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、p_j10And p_j11, determine a little p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、p_j10And p_j11Corresponding window is respectively For W_j1[p_j1-169,p_j1]、W_j2[p_j2-169,p_j2]、W_j3[p_j3-169,p_j3]、W_j4[p_j4-169, p_j4]、W_j5[p_j5-169,p_j5]、W_j6[p_j6-169,p_j6]、W_j7[p_j7-169,p_j7]、W_j8[p_j8 -169,p_j8]、W_j9[p_j9-169,p_j9]、W_j10[p_j10-169,p_j10] and W_j11[p_j11-169,p_j11]。 Wherein, p_jxWith potential cut-point k_jSpacing d_xIndividual byte, concrete, p_j1With k_jSpacing 2 Individual byte, p_j2With k_jSpacing 3 bytes, p_j3With k_jSpacing 4 bytes, p_j4With k_jSpacing 5 Individual byte, p_j5With k_jSpacing 6 bytes, p_j6With k_jSpacing 7 bytes, p_j7With k_jSpacing 8 Individual byte, p_j8With k_jSpacing 9 bytes, p_j9With k_jSpacing 10 bytes, p_j10With k_jSpacing 1 Individual byte, p_j11With k_j0 byte of spacing, and p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、 p_j8、p_j9And p_j10Relative to potential cut-point k_jIt is respectively positioned on data flow point cutpoint and searches opposite direction. Judge W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁, judge W_j2 [p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂, judge W_j3[p_j3-169, p_j3In], whether at least part of data meet predetermined condition C₃, judge W_j4[p_j4-169,p_j4In] extremely Whether small part data meet predetermined condition C₄, judge W_j5[p_j5-169,p_j5At least partly count in] According to whether meeting predetermined condition C₅, judge W_j6[p_j6-169,p_j6In], whether at least part of data Meet predetermined condition C₆, judge W_j7[p_j7-169,p_j7In], whether at least part of data meet pre- Fixed condition C₇, judge W_j8[p_j8-169,p_j8In], whether at least part of data meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In], whether at least part of data meet predetermined condition C₉, sentence Disconnected W_j10[p_j10-169,p_j10In], whether at least part of data meet predetermined condition C₁₀And judgement W_j11[p_j11-169,p_j11In], whether at least part of data meet predetermined condition C₁₁.Certainly at this In inventive embodiments, it is judged that potential cut-point k_aWhen whether being data flow point cutpoint former also in compliance with this Then, implement and no longer describe, be referred to judge potential cut-point k_iDescription.Work as judgement Window W_j1In at least partly data meet predetermined condition C₁, window W_j2In at least partly data full Foot predetermined condition C₂, window W_j3In at least partly data meet predetermined condition C₃, window W_j4In At least partly data meet predetermined condition C₄, window W_j5In at least partly data meet predetermined bar Part C₅, window W_j6In at least partly data meet predetermined condition C₆, window W_j7In at least partly Data meet predetermined condition C₇, window W_j8In at least partly data meet predetermined condition C₈, window Mouth W_j9In at least partly data meet predetermined condition C₉, window W_j10In at least partly data meet Predetermined condition C₁₀With window W_j11In at least partly data meet predetermined condition C₁₁Time, the most currently dive At cut-point k_jFor data flow point cutpoint, k_jWith k_aBetween data constitute 1 data block, with Time according to k_aIdentical mode skips minimum piecemeal size 4KB, it is thus achieved that next potential segmentation Point, and according to the rule preset on duplicate removal server 103, it is judged that next potential cut-point Whether it is data flow point cutpoints.When judging potential cut-point k_jWhen not being data flow point cutpoint, press According to k_iJump 11 bytes of identical mode obtain next potential cut-points, and according to going The rule preset on weight server 103 and said method judge that whether next potential cut-point is Data flow point cutpoint.When the maximum data block exceeding setting does not the most find data flow point cutpoint Time, then from the end position of maximum data block as force-splitting point.Certainly the enforcement of the method By maximum data block length and the size constraint of the file constituting this data stream, do not repeat them here.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, at the embodiment shown in Fig. 9 In, duplicate removal server 103 is preset with rule, described rule is: true for potential cut-point k Fixed 11 some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] Corresponding predetermined condition C_x, wherein A₁=A₂=A₃=A₄=A₅=A₆=A₇=A₈=A₉=A₁₀=A₁₁=169, B₁=B₂=B₃=B₄=B₅=B₆=B₇=B₈=B₉=B₁₀=B₁₁=0, and C₁=C₂=C₃=C₄=C₅=C₆= C₇=C₈=C₉=C₁₀=C₁₁.Wherein, p_xSpacing d with potential cut-point k_xIndividual byte, tool Body, p₁With 3 bytes of spacing of potential cut-point k, p₂With 2 bytes of spacing of k, p₃With 1 byte of spacing of k, p₄With 0 byte of spacing of k, p₅Spacing 1 with k Individual byte, p₆With 2 bytes of spacing of k, p₇With 3 bytes of spacing of k, p₈With k it 4 bytes of spacing, p₉With 5 bytes of spacing of k, p₁₀With 6 bytes of spacing of k, p₁₁With 7 bytes of spacing of k, and p₅、p₆、p₇、p₈、p₉、p₁₀And p₁₁Relative to Potential cut-point k is respectively positioned on data flow point cutpoint and searches opposite direction, p₁、p₂And p₃Relative to latent It is respectively positioned on data flow point cutpoint search direction at cut-point k.k_aFor data flow point cutpoint, Fig. 9 Shown in data flow point cutpoint search direction be from left to right, from data flow point cutpoint k_aSkip After small data block 4KB, minimum data block 4KB end position is as next potential cut-point k_i, For potential cut-point k_iDetermine a p_ix, in the present embodiment, according on duplicate removal server 103 The rule preset, x is respectively 1 to 11 continuous print natural numbers.In the embodiment shown in Fig. 9, For potential cut-point k_iThe point determined is 11, respectively p_i1、p_i2、p_i3、p_i4、p_i5、p_i6、 p_i7、p_i8、p_i9、p_i10And p_i11, put p_i1、p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、 p_i9、p_i10And p_i11Corresponding window is respectively W_i1[p_i1-169,p_i1]、W_i2[p_i2-169,p_i2]、W_i3 [p_i3-169,p_i3]、W_i4[p_i4-169,p_i4]、W_i5[p_i5-169,p_i5]、W_i6[p_i6-169,p_i6]、W_i7 [p_i7-169,p_i7]、W_i8[p_i8-169,p_i8]、W_i9[p_i9-169,p_i9]、W_i10[p_i10-169,p_i10] and W_i11 [p_i11-169,p_i11].Wherein, p_ixWith potential cut-point k_iSpacing d_xIndividual byte, concrete, p_i1With k_iSpacing 3 bytes, p_i2With k_iSpacing 2 bytes, p_i3With k_iSpacing 1 byte, p_i4 With k_iSpacing 0 byte, p_i5With k_iSpacing 1 byte, p_i6With k_iSpacing 2 bytes, p_i7With k_iSpacing 3 bytes, p_i8With k_iSpacing 4 bytes, p_i9With k_iSpacing 5 bytes, p_i10With k_i 6 bytes of spacing, p_i11With k_i7 bytes of spacing, and p_i5、p_i6、p_i7、p_i8、p_i9、p_i10 And p_i11Relative to potential cut-point k_iIt is respectively positioned on data flow point cutpoint and searches opposite direction, p_i1、p_i2With p_i3Relative to potential cut-point k_iIt is respectively positioned on data flow point cutpoint search direction.Judge W_i1[p_i1 -169,p_i1In], whether at least part of data meet predetermined condition C₁, judge W_i2[p_i2-169,p_i2] In at least partly data whether meet predetermined condition C₂, judge W_i3[p_i3-169,p_i3At least portion in] Whether divided data meets predetermined condition C₃, judge W_i4[p_i4-169,p_i4In], at least part of data are No meet predetermined condition C₄, judge W_i5[p_i5-169,p_i5In], whether at least part of data meet pre- Fixed condition C₅, judge W_i6[p_i6-169,p_i6In], whether at least part of data meet predetermined condition C₆、 Judge W_i7[p_i7-169,p_i7In], whether at least part of data meet predetermined condition C₇, judge W_i8 [p_i8-169,p_i8In], whether at least part of data meet predetermined condition C₈, judge W_i9[p_i9-169, p_i9In], whether at least part of data meet predetermined condition C₉, judge W_i10[p_i10-169,p_i10In] extremely Whether small part data meet predetermined condition C₁₀With judge W_i11[p_i11-169,p_i11In] at least partly Whether data meet predetermined condition C₁₁.When judging window W_i1In at least partly data meet predetermined Condition C₁, window W_i2In at least partly data meet predetermined condition C₂, window W_i3In at least portion Divided data meets predetermined condition C₃, window W_i4In at least partly data meet predetermined condition C₄、 Window W_i5In at least partly data meet predetermined condition C₅, window W_i6In at least partly data full Foot predetermined condition C₆, window W_i7In at least partly data meet predetermined condition C₇, window W_i8In At least partly data meet predetermined condition C₈, window W_i9In at least partly data meet predetermined bar Part C₉, window W_i10In at least partly data meet predetermined condition C₁₀With window W_i11In at least portion Divided data meets predetermined condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint.When When in 11 windows, in any one window, at least part of data are unsatisfactory for the predetermined condition of correspondence, as Shown in Figure 10, W_i7[p_i7-169,p_i7In], at least part of data are unsatisfactory for the predetermined condition of correspondence, Then from a p_i7Along the data flow point cutpoint search direction N number of byte of jump, the most N number of byte is not More than ‖ B₄‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), in the embodiment shown in Figure 10, Jump N number of byte, be specially and be not more than 179 bytes, in the present embodiment, specifically take N=8, Obtain new potential cut-point, for potential cut-point k_iDifference, here by new potential segmentation Point is expressed as k_j, according to the rule preset on duplicate removal server 103 in the embodiment shown in Fig. 9 Then, for potential cut-point k_jThe point determined is 11, respectively p_j1、p_j2、p_j3、p_j4、p_j5、 p_j6、p_j7、p_j8、p_j9、p_j10And p_j11, determine a p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、 p_j8、p_j9、p_j10And p_j11Corresponding window is respectively W_j1[p_j1-169,p_j1]、W_j2[p_j2-169, p_j2]、W_j3[p_j3-169,p_j3]、W_j4[p_j4-169,p_j4]、W_j5[p_j5-169,p_j5]、W_j6[p_j6-169, p_j6]、W_j7[p_j7-169,p_j7]、W_j8[p_j8-169,p_j8]、W_j9[p_j9-169,p_j9]、W_j10[p_j10 -169,p_j10] and W_j11[p_j11-169,p_j11].Wherein, p_jxWith potential cut-point k_jSpacing d_xIndividual byte, concrete, p_j1With k_jSpacing 3 bytes, p_j2With k_jSpacing 2 bytes, p_j3 With k_jSpacing 1 byte, p_j4With k_jSpacing 0 byte, p_j5With k_jSpacing 1 byte, p_j6With k_jSpacing 2 bytes, p_j7With k_jSpacing 3 bytes, p_j8With k_jSpacing 4 bytes, p_j9With k_j Spacing 5 bytes, p_j10With k_j6 bytes of spacing, p_j11With k_j7 bytes of spacing, and p_j5、 p_j6、p_j7、p_j8、p_j9、p_j10And p_j11Relative to potential cut-point k_jIt is respectively positioned on the segmentation of data stream Point searches opposite direction, p_j1、p_j2And p_j3Relative to potential cut-point k_jIt is respectively positioned on the segmentation of data stream Point search direction.Judge W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁, judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂, judge W_j3[p_j3-169,p_j3In], whether at least part of data meet predetermined condition C₃, judge W_j4[p_j4 -169,p_j4In], whether at least part of data meet predetermined condition C₄, judge W_j5[p_j5-169,p_j5] In at least partly data whether meet predetermined condition C₅, judge W_j6[p_j6-169,p_j6In] at least Whether part data meet predetermined condition C₆, judge W_j7[p_j7-169,p_j7At least partly count in] According to whether meeting predetermined condition C₇, judge W_j8[p_j8-169,p_j8In], whether at least part of data Meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In], whether at least part of data meet pre- Fixed condition C₉, judge W_j10[p_j10-169,p_j10In], whether at least part of data meet predetermined bar Part C₁₀With judge W_j11[p_j11-169,p_j11In], whether at least part of data meet predetermined condition C₁₁。 The most in embodiments of the present invention, it is judged that potential cut-point k_aWhen whether being data flow point cutpoint also Follow this principle, implement and no longer describe, be referred to judge potential cut-point k_iDescription. When judging window W_j1In at least partly data meet predetermined condition C₁, window W_j2In at least partly Data meet predetermined condition C₂, window W_j3In at least partly data meet predetermined condition C₃, window Mouth W_j4In at least partly data meet predetermined condition C₄, window W_j5In at least partly data meet Predetermined condition C₅, window W_j6In at least partly data meet predetermined condition C₆, window W_j7In extremely Small part data meet predetermined condition C₇, window W_j8In at least partly data meet predetermined condition C₈, window W_j9In at least partly data meet predetermined condition C₉, window W_j10In at least partly count According to meeting predetermined condition C₁₀With window W_j11In at least partly data meet predetermined condition C₁₁Time, then Current potential cut-point k_jFor data flow point cutpoint, k_jWith k_aBetween data constitute 1 data Block, simultaneously according to k_aIdentical mode skips minimum piecemeal size 4KB, it is thus achieved that next latent At cut-point, and according to the rule preset on duplicate removal server 103, it is judged that next potential Whether cut-point is data flow point cutpoints.When judging potential cut-point k_jIt not data flow point cutpoints Time, according to k_iJump 8 bytes of identical mode obtain next potential cut-points, and press Impinge upon the rule preset on duplicate removal server 103 and said method judges next potential cut-point Whether it is data flow point cutpoints.When the maximum data block exceeding setting does not the most find data stream During cut-point, then from the end position of maximum data block as force-splitting point.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, the embodiment party shown in Figure 11 In formula, being preset with rule on duplicate removal server 103, described rule is: for potential cut-point k Determine 11 some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+ B_x] corresponding predetermined condition C_x, wherein A₁=A₂=A₃=A₄=A₅=A₆=A₇=A₈=A₉=A₁₀ =169, A₁₁=182, B₁=B₂=B₃=B₄=B₅=B₆=B₇=B₈=B₉=B₁₀=B₁₁=0, and C₁=C₂ =C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀≠C₁₁.Wherein, p_xSpacing with potential cut-point k From d_xIndividual byte, concrete, p₁With 0 byte of spacing of potential cut-point k, p₂With k it 1 byte of spacing, p₃With 2 bytes of spacing of k, p₄With 3 bytes of spacing of k, p₅With 4 bytes of spacing of k, p₆With 5 bytes of spacing of k, p₇Spacing 6 with k Individual byte, p₈With 7 bytes of spacing of k, p₉With 8 bytes of spacing of k, p₁₀With k it 1 byte of spacing, p₁₁With 3 bytes of spacing of k, and, p₂、p₃、p₄、p₅、 p₆、p₇、p₈And p₉It is respectively positioned on data flow point cutpoint relative to potential cut-point k and searches opposite direction, p₁₀And p₁₁It is respectively positioned on data flow point cutpoint search direction relative to potential cut-point k.k_aFor number According to flow point cutpoint, the cutpoint search direction of data flow point shown in Figure 11 is from left to right, from data Flow point cutpoint k_aAfter skipping minimum data block 4KB, minimum data block 4KB end position as under One potential cut-point k_i, for potential cut-point k_iDetermine a p_ix, in the present embodiment, according to The rule preset on duplicate removal server 103, x is respectively 1 to 11 continuous print natural numbers.At figure In embodiment shown in 11, for potential cut-point k_iThe point determined is 11, respectively p_i1、 p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11, put p_i1、p_i2、p_i3、 p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11Corresponding window is respectively W_i1[p_i1-169, p_i1]、W_i2[p_i2-169,p_i2]、W_i3[p_i3-169,p_i3]、W_i4[p_i4-169,p_i4]、W_i5[p_i5-169, p_i5]、W_i6[p_i6-169,p_i6]、W_i7[p_i7-169,p_i7]、W_i8[p_i8-169,p_i8]、W_i9[p_i9-169, p_i9]、W_i10[p_i10-169,p_i10] and W_i11[p_i11-182,p_i11].Wherein, p_ixWith potential cut-point k_iIt Spacing d_xIndividual byte, concrete, p_i1With k_iSpacing 0 byte, p_i2With k_i1 word of spacing Joint, p_i3With k_iSpacing 2 bytes, p_i4With k_iSpacing 3 bytes, p_i5With k_i4 bytes of spacing, p_i6With k_iSpacing 5 bytes, p_i7With k_iSpacing 6 bytes, p_i8With k_iSpacing 7 bytes, p_i9 With k_iSpacing 8 bytes, p_i10With k_i1 byte of spacing, p_i11With k_i3 bytes of spacing, and p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8And p_i9Relative to potential cut-point k_iIt is respectively positioned on number Opposite direction, p is searched according to flow point cutpoint_i10And p_i11Relative to potential cut-point k_iIt is respectively positioned on data stream Cut-point search direction.Judge W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined Condition C₁, judge W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂、 Judge W_i3[p_i3-169,p_i3In], whether at least part of data meet predetermined condition C₃, judge W_i4 [p_i4-169,p_i4In], whether at least part of data meet predetermined condition C₄, judge W_i5[p_i5-169, p_i5In], whether at least part of data meet predetermined condition C₅, judge W_i6[p_i6-169,p_i6In] extremely Whether small part data meet predetermined condition C₆, judge W_i7[p_i7-169,p_i7At least partly count in] According to whether meeting predetermined condition C₇, judge W_i8[p_i8-169,p_i8In], at least part of data are the fullest Foot predetermined condition C₈, judge W_i9[p_i9-169,p_i9In], whether at least part of data meet predetermined bar Part C₉, judge W_i10[p_i10-169,p_i10In], whether at least part of data meet predetermined condition C₁₀With Judge W_i11[p_i11-169,p_i11In], whether at least part of data meet predetermined condition C₁₁.Work as judgement Window W_i1In at least partly data meet predetermined condition C₁, window W_i2In at least partly data full Foot predetermined condition C₂, window W_i3In at least partly data meet predetermined condition C₃, window W_i4In At least partly data meet predetermined condition C₄, window W_i5In at least partly data meet predetermined bar Part C₅, window W_i6In at least partly data meet predetermined condition C₆, window W_i7In at least partly Data meet predetermined condition C₇, window W_i8In at least partly data meet predetermined condition C₈, window Mouth W_i9In at least partly data meet predetermined condition C₉, window W_i10In at least partly data meet Predetermined condition C₁₀With window W_i11In at least partly data meet predetermined condition C₁₁Time, the most currently dive At cut-point k_iFor data flow point cutpoint.When judging window W_i11In at least partly data be unsatisfactory for Predetermined condition C₁₁Time, then from potential cut-point k_iAlong data flow point cutpoint search direction jump 1 Individual byte, obtains new potential cut-point, for potential cut-point k_iDifference, here by new Potential cut-point is expressed as k_j.Work as W_i1、W_i2、W_i3、W_i4、W_i5、W_i6、W_i7、W_i8、W_i9 And W_i10In 10 windows, in any one window, at least part of data are unsatisfactory for the predetermined condition of correspondence Time, as shown in figure 12, W_i4[p_i4-169,p_i4], then from a p_i4Look into along data flow point cutpoint Looking for the N number of byte of direction jump, the most N number of byte is not more than ‖ B₄‖+max_x(‖ A_x‖+‖(k_i-p_ix) ‖), in the embodiment shown in Figure 12, N number of byte of jumping, specifically For no more than 179, in the present embodiment, specifically take N=9, obtain new potential cut-point, For with potential cut-point k_iDifference, is expressed as k by new potential cut-point here_j, according to Figure 11 The rule preset on duplicate removal server 103 in shown embodiment, for potential cut-point k_j The point determined is 11, respectively p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、p_j10 And p_j11, determine a p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、p_j10And p_j11Right The window answered is respectively W_j1[p_j1-169,p_j1]、W_j2[p_j2-169,p_j2]、W_j3[p_j3-169,p_j3]、 W_j4[p_j4-169,p_j4]、W_j5[p_j5-169,p_j5]、W_j6[p_j6-169,p_j6]、W_j7[p_j7-169, p_j7]、W_j8[p_j8-169,p_j8]、W_j9[p_j9-169,p_j9]、W_j10[p_j10-169,p_j10] and W_j11 [p_j8-182,p_j8].Wherein, p_jxWith potential cut-point k_jSpacing d_xIndividual byte, specifically , p_j1With k_jSpacing 0 byte, p_j2With k_jSpacing 1 byte, p_j3With k_j2 words of spacing Joint, p_j4With k_jSpacing 3 bytes, p_j5With k_jSpacing 4 bytes, p_j6With k_j5 words of spacing Joint, p_j7With k_jSpacing 6 bytes, p_j8With k_jSpacing 7 bytes, p_j9With k_j8 words of spacing Joint, p_j10With k_j1 byte of spacing, p_j11With k_j3 bytes of spacing, and p_j2、p_j3、p_j4、 p_j5、p_j6、p_j7、p_j8And p_j9Relative to potential cut-point k_jIt is respectively positioned on data flow point cutpoint to search Opposite direction, p_j10And p_j11Relative to potential cut-point k_jIt is respectively positioned on data flow point cutpoint search direction. Judge W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁, judge W_j2 [p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂, judge W_j3[p_j3-169, p_j3In], whether at least part of data meet predetermined condition C₃, judge W_j4[p_j4-169,p_j4In] extremely Whether small part data meet predetermined condition C₄, judge W_j5[p_j5-169,p_j5At least partly count in] According to whether meeting predetermined condition C₅, judge W_j6[p_j6-169,p_j6In], whether at least part of data Meet predetermined condition C₆, judge W_j7[p_j7-169,p_j7In], whether at least part of data meet pre- Fixed condition C₇, judge W_j8[p_j8-169,p_j8In], whether at least part of data meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In], whether at least part of data meet predetermined condition C₉, sentence Disconnected W_j10[p_j10-169,p_j10In], whether at least part of data meet predetermined condition C₁₀And judgement W_j11[p_j11-182,p_j11In], whether at least part of data meet predetermined condition C₁₁.Certainly at this In bright embodiment, it is judged that potential cut-point k_aAlso in compliance with this principle when whether being data flow point cutpoint, Implement and no longer describe, be referred to judge potential cut-point k_iDescription.When judging window W_j1In at least partly data meet predetermined condition C₁, window W_j2In at least partly data meet pre- Fixed condition C₂, window W_j3In at least partly data meet predetermined condition C₃, window W_j4In at least Part data meet predetermined condition C₄, window W_j5In at least partly data meet predetermined condition C₅、 Window W_j6In at least partly data meet predetermined condition C₆, window W_j7In at least partly data full Foot predetermined condition C₇, window W_j8In at least partly data meet predetermined condition C₈, window W_j9In At least partly data meet predetermined condition C₉, window W_j10In at least partly data meet predetermined bar Part C₁₀With window W_j11In at least partly data meet predetermined condition C₁₁Time, the most current potential segmentation Point k_jFor data flow point cutpoint, k_jWith k_aBetween data constitute 1 data block, simultaneously according to With k_aIdentical mode skips minimum piecemeal size 4KB, it is thus achieved that next potential cut-point, and According to the rule preset on duplicate removal server 103, it is judged that whether next potential cut-point is Data flow point cutpoint.When judging potential cut-point k_jWhen not being data flow point cutpoint, according to k_i Identical mode obtains next potential cut-point, and presets according on duplicate removal server 103 Rule and said method judge whether next potential cut-point is data flow point cutpoints.When super Cross the maximum data block set when the most not finding data flow point cutpoint, then from maximum data block End position as force-splitting point.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, the embodiment party shown in Figure 13 In formula, being preset with rule on duplicate removal server 103 is: determine 11 points for potential cut-point k p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] corresponding Predetermined condition C_x, x is respectively 1 to 11 continuous print natural numbers, wherein, puts p_xCorresponding window W_x [p_x-A_x,p_x+B_xThe probability that in], at least part of data meet predetermined condition is 1/2, and A₁=A₂ =A₃=A₄=A₅=A₆=A₇=A₈=A₉=A₁₀=A₁₁=169, B₁=B₂=B₃=B₄=B₅=B₆=B₇=B₈=B₉ =B₁₀=B₁₁=0, and C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁, wherein, p_x Spacing d with potential cut-point k_xIndividual byte, concrete, p₁And between potential cut-point k 0 byte of distance, p₂With 2 bytes of spacing of k, p₃With 4 bytes of spacing of k, p₄ With 6 bytes of spacing of k, p₅With 8 bytes of spacing of k, p₆Spacing 10 with k Individual byte, p₇With 12 bytes of spacing of k, p₈With 14 bytes of spacing of k, p₉With k 16 bytes of spacing, p₁₀With 18 bytes of spacing of k, p₁₁Spacing 20 with k Byte, and p₂、p₃、p₄、p₅、p₆、p₇、p₈、p₉、p₁₀And p₁₁Relative to potential Cut-point k is respectively positioned on data flow point cutpoint and searches opposite direction.k_aFor data flow point cutpoint, Figure 13 Shown in data flow point cutpoint search direction be from left to right, from data flow point cutpoint k_aSkip After small data block 4KB, at minimum data block 4KB end position as next potential cut-point k_i, for potential cut-point k_iDetermine a p_ix, in the present embodiment, according at duplicate removal server 103 The upper rule preset, x is respectively 1 to 11 continuous print natural numbers.At the embodiment shown in Figure 13 In, according to pre-defined rule, for potential cut-point k_iThe point determined is 11, respectively p_i1、p_i2、 p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11, put p_i1、p_i2、p_i3、p_i4、 p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11Corresponding window is respectively W_i1[p_i1-169,p_i1]、 W_i2[p_i2-169,p_i2]、W_i3[p_i3-169,p_i3]、W_i4[p_i4-169,p_i4]、W_i5[p_i5-169,p_i5]、 W_i6[p_i6-169,p_i6]、W_i7[p_i7-169,p_i7]、W_i8[p_i8-169,p_i8]、W_i9[p_i9-169,p_i9]、 W_i10[p_i10-169,p_i10] and W_i11[p_i11-169,p_i11].Wherein, p_ixWith potential cut-point k_iSpacing From d_xIndividual byte, concrete, p_i1With k_iSpacing 0 byte, p_i2With k_iSpacing 2 bytes, p_i3 With k_iSpacing 4 bytes, p_i4With k_iSpacing 6 bytes, p_i5With k_iSpacing 8 bytes, p_i6With k_iSpacing 10 bytes, p_i7With k_iSpacing 12 bytes, p_i8With k_iSpacing 14 bytes, p_i9With k_iSpacing 16 bytes, p_i10With k_i18 bytes of spacing, p_i11With k_i20 bytes of spacing, and And p_i2、p_i3、p_i4、p_i5、p_i6、p_i7、p_i8、p_i9、p_i10And p_i11Relative to potential segmentation Point k_iIt is respectively positioned on data flow point cutpoint and searches opposite direction.Judge W_i1[p_i1-169,p_i1In] at least partly Whether data meet predetermined condition C₁, judge W_i2[p_i2-169,p_i2In], whether at least part of data Meet predetermined condition C₂, judge W_i3[p_i3-169,p_i3In], whether at least part of data meet predetermined Condition C₃, judge W_i4[p_i4-169,p_i4In], whether at least part of data meet predetermined condition C₄、 Judge W_i5[p_i5-169,p_i5In], whether at least part of data meet predetermined condition C₅, judge W_i6 [p_i6-169,p_i6In], whether at least part of data meet predetermined condition C₆, judge W_i7[p_i7-169, p_i7In], whether at least part of data meet predetermined condition C₇, judge W_i8[p_i8-169,p_i8In] at least Whether part data meet predetermined condition C₈, judge W_i9[p_i9-169,p_i9At least part of data in] Whether meet predetermined condition C₉, judge W_i10[p_i10-169,p_i10In], whether at least part of data meet Predetermined condition C₁₀With judge W_i11[p_i11-169,p_i11In], whether at least part of data meet predetermined bar Part C₁₁.When judging window W_i1In at least partly data meet predetermined condition C₁, window W_i2In extremely Small part data meet predetermined condition C₂, window W_i3In at least partly data meet predetermined condition C₃, window W_i4In at least partly data meet predetermined condition C₄, window W_i5In at least partly count According to meeting predetermined condition C₅, window W_i6In at least partly data meet predetermined condition C₆, window W_i7In at least partly data meet predetermined condition C₇, window W_i8In at least partly data meet pre- Fixed condition C₈, window W_i9In at least partly data meet predetermined condition C₉, window W_i10In at least Part data meet predetermined condition C₁₀With window W_i11In at least partly data meet predetermined condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint.When any one window in 11 windows When in Kou, at least part of data are unsatisfactory for the predetermined condition of correspondence, as shown in figure 14, W_i4[p_i4 -169,p_i4In], at least part of data are unsatisfactory for predetermined condition C₄, then next potential segmentation is selected Point, for potential cut-point k_iDifference, here shown as k_j, k_jIt is positioned at k_iThe right, and k_j With k_i1 byte of spacing.As shown in figure 14, according to the rule preset on duplicate removal server 103, For potential cut-point k_jDetermine 11 points, respectively p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、 p_j8、p_j9、p_j10And p_j11, determine a p_j1、p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、 p_j10And p_j11Corresponding window is respectively W_j1[p_j1-169,p_j1]、W_j2[p_j2-169,p_j2]、W_j3 [p_j3-169,p_j3]、W_j4[p_j4-169,p_j4]、W_j5[p_j5-169,p_j5]、W_j6[p_j6-169,p_j6]、 W_j7[p_j7-169,p_j7]、W_j8[p_j8-169,p_j8]、W_j9[p_j9-169,p_j9]、W_j10[p_j10-169, p_j10] and W_j11[p_j11-169,p_j11], wherein, A₁=A₂=A₃=A₄=A₅=A₆=A₇=A₈=A₉=A₁₀= A₁₁=169, B₁=B₂=B₃=B₄=B₅=B₆=B₇=B₈=B₉=B₁₀=B₁₁=0, and C₁=C₂=C₃=C₄ =C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁.Wherein, p_jxWith potential cut-point k_jSpacing d_xIndividual Byte, concrete, p_j1With k_jSpacing 0 byte, p_j2With k_jSpacing 2 bytes, p_j3With k_j Spacing 4 bytes, p_j4With k_jSpacing 6 bytes, p_j5With k_jSpacing 8 bytes, p_j6With k_jBetween Away from 10 bytes, p_j7With k_jSpacing 12 bytes, p_j8With k_jSpacing 14 bytes, p_j9With k_j Spacing 16 bytes, p_j10With k_j18 bytes of spacing, p_j11With k_j20 bytes of spacing, and p_j2、p_j3、p_j4、p_j5、p_j6、p_j7、p_j8、p_j9、p_j10And p_j11Relative to potential cut-point k_jAll It is positioned at data flow point cutpoint and searches opposite direction.Judge W_j1[p_j1-169,p_j1At least part of data in] Whether meet predetermined condition C₁, judge W_j2[p_j2-169,p_j2In], whether at least part of data meet Predetermined condition C₂, judge W_j3[p_j3-169,p_j3In], whether at least part of data meet predetermined condition C₃, judge W_j4[p_j4-169,p_j4In], whether at least part of data meet predetermined condition C₄, judge W_j5[p_j5-169,p_j5In], whether at least part of data meet predetermined condition C₅, judge W_j6[p_j6 -169,p_j6In], whether at least part of data meet predetermined condition C₆, judge W_j7[p_j7-169, p_j7In], whether at least part of data meet predetermined condition C₇, judge W_j8[p_j8-169,p_j8In] extremely Whether small part data meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In] at least partly Whether data meet predetermined condition C₉, judge W_j10[p_j10-169,p_j10In], at least part of data are No meet predetermined condition C₁₀With judge W_j11[p_j11-169,p_j11In], at least part of data are the fullest Foot predetermined condition C₁₁.When judging window W_j1In at least partly data meet predetermined condition C₁, window Mouth W_j2In at least partly data meet predetermined condition C₂, window W_j3In at least partly data meet Predetermined condition C₃, window W_j4In at least partly data meet predetermined condition C₄, window W_j5In extremely Small part data meet predetermined condition C₅, window W_j6In at least partly data meet predetermined condition C₆, window W_j7In at least partly data meet predetermined condition C₇, window W_j8In at least partly count According to meeting predetermined condition C₈, window W_i9In at least partly data meet predetermined condition C₉, window W_j10In at least partly data meet predetermined condition C₁₀With window W_j11In at least partly data meet Predetermined condition C₁₁Time, the most current potential cut-point k_jFor data flow point cutpoint.When judging window W_j1、W_j2、W_j3、W_j4、W_j5、W_j6、W_j7、W_j8、W_j9、W_j10And W_j11In any one When in window, at least part of data are unsatisfactory for predetermined condition, as shown in figure 15, W_j3[p_j3-169, p_j3In], at least part of data are unsatisfactory for predetermined condition C₃Time, put p_i4Relative to data flow point cutpoint Search direction is positioned at a p_j3The left side, from a p_i4Along data flow point cutpoint search direction jump 21 Individual byte, it is thus achieved that next potential cut-point, for potential cut-point k_i、k_jDistinguish, table It is shown as k_l.According in Figure 13 institute embodiment on duplicate removal server 103 preset rule, for Potential cut-point k_lThe point determined is 11, respectively p_l1、p_l2、p_l3、p_l4、p_l5、p_l6、 p_l7、p_l8、p_l9、p_l10And p_l11, put p_l1、p_l2、p_l3、p_l4、p_l5、p_l6、p_l7、p_l8、 p_l9、p_l10And p_l11Corresponding window is respectively W_l1[p_l1-169,p_l1]、W_l2[p_l2-169,p_l2]、 W_l3[p_l3-169,p_l3]、W_l4[p_l4-169,p_l4]、W_l5[p_l5-169,p_l5]、W_l6[p_l6-169, p_l6]、W_l7[p_l7-169,p_l7]、W_l8[p_l8-169,p_l8]、W_l9[p_l9-169,p_l9]、W_l10[p_l10 -169,p_l10] and W_l11[p_l11-169,p_l11], wherein, p_lxWith potential cut-point k_lSpacing d_x Individual byte, concrete, p_l1With potential cut-point k_l0 byte of spacing, p_l2With k_lBetween 2 bytes of distance, p_l3With k_l4 bytes of spacing, p_l4With k_l6 bytes of spacing, p_l5With k_l8 bytes of spacing, p_l6With k_l10 bytes of spacing, p_l7With k_lSpacing From 12 bytes, p_l8With k_l14 bytes of spacing, p_l9With k_l16 bytes of spacing, p_l10With k_l18 bytes of spacing, p_l11With k_l20 bytes of spacing, and p_l2、p_l3、 p_l4、p_l5、p_l6、p_l7、p_l8、p_l9、p_l10And p_l11Relative to potential cut-point k_lIt is respectively positioned on Data flow point cutpoint searches opposite direction.Judge W_l1[p_l1-169,p_l1In], whether at least part of data Meet predetermined condition C₁, judge W_l2[p_l2-169,p_l2In], whether at least part of data meet pre- Fixed condition C₂, judge W_l3[p_l3-169,p_l3In], whether at least part of data meet predetermined condition C₃, judge W_l4[p_l4-169,p_l4In], whether at least part of data meet predetermined condition C₄, sentence Disconnected W_l5[p_l5-169,p_l5In], whether at least part of data meet predetermined condition C₅, judge W_l6[p_l6 -169,p_l6In], whether at least part of data meet predetermined condition C₆, judge W_l7[p_l7-169,p_l7] In at least partly data whether meet predetermined condition C₇, judge W_l8[p_l8-169,p_l8In] at least Whether part data meet predetermined condition C₈, judge W_l9[p_l9-169,p_l9At least partly count in] According to whether meeting predetermined condition C₉, judge W_l10[p_l10-169,p_l10In], whether at least part of data Meet predetermined condition C₁₀With judge W_l11[p_l11-169,p_l11In], whether at least part of data meet Predetermined condition C₁₁.When judging window W_l1In at least partly data meet predetermined condition C₁, window W_l2In at least partly data meet predetermined condition C₂, window W_l3In at least partly data meet pre- Fixed condition C₃, window W_l4In at least partly data meet predetermined condition C₄, window W_l5In at least Part data meet predetermined condition C₅, window W_l6In at least partly data meet predetermined condition C₆、 Window W_l7In at least partly data meet predetermined condition C₇, window W_l8In at least partly data full Foot predetermined condition C₈, window W_l9In at least partly data meet predetermined condition C₉, window W_l10In At least partly data meet predetermined condition C₁₀With window W_l11In at least partly data meet predetermined Condition C₁₁Time, the most current potential cut-point k_lFor data flow point cutpoint.As window W_l1、W_l2、 W_l3、W_l4、W_l5、W_l6、W_l7、W_l8、W_l9、W_l10And W_l11At least portion in middle either window When divided data is unsatisfactory for predetermined condition, select next potential cut-point, for potential cut-point k_i、k_jAnd k_lDifference, is expressed as k_m, k_mIt is positioned at k_lThe right, and k_mWith k_l1 byte of spacing. The rule preset on duplicate removal server 103 according to embodiment illustrated in fig. 13, for potential cut-point k_mThe point determined is 11, respectively p_m1、p_m2、p_m3、p_m4、p_m5、p_m6、p_m7、p_m8、 p_m9、p_m10And p_m11, put p_m1、p_m2、p_m3、p_m4、p_m5、p_m6、p_m7、p_m8、p_m9、 p_m10And p_m11Corresponding window is respectively W_m1[p_m1-169,p_m1]、W_m2[p_m2-169,p_m2]、 W_m3[p_m3-169,p_m3]、W_m4[p_m4-169,p_m4]、W_m5[p_m5-169,p_m5]、W_m6[p_m6-169, p_m6]、W_m7[p_m7-169,p_m7]、W_m8[p_m8-169,p_m8]、W_m9[p_m9-169,p_m9]、W_m10 [p_m10-169,p_m10] and W_m11[p_m11-169,p_m11], wherein, p_mxWith potential cut-point k_mIt Spacing d_xIndividual byte, concrete, p_m1With potential cut-point k_m0 byte of spacing, p_m2 With k_m2 bytes of spacing, p_m3With k_m4 bytes of spacing, p_m4With k_mSpacing 6 bytes, p_m5With k_m8 bytes of spacing, p_m6With k_m10 bytes of spacing, p_m7 With k_m12 bytes of spacing, p_m8With k_m14 bytes of spacing, p_m9With k_mSpacing From 16 bytes, p_m10With k_m18 bytes of spacing, p_m11With k_m20 words of spacing Joint, and p_m2、p_m3、p_m4、p_m5、p_m6、p_m7、p_m8、p_m9、p_m10And p_m11Relatively In potential cut-point k_mIt is respectively positioned on data flow point cutpoint and searches opposite direction.Judge W_m1[p_m1-169, p_m1In], whether at least part of data meet predetermined condition C₁, judge W_m2[p_m2-169,p_m2In] At least partly whether data meet predetermined condition C₂, judge W_m3[p_m3-169,p_m3At least portion in] Whether divided data meets predetermined condition C₃, judge W_m4[p_m4-169,p_m4At least part of data in] Whether meet predetermined condition C₄, judge W_m5[p_m5-169,p_m5In], at least part of data are the fullest Foot predetermined condition C₅, judge W_m6[p_m6-169,p_m6In], whether at least part of data meet predetermined Condition C₆, judge W_m7[p_m7-169,p_m7In], whether at least part of data meet predetermined condition C₇、 Judge W_m8[p_m8-169,p_m8In], whether at least part of data meet predetermined condition C₈, judge W_m9 [p_m9-169,p_m9In], whether at least part of data meet predetermined condition C₉, judge W_m10[p_m10 -169,p_m10In], whether at least part of data meet predetermined condition C₁₀With judge W_m11[p_m11-169, p_m11In], whether at least part of data meet predetermined condition C₁₁.When judging window W_m1In at least portion Divided data meets predetermined condition C₁, window W_m2In at least partly data meet predetermined condition C₂、 Window W_m3In at least partly data meet predetermined condition C₃, window W_m4In at least partly data full Foot predetermined condition C₄, window W_m5In at least partly data meet predetermined condition C₅, window W_m6In At least partly data meet predetermined condition C₆, window W_m7In at least partly data meet predetermined bar Part C₇, window W_m8In at least partly data meet predetermined condition C₈, window W_m9In at least partly Data meet predetermined condition C₉, window W_m10In at least partly data meet predetermined condition C₁₀And window Mouth W_m11In at least partly data meet predetermined condition C₁₁Time, the most current potential cut-point k_mFor number According to flow point cutpoint.When data at least part of in any one window are unsatisfactory for predetermined condition, then press Jump is performed, to obtain next potential cut-point and to determine whether according to previously described scheme Data flow point cutpoint.

Embodiments provide one and judge window W_iz[p_iz-A_z,p_iz+B_zAt least portion in] Whether divided data meets predetermined condition C_zMethod, in the present embodiment use random function judge window Mouth W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet predetermined condition C_z, with Fig. 5 As a example by shown embodiment, according to the rule preset on duplicate removal server 103, for potential Cut-point k_iDetermine a p_i1And some p_i1Corresponding window W_i1[p_i1-169,p_i1], it is judged that W_i1[p_i1-169, p_i1In], whether at least part of data meet predetermined condition C₁, as shown in figure 16, W_i1Represent window Mouth W_i1[p_i1-169,p_i1], for judging W_i1[p_i1-169,p_i1In], whether at least part of data meet pre- Fixed condition C₁, select 5 bytes, 1 byte that in Figure 16, " ■ " expression selects, adjacent two 42 bytes are differed between the byte selected.5 byte datas selected are recycled 51 times, Obtain 255 bytes altogether, to increase randomness.The most each byte is formed by 8, is designated as a_m,1… a_m,8, represent in 255 bytes the 1st to the 8th of m-th byte, therefore, 255 bytes pair The position answered can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{255,1}& a_{255,2}& \mathrm{...}& a_{255,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n=1, when a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, 255 bytes pair The position answered is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, can be expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{a255,1}& V_{a255,2}& \mathrm{...}& V_{a255,8}\end{array}\right).$ Choose a large amount of random number, form matrix, by random data The matrix of composition once forms, and keeps constant, as (divided with normal state here from obedience specific distribution As a example by cloth) random number in select 255*8 random number to form matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_aThe random number phase of m row and the m row of matrix R Taking advantage of, then summation obtains a value, is embodied as S_am=V_am,1*h_m,1+V_am,2*h_m,2+…+V_am,8 *h_m,8.According to the method, it is thus achieved that S_a1、S_a2... to S_a255, add up S_a1、S_a2... to S_a255In Meet number K of the value of specified conditions (here as a example by more than 0).Owing to matrix R is just obeying State is distributed, then S_amAs matrix R, still Normal Distribution, according to theory of probability, normal state The distribution random numbers probability more than 0 is 1/2, at S_a1、S_a2... to S_a255In, each value is more than 0 Probability be 1/2, so K meets binomial distribution: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistical result, it is judged that S_a1、S_a2... to S_a255Value more than 0 number K whether be even number, The random number of binomial distribution be the probability of even number for for 1/2, so K meets bar with the probability of 1/2 Part.When K is even number, show W_i1[p_i1-169,p_i1In], at least part of data meet predetermined condition C₁；When K is odd number, show W_i1[p_i1-169,p_i1In], at least part of data are unsatisfactory for making a reservation for Condition C₁, C here₁I.e. refer to the S obtained according to aforesaid way_a1、S_a2... to S_a255Value more than 0 Number K be even number.In the embodiment shown in Fig. 5, at W_i1[p_i1-169,p_i1]、W_i2[p_i2 -169,p_i2]、W_i3[p_i3-169,p_i3]、W_i4[p_i4-169,p_i4]、W_i5[p_i5-169,p_i5]、W_i6[p_i6 -169,p_i6]、W_i7[p_i7-169,p_i7]、W_i8[p_i8-169,p_i8]、W_i9[p_i9-169,p_i9]、W_i10[p_i10 -169,p_i10] and W_i11[p_i11-169,p_i11In], each window size is identical, i.e. window size is 169 Byte, judges that the mode that in window, whether at least part of data meet predetermined condition is the most identical simultaneously, It is specifically shown in above-mentioned judgement W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁ Description.Therefore, as shown in figure 16,Represent and judge window W_i2[p_i2-169,p_i2In] extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select 42 bytes are differed between byte.5 byte datas selected are recycled 51 times, obtains altogether 255 bytes, to increase randomness.The most each byte is formed by 8, is designated as b_m,1…b_m,8, Representing in 255 bytes the 1st to the 8th of m-th byte, therefore, 255 bytes are corresponding Position can be expressed as: $\left(\begin{array}{cccc}{b}_{1,1}& b_{1,2}& \mathrm{...}& b_{1,8}\\ b_{2,1}& b_{2,2}& \mathrm{...}& b_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ b_{255,1}& b_{255,2}& \mathrm{...}& b_{255,8}\end{array}\right),$ Work as b_m,nWhen=1, V_bm,n=1, work as b_m,n=0 Time, V_bm,n=-1, wherein b_m,nRepresent b_m,1…b_m,8In any one, the position that 255 bytes are corresponding According to b_m,nWith V_bm,nTransformational relation obtain matrix V_b, can be expressed as: $\left(\begin{array}{cccc}{V}_{b1,1}& V_{b1,2}& \mathrm{...}& V_{b1,8}\\ V_{b2,1}& V_{b2,2}& \mathrm{...}& V_{b2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{b255,1}& V_{b255,2}& \mathrm{...}& V_{b255,8}\end{array}\right),$ Judge W_i1[p_i1-169,p_i1In], at least part of data are the fullest Foot predetermined condition mode with judge window W_i2[p_i2-169,p_i2In], whether at least part of data The mode meeting predetermined condition is identical, therefore use matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_bM row be multiplied with the random number of the m row of matrix R, then summation obtain one Individual value, is embodied as S_bm=V_bm,1*h_m,1+V_bm,2*h_m,2+…+V_bm,8*h_m,8.According to the method, Obtain S_b1、S_b2... to S_b255, add up S_b1、S_b2... to S_b255In meet specified conditions (here As a example by more than 0) number K of value.Due to matrix R Normal Distribution, then S_bmWith square R is the same for battle array, still Normal Distribution, and according to theory of probability, normal distribution random number is more than 0 Probability be 1/2, at S_b1、S_b2... to S_b255In, each value probability more than 0 is 1/2, institute Binomial distribution is met with K: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistical result, Judge S_b1、S_b2... to S_b255Value more than 0 number K whether be even number, binomial distribution with Machine number be the probability of even number for for 1/2, so K meets condition with the probability of 1/2.When K is even number Time, show W_i2[p_i2-169,p_i2In], at least part of data meet predetermined condition C₂；When K is strange During number, show W_i2[p_i2-169,p_i2In], at least part of data are unsatisfactory for predetermined condition C₂, C here₂ I.e. refer to the S obtained according to aforesaid way_b1、S_b2... to S_b255Value more than 0 number K be even number. In embodiment shown in Fig. 3, W_i2[p_i2-169,p_i2In], at least part of data meet predetermined condition C₂。

Therefore, as shown in figure 16,Represent and judge window W_i3[p_i3-169,p_i3At least portion in] Whether divided data meets predetermined condition C₃Time select 1 byte, adjacent two select bytes Between differ 42 bytes.5 byte datas selected are recycled 51 times, obtains 255 altogether Byte, to increase randomness.Then use and judge window W_i1[p_i1-169,p_i1] and W_i2[p_i2-169, p_i2The method that in], whether at least part of data meet predetermined condition, it is judged that W_i3[p_i3-169,p_i3] In at least data whether meet predetermined condition C₃.In embodiment shown in Fig. 5, W_i3[p_i3-169, p_i3In], at least part of data meet predetermined condition.As shown in figure 16,Represent and judge window W_i4[p_i4-169,p_i4In], whether at least part of data meet predetermined condition C₄Time select 1 word Joint, differs 42 bytes between adjacent two bytes selected.By anti-for 5 byte datas of selection Utilize again 51 times, obtain 255 bytes altogether, to increase randomness.Then use and judge window W_i1 [p_i1-169,p_i1]、W_i2[p_i2-169,p_i2] and W_i3[p_i3-169,p_i3In], at least part of data are the fullest The method of foot predetermined condition, it is judged that W_i4[p_i4-169,p_i4In], whether at least part of data meet pre- Fixed condition C₄.In embodiment shown in Fig. 5, W_i4[p_i4-169,p_i4At least part of data in] Meet predetermined condition C₄.As shown in figure 16,Represent and judge window W_i5[p_i5-169,p_i5In] At least partly whether data meet predetermined condition C₅Time select 1 byte, adjacent two selections Byte between differ 42 bytes.5 byte datas selected are recycled 51 times, obtains altogether Obtain 255 bytes, to increase randomness.Then use and judge window W_i1[p_i1-169,p_i1]、W_i2[p_i2 -169,p_i2]、W_i3[p_i3-169,p_i3] and W_i4[p_i4-169,p_i4In], whether at least part of data meet The method of predetermined condition, it is judged that W_i5[p_i5-169,p_i5In], at least whether data meet predetermined condition C₅.In embodiment shown in Fig. 5, W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for pre- Fixed condition C₅。

Work as W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for C during predetermined condition₅, from a p_i5 Along data flow point cutpoint search direction 11 bytes of jump, at the end position of the 11st byte Obtain next potential cut-point k_j, as shown in Figure 6, preset according on duplicate removal server 103 Rule, for potential cut-point k_jDetermine a p_j1, some p_j1Corresponding window W_j1[p_j1-169,p_j1], Judge window W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁Side Formula with judge window W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁'s Mode is identical, the most as shown in figure 17, and W_j1Represent window W_j1[p_j1-169,p_j1], for judging W_j1 [p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁, select 5 bytes, figure In 17, " ■ " represents 1 byte selected, and differs 42 words between adjacent two bytes selected Joint.5 byte datas selected are recycled 51 times, obtains 255 bytes altogether, random to increase Property.The most each byte is formed by 8, is designated as a_m,1'…a_m,8', represent m in 255 bytes The 1st of individual byte to the 8th, therefore, position corresponding to 255 bytes can be expressed as: $\left(\begin{array}{cccc}{a_{1,1}}^{\′}& {a_{1,2}}^{\′}& \mathrm{...}& {a_{1,8}}^{\′}\\ {a_{2,1}}^{\′}& {a_{2,2}}^{\′}& \mathrm{...}& {a_{2,8}}^{\′}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ {a_{255,1}}^{\′}& {a_{255,2}}^{\′}& \mathrm{...}& {a_{255,8}}^{\′}\end{array}\right),$ Work as a_m,nDuring '=1, V_am,n'=1, works as a_m,nDuring '=0, V_am,n'=-1, Wherein a_m,n' represent a_m,1'…a_m,8Any one in ', position corresponding to 255 bytes is according to a_m,n' with V_am,n' transformational relation obtain matrix V_a', can be expressed as: $\left(\begin{array}{cccc}{V_{a1,1}}^{\′}& {V_{a1,2}}^{\′}& \mathrm{...}& {V_{a1,8}}^{\′}\\ {V_{a2,1}}^{\′}& {V_{a2,2}}^{\′}& \mathrm{...}& {V_{a2,8}}^{\′}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ {V_{a255,1}}^{\′}& {V_{a255,2}}^{\′}& \mathrm{...}& {V_{a255,8}}^{\′}\end{array}\right).$ Judge window W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition and sentence Disconnected window W_i1[p_i1-169,p_i1In], whether at least part of data meet the mode phase of predetermined condition With, therefore use matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_a' m row and matrix The random number of the m row of R is multiplied, and then summation obtains a value, is embodied as S_am'=V_am,1' *h_m,1+V_am,2'*h_m,2+…+V_am,8'*h_m,8.According to the method, it is thus achieved that S_a1'、S_a2' ... to S_a255', Statistics S_a1'、S_a2' ... to S_a255The value of specified conditions (here as a example by more than 0) is met in ' Number K.Due to matrix R Normal Distribution, then S_am' as matrix R, just still obeying State is distributed, and according to theory of probability, the normal distribution random number probability more than 0 is 1/2, at S_a1'、 S_a2' ... to S_a255In ', each value probability more than 0 is 1/2, so K meets binomial distribution: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistical result, it is judged that S_a1'、S_a2' ... arrive S_a255' value more than 0 number K whether be even number, the random number of binomial distribution is the general of even number Rate is 1/2, so K meets condition with the probability of 1/2.When K is even number, show W_j1[p_j1-169, p_j1In], at least part of data meet predetermined condition C₁；When K is odd number, show W_j1[p_j1-169, p_j1In], at least part of data are unsatisfactory for predetermined condition C₁。

Judge W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 17,Represent and judge window W_j2[p_j2-169,p_j2In] extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select 42 bytes are differed between byte.5 byte datas selected are recycled 51 times, obtains altogether 255 bytes, to increase randomness.The most each byte is formed by 8, is designated as b_m,1'…b_m,8', Representing in 255 bytes the 1st to the 8th of m-th byte, therefore, 255 bytes are corresponding Position can be expressed as: $\left(\begin{array}{cccc}{b_{1,1}}^{\′}& {b_{1,2}}^{\′}& \mathrm{...}& {b_{1,8}}^{\′}\\ {b_{2,1}}^{\′}& {b_{2,2}}^{\′}& \mathrm{...}& {b_{2,8}}^{\′}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ {b_{255,1}}^{\′}& {b_{255,2}}^{\′}& \mathrm{...}& {b_{255,8}}^{\′}\end{array}\right),$ Work as b_m,nDuring '=1, V_bm,n'=1, works as b_m,n' When=0, V_bm,n'=-1, wherein b_m,n' represent b_m,1'…b_m,8Any one in ', 255 byte correspondences Position according to b_m,n' and V_bm,n' transformational relation obtain matrix V_b', can be expressed as: $\left(\begin{array}{cccc}{V}_{b1,1}& V_{b1,2}& \mathrm{...}& V_{b1,8}\\ V_{b2,1}& V_{b2,2}& \mathrm{...}& V_{b2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{b255,1}& V_{b255,2}& \mathrm{...}& V_{b255,8}\end{array}\right).$ Window W₂[p₂-169,p₂] and W₂[q₂-169,q₂In] at least The mode whether part data meet predetermined condition is identical, the most still uses matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_b' the random number of m row of m row and matrix R Being multiplied, then summation obtains a value, is embodied as S_bm'=V_bm,1'*h_m,1+V_bm,2'*h_m,2 +…+V_bm,8'*h_m,8.According to the method, it is thus achieved that S_b1'、S_b2' ... to S_b255', add up S_b1'、S_b2'… To S_b255Number K of the value of specified conditions (here as a example by more than 0) is met in '.Due to square Battle array R Normal Distribution, then S_bm' as matrix R, still Normal Distribution, according to generally Rate opinion, the normal distribution random number probability more than 0 is 1/2, at S_b1'、S_b2' ... to S_b255In ', Each value probability more than 0 is 1/2, so K meets binomial distribution: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistical result, it is judged that S_b1'、S_b2' ... arrive S_b255' value more than 0 number K whether be even number, the random number of binomial distribution is the general of even number Rate is for for 1/2, so K meets condition with the probability of 1/2.When K is even number, show W_j2[p_j2 -169,p_j2In], at least part of data meet predetermined condition C₂；When K is odd number, show W_j2[p_j2 -169,p_j2In], at least part of data are unsatisfactory for predetermined condition C₂.In like manner, it is judged that W_i3[p_i3-169, p_i3In], whether at least part of data meet predetermined condition C₃Mode with judge W_j3[p_j3-169, p_j3In], whether at least part of data meet predetermined condition C₃Mode identical, in like manner, it is judged that W_j4 [p_j4-169,p_j4In], whether at least part of data meet predetermined condition C₄, judge W_j5[p_j5-169, p_j5In], whether at least part of data meet predetermined condition C₅, judge W_j6[p_j6-169,p_j6In] extremely Whether small part data meet predetermined condition C₆, judge W_j7[p_j7-169,p_j7In] at least partly Whether data meet predetermined condition C₇, judge W_j8[p_j8-169,p_j8In], at least part of data are No meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In], whether at least part of data meet Predetermined condition C₉, judge W_j10[p_j10-169,p_j10In], whether at least part of data meet predetermined bar Part C₁₀With judge W_j11[p_j11-169,p_j11In], whether at least part of data meet predetermined condition C₁₁, Do not repeat them here.

Still as a example by Fig. 5 illustrated embodiment, it is provided that one judges window W_iz[p_iz-A_z, p_iz+B_zIn], whether at least part of data meet predetermined condition C_zMethod, the present embodiment makes Window W is judged with random function_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet pre- Fixed condition C_z, according to the rule preset on duplicate removal server 103, for potential cut-point k_iReally Fixed point p_i1And p_i1Corresponding window W_i1[p_i1-169,p_i1], it is judged that W_i1[p_i1-169,p_i1At least portion in] Whether divided data meets predetermined condition C₁, as shown in figure 16, W_i1Represent window W_i1[p_i1-169, p_i1], for judging W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁, choosing Selecting 5 bytes, in Figure 16, " ■ " represents 1 byte selected, the word of adjacent two selections " ■ " 42 bytes are differed between joint.One of which implementation selects for using HASH function to calculate 5 bytes, use the calculated numerical value of HASH function to be one and fixing be uniformly distributed, If using the calculated numerical value of HASH function is even number, then judge W_i1[p_i1-169,p_i1] In at least partly data meet predetermined condition C₁, i.e. C₁Represent and use HASH according to aforesaid way The calculated numerical value of function is even number.Therefore, W_i1[p_i1-169,p_i1At least part of data in] The probability whether meeting predetermined condition is 1/2.In the embodiment shown in Fig. 5, use Hash Function judges W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂、W_i3 [p_i3-169,p_i3In], whether at least part of data meet predetermined condition C₃、W_i4[p_i4-169,p_i4In] At least partly whether data meet predetermined condition C₄And W_i5[p_i5-169,p_i5At least part of data in] Whether meet predetermined condition C₅, implement and refer to describe the use of Fig. 5 illustrated embodiment Hash function judges W_i1[p_i1-169,p_i1In], whether at least part of data meet the side of predetermined condition Formula C₁, do not repeat them here.

Work as W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for predetermined condition C₅Time, from a p_i5 Along data flow point cutpoint search direction 11 bytes of jump, at the end position of the 11st byte Obtain current potential cut-point k_j, as shown in Figure 6, according to preset on duplicate removal server 103 Rule, for potential cut-point k_jDetermine a p_j1, some p_j1Corresponding window W_j1[p_j1-169,p_j1], Judge window W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁Side Formula with judge window W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁'s Mode is identical, the most as shown in figure 17, and W_j1Represent window W_j1[p_j1-169,p_j1], for judging W_j1 [p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁, select 5 bytes, figure In 17, " ■ " represents 1 byte selected, and differs 42 between adjacent two bytes " ■ " selected Individual byte.Hash function is used to calculate from window W_j1[p_j1-169,p_j15 bytes chosen in], If the numerical value obtained is even number, then W_j1[p_j1-169,p_j1In], at least part of data meet predetermined Condition C₁.In Figure 17, it is judged that W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined Condition C₂Mode and judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined Condition C₂Mode identical, therefore, as shown in figure 17,Represent and judge window W_j2[p_j2-169, p_j2In], whether at least part of data meet predetermined condition C₂Time select 1 byte, adjacent two The byte of individual selectionBetween differ 42 bytes.Hash function is used to calculate 5 selected Byte, if the numerical value obtained is even number, then W_j2[p_j2-169,p_j2In], at least part of data are full Foot predetermined condition C₂.In Figure 17, it is judged that W_i3[p_i3-169,p_i3In], at least part of data are the fullest Foot predetermined condition C₃Mode with judge W_j3[p_j3-169,p_j3In], at least part of data are the fullest Foot predetermined condition C₃Mode identical, therefore, as shown in figure 17,Represent and judge window W_j3 [p_j3-169,p_j3In], whether at least part of data meet predetermined condition C₃Time select 1 byte, Adjacent two bytes selectedBetween differ 42 bytes.Use Hash function to calculate to select 5 bytes, the numerical value obtained is even number, then W_j3[p_j3-169,p_j3At least part of data in] Meet predetermined condition C₃.In Figure 17, it is judged that W_j4[p_j4-169,p_j4In], whether at least part of data Meet predetermined condition C₄Mode and judge window W_i4[p_i4-169,p_i4At least part of data in] Whether meet predetermined condition C₄Mode, therefore, as shown in figure 17,Represent and judge window Mouth W_j4[p_j4-169,p_j4In], whether at least part of data meet predetermined condition C₄Time select 1 Byte, adjacent two bytes selectedBetween differ 42 bytes.Use Hash function meter Calculating 5 bytes selected, the numerical value obtained is even number, then W_j4[p_j4-169,p_j4At least portion in] Divided data meets predetermined condition C₄.According to said method, it is judged that W_j5[p_j5-169,p_j5In] at least Whether part data meet predetermined condition C₅, judge W_j6[p_j6-169,p_j6At least partly count in] According to whether meeting predetermined condition C₆, judge W_j7[p_j7-169,p_j7In], whether at least part of data Meet predetermined condition C₇, judge W_j8[p_j8-169,p_j8In], whether at least part of data meet pre- Fixed condition C₈, judge W_j9[p_j9-169,p_j9In], whether at least part of data meet predetermined condition C₉, judge W_j10[p_j10-169,p_j10In], whether at least part of data meet predetermined condition C₁₀With Judge W_j11[p_j11-169,p_j11In], whether at least part of data meet predetermined condition C₁₁, at this not Repeat again.

As a example by the embodiment shown in Fig. 5, it is provided that one judges window W_iz[p_iz-A_z,p_iz +B_zIn], whether at least part of data meet predetermined condition C_zMethod, in the present embodiment use Random function judges window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet predetermined Condition C_z, according to the rule preset on duplicate removal server 103, for potential cut-point k_iDetermine Point p_i1And p_i1Corresponding window W_i1[p_i1-169,p_i1], it is judged that W_i1[p_i1-169,p_i1In] at least partly Whether data meet predetermined condition C₁, as shown in figure 16, W_i1Represent window W_i1[p_i1-169,p_i1], For judging W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁, select 5 Byte, in Figure 16, the byte " ■ " of serial number 169,127,85,43 and 1 represents selection respectively 1 byte, adjacent two select bytes between differ 42 bytes.By serial number 169, 127, the byte " ■ " of 85,43 and 1 is converted into a decimal value respectively, represents respectively For a₁、a₂、a₃、a₄And a₅.Because 1 byte is formed by 8, so each byte " ■ " As numerical value, then an a₁、a₂、a₃、a₄And a₅In any one a_rIt is satisfied by 0≤a_r≤255。 a₁、a₂、a₃、a₄And a₅The matrix of composition 1*5.Select from the random number obeying binomial distribution Select 256*5 random number, form matrix R, be expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right),$

According to a₁Value and the row at place, search from matrix R correspondence value, such as a₁=36, a₁ It is positioned at the 1st row, then searches h_36,1Corresponding value；According to a₂Value and the row at place, from matrix R The middle value searching correspondence, such as a₂=48, a₂It is positioned at the 2nd row, then searches h_48,2Corresponding value；Root According to a₃Value and the row at place, search from matrix R correspondence value, such as a₃=26, a₃It is positioned at 3rd row, then search h_26,3Corresponding value；According to a₄Value and the row at place, look into from matrix R Look for the value of correspondence, such as a₄=26, a₄It is positioned at the 4th row, then searches h_26,4Corresponding value；According to a₅ Value and the row at place, search from matrix R correspondence value, such as a₅=88, a₅It is positioned at the 5th row, Then search h_88,5Corresponding value.S₁=h_36,1+h_48,2+h_26,3+h_26,4+h_88,5, because matrix R obeys two Item distribution, therefore, S₁Also binomial distribution is obeyed.Work as S₁For even number, then W_i1[p_i1-169,p_i1] In at least partly data meet predetermined condition C₁, work as S₁For odd number, then W_i1[p_i1-169,p_i1In] At least partly data are unsatisfactory for predetermined condition C₁, S₁Probability for even number is 1/2, C₁Expression is pressed Aforesaid way calculates S₁For even number.In the embodiment shown in fig. 5, W_i1[p_i1-169,p_i1In] at least Part data meet predetermined condition C₁.As shown in figure 16,Represent and judge window W_i2[p_i2-169, p_i2In], whether at least part of data meet predetermined condition C₂Time 1 byte selecting respectively, at figure In 16, represent by sequence number 170,128,86,44 and 2 respectively, adjacent two bytes selected Between differ 42 bytes.Byte by sequence number 170,128,86,44 and 2Turn respectively Change a decimal value into, be expressed as b₁、b₂、b₃、b₄And b₅.Because 1 byte Formed by 8, so each byteAs numerical value, then a b₁、b₂、b₃、b₄And b₅ In any one b_rIt is satisfied by 0≤b_r≤255。b₁、b₂、b₃、b₄And b₅The matrix of composition 1*5. In present embodiment, it is judged that W_i1And W_i2In at least partly data whether meet the side of predetermined condition Formula is identical, the most still uses matrix R, according to b₁Value and the row at place, from matrix R Search corresponding value, such as b₁=66, b₁It is positioned at the 1st row, then searches h_66,1Corresponding value；According to b₂ Value and the row at place, search from matrix R correspondence value, such as b₂=48, b₂It is positioned at the 2nd row, Then search h_48,2Corresponding value；According to b₃Value and the row at place, search corresponding from matrix R Value, such as b₃=99, b₃It is positioned at the 3rd row, then searches h_99,3Corresponding value；According to b₄Value and institute Row, search from matrix R correspondence value, such as b₄=26, b₄It is positioned at the 4th row, then searches h_26,4 Corresponding value；According to b₅Value and the row at place, search from matrix R correspondence value, such as b₅=90, b₅It is positioned at the 5th row, then searches h_90,5Corresponding value.S₂=h_66,1+h_48,2+h_99,3+h_26,4+h_90,5, because Matrix R obeys binomial distribution, therefore, S₂Also binomial distribution is obeyed.Work as S₂For even number, then W_i2 [p_i2-169,p_i2In], at least part of data meet predetermined condition C₂, work as S₂For odd number, then W_i2[p_i2 -169,p_i2In], at least part of data are unsatisfactory for predetermined condition C₂, S₂Probability for even number is 1/2. In the embodiment shown in fig. 5, W_i2[p_i2-169,p_i2In], at least part of data meet predetermined condition C₂.Use same rule, judge W respectively_i3[p_i3-169,p_i3In], whether at least part of data Meet predetermined condition C₃, judge W_i4[p_i4-169,p_i4In], whether at least part of data meet predetermined Condition C₄, judge W_i5[p_i5-169,p_i5In], whether at least part of data meet predetermined condition C₅、 Judge W_i6[p_i6-169,p_i6In], whether at least part of data meet predetermined condition C₆, judge W_i7 [p_i7-169,p_i7In], whether at least part of data meet predetermined condition C₇, judge W_i8[p_i8-169, p_i8In], whether at least part of data meet predetermined condition C₈, judge W_i9[p_i9-169,p_i9In] at least Whether part data meet predetermined condition C₉, judge W_i10[p_i10-169,p_i10At least part of data in] Whether meet predetermined condition C₁₀With judge W_i11[p_i11-169,p_i11In], at least part of data are the fullest Foot predetermined condition C₁₁.In embodiment shown in Fig. 5, W_i5[p_i5-169,p_i5At least partly count in] According to being unsatisfactory for predetermined condition C₅, from a p_i5Along data flow point cutpoint search direction 11 words of jump Joint, the end position the 11st byte obtains current potential cut-point k_j, as shown in Figure 6, According to the rule preset on duplicate removal server 103, for potential cut-point k_jDetermine a p_j1, point p_j1Corresponding window W_j1[p_j1-169,p_j1], it is judged that window W_j1[p_j1-169,p_j1In] at least partly Whether data meet predetermined condition C₁Mode with judge window W_i1[p_i1-169,p_i1At least portion in] Whether divided data meets predetermined condition C₁Mode identical, the most as shown in figure 17, W_j1Represent Window W_j1[p_j1-169,p_j1], for judging W_j1[p_j1-169,p_j1In], at least part of data are the fullest Foot predetermined condition C₁, in Figure 17, the byte " ■ " of serial number 169,127,85,43 and 1 is respectively Represent 1 byte selected, between adjacent two bytes selected, differ 42 bytes.By sequence Number be 169,127,85,43 and 1 byte " ■ " be converted into a decimal value respectively, It is expressed as a₁'、a₂'、a₃'、a₄' and a₅'.Because 1 byte is formed by 8, so often Individual byte " ■ " is as numerical value, then an a₁'、a₂'、a₃'、a₄' and a₅Any one a in '_r' all Meet 0≤a_r'≤255。a₁'、a₂'、a₃'、a₄' and a₅' composition 1*5 matrix.Judge window W_j1 [p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁Mode with judge window W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁Mode identical, because of This, still use matrix R, be expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right),$

According to a₁' value and the row at place, search from matrix R correspondence value, such as a₁'=16, a₁' position In the 1st row, then search h_16,1Corresponding value；According to a₂' value and the row at place, from matrix R Search corresponding value, such as a₂'=98, a₂' be positioned at the 2nd row, then search h_98,2Corresponding value；According to a₃' value and the row at place, search from matrix R correspondence value, such as a₃'=56, a₃' it is positioned at 3 row, then search h_56,3Corresponding value；According to a₄' value and the row at place, from matrix R search Corresponding value, such as a₄'=36, a₄' be positioned at the 4th row, then search h_36,4Corresponding value；According to a₅' Value and the row at place, search the value of correspondence, such as a from matrix R₅'=99, a₅' it is positioned at the 5th row, Then search h_99,5Corresponding value.S₁'=h_16,1+h_98,2+h_56,3+h_36,4+h_99,5, because matrix R obeys binomial Distribution, therefore, S₁' also obey binomial distribution.Work as S₁' for even number, then W_j1[p_j1-169,p_j1] In at least partly data meet predetermined condition C₁, work as S₁' for odd number, then W_j1[p_j1-169,p_j1] In at least partly data be unsatisfactory for predetermined condition C₁, S₁' it is 1/2 for the probability of even number.

Judge W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 17,Represent and judge window W_j2[p_j2-169,p_j2In] extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select Differ 42 bytes between byte, represent by sequence number 170,128,86,44 and 2 respectively, phase 42 bytes are differed between adjacent two bytes selected.By sequence number 170,128,86,44 and 2 ByteIt is converted into a decimal value respectively, is expressed as b₁'、b₂'、b₃'、b₄' And b₅'.Because 1 byte is formed by 8, so each byteAs a numerical value, then b₁'、b₂'、b₃'、b₄' and b₅Any one b in '_r' it is satisfied by 0≤b_r'≤255。b₁'、b₂'、b₃'、b₄' And b₅' composition 1*5 matrix.With judge window W_i2[p_i2-169,p_i2In], at least part of data are No meet predetermined condition C₂Use identical matrix R, according to b₁' value and the row at place, from square Battle array R searches the value of correspondence, such as b₁'=210, b₁' be positioned at the 1st row, then search h_210,1Corresponding value； According to b₂' value and the row at place, search from matrix R correspondence value, such as b₂'=156, b₂' position In the 2nd row, then search h_156,2Corresponding value；According to b₃' value and the row at place, from matrix R Search corresponding value, such as b₃'=144, b₃' be positioned at the 3rd row, then search h_144,3Corresponding value；Root According to b₄' value and the row at place, search from matrix R correspondence value, such as b₄'=60, b₄' be positioned at 4th row, then search h_60,4Corresponding value；According to b₅' value and the row at place, look into from matrix R Look for the value of correspondence, such as b₅'=90, b₅' be positioned at the 5th row, then search h_90,5Corresponding value.S₂'=h_210,1 +h_156,2+h_144,3+h_60,4+h_90,5, with S₂Rule of judgment identical, work as S₂' for even number, then W_j2[p_j2 -169,p_j2In], at least part of data meet predetermined condition C₂, work as S₂' for odd number, then W_j2[p_j2 -169,p_j2In], at least part of data are unsatisfactory for predetermined condition C₂, S₂' it is 1/2 for the probability of even number.

In like manner, it is judged that W_i3[p_i3-169,p_i3In], whether at least part of data meet predetermined condition C₃ Mode with judge W_j3[p_j3-169,p_j3In], whether at least part of data meet predetermined condition C₃ Mode identical, in like manner, it is judged that W_j4[p_j4-169,p_j4In], whether at least part of data meet pre- Fixed condition C₄, judge W_j5[p_j5-169,p_j5In], whether at least part of data meet predetermined condition C₅、 Judge W_j6[p_j6-169,p_j6In], whether at least part of data meet predetermined condition C₆, judge W_j7 [p_j7-169,p_j7In], whether at least part of data meet predetermined condition C₇, judge W_j8[p_j8-169, p_j8In], whether at least part of data meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In] extremely Whether small part data meet predetermined condition C₉, judge W_j10[p_j10-169,p_j10In] at least partly Whether data meet predetermined condition C₁₀With judge W_j11[p_j11-169,p_j11At least part of data in] Whether meet predetermined condition C₁₁, do not repeat them here.

As a example by the embodiment shown in Fig. 5, it is provided that one judges window W_iz[p_iz-A_z,p_iz +B_zIn], whether at least part of data meet predetermined condition C_zMethod, in the present embodiment use Random function judges window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet predetermined Condition C_z, according to the rule preset on duplicate removal server 103, for potential cut-point k_iDetermine Point p_i1And p_i1Corresponding window W_i1[p_i1-169,p_i1], it is judged that W_i1[p_i1-169,p_i1In] at least partly Whether data meet predetermined condition C₁, as shown in figure 16, W_i1Represent window W_i1[p_i1-169,p_i1], For judging W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁, select 5 Byte, in Figure 16, the byte " ■ " of serial number 169,127,85,43 and 1 represents selection respectively 1 byte, adjacent two select bytes between differ 42 bytes.By serial number 169, 127, the byte " ■ " of 85,43 and 1 is converted into a decimal value respectively, represents respectively For a₁、a₂、a₃、a₄And a₅.Because 1 byte is formed by 8, so each byte " ■ " As numerical value, then an a₁、a₂、a₃、a₄And a₅In any one a_sIt is satisfied by 0≤a_s≤255。 a₁、a₂、a₃、a₄And a₅The matrix of composition 1*5.Select from the random number obeying binomial distribution Select 256*5 random number, form matrix R, be expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right),$ From clothes From the random number of binomial distribution, select 256*5 random number, form matrix G, be expressed as: $\left(\begin{array}{cccc}{g}_{0,1}& g_{0,2}& \mathrm{...}& g_{0,5}\\ g_{1,1}& g_{1,2}& \mathrm{...}& g_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ g_{255,5}& g_{255,5}& \mathrm{...}& g_{255,5}\end{array}\right).$

According to a₁Value and the row at place, such as a₁=36, a₁It is positioned at the 1st row, then from matrix R Search h_36,1Corresponding value, searches g from matrix G_36,1Corresponding value；According to a₂Value and The row at place, such as a₂=48, a₂It is positioned at the 2nd row, then from matrix R, looks into h_48,2Corresponding value, G is searched from matrix G_48,2Corresponding value；According to a₃Value and the row at place, such as a₃=26, a₃ It is positioned at the 3rd row, then from matrix R, searches h_26,3Corresponding value, searches g from matrix G_26,3Right The value answered；According to a₄Value and the row at place, such as a₄=26, a₄It is positioned at the 4th row, then from matrix R searches h_26,4Corresponding value, searches g from matrix G_26,4Corresponding value；According to a₅Value and The row at place, such as a₅=88, a₅It is positioned at the 5th row, then from matrix R, searches h_88,5Corresponding value, G is searched from matrix G_88,5Corresponding value.S_1h=h_36,1+h_48,2+h_26,3+h_26,4+h_88,5, because matrix R obeys binomial distribution, therefore, S_1hAlso binomial distribution is obeyed；S_1g=g_36,1+g_48,2+g_26,3+g_26,4+ g_88,5, because matrix G obeys binomial distribution, therefore S_1gAlso binomial distribution is obeyed.Work as S_1hAnd S_1g In have 1 for even number, then W_i1[p_i1-169,p_i1In], at least part of data meet predetermined condition C₁, Work as S_1hAnd S_1gIt is odd number, then W_i1[p_i1-169,p_i1In], at least part of data are unsatisfactory for predetermined bar Part C₁, C₁The S that statement obtains according to the method described above_1hAnd S_1gIn have 1 for even number.Because S_1hWith S_1gAll obey binomial distribution, therefore S_1hProbability for even number is 1/2, S_1gProbability for even number is 1/2, S_1hAnd S_1gIn to have 1 probability for even number be 1-1/4=3/4, therefore, W_i1[p_i1-169,p_i1] In at least partly data meet predetermined condition C₁Probability be 3/4.In the embodiment shown in fig. 5, W_i1[p_i1-169,p_i1In], at least part of data meet predetermined condition C₁.The embodiment party shown in Fig. 5 In formula, at W_i1[p_i1-169,p_i1]、W_i2[p_i2-169,p_i2]、W_i3[p_i3-169,p_i3]、W_i4[p_i4-169, p_i4]、W_i5[p_i5-169,p_i5]、W_i6[p_i6-169,p_i6]、W_i7[p_i7-169,p_i7]、W_i8[p_i8-169, p_i8]、W_i9[p_i9-169,p_i9]、W_i10[p_i10-169,p_i10] and W_i11[p_i11-169,p_i11In], each window Size is identical, i.e. window size is 169 bytes, judges that in window, at least part of data are simultaneously The no mode meeting predetermined condition is the most identical, is specifically shown in above-mentioned judgement W_i1[p_i1-169,p_i1In] at least Whether part data meet predetermined condition C₁Description.Therefore, as shown in figure 16,Represent Judge window W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂Time-division 1 byte not selected, in figure 16, represents by sequence number 170,128,86,44 and 2 respectively, 42 bytes are differed between adjacent two bytes selected.By sequence number 170,128,86,44 With 2 byteIt is converted into a decimal value respectively, is expressed as b₁、b₂、b₃、 b₄And b₅.Because 1 byte is formed by 8, so each byteAs a numerical value, Then b₁、b₂、b₃、b₄And b₅In any one b_sIt is satisfied by 0≤b_s≤255。b₁、b₂、b₃、b₄ And b₅The matrix of composition 1*5.In present embodiment, it is judged that in each window, at least part of data are The no mode meeting predetermined condition is identical, the most still uses same matrix R and G.According to b₁'s Value and the row at place, such as b₁=66, b₁It is positioned at the 1st row, then from matrix R, searches h_66,1Corresponding Value, searches g from matrix G_66,1Corresponding value；According to b₂Value and the row at place, such as b₂=48, b₂It is positioned at the 2nd row, then from matrix R, searches h_48,2Corresponding value, searches g from matrix G_48,2Right The value answered；According to b₃Value and the row at place, such as b₃=99, b₃It is positioned at the 3rd row, then from matrix R searches h_99,3Corresponding value, searches g from matrix G_99,3Corresponding value；According to b₄Value and The row at place, such as b₄=26, b₄It is positioned at the 4th row, then from matrix R, searches h_26,4Corresponding value, G is searched from matrix G_26,4Corresponding value；According to b₅Value and the row at place, such as b₅=90, b₅Position In the 5th row, then from matrix R, search h_90,5Corresponding value, searches g from matrix G_90,5Corresponding Value.S_2h=h_66,1+h_48,2+h_99,3+h_26,4+h_90,5, because matrix R obedience binomial distribution, therefore, S_2hAlso binomial distribution is obeyed.S_2g=g_66,1+g_48,2+g_99,3+g_26,4+g_90,5, because matrix G obeys Binomial distribution, therefore, S_2gAlso binomial distribution is obeyed.Work as S_2hAnd S_2gIn have 1 for even number, Then W_i2[p_i2-169,p_i2In], at least part of data meet predetermined condition C₂, work as S_2hAnd S_2gIt is Odd number, then W_i2[p_i2-169,p_i2In], at least part of data are unsatisfactory for predetermined condition C₂, S_2hAnd S_2g In to have 1 probability for even number be 3/4.In the embodiment shown in fig. 5, W_i2[p_i2-169,p_i2] In at least partly data meet predetermined condition C₂.Use same rule, judge W respectively_i3[p_i3 -169,p_i3In], whether at least part of data meet predetermined condition C₃, judge W_i4[p_i4-169,p_i4] In at least partly data whether meet predetermined condition C₄, judge W_i5[p_i5-169,p_i5At least portion in] Whether divided data meets predetermined condition C₅, judge W_i6[p_i6-169,p_i6In], at least part of data are No meet predetermined condition C₆, judge W_i7[p_i7-169,p_i7In], whether at least part of data meet pre- Fixed condition C₇, judge W_i8[p_i8-169,p_i8In], whether at least part of data meet predetermined condition C₈、 Judge W_i9[p_i9-169,p_i9In], whether at least part of data meet predetermined condition C₉, judge W_i10 [p_i10-169,p_i10In], whether at least part of data meet predetermined condition C₁₀With judge W_i11[p_i11 -169,p_i11In], whether at least part of data meet predetermined condition C₁₁.Embodiment shown in Fig. 5 In, W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for predetermined condition C₅, from a p_i5Along Data flow point cutpoint search direction 11 bytes of jump, the end position the 11st byte obtains Current potential cut-point k_j, as shown in Figure 6, according to the rule preset on duplicate removal server 103, For potential cut-point k_jDetermine a p_j1, some p_j1Corresponding window W_j1[p_j1-169,p_j1], it is judged that Window W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁Mode with Judge window W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁Mode Identical, the most as shown in figure 17, W_j1Represent window W_j1[p_j1-169,p_j1], for judging W_j1[p_j1 -169,p_j1In], whether at least part of data meet predetermined condition C₁, serial number 169 in Figure 17, 127, the byte " ■ " of 85,43 and 1 represents 1 byte of selection, adjacent two selections respectively Byte between differ 42 bytes.By the byte " ■ " of serial number 169,127,85,43 and 1 It is converted into a decimal value respectively, is expressed as a₁'、a₂'、a₃'、a₄' and a₅'。 Because 1 byte is formed by 8, so each byte " ■ " is as numerical value, then an a₁'、a₂'、 a₃'、a₄' and a₅Any one a in '_s' it is satisfied by 0≤a_s'≤255。a₁'、a₂'、a₃'、a₄' and a₅' composition 1*5 matrix.Use and judge window W_i1[p_i1-169,p_i1In], at least part of data are No meet predetermined condition C₁Identical matrix R and G, is expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right)$ With $\left(\begin{array}{cccc}{g}_{0,1}& g_{0,2}& \mathrm{...}& g_{0,5}\\ g_{1,1}& g_{1,2}& \mathrm{...}& g_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ g_{255,5}& g_{255,5}& \mathrm{...}& g_{255,5}\end{array}\right).$

According to a₁' value and the row at place, such as a₁'=16, a₁' be positioned at the 1st row, then look into from matrix R Look for h_16,1Corresponding value, searches g from matrix G_16,1Corresponding value；According to a₂' value and place Row, such as a₂'=98, a₂' be positioned at the 2nd row, then from matrix R, search h_98,2Corresponding value, from square Battle array G searches g_98,2Corresponding value；According to a₃' value and the row at place, such as a₃'=56, a₃' position In the 3rd row, then from matrix R, search h_56,3Corresponding value, searches g from matrix G_56,3Corresponding Value；According to a₄' value and the row at place, such as a₄'=36, a₄' it is positioned at the 4th row, then from matrix R Middle lookup h_36,4Corresponding value, searches g from matrix G_36,4Corresponding value；According to a₅' value and The row at place, such as a₅'=99, a₅' be positioned at the 5th row, then from matrix R, search h_99,5Corresponding Value, searches g from matrix G_99,5Corresponding value.S_1h'=h_16,1+h_98,2+h_56,3+h_36,4+h_99,5, because of Binomial distribution, therefore, S is obeyed for matrix R_1h' also obey binomial distribution；S_1g'=g_16,1+g_98,2+ g_56,3+g_36,4+g_99,5, because matrix G obeys binomial distribution, therefore S_1g' also obey binomial distribution. Work as S_1h' and S_1g1 is had for even number, then W in '_j1[p_j1-169,p_j1In], at least part of data meet Predetermined condition C₁, work as S_1h' and S_1g' be odd number, then W_j1[p_j1-169,p_j1At least partly count in] According to being unsatisfactory for predetermined condition C₁, S_1h' and S_1g' to have 1 probability for even number be 3/4.

Judge W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 17,Represent and judge window W_j2[p_j2-169,p_j2In] extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select 42 bytes are differed between byte.In fig. 17, respectively by sequence number 170,128,86,44 Represent with 2, between adjacent two bytes selected, differ 42 bytes.By sequence number 170,128, 86, the byte of 44 and 2It is converted into a decimal value respectively, is expressed as b₁'、 b₂'、b₃'、b₄' and b₅'.Because 1 byte is formed by 8, so each byteAs One numerical value, then b₁'、b₂'、b₃'、b₄' and b₅Any one b in '_s' it is satisfied by 0≤b_s'≤255。 b₁'、b₂'、b₃'、b₄' and b₅' composition 1*5 matrix.Use and judge window W_i2[p_i2-169,p_i2] In at least partly data whether meet predetermined condition C₂Identical matrix R and G, according to b₁' value With the row at place, such as b₁'=210, b₁' be positioned at the 1st row, then from matrix R, search h_210,1Corresponding Value, from matrix G search g_210,1Corresponding value；According to b₂' value and the row at place, such as b₂' =156, b₂' be positioned at the 2nd row, then from matrix R, search h_156,2Corresponding value, looks into from matrix G Look for g_156,2Corresponding value；According to b₃' value and the row at place, such as b₃'=144, b₃' it is positioned at the 3rd Row, then search h from matrix R_144,3Corresponding value, searches g from matrix G_144,3Corresponding value； According to b₄' value and the row at place, such as b₄'=60, b₄' be positioned at the 4th row, then look into from matrix R Look for h_60,4Corresponding value, searches g from matrix G_60,4Corresponding value；According to b₅' value and place Row, such as b₅'=90, b₅' be positioned at the 5th row, then from matrix R, search h_90,5Corresponding value, from square Battle array G searches g_90,5Corresponding value.S_2h'=h_210,1+h_156,2+h_144,3+h_60,4+h_90,5,S_2g'=g_210,1+ g_156,2+g_144,3+g_60,4+g_90,5.Work as S_2h' and S_2g1 is had for even number, then W in '_j2[p_j2-169,p_j2] In at least partly data meet predetermined condition C₂, work as S_2h' and S_2g' be odd number, then W_j2[p_j2 -169,p_j2In], at least part of data are unsatisfactory for predetermined condition C₂, S_2h' and S_2g1 is had for even in ' The probability of number is 3/4.

In like manner, it is judged that W_i3[p_i3-169,p_i3In], whether at least part of data meet predetermined condition C₃ Mode with judge W_j3[p_j3-169,p_j3In], whether at least part of data meet predetermined condition C₃ Mode identical, in like manner, it is judged that W_j4[p_j4-169,p_j4In], whether at least part of data meet pre- Fixed condition C₄, judge W_j5[p_j5-169,p_j5In], whether at least part of data meet predetermined condition C₅、 Judge W_j6[p_j6-169,p_j6In], whether at least part of data meet predetermined condition C₆, judge W_j7 [p_j7-169,p_j7In], whether at least part of data meet predetermined condition C₇, judge W_j8[p_j8-169, p_j8In], whether at least part of data meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In] extremely Whether small part data meet predetermined condition C₉, judge W_j10[p_j10-169,p_j10In] at least partly Whether data meet predetermined condition C₁₀With judge W_j11[p_j11-169,p_j11At least part of data in] Whether meet predetermined condition C₁₁, do not repeat them here.

As a example by the embodiment shown in Fig. 5, it is provided that one judges window W_iz[p_iz-A_z,p_iz +B_zIn], whether at least part of data meet predetermined condition C_zMethod, in the present embodiment use Random function judges window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet predetermined Condition C_z, according to the rule preset on duplicate removal server 103, for potential cut-point k_iDetermine Point p_i1And p_i1Corresponding window W_i1[p_i1-169,p_i1], it is judged that W_i1[p_i1-169,p_i1In] at least partly Whether data meet predetermined condition C₁, as shown in figure 16, W_i1Represent window W_i1[p_i1-169,p_i1], For judging W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁, select 5 Byte, in Figure 16, the byte " ■ " of serial number 169,127,85,43 and 1 represents selection respectively 1 byte, adjacent two select bytes between differ 42 bytes.By serial number 169, 127, the byte " ■ " of 85,43 and 1 regards 40 positions successively as, is expressed as a₁、a₂、a₃、 a₄…a₄₀。a₁、a₂、a₃、a₄…a₄₀In arbitrary a_t, work as a_tWhen=0, V_at=-1, when a_tWhen=1, V_at=1, according to a_tWith V_atCorresponding relation, generates V_a1、V_a2、V_a3、V_a4…V_a40。 From the random number of Normal Distribution, select 40 randoms number, be expressed as: h₁、h₂、 h₃、h₄...h₄₀。S_a=V_a1*h₁+V_a2*h₂+V_a3*h₃+V_a4*h₄+…+V_a40*h₄₀.Because h₁、 h₂、h₃、h₄...h₄₀Normal Distribution, therefore, S_aAlso Normal Distribution.Work as S_aFor Positive number, then W_i1[p_i1-169,p_i1In], at least part of data meet predetermined condition C₁, work as S_aIt is negative Number or 0, then W_i1[p_i1-169,p_i1In], at least part of data are unsatisfactory for predetermined condition C₁, S_aFor just The probability of number is 1/2.In the embodiment shown in fig. 5, W_i1[p_i1-169,p_i1At least part of data in] Meet predetermined condition C₁.As shown in figure 16,Represent and judge window W_i2[p_i2-169,p_i2In] At least partly whether data meet predetermined condition C₂Time 1 byte selecting respectively, in figure 16, Represent by sequence number 170,128,86,44 and 2 respectively, phase between adjacent two bytes selected Differ from 42 bytes.Byte by sequence number 170,128,86,44 and 2Regard 40 successively as Individual position, is expressed as b₁、b₂、b₃、b₄…b₄₀。b₁、b₂、b₃、b₄…b₄₀In appoint One b_t, work as b_tWhen=0, V_bt=-1, works as b_tWhen=1, V_bt=1, according to b_tWith V_btCorresponding relation, Generate V_b1、V_b2、V_b3、V_b4…V_b40.Judge window W_i1[p_i1-169,p_i1At least partly count in] According to whether meeting predetermined condition C₁Mode with judge window W_i2[p_i2-169,p_i2At least portion in] Whether divided data meets predetermined condition C₂Mode identical, therefore, use identical random number: h₁、h₂、h₃、h₄...h₄₀, S_b=V_b1*h₁+V_b2*h₂+V_b3*h₃+V_b4*h₄+…+V_b40*h₄₀。 Because h₁、h₂、h₃、h₄...h₄₀Normal Distribution, therefore, S_bAlso Normal Distribution. Work as S_bFor positive number, then W_i2[p_i2-169,p_i2In], at least part of data meet predetermined condition C₂, when S_bFor negative or 0, then W_i2[p_i2-169,p_i2In], at least part of data are unsatisfactory for predetermined condition C₂, S_bProbability for positive number is 1/2.In the embodiment shown in fig. 5, W_i2[p_i2-169,p_i2In] at least Part data meet predetermined condition C₂.Use same rule, judge W respectively_i3[p_i3-169,p_i3] In at least partly data whether meet predetermined condition C₃, judge W_i4[p_i4-169,p_i4At least portion in] Whether divided data meets predetermined condition C₄, judge W_i5[p_i5-169,p_i5In], at least part of data are No meet predetermined condition C₅, judge W_i6[p_i6-169,p_i6In], whether at least part of data meet pre- Fixed condition C₆, judge W_i7[p_i7-169,p_i7In], whether at least part of data meet predetermined condition C₇、 Judge W_i8[p_i8-169,p_i8In], whether at least part of data meet predetermined condition C₈, judge W_i9 [p_i9-169,p_i9In], whether at least part of data meet predetermined condition C₉, judge W_i10[p_i10-169, p_i10In], whether at least part of data meet predetermined condition C₁₀With judge W_i11[p_i11-169,p_i11In] At least partly whether data meet predetermined condition C₁₁.In embodiment shown in Fig. 5, W_i5[p_i5 -169,p_i5In], at least part of data are unsatisfactory for predetermined condition C₅, from a p_i5Split along data stream Point search direction 11 bytes of jump, the end position the 11st byte obtains the most potential point Cutpoint k_j, as shown in Figure 6, according to the rule preset on duplicate removal server 103, for potential point Cutpoint k_jDetermine a p_j1, some p_j1Corresponding window W_j1[p_j1-169,p_j1], it is judged that window W_j1[p_j1 -169,p_j1In], whether at least part of data meet predetermined condition C₁Mode with judge window W_i1 [p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁Mode identical, the most such as Shown in Figure 17, W_j1Represent window W_j1[p_j1-169,p_j1], for judging W_j1[p_j1-169,p_j1In] extremely Whether small part data meet predetermined condition C₁, for judging W_j1[p_j1-169,p_j1In] at least partly Whether data meet predetermined condition C₁, select 5 bytes, serial number 169 in Figure 17,127, 85, the byte " ■ " of 43 and 1 represents 1 byte of selection, adjacent two words selected respectively 42 bytes are differed between joint.The byte " ■ " of serial number 169,127,85,43 and 1 is depended on Secondary regard 40 positions as, be expressed as a₁'、a₂'、a₃'、a₄'…a₄₀'。a₁'、a₂'、a₃'、 a₄'…a₄₀Arbitrary a in '_t', work as a_tDuring '=0, V_at'=-1, works as a_tDuring '=1, V_at'=1, according to a_t' and V_at' corresponding relation, generate V_a1'、V_a2'、V_a3'、V_a4'…V_a40'.Judge window W_j1[p_j1 -169,p_j1In], whether at least part of data meet predetermined condition C₁Mode with judge window W_i1 [p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁Mode identical, therefore make Random number with identical: h₁、h₂、h₃、h₄...h₄₀。S_a'=V_a1'*h₁+V_a2'*h₂+V_a3'*h₃ +V_a4'*h₄+…+V_a40'*h₄₀.Because h₁、h₂、h₃、h₄...h₄₀Normal Distribution, because of This, S_a' also Normal Distribution.Work as S_a' for positive number, then W_j1[p_j1-169,p_j1At least portion in] Divided data meets predetermined condition C₁, work as S_a' for negative or 0, then W_j1[p_j1-169,p_j1In] at least Part data are unsatisfactory for predetermined condition C₁, S_a' it is 1/2 for the probability of positive number.

Judge W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 17,Represent and judge window W_j2[p_j2-169,p_j2In] extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select 42 bytes are differed between byte.In fig. 17, respectively by sequence number 170,128,86,44 Represent with 2, between adjacent two bytes selected, differ 42 bytes.By sequence number 170,128, 86, the byte of 44 and 2Regard 40 positions successively as, be expressed as b₁'、b₂'、b₃'、b₄'… b₄₀'。b₁'、b₂'、b₃'、b₄'…b₄₀Arbitrary b in '_t', work as b_tDuring '=0, V_bt'=-1, works as b_t'=1 Time, V_bt'=1, according to b_t' and V_bt' corresponding relation, generate V_b1'、V_b2'、V_b3'、V_b4'…V_b40'。 Judge W_i2[p_i2-169,p_i2In], whether at least part of data meet predetermined condition C₂Mode and Judge W_j2[p_j2-169,p_j2In], whether at least part of data meet predetermined condition C₂Mode phase With, therefore, use identical random number: h₁、h₂、h₃、h₄...h₄₀, S_b'=V_b1'*h₁+V_b2' *h₂+V_b3'*h₃+V_b4'*h₄+…+V_b40'*h₄₀.Because h₁、h₂、h₃、h₄...h₄₀Just obey State is distributed, therefore, and S_b' also Normal Distribution.Work as S_b' for positive number, then W_j2[p_j2-169,p_j2] In at least partly data meet predetermined condition C₂, work as S_b' for negative or 0, then W_j2[p_j2-169,p_j2] In at least partly data be unsatisfactory for predetermined condition C₂, S_b' it is 1/2 for the probability of positive number.

In like manner, it is judged that W_i3[p_i3-169,p_i3In], whether at least part of data meet predetermined condition C₃ Mode with judge W_j3[p_j3-169,p_j3In], whether at least part of data meet predetermined condition C₃ Mode identical, in like manner, it is judged that W_j4[p_j4-169,p_j4In], whether at least part of data meet pre- Fixed condition C₄, judge W_j5[p_j5-169,p_j5In], whether at least part of data meet predetermined condition C₅、 Judge W_j6[p_j6-169,p_j6In], whether at least part of data meet predetermined condition C₆, judge W_j7 [p_j7-169,p_j7In], whether at least part of data meet predetermined condition C₇, judge W_j8[p_j8-169, p_j8In], whether at least part of data meet predetermined condition C₈, judge W_j9[p_j9-169,p_j9In] extremely Whether small part data meet predetermined condition C₉, judge W_j10[p_j10-169,p_j10In] at least partly Whether data meet predetermined condition C₁₀With judge W_j11[p_j11-169,p_j11At least part of data in] Whether meet predetermined condition C₁₁, do not repeat them here.

Still as a example by Fig. 5 illustrated embodiment, it is provided that one judges window W_iz[p_iz-A_z, p_iz+B_zIn], whether at least part of data meet predetermined condition C_zMethod, the present embodiment makes Window W is judged with random function_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet pre- Fixed condition C_z, according to the rule preset on duplicate removal server 103, for potential cut-point k_iReally Fixed point p_i1And p_i1Corresponding window W_i1[p_i1-169,p_i1], it is judged that W_i1[p_i1-169,p_i1At least portion in] Whether divided data meets predetermined condition C₁, as shown in figure 16, W_i1Represent window W_i1[p_i1-169,p_i1], For judging W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁, select 5 Byte, in Figure 16, the byte " ■ " of serial number 169,127,85,43 and 1 represents selection respectively 1 byte, adjacent two select bytes between differ 42 bytes.By serial number 169, 127, the byte " ■ " of 85,43 and 1 is converted into 1 decimal number, and scope is 0-(2^40-1), Using uniform random number maker is that each decimal number in 0-(2^40-1) generates 1 Individual designated value, right between each decimal number and the designated value in record 0-(2^40-1) Should be related to R, once specify, the designated value that this decimal number is corresponding is the most constant, and this designated value takes From being uniformly distributed, if this designated value is even number, then W_i1[p_i1-169,p_i1At least partly count in] According to meeting predetermined condition C₁If this designated value is odd number, then W_i1[p_i1-169,p_i1In] at least Part data are unsatisfactory for predetermined condition C₁, C₁Represent that the designated value obtained according to the method described above is for even Number.Because the probability that equally distributed random number is even number is 1/2, therefore, [p_i1-169,p_i1] In at least partly data meet predetermined condition C₁Probability be 1/2.At the embodiment shown in Fig. 5 In, use same rule, judge W respectively_i2[p_i2-169,p_i2In], whether at least part of data Meet predetermined condition C₂, it is judged that W_i3[p_i3-169,p_i3In], whether at least part of data meet predetermined Condition C₃, judge W_i4[p_i4-169,p_i4In], whether at least part of data meet predetermined condition C₄、 Judge W_i5[p_i5-169,p_i5In], whether at least part of data meet predetermined condition C₅, at this no longer Repeat.

Work as W_i5[p_i5-169,p_i5In], at least part of data are unsatisfactory for predetermined condition C₅, from a p_i5Edge Data flow point cutpoint search direction 11 bytes of jump, the end position the 11st byte obtains Obtain current potential cut-point k_j, as shown in Figure 6, according to the rule preset on duplicate removal server 103 Then, for potential cut-point k_jDetermine a p_j1, some p_j1Corresponding window W_j1[p_j1-169,p_j1], Judge window W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁Side Formula with judge window W_i1[p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁'s Mode is identical, therefore, uses each decimal number in identical 0-(2^40-1) and finger Corresponding relation R between definite value, as shown in figure 17, W_j1Represent window W_j1[p_j1-169,p_j1], For judging W_j1[p_j1-169,p_j1In], whether at least part of data meet predetermined condition C₁, select 5 Individual byte, in Figure 17, " ■ " represents 1 byte selected, adjacent two bytes " ■ " selected Between differ 42 bytes.The byte " ■ " of serial number 169,127,85,43 and 1 is changed Become 1 decimal number, search, at R, the designated value that this decimal number is corresponding, if this designated value is Even number, then W_j1[p_j1-169,p_j1In], at least part of data meet predetermined condition C₁If this refers to Definite value is odd number, then W_j1[p_j1-169,p_j1In], at least part of data are unsatisfactory for predetermined condition C₁, Because the probability that equally distributed random number is even number is 1/2, therefore, W_j1[p_j1-169,p_j1] In at least partly data meet predetermined condition C₁Probability be 1/2.In like manner, it is judged that W_i2[p_i2-169, p_i2In], whether at least part of data meet predetermined condition C₂Mode and judge W_j2[p_j2-169, p_j2In], whether at least part of data meet predetermined condition C₂Mode identical, it is judged that W_i3[p_i3 -169,p_i3In], whether at least part of data meet predetermined condition C₃Mode with judge W_j3[p_j3 -169,p_j3In], whether at least part of data meet predetermined condition C₃Mode identical, in like manner, sentence Disconnected W_j4[p_j4-169,p_j4In], whether at least part of data meet predetermined condition C₄, judge W_j5[p_j5 -169,p_j5In], whether at least part of data meet predetermined condition C₅, judge W_j6[p_j6-169,p_j6] In at least partly data whether meet predetermined condition C₆, judge W_j7[p_j7-169,p_j7In] at least Whether part data meet predetermined condition C₇, judge W_j8[p_j8-169,p_j8At least partly count in] According to whether meeting predetermined condition C₈, judge W_j9[p_j9-169,p_j9In], whether at least part of data Meet predetermined condition C₉, judge W_j10[p_j10-169,p_j10In], whether at least part of data meet pre- Fixed condition C₁₀With judge W_j11[p_j11-169,p_j11In], whether at least part of data meet predetermined bar Part C₁₁, do not repeat them here.

Duplicate removal server 103 in the embodiment of the present invention shown in Fig. 1, refers to realize this The device of the technical scheme described by bright embodiment, as shown in figure 18, generally includes central authorities' process Unit, main storage and input/output interface.CPU, main storage and input The intercommunication of output interface, main memory store executable instruction, CPU is held The executable instruction of storage in row main storage, thus perform specific function, as the present invention is real Execute the lookup data flow point cutpoint described by illustration 4 to Figure 17.Therefore, as shown in figure 19, root According to the embodiment of the present invention shown in Fig. 4 to Figure 17, duplicate removal server 103, at duplicate removal server 103 On be preset with rule, described rule is: for potential cut-point k determine M some p_x, some p_xRight The window W answered_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] corresponding predetermined condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；Duplicate removal takes Business device 103 includes determining unit 1901 and judging processing unit 1902.Wherein it is determined that unit 1901, For being used for performing step a): be a) current potential cut-point k according to described rule_iDetermine a p_iz And described some p_izCorresponding window W_iz[p_iz-A_z,p_iz+B_z], i and z is integer, and 1≤z ≤M；Judge processing unit 1902, for described window W_iz[p_iz-A_z,p_iz+B_zAt least portion in] Whether divided data meets predetermined condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined Condition C_z, from described some p_izAlong the described data flow point cutpoint search direction N number of data flow point of jump Cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), Obtain new potential cut-point, the most described determine that unit is that described new potential cut-point performs step A)；As described current potential cut-point k_iM window in each window W_ix[p_ix- A_x,p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential segmentation Point k_iFor data flow point cutpoint.

Further, described rule also includes: at least two point p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f.Further, described rule also includes: described at least two point p_eWith p_f, relative to described potential cut-point k, search in the reverse direction at described data flow point cutpoint.

Further, described rule also includes: described at least two point p_eAnd p_fBetween distance It is 1 U.

Further, described judgement processing unit 1902 is specifically for using random function to judge institute State window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z.Specifically, described judgement processing unit 1902 is described specifically for using hash function to judge Window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z。 Specifically, described judgement processing unit 1902 is specifically for using random function to judge described window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z, specifically Including:

At described window W_iz[p_iz-A_z,p_iz+B_zF byte is selected, by described F byte in] Recycling H time, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[p_iz-A_z, p_iz+B_zIn], at least part of data meet described predetermined condition C_z。

Further, described judgement processing unit 1902 is for as described window W_iz[p_iz-A_z,p_iz+ B_zIn], at least part of data are unsatisfactory for described predetermined condition C_z, from described some p_izAlong described data Flow point cutpoint search direction jump N number of data flow point cutpoint minimum search unit U, it is thus achieved that described newly Potential cut-point, described determine that unit 1901 is that described new potential cut-point performs step a), According to described rule, the some p determined for described new potential cut-point_icCorresponding window W_ic [p_ic-A_c,p_ic+B_c] left margin and described window W_iz[p_iz-A_z,p_iz+B_z] right margin overlap Or the described window W determined for described new potential cut-point_ic[p_ic-A_c,p_ic+B_c] a left side Border is positioned at described window W_iz[p_iz-A_z,p_iz+B_zWithin the scope of]；Wherein, for described new diving At the described window W that cut-point determines_ic[p_ic-A_c,p_ic+B_c] it is according to described rule, for described M the point that new potential cut-point determines is arranged according in the sequence of data stream search direction acquisition The point of sequence first.

According to shown in Fig. 4 to Figure 17 the embodiment of the present invention provide based on whois lookup data In the method for flow point cutpoint, for potential cut-point k_iDetermine a p_ixAnd some p_ixWindow W_ix[p_ix- A_x, p_ix+B_x], wherein, x is respectively 1 and arrives M continuous print natural number, M >=2, can sentence parallel In disconnected M window, in each window, at least partly whether data meet predetermined condition C_x, or Judge in window, whether at least part of data meet predetermined condition successively, it is also possible to judge window W_i1[p_i1-A₁, p_i1+B₁In], at least part of data meet predetermined condition C₁Time, then judge W_i2 [p_i2-A₂, p_i2+B₂In], at least part of data meet predetermined condition C₂Time, until judging W_im [p_im-A_m, p_im+B_mIn], at least part of data meet predetermined condition C_m.Other windows in embodiment The judgement of mouth is identical with this, repeats no more.

It addition, according to the embodiment of the present invention shown according to Fig. 4 to Figure 17, in actual application, Being preset with rule on duplicate removal server 103, described rule is: determine M for potential cut-point k Individual some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] Corresponding predetermined condition C_x, x is respectively 1 and arrives M continuous print natural number, M >=2, presets rule at this In then, A₁、A₂、A₃…A_mCan not be the most equal, B₁、B₂、B₃…B_mCan not be complete Portion is equal, C₁、C₂、C₃…C_MCan not also be the most identical.At the embodiment shown in Fig. 5 In, at window W_i1[p_i1-169,p_i1]、W_i2[p_i2-169,p_i2]、W_i3[p_i3-169,p_i3]、W_i4 [p_i4-169,p_i4]、W_i5[p_i5-169,p_i5]、W_i6[p_i6-169,p_i6]、W_i7[p_i7-169,p_i7]、 W_i8[p_i8-169,p_i8]、W_i9[p_i9-169,p_i9]、W_i10[p_i10-169,p_i10] and W_i11[p_i11-169, p_i11In], each window size is identical, i.e. window size is 169 bytes, judges in window simultaneously The mode whether at least part of data meet predetermined condition is the most identical, is specifically shown in above-mentioned judgement W_i1 [p_i1-169,p_i1In], whether at least part of data meet predetermined condition C₁Description, but at Figure 11 In shown embodiment, W_i1[p_i1-169,p_i1]、W_i2[p_i2-169,p_i2]、W_i3[p_i3-169, p_i3]、W_i4[p_i4-169,p_i4]、W_i5[p_i5-169,p_i5]、W_i6[p_i6-169,p_i6]、W_i7[p_i7-169, p_i7]、W_i8[p_i8-169,p_i8]、W_i9[p_i9-169,p_i9]、W_i10[p_i10-169,p_i10] and W_i11[p_i11 -182,p_i11] window size can differ, and judges that in window, at least part of data are the fullest simultaneously The mode of foot predetermined condition can also differ.In all embodiments, according in duplicate removal service The rule preset on device 103, it is judged that window W_i1In at least partly data whether meet predetermined condition C₁Mode with judge window W_j1In at least partly data whether meet predetermined condition C₁Mode Inevitable identical, it is judged that W_i2In at least partly data whether meet predetermined condition C₂Mode with sentence Disconnected W_j2In at least partly data whether meet predetermined condition C₂Mode inevitable the most identical ... judge window Mouth W_iMIn at least partly data whether meet predetermined condition C_MMode with judge window W_jMIn At least partly whether data meet predetermined condition C_MMode inevitable the most identical.Do not repeat them here, Simultaneously according to the embodiment of the present invention shown in Fig. 4 to Figure 17, although all as a example by M=11, but root According to being actually needed, the value of M is not limited to 11, and those skilled in the art implement according to the present invention Description in example, determines the value of M.

According to the embodiment of the present invention shown in Fig. 4 to Figure 17, duplicate removal server 103 is preset with Rule, k_a、k_i、k_j、k_lAnd k_mFor searching cut-point along data flow point cutpoint search direction Time obtain potential cut-point, k_a、k_i、k_j、k_lAnd k_mAll according to this rule.The present invention is real Execute the window W in example_x[p_x-A_x,p_x+B_x] represent a particular range, select at this particular range Select data to judge whether these data meet predetermined condition C_x, specifically, can be specific at this In the range of select part data, it is also possible to select total data to judge whether these data meet Predetermined condition C_x.Window concept specifically used in the embodiment of the present invention can refer to window W_x[p_x -A_x,p_x+B_x], do not repeat them here.

According to the embodiment of the present invention shown in Fig. 4 to Figure 17, window W_x[p_x-A_x,p_x+B_xIn], (p_x-A_x) and (p_x+B_x) represent this window W_x[p_x-A_x,p_x+B_x] two borders, Wherein (p_x-A_x) represent window W_x[p_x-A_x,p_x+B_x] relative to a p_xIt is positioned at data flow point Cutpoint searches reciprocal border, (p_x+B_x) represent window W_x[p_x-A_x,p_x+B_x] relatively In a p_xIt is positioned at the border of data flow point cutpoint search direction.Specifically, in the embodiment of the present invention In, it is from left to right in the data flow point cutpoint search direction shown in Fig. 3 to Figure 15, wherein (p_x-A_x) represent window W_x[p_x-A_x,p_x+B_x] relative to a p_xIt is positioned at data flow point cutpoint Search reciprocal border (i.e. left margin), (p_x+B_x) represent window W_x[p_x-A_x,p_x +B_x] relative to a p_xIt is positioned at the border (i.e. right margin) of data flow point cutpoint search direction. If being from right to left in the data flow point cutpoint search direction shown in Fig. 3 to Figure 15, wherein (p_x-A_x) represent window W_x[p_x-A_x,p_x+B_x] relative to a p_xIt is positioned at data flow point cutpoint Search reciprocal border (i.e. right margin), (p_x+B_x) represent window W_x[p_x-A_x,p_x +B_x] relative to a p_xIt is positioned at the border (i.e. left margin) of data flow point cutpoint search direction.

Those of ordinary skill in the art are it is to be appreciated that combine respectively showing of embodiment of the present invention description The unit of example and algorithm steps, the key feature of the embodiment of the present invention can be tied mutually with other technologies Close, present with increasingly complex form, but still the key feature of the present invention can be comprised.Truly May use standby cut-point in environment, such as one embodiment is, according in duplicate removal service The rule preset on device 103, for potential cut-point k_iDetermine 11 some p_x, x is 1 to 11 continuous Natural number, determine p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+ B_x] corresponding predetermined condition C_x, as each window W in 11 windows_x[p_x-A_x,p_x+B_x] In at least partly data be satisfied by predetermined condition C_x, the most potential cut-point k_iFor data flow point cutpoint, When exceeding the maximum data block of setting, do not find cut-point yet, at this moment may use standby Preset rules, standby preset rules with on duplicate removal server 103 preset rule similar, Standby preset rules is: the most potential cut-point k_iDetermine 10 some p_x, x is 1 to 10 Continuous print natural number, determines p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x, p_x+B_x] corresponding predetermined condition C_x, as each window W in 10 windows_x[p_x-A_x,p_x+ B_xIn], at least part of data are satisfied by predetermined condition C_x, the most potential cut-point k_iFor data flow point Cutpoint, when exceeding the maximum data block of setting, when not finding data flow point cutpoint yet, from The end position of maximum data block is as force-splitting point.

Duplicate removal server 103 is preset with rule, described rule is potential cut-point k Determine M point, be not necessarily to first there is a potential cut-point k, can be by really M fixed point judges potential cut-point k.

The embodiment of the present invention provides a kind of side based on duplicate removal whois lookup data flow point cutpoint Method, as shown in figure 20, including:

Being preset with rule on duplicate removal server 103, described rule is: true for potential cut-point k Determine M window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, Wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；Shown in Fig. 3 Embodiment in, about the value of M, one of which implementation, M*U value is not more than Ultimate range between two the adjacent data flow point cutpoints preset, the data block i.e. preset is Long length.Judge window W_z[k-A_z, k+B_zIn], whether at least part of data meet predetermined bar Part C_z, wherein, z is integer, 1≤z≤M, (k-A_z) and (k+B_z) represent window respectively W_zTwo borders.When judging any one window W_z[k-A_z, k+B_zAt least partly count in] According to being unsatisfactory for predetermined condition C_z, then jump along data flow point cutpoint search direction from potential cut-point k Jump N number of byte, N≤‖ B_z‖+max_x(‖A_x‖).Wherein, ‖ B_z‖ represents W_z[k-A_z, k+B_zB in]_zAbsolute value, max_x(‖A_x‖) represent A in M window_xIn absolute value Big value, will specifically introduce the principle of N value in embodiment below.When judging in M window Each window W_x[k-A_x,k+B_xIn], at least part of data meet predetermined condition C_x, then dive It is data flow point cutpoints at cut-point k.

Specifically, to current potential cut-point k_i, according to described rule, perform following steps:

Step 2001: be current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz [k_i-A_z,k_i+B_z], i and z is integer, and 1≤z≤M；

Step 2002: judge described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are the fullest Foot predetermined condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described current potential cut-point k_iAlong described data flow point cutpoint search direction jump N Individual data flow point cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖), Obtain new potential cut-point, perform step 2001；

As described current potential cut-point k_iM window in each window W_ix[k_i-A_x,k_i +B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor number According to flow point cutpoint.

Further, described rule also includes: at least two window W_ie[k_i-A_e,k_i+B_e] and W_if [k_i-A_f,k_i+B_f], meet condition: | A_e+B_e|=| A_f+B_f|, C_e=C_f；Further, institute State rule also to include: A_eAnd A_fFor positive integer；Further, described rule also includes: A_e-1= A_f, B_e+ 1=B_f.Wherein, | A_e+B_e| represent window W_ieSize, | A_f+B_f| represent window W_ifSize.

Further, described window W is judged_iz[k_i-A_z,k_i+B_zIn], whether at least part of data Meet described predetermined condition C_z, specifically include: use random function to judge described window W_iz[k_i- A_z,k_i+B_zIn], whether at least part of data meet described predetermined condition C_z；Further, institute State use random function and judge described window W_iz[k_i-A_z,k_i+B_zIn], whether at least part of data Meet described predetermined condition C_z, it is specially and uses hash function to judge described window W_iz[k_i-A_z,k_i +B_zIn], whether at least part of data meet described predetermined condition C_z。

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined Condition C_z, from described current potential cut-point k_iJump along described data flow point cutpoint search direction N number of data flow point cutpoint minimum searches unit U, it is thus achieved that described new potential cut-point, according to institute State rule, the window W determined for described new potential cut-point_ic[k_i-A_c,k_i+B_c] the left side Boundary and described window W_iz[k_i-A_z,k_i+B_z] right margin overlap or be described newly potential The described window W that cut-point determines_ic[k_i-A_c,k_i+B_c] left margin be positioned at described window W_iz [k_i-A_z,k_i+B_zWithin the scope of]；Wherein, described in determining for described new potential cut-point Window W_ic[k_i-A_c,k_i+B_c] it is according to described rule, determine for described new potential cut-point The sequence that obtains according to data stream search direction of M window in sort first window.

The embodiment of the present invention at least partly counts in some window in M window by judging According to whether meeting predetermined condition, search data flow point cutpoint, when at least portion in some window Divided data is unsatisfactory for predetermined condition, then skip N*U length, and wherein, N*U is not more than ‖ B_z‖+max_x(‖A_x‖), it is thus achieved that next potential cut-point, improve data flow point cutpoint Search efficiency.

During data de-duplication, for ensureing that data block size is uniform, average can be considered According to block (being referred to as average piecemeal) size, i.e. meeting minimum data block size and maximum While data block size limits, can determine whether average data block size, to ensure the data obtained Block size is uniform.Window W_x[k-A_x, k+B_x] number M and window W_x[k-A_x, k+B_x] In at least partly data meet pre-conditioned probability the two factor and determine and find data stream The probability (representing with P (n)) of cut-point, the former affects the length of jump, and the latter affects jump Probability, the two joint effect average mark block size.It is said that in general, it is solid at average mark block size Regularly, W_x[k-A_x, k+B_x] number increase, then single window W_x[k-A_x, k+B_xIn] extremely Small part data meet the probability of predetermined condition also to be increased, such as pre-on duplicate removal server 103 Being provided with rule, described rule is: determine 11 window W for potential cut-point k_x[k-A_x, k+B_x], X is respectively 1 to 11 continuous print natural numbers, any one window W in 11 windows_x[k-A_x, k+B_x] In at least partly data meet pre-conditioned probability is 1/2.And it is pre-on duplicate removal server 103 If another group rule be: determine 24 window W for potential cut-point k_x[k-A_x, k+B_x], X is respectively 1 to 24 continuous print natural numbers, any one window W in 24 windows_x[k-A_x, k+B_x] In at least partly data meet pre-conditioned probability 3/4, concrete window W_x[k-A_x, k+B_x] In at least partly data meet pre-conditioned probability and set can be found in and judge window W_x[k-A_x, k+B_xIn], whether at least part of data meet the description of pre-conditioned part.Window W_x[k-A_x, k+B_x] number M and window W_x[k-A_x, k+B_xThe default bar that in], at least part of data meet The probability the two factor of part determines P (n), and P (n) represents: from data stream original position or Search after n data flow point cutpoint minimum searches unit from a upper data flow point cutpoint and do not find number Probability according to flow point cutpoint.The calculating process of P (n) is determined, actually about the two factor Step-length Fibonacci ordered series of numbers more than, after will be described in detail.After obtaining P (n), 1-P (n) Being the distribution function of data flow point cutpoint, (1-P (n))-(1-P (n-1))=P (n-1)-P (n) is N data flow point cutpoint minimum is searched unit and is found data flow point cutpoint probability, namely data The density function of flow point cutpoint, the density function according to data flow point cutpoint just can be with integrationThus try to achieve the desired length of data flow point cutpoint, i.e. average mark Block size, wherein, 4*1024 (byte) represents minimum data block length, 12*1024 (byte) Represent maximum data block length.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, the embodiment party shown in Figure 21 In formula, being preset with rule on duplicate removal server 103, described rule is: for potential cut-point k Determine 11 window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, wherein, x is 1 to 11 continuous print natural numbers, A_xAnd B_xFor integer.Wherein, A₁=169, B₁ =0；A₂=170, B₂=-1；A₃=171, B₃=-2；A₄=172, B₄=-3；A₅=173, B₅=-4；A₆=174, B₆=-5；A₇=175, B₇=-6；A₈=176, B₈=-7；A₉=177, B₉=-8；A₁₀=178, B₁₀=-9；A₁₁=179, B₁₁=-10, and C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁, then 11 windows It is respectively W₁[k-169,k]、W₂[k-170,k-1]、W₃[k-171,k-2]、W₄[k-172,k-3]、 W₅[k-173,k-4]、W₆[k-174,k-5]、W₇[k-175,k-6]、W₈[k-176,k-7]、W₉[k-177, k-8]、W₁₀[k-178, k-9] and W₁₁[k-179,k-10]。k_aFor data flow point cutpoint, Tu21Zhong Shown data flow point cutpoint search direction is from left to right, from data flow point cutpoint k_aSkip minimum After data block 4KB, minimum data block 4KB end position is as next potential cut-point k_i, According to the rule preset for duplicate removal server 103, for potential cut-point k_iDetermine window W_ix[k_i- A_x,k_i+B_x], in the present embodiment, x is respectively 1 to 11 continuous print natural numbers.Shown in Figure 21 Embodiment in, for potential cut-point k_iThe window determined is 11, respectively W_i1[k_i-169, k_i]、W_i2[k_i-170,k_i-1]、W_i3[k_i-171,k_i-2]、W_i4[k_i-172,k_i-3]、W_i5[k_i-173, k_i-4]、W_i6[k_i-174,k_i-5]、W_i7[k_i-175,k_i-6]、W_i8[k_i-176,k_i-7]、W_i9[k_i-177, k_i-8]、W_i10[k_i-178,k_i-9] and W_i11[k_i-179,k_i-10].Judge W_i1[k_i-169,k_iIn] extremely Whether small part data meet predetermined condition C₁, judge W_i2[k_i-170,k_i-1] at least partly count in According to whether meeting predetermined condition C₂, judge W_i3[k_i-171,k_i-2] in, at least part of data are the fullest Foot predetermined condition C₃, judge W_i4[k_i-172,k_i-3] in, whether at least part of data meet predetermined bar Part C₄, judge W_i5[k_i-173,k_i-4] in, whether at least part of data meet predetermined condition C₅, sentence Disconnected W_i6[k_i-174,k_i-5] in, whether at least part of data meet predetermined condition C₆, judge W_i7[k_i -175,k_i-6] in, whether at least part of data meet predetermined condition C₇, judge W_i8[k_i-176,k_i-7] In at least partly data whether meet predetermined condition C₈, judge W_i9[k_i-177,k_i-8] at least portion in Whether divided data meets predetermined condition C₉, judge W_i10[k_i-178,k_i-9] in, at least part of data are No meet predetermined condition C₁₀With judge W_i11[k_i-179,k_i-10] in, at least part of data are the fullest Foot predetermined condition C₁₁.When judging window W_i1In at least partly data meet predetermined condition C₁, window Mouth W_i2In at least partly data meet predetermined condition C₂, window W_i3In at least partly data meet Predetermined condition C₃, window W_i4In at least partly data meet predetermined condition C₄, window W_i5In extremely Small part data meet predetermined condition C₅, window W_i6In at least partly data meet predetermined condition C₆, window W_i7In at least partly data meet predetermined condition C₇, window W_i8In at least partly count According to meeting predetermined condition C₈, window W_i9In at least partly data meet predetermined condition C₉, window W_i10In at least partly data meet predetermined condition C₁₀With window W_i11In at least partly data meet Predetermined condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint.When in 11 windows When in any one window, at least part of data are unsatisfactory for the predetermined condition of correspondence, as shown in figure 22, W_i5[k_i-173,k_i-4], then from potential cut-point k_iJump along data flow point cutpoint search direction N number of byte, the most N number of byte is not more than ‖ B₅‖+max_x(‖A_x‖), shown in Figure 22 In embodiment, N number of byte of jumping is not more than 183 bytes, in the present embodiment, N=7, Obtain new potential cut-point, for potential cut-point k_iDifference, here by new potential segmentation Point is expressed as k_j.According in the embodiment shown in Figure 21, duplicate removal server 103 is preset Regular, described rule is: for potential cut-point k_jDetermine window W_jx[k_j-A_x,k_j+B_x], In the present embodiment, x is respectively 1 to 11 continuous print natural numbers.For potential cut-point k_jDetermine Window be 11, respectively W_j1[k_j-169,k_j]、W_j2[k_j-170,k_j-1]、W_j3[k_j-171, k_j-2]、W_j4[k_j-172,k_j-3]、W_j5[k_j-173,k_j-4]、W_j6[k_j-174,k_j-5]、W_j7[k_j -175,k_j-6]、W_j8[k_j-176,k_j-7]、W_j9[k_j-177,k_j-8]、W_j10[k_j-178,k_j-9] and W_j11 [k_j-179,k_j-10].As shown in figure 22, the 11st the window W determined for potential cut-point_j11[k_j -179,k_j-10], potential cut-point k is being ensured_iWith potential cut-point k_jBetween scope all sentencing Within the scope of Duan, the most in the present embodiment, it is necessary to assure window W_j11[k_j-179,k_j-10] Left margin and window W_i5[k_i-173,k_i-4] right margin (k_i-4) overlap, or be positioned at window W_i5 [k_i-173,k_i-4] within the scope of, described window W_j11[k_j-179,k_j-10] it is according to described rule, For described potential cut-point k_jThe sequence that M the window determined obtains according to data stream search direction The window of sequence first in row.Therefore, in this restriction, as window W_i5[k_i-173,k_i-4] In at least partly data be unsatisfactory for predetermined condition C₅, from potential cut-point k_iAlong data flow point cutpoint The distance that search direction is jumped is not more than ‖ B₅‖+max_x(‖A_x‖).Judge W_j1[k_j-169, k_jIn], whether at least part of data meet predetermined condition C₁, judge W_j2[k_j-170,k_j-1] in extremely Whether small part data meet predetermined condition C₂, judge W_j3[k_j-171,k_j-2] at least partly count in According to whether meeting predetermined condition C₃, judge W_j4[k_j-172,k_j-3] in, at least part of data are the fullest Foot predetermined condition C₄, judge W_j5[k_j-173,k_j-4] in, whether at least part of data meet predetermined bar Part C₅, judge W_j6[k_j-174,k_j-5] in, whether at least part of data meet predetermined condition C₆, sentence Disconnected W_j7[k_j-175,k_j-6] in, whether at least part of data meet predetermined condition C₇, judge W_j8[k_j -176,k_j-7] in, whether at least part of data meet predetermined condition C₈, judge W_j9[k_j-177,k_j-8] In at least partly data whether meet predetermined condition C₉, judge W_j10[k_j-178,k_j-9] at least Whether part data meet predetermined condition C₁₀With judge W_j11[k_j-179,k_j-10] at least partly Whether data meet predetermined condition C₁₁.When judging window W_j1In at least partly data meet predetermined Condition C₁, window W_j2In at least partly data meet predetermined condition C₂, window W_j3In at least portion Divided data meets predetermined condition C₃, window W_j4In at least partly data meet predetermined condition C₄、 Window W_j5In at least partly data meet predetermined condition C₅, window W_j6In at least partly data full Foot predetermined condition C₆, window W_j7In at least partly data meet predetermined condition C₇, window W_j8In At least partly data meet predetermined condition C₈, window W_j9In at least partly data meet predetermined bar Part C₉, window W_j10In at least partly data meet predetermined condition C₁₀With window W_j11In at least portion Divided data meets predetermined condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint, k_j With k_aBetween data constitute 1 data block, simultaneously according to k_aIdentical mode skips minimum Piecemeal size 4KB, it is thus achieved that next potential cut-point, and according on duplicate removal server 103 The rule preset, it is judged that whether next potential cut-point is data flow point cutpoints.Latent when judging At cut-point k_jWhen not being data flow point cutpoint, according to k_iIdentical mode obtains next latent At cut-point, and according under the rule preset on duplicate removal server 103 and said method judgement Whether one potential cut-point is data flow point cutpoints.When exceeding the maximum data block of setting still When not finding data flow point cutpoint, then from the end position of maximum data block as force-splitting Point.

In embodiment as shown in figure 21, according to the rule preset on duplicate removal server 103 Then, from judging W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁Start, When judging W_i1[k_i-169,k_i]、W_i2[k_i-170,k_i-1]、W_i3[k_i-171,k_i-2] and W_i4[k_i-172, k_i-3] in, at least part of data, at least part of data meet predetermined condition C respectively₁、C₂、C₃With C₄, it is judged that W_i5[k_i-173,k_i-4] in, at least part of data are unsatisfactory for predetermined condition C₅Time, from latent At cut-point k_iAlong data flow point cutpoint search direction 6 bytes of jump, the 6th byte End position obtains new potential cut-point, for distinguishing with other potential cut-points, shown herein as For k_g, according to the rule preset on duplicate removal server 103, for potential cut-point k_gDetermine 11 Individual window, respectively W_g1[k_g-169,k_g]、W_g2[k_g-170,k_g-1]、W_g3[k_g-171,k_g-2]、 W_g4[k_g-172,k_g-3]、W_g5[k_g-173,k_g-4]、W_g6[k_g-174,k_g-5]、W_g7[k_g-175,k_g -6]、W_g8[k_g-176,k_g-7]、W_g9[k_g-177,k_g-8]、W_g10[k_g-178,k_g-9] and W_g11[k_g -179,k_g-10].Judge W_g1[k_g-169,k_gIn], whether at least part of data meet predetermined condition C₁、 Judge W_g2[k_g-170,k_g-1] in, whether at least part of data meet predetermined condition C₂, judge W_g3 [k_g-171,k_g-2] in, whether at least part of data meet predetermined condition C₃, judge W_g4[k_g-172, k_g-3] in, whether at least part of data meet predetermined condition C₄, judge W_g5[k_g-173,k_g-4] in At least partly whether data meet predetermined condition C₅, judge W_g6[k_g-174,k_g-5] at least portion in Whether divided data meets predetermined condition C₆, judge W_g7[k_g-175,k_g-6] at least part of data in Whether meet predetermined condition C₇, judge W_g8[k_g-176,k_g-7] in, at least part of data are the fullest Foot predetermined condition C₈, judge W_g9[k_g-177,k_g-8] in, whether at least part of data meet predetermined Condition C₉, judge W_g10[k_g-178,k_g-9] in, whether at least part of data meet predetermined condition C₁₀ With judge W_g11[k_g-179,k_g-10] in, whether at least part of data meet predetermined condition C₁₁.Window W_g11[k_g-179,k_g-10] with window W_i5[k_i-173,k_i-4] overlap, and C₅=C₁₁, therefore, When judging W_i5[k_i-173,k_i-4] in, at least part of data are unsatisfactory for predetermined condition C₅Time, from potential Cut-point k_iAlong data flow point cutpoint T byte of search direction jump, it is thus achieved that potential segmentation Point k_gStill the condition as data flow point cutpoint is not met.Therefore, if from potential cut-point k_iDouble counting can be there is along data flow point cutpoint search direction 6 bytes of jumping, therefore, From potential cut-point k_iWeight can be reduced along data flow point cutpoint search direction 7 bytes of jump Multiple calculating, in hgher efficiency.Therefore improve the speed searching data flow point cutpoint.When default rule Window W in fixed_x[k-A_x,k+B_xIn], at least part of data meet predetermined condition C_xProbability be When 1/2, i other words perform jump with the probability of 1/2, the most at most can jump ‖ B₁₁‖+‖ A₁₁‖=189 byte.

In the present embodiment, pre-defined rule is: determine 11 window W for potential cut-point k_x [k-A_x,k+B_x] and window W_x[k-A_x,k+B_xIn], at least part of data meet pre-conditioned C_x, Wherein W_x[k-A_x,k+B_xIn], at least part of data meet pre-conditioned C_xProbability be 1/2, x It is respectively 1 to 11 continuous print natural number and A_xAnd B_xFor integer.Wherein, A₁=169, B₁=0； A₂=170, B₂=-1；A₃=171, B₃=-2；A₄=172, B₄=-3；A₅=173, B₅=-4；A₆=174, B₆=-5； A₇=175, B₇=-6；A₈=176, B₈=-7；A₉=177, B₉=-8；A₁₀=178, B₁₀=-9；A₁₁=179, B₁₁=-10, and C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁.It is potential point Cutpoint k selects 11 windows, and is continuous 11 windows, can be counted by the two factor Calculate P (n).The selection mode of 11 windows and judge in each window in 11 windows at least Part data meet predetermined condition C_xFollow the rule preset on duplicate removal server 103, therefore Whether there are in continuous 11 windows at least part of data in each window and meet predetermined condition C_xJust determine whether potential cut-point k is data flow point cutpoints.We claim between two bytes Gap is a point.P (n) represents: there is not continuous print 11 in n window of continuous print full , the most there is not the probability of data flow point cutpoint in the probability of the window of foot condition.From file header/ One cut-point jumps after minimum piecemeal size 4KB, searches opposite direction rollback to data flow point cutpoint 10 bytes, find the 4086th point, the most there is not data flow point cutpoint, so P (4086)=1, the like, P (4087)=1 ... P (4095)=1.The 4096th At individual point, i.e. at minimum piecemeal size, with every in these 11 windows of probability of (1/2) ^11 In one window, at least part of data meet predetermined condition C_x, therefore with the probability of (1/2) ^11 There is data flow point cutpoint, there is not data flow point cutpoint, institute with the probability of 1-(1/2) ^11 With P (4096)=1-(1/2) ^11.

At the n-th window, 12 kinds of situations can be divided into carry out recursion P (n).

In situation 1: the n-th window, at least part of data are unsatisfactory for predetermined condition with the probability of 1/2, Now there is not continuous print 11 with the probability of P (n-1) in n-1 window before the n-th window In window, at least part of data of each window are satisfied by predetermined condition, and therefore P (n) comprises 1/2* P(n-1).In n-th window, at least part of data are unsatisfactory for predetermined condition, and while n-th There are at least part of data in 11 each windows of window of continuous print in some n-1 window above The situation being satisfied by predetermined condition is unrelated with P (n).

In situation 2: the n-th window, at least part of data meet predetermined condition with the probability of 1/2, In (n-1)th window, at least part of data are unsatisfactory for predetermined condition with the probability of 1/2, and now (n-1)th N-2 window before individual window does not exist in 11 windows of continuous print with the probability of P (n-2) In each window, at least part of data are satisfied by predetermined condition, and therefore P (n) comprises 1/2*1/2*P (n-2).In n-th window, at least part of data meet predetermined condition, in (n-1)th some window N-2 the window that at least partly data are unsatisfactory for before predetermined condition, and (n-1)th window is deposited In 11 windows of continuous print, at least part of data of each window meet the situation of predetermined condition Unrelated with P (n).

According to foregoing description, in 11: the n-th to n-9 window of situation, at least part of data are with (1/2) The probability of ^10 meets predetermined condition, in the (n-1)th 0 windows at least partly data with 1/2 probability Being unsatisfactory for predetermined condition, now n-11 window before the (n-1)th 0 windows is with P's (n-11) There are not in 11 windows of continuous print at least part of data in each window and be satisfied by pre-in probability Fixed condition, therefore P (n) comprises (1/2) ^10*1/2*P (n-11).The n-th to n-9 window In Kou, at least part of data are satisfied by predetermined condition, and in the (n-1)th 0 windows, at least partly data are not Meet predetermined condition, and n-11 window before the (n-1)th 0 windows exists continuous print 11 In window, in each window, at least part of data are satisfied by situation and P (n) nothing of predetermined condition Close.

In the window that situation is 12: the n-th to n-10, at least part of data are with the probability of (1/2) ^11 Meeting predetermined condition, this situation is unrelated with P (n).

Therefore, P (n)=1/2*P (n-1)+(1/2) ^2*P (n-2)+...+(1/2) ^11*P(n-11).Another kind of preset rules: determine 24 window W for potential cut-point k_x[k -A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, wherein, x is 1 to 11 Continuous print natural number, A₁=169, B₁=0；A₂=170, B₂=-1；A₃=171, B₃=-2；A₄=172, B₄ =-3；A₅=173, B₅=-4；A₆=174, B₆=-5；A₇=175, B₇=-6；A₈=176, B₈=-7；A₉=177, B₉=-8；A₁₀=178, B₁₀=-9；A₁₁=179, B₁₁=-10 ... A₂₄=192, B₂₄=-23, and C₁=C₂ =C₃=C₄=C₅=C₆=C₇=C₈=C₉=...=C₂₄, window W_x[k-A_x,k+B_xIn] at least partly Data meet predetermined condition C_xProbability be 3/4, P (n) can be calculated by the two factor.

The most whether there are at least part of data in each window in continuous 24 windows equal Meet predetermined condition C_xJust determine whether potential cut-point k is data flow point cutpoints, can pass through Equation below calculates:

P (1)=1, P (2) ... P (23)=1, P (24)=1-(3/4) ^24,

P (n)=1/4*P (n-1)+1/4* (3/4) * P (n-2)+...+1/4* (3/4) ^23*P(n-24)。

Through calculating, P (5*1024)=0.78, P (11*1024)=0.17, P (12*1024)=0.13, I.e. from data stream original position/a data flow point cutpoint find after 12KB the probability with 13% Do not find data flow point cutpoint yet, force to split.By this probability, try to achieve data stream The density function of cut-point, through integration try to achieve about averagely from data stream original position/on One data flow point cutpoint finds data flow point cutpoint when searching 7.6KB, i.e. average mark block length is big It is about 7.6KB.At least part of data meet predetermined with the probability of 1/2 with 11 windows of continuous print Condition is different, when tradition CDC algorithm uses a window to meet condition with the probability of 1/2^12, The effect of average mark block length 7.6KB can be reached.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, the embodiment party shown in Figure 23 In formula, being preset with rule on duplicate removal server 103, described rule is: for potential cut-point k Determine 11 window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, wherein, x is 1 to 11 continuous print natural numbers, A_xAnd B_xFor integer.Wherein, window W_x [k-A_x,k+B_xIn], at least part of data meet predetermined condition C_xProbability be 1/2, A₁=171, B₁ =-2；A₂=172, B₂=-3；A₃=173, B₃=-4；A₄=174, B₄=-5；A₅=175, B₅=-6；A₆=176, B₆=-7；A₇=177, B₇=-8；A₈=178, B₈=-9；A₉=179, B₉=-10；A₁₀=170, B₁₀=-1；A₁₁ =169, B₁₁=0, and C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁。k_aFor number According to flow point cutpoint, the cutpoint search direction of data flow point shown in Figure 23 is from left to right, from data Flow point cutpoint k_aAfter skipping minimum data block 4KB, in minimum data block 4KB end position conduct Next potential cut-point k_i, according to the rule preset on duplicate removal server 103, for potential Cut-point k_iDetermine W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding pre-conditioned C_x, wherein x is 1 to 11 continuous print natural numbers.11 windows determined are respectively W_i1[k_i-171, k_i-2]、W_i2[k_i-172,k_i-3]、W_i3[k_i-173,k_i-4]、W_i4[k_i-174,k_i-5]、W_i5[k_i-175, k_i-6]、W_i6[k_i-176,k_i-7]、W_i7[k_i-177,k_i-8]、W_i8[k_i-178,k_i-9]、W_i9[k_i-179, k_i-10]、W_i10[k_i-170,k_i-1] and W_i11[k_i-169,k_i].Judge W_i1[k_i-171,k_i-2] in extremely Whether small part data meet predetermined condition C₁, judge W_i2[k_i-172,k_i-3] at least partly count in According to whether meeting predetermined condition C₂, judge W_i3[k_i-173,k_i-4] in, at least part of data are the fullest Foot predetermined condition C₃, judge W_i4[k_i-174,k_i-5] in, whether at least part of data meet predetermined bar Part C₄, judge W_i5[k_i-175,k_i-6] in, whether at least part of data meet predetermined condition C₅, sentence Disconnected W_i6[k_i-176,k_i-7] in, whether at least part of data meet predetermined condition C₆, judge W_i7[k_i -177,k_i-8] in, whether at least part of data meet predetermined condition C₇, judge W_i8[k_i-178,k_i-9] In at least partly data whether meet predetermined condition C₈, judge W_i9[k_i-179,k_i-10] at least Whether part data meet predetermined condition C₉, judge W_i10[k_i-170,k_i-1] at least part of data in Whether meet predetermined condition C₁₀With judge W_i11[k_i-169,k_iIn], whether at least part of data meet Predetermined condition C₁₁.When judging window W_i1In at least partly data meet predetermined condition C₁, window W_i2In at least partly data meet predetermined condition C₂, window W_i3In at least partly data meet pre- Fixed condition C₃, window W_i4In at least partly data meet predetermined condition C₄, window W_i5In at least Part data meet predetermined condition C₅, window W_i6In at least partly data meet predetermined condition C₆、 Window W_i7In at least partly data meet predetermined condition C₇, window W_i8In at least partly data full Foot predetermined condition C₈, window W_i9In at least partly data meet predetermined condition C₉, window W_i10In At least partly data meet predetermined condition C₁₀With window W_i11In at least partly data meet predetermined Condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint.When arbitrary in 11 windows When in individual window, at least part of data are unsatisfactory for the predetermined condition of correspondence, as shown in figure 24, W_i3 [p_i3-169,p_i3In], at least part of data are unsatisfactory for predetermined condition C₃, put p_i3Along data flow point It is described as a example by cutpoint search direction 11 bytes of jump.As shown in figure 24, when judging W₃No Meet predetermined condition C₃Time, with k_iFor starting point, jump along data flow point cutpoint search direction N number of byte, the most N number of byte is not more than ‖ B₃‖+max_x(‖A_x‖), in the present embodiment, N=7, at the end position of the 7th byte, it is thus achieved that next potential cut-point, for potential Cut-point k_iDifference, is expressed as k by new potential cut-point here_j, according at duplicate removal server The rule preset on 103, for potential cut-point k_jDetermine 11 window W_jx[k_j-A_x,k_j+B_x], It is respectively W_j1[k_j-171,k_j-2]、W_j2[k_j-172,k_j-3]、W_j3[k_j-173,k_j-4]、W_j4[k_j -174,k_j-5]、W_j5[k_j-175,k_j-6]、W_j6[k_j-176,k_j-7]、W_j7[k_j-177,k_j-8]、W_j8 [k_j-178,k_j-9]、W_j9[k_j-179,k_j-10]、W_j10[k_j-170,k_j-1] and W_j11[k_j-169,k_j]。 Judge W_j1[k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₁, judge W_j2 [k_j-172,k_j-3] in, whether at least part of data meet predetermined condition C₂, judge W_j3[k_j-173, k_j-4] in, whether at least part of data meet predetermined condition C₃, judge W_j4[k_j-174,k_j-5] in At least partly whether data meet predetermined condition C₄, judge W_j5[k_j-175,k_j-6] at least partly Whether data meet predetermined condition C₅, judge W_j6[k_j-176,k_j-7] in, whether at least part of data Meet predetermined condition C₆, judge W_j7[k_j-177,k_j-8] in, whether at least part of data meet predetermined Condition C₇, judge W_j8[k_j-178,k_j-9] in, whether at least part of data meet predetermined condition C₈、 Judge W_j9[k_j-179,k_j-10] in, whether at least part of data meet predetermined condition C₉, judge W_j10[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₁₀With judge W_j11[k_j -169,k_jIn], whether at least part of data meet predetermined condition C₁₁.Certainly in the embodiment of the present invention In, it is judged that potential cut-point k_aAlso in compliance with this principle when whether being data flow point cutpoint, specifically real The most no longer describe, be referred to judge potential cut-point k_iDescription.When judging window W_j1In At least partly data meet predetermined condition C₁, window W_j2In at least partly data meet predetermined bar Part C₂, window W_j3In at least partly data meet predetermined condition C₃, window W_j4In at least partly Data meet predetermined condition C₄, window W_j5In at least partly data meet predetermined condition C₅, window Mouth W_j6In at least partly data meet predetermined condition C₆, window W_j7In at least partly data meet Predetermined condition C₇, window W_j8In at least partly data meet predetermined condition C₈, window W_j9In extremely Small part data meet predetermined condition C₉, window W_j10In at least partly data meet predetermined condition C₁₀With window W_j11In at least partly data meet predetermined condition C₁₁Time, the most current potential segmentation Point k_jFor data flow point cutpoint, k_jWith k_aBetween data constitute 1 data block, simultaneously according to With k_aIdentical mode skips minimum piecemeal size 4KB, it is thus achieved that next potential cut-point, and According to the rule preset on duplicate removal server 103, it is judged that whether next potential cut-point is Data flow point cutpoint.When judging potential cut-point k_jWhen not being data flow point cutpoint, according to k_i Identical mode obtains next potential cut-point, and presets according on duplicate removal server 103 Rule and said method judge whether next potential cut-point is data flow point cutpoints.When super Cross the maximum data block set when the most not finding data flow point cutpoint, then from maximum data block End position as force-splitting point.Certainly the enforcement of the method by maximum data block length and Constitute the size constraint of the file of this data stream, do not repeat them here.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, the embodiment party shown in Figure 25 In formula, being preset with rule on duplicate removal server 103, described rule is: for potential cut-point k Determine 11 window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, wherein x is 1 to 11 consecution natural numbers, A₁=166, B₁=3；A₂=167, B₂=2；A₃=168, B₃ =1；A₄=169, B₄=0；A₅=170, B₅=-1；A₆=171, B₆=-2；A₇=172, B₇=-3；A₈=173, B₈=-4；A₉=174, B₉=-5；A₁₀=175, B₁₀=-6；A₁₁=176, B₁₁=-7；And C₁=C₂=C₃=C₄ =C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁, then 11 windows are respectively W₁[k-166,k+3]、W₂ [k-167,k+2]、W₃[k-168,k+1]、W₄[k-169,k]、W₅[k-170,k-1]、W₆[k-171, k-2]、W₇[k-172,k-3]、W₈[k-173,k-4]、W₉[k-174,k-5]、W₁₀[k-175,k-6] And W₁₁[k-176,k-7]。k_aFor data flow point cutpoint, the cutpoint of data flow point shown in Figure 25 is looked into Looking for direction is from left to right, from data flow point cutpoint k_aAfter skipping minimum data block 4KB, minimum Data block 4KB end position is as next potential cut-point k_i, in the present embodiment, according to The rule preset on duplicate removal server 103, for potential cut-point k_iDetermine 11 window W_ix[k- A_x,k+B_x] and window W_ix[k-A_x,k+B_x] corresponding predetermined condition C_x, x is respectively 1 to 11 even Continuous natural number.In the embodiment shown in Figure 25, for potential cut-point k_iDetermine 11 windows Mouthful, respectively W_i1[k_i-166,k_i+3]、W_i2[k_i-167,k_i+2]、W_i3[k_i-168,k_i+1]、 W_i4[k_i-169,k_i]、W_i5[k_i-170,k_i-1]、W_i6[k_i-171,k_i-2]、W_i7[k_i-172,k_i-3]、 W_i8[k_i-173,k_i-4]、W_i9[k_i-174,k_i-5]、W_i10[k_i-175,k_i-6] and W_i11[k_i-176,k_i-7]。 Judge W_i1[k_i-166,k_i+ 3] in, whether at least part of data meet predetermined condition C₁, judge W_i2 [k_i-167,k_i+ 2] in, whether at least part of data meet predetermined condition C₂, judge W_i3[k_i-168, k_i+ 1] in, whether at least part of data meet predetermined condition C₃, judge W_i4[k_i-169,k_iIn] extremely Whether small part data meet predetermined condition C₄, judge W_i5[k_i-170,k_i-1] at least partly count in According to whether meeting predetermined condition C₅, judge W_i6[k_i-171,k_i-2] in, at least part of data are the fullest Foot predetermined condition C₆, judge W_i7W_i7[k_i-172,k_i-3] in, whether at least part of data meet pre- Fixed condition C₇, judge W_i8[k_i-173,k_i-4] in, whether at least part of data meet predetermined condition C₈、 Judge W_i9[k_i-174,k_i-5] in, whether at least part of data meet predetermined condition C₉, judge W_i10 [k_i-175,k_i-6] in, whether at least part of data meet predetermined condition C₁₀With judge W_i11[k_i-176, k_i-7] in, whether at least part of data meet predetermined condition C₁₁.When judging window W_i1In at least portion Divided data meets predetermined condition C₁, window W_i2In at least partly data meet predetermined condition C₂、 Window W_i3In at least partly data meet predetermined condition C₃, window W_i4In at least partly data full Foot predetermined condition C₄, window W_i5In at least partly data meet predetermined condition C₅, window W_i6In At least partly data meet predetermined condition C₆, window W_i7In at least partly data meet predetermined bar Part C₇, window W_i8In at least partly data meet predetermined condition C₈, window W_i9In at least partly Data meet predetermined condition C₉, window W_i10In at least partly data meet predetermined condition C₁₀And window Mouth W_i11In at least partly data meet predetermined condition C₁₁Time, the most current potential cut-point k_iFor number According to flow point cutpoint.When data at least part of in any one window in 11 windows are unsatisfactory for correspondence During predetermined condition, as shown in figure 26, W_i7[k_i-172,k_i-3], then from potential cut-point k_iAlong The data flow point cutpoint search direction N number of byte of jump, the most N number of byte is not more than ‖ B₇‖+ max_x(‖A_x‖), in the embodiment shown in Figure 26, N number of byte of jumping is not more than 185 Individual byte, in the present embodiment, N=5, obtain new potential cut-point, for potential segmentation Point k_iDifference, is expressed as k by new potential cut-point here_j, according to the embodiment party shown in Figure 25 The rule preset on duplicate removal server 103 in formula, for potential cut-point k_jThe window determined is 11, respectively W_j1[k_j-166,k_j+3]、W_j2[k_j-167,k_j+2]、W_j3[k_j-168,k_j+1]、 W_j4[k_j-169,k_j]、W_j5[k_j-170,k_j-1]、W_j6[k_j-171,k_j-2]、W_j7[k_j-172,k_j-3]、 W_j8[k_j-173,k_j-4]、W_j9[k_j-174,k_j-5]、W_j10[k_j-175,k_j-6] and W_j11[k_j-176,k_j -7].Judge W_j1[k_j-166,k_j+ 3] in, whether at least part of data meet predetermined condition C₁, sentence Disconnected W_j2[k_j-167,k_j+ 2] in, whether at least part of data meet predetermined condition C₂, judge W_j3[k_j -168,k_j+ 1] in, whether at least part of data meet predetermined condition C₃, judge W_j4[k_j-169,k_j] In at least partly data whether meet predetermined condition C₄, judge W_j5[k_j-170,k_j-1] at least portion in Whether divided data meets predetermined condition C₅, judge W_j6[k_j-171,k_j-2] in, at least part of data are No meet predetermined condition C₆, judge W_j7[k_j-172,k_j-3] in, whether at least part of data meet pre- Fixed condition C₇, judge W_j8[k_j-173,k_j-4] in, whether at least part of data meet predetermined condition C₈、 Judge W_j9[k_j-174,k_j-5] in, whether at least part of data meet predetermined condition C₉, judge W_j10 [k_j-175,k_j-6] in, whether at least part of data meet predetermined condition C₁₀With judge W_j11[k_j-176, k_j-7] in, whether at least part of data meet predetermined condition C₁₁.The most in embodiments of the present invention, Judge potential cut-point k_aAlso in compliance with this principle when whether being data flow point cutpoint, implement not Describe again, be referred to judge potential cut-point k_iDescription.When judging window W_j1In at least Part data meet predetermined condition C₁, window W_j2In at least partly data meet predetermined condition C₂、 Window W_j3In at least partly data meet predetermined condition C₃, window W_j4In at least partly data full Foot predetermined condition C₄, window W_j5In at least partly data meet predetermined condition C₅, window W_j6In At least partly data meet predetermined condition C₆, window W_j7In at least partly data meet predetermined bar Part C₇, window W_j8In at least partly data meet predetermined condition C₈, window W_j9In at least partly Data meet predetermined condition C₉, window W_j10In at least partly data meet predetermined condition C₁₀And window Mouth W_j11In at least partly data meet predetermined condition C₁₁Time, the most current potential cut-point k_jFor number According to flow point cutpoint, k_jWith k_aBetween data constitute 1 data block, simultaneously according to k_aIdentical Mode skip minimum piecemeal size 4KB, it is thus achieved that next potential cut-point, and according to going The rule preset on weight server 103, it is judged that whether next potential cut-point is data flow point Cutpoint.When judging potential cut-point k_jWhen not being data flow point cutpoint, according to k_iIdentical side Formula obtain next potential cut-point, and according on duplicate removal server 103 preset rule and Said method judges whether next potential cut-point is data flow point cutpoints.When exceeding setting When maximum data block does not the most find data flow point cutpoint, then from the stop bits of maximum data block Put as force-splitting point.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, in the reality shown in Figure 27 Executing in mode, be preset with rule on duplicate removal server 103, described rule is: be latent 11 window W are determined at cut-point k_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] right Predetermined condition C answered_x, wherein x is 1 to 11 continuous print natural numbers, A₁=169, B₁=0； A₂=170, B₂=-1；A₃=171, B₃=-2；A₄=172, B₄=-3；A₅=173, B₅=-4； A₆=174, B₆=-5；A₇=175, B₇=-6；A₈=176, B₈=-7；A₉=177, B₉=-8； A₁₀=168, B₁₀=1；A₁₁=179, B₁₁=3；And C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈ =C₉=C₁₀≠C₁₁, then 11 windows are respectively W₁[k-169,k]、W₂[k-170,k-1]、W₃ [k-171,k-2]、W₄[k-172,k-3]、W₅[k-173,k-4]、W₆[k-174,k-5]、W₇ [k-175,k-6]、W₈[k-176,k-7]、W₉[k-177,k-8]、W₁₀[k-168, k+1] and W₁₁ [k-179,k+3]。k_aFor data flow point cutpoint, the cutpoint of data flow point shown in Figure 27 is searched Direction is from left to right, from data flow point cutpoint k_aAfter skipping minimum data block 4KB, Small data block 4KB end position is as next potential cut-point k_i, in the present embodiment, According to the rule preset on duplicate removal server 103, for potential cut-point k_iDetermine window W_ix [k_i-A_x, k_i+B_x], x is respectively 1 to 11 continuous print natural numbers, shown in Figure 27 In embodiment, for potential cut-point k_iDetermine that 11 windows are respectively W_i1[k_i-169,k_i]、 W_i2[k_i-170,k_i-1]、W_i3[k_i-171,k_i-2]、W_i4[k_i-172,k_i-3]、W_i5[k_i-173, k_i-4]、W_i6[k_i-174,k_i-5]、W_i7[k_i-175,k_i-6]、W_i8[k_i-176,k_i-7]、W_i9 [k_i-177,k_i-8]、W_i10[k_i-168,k_i+ 1] and W_i11[k_i-179,k_i+3].Judge W_i1[k_i -169,k_iIn], whether at least part of data meet predetermined condition C₁, judge W_i2[k_i-170,k_i -1] in, whether at least part of data meet predetermined condition C₂, judge W_i3[k_i-171,k_i-2] in At least partly whether data meet predetermined condition C₃, judge W_i4[k_i-172,k_i-3] at least Whether part data meet predetermined condition C₄, judge W_i5[k_i-173,k_i-4] at least partly Whether data meet predetermined condition C₅, judge W_i6[k_i-174,k_i-5] at least part of data in Whether meet predetermined condition C₆, judge W_i7[k_i-175,k_i-6] in, whether at least part of data Meet predetermined condition C₇, judge W_i8[k_i-176,k_i-7] in, whether at least part of data meet Predetermined condition C₈, judge W_i9[k_i-177,k_i-8] in, whether at least part of data meet predetermined Condition C₉, judge W_i10[k_i-168,k_i+ 1] in, whether at least part of data meet predetermined condition C₁₀With judge W_i11[k_i-179,k_i+ 3] in, whether at least part of data meet predetermined condition C₁₁。 When judging window W_i1In at least partly data meet predetermined condition C₁, window W_i2In at least portion Divided data meets predetermined condition C₂, window W_i3In at least partly data meet predetermined condition C₃、 Window W_i4In at least partly data meet predetermined condition C₄, window W_i5In at least part of data Meet predetermined condition C₅, window W_i6In at least partly data meet predetermined condition C₆, window W_i7In at least partly data meet predetermined condition C₇, window W_i8In at least partly data meet Predetermined condition C₈, window W_i9In at least partly data meet predetermined condition C₉, window W_i10In At least partly data meet predetermined condition C₁₀With window W_i11In at least partly data meet pre- Fixed condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint.When judging window W_i11In at least partly data be unsatisfactory for predetermined condition C₁₁Time, then from potential cut-point k_iAlong Data flow point cutpoint search direction 1 byte of jump, obtains new potential cut-point, for With potential cut-point k_iDifference, is expressed as k by new potential cut-point here_j.Work as W_i1、W_i2、 W_i3、W_i4、W_i5、W_i6、W_i7、W_i8、W_i9And W_i10Any one window in 10 windows In time at least partly data are unsatisfactory for the predetermined condition of correspondence, as shown in figure 28, W_i4[k_i -172,k_i-3], then from a k_iAlong the data flow point cutpoint search direction N number of byte of jump, The most N number of byte is not more than ‖ B₄‖+max_x(‖A_x‖), in the enforcement shown in Figure 28 In mode, N number of byte of jumping is not more than 182 bytes, in the present embodiment, N=6, Obtain new potential cut-point, for potential cut-point k_iDifference, here by new potential point Cutpoint is expressed as k_j, according in the embodiment shown in Figure 27 on duplicate removal server 103 The rule preset, for potential cut-point k_jThe window determined is respectively W_j1[k_j-169,k_j]、 W_j2[k_j-170,k_j-1]、W_j3[k_j-171,k_j-2]、W_j4[k_j-172,k_j-3]、W_j5[k_j-173, k_j-4]、W_j6[k_j-174,k_j-5]、W_j7[k_j-175,k_j-6]、W_j8[k_j-176,k_j-7]、W_j9 [k_j-177,k_j-8]、W_j10[k_j-168,k_j+ 1] and W_j11[k_j-179,k_j+3].Judge W_j1 [k_j-169,k_jIn], whether at least part of data meet predetermined condition C₁, judge W_j2[k_j-170, k_j-1] in, whether at least part of data meet predetermined condition C₂, judge W_j3[k_j-171,k_j-2] In at least partly data whether meet predetermined condition C₃, judge W_j4[k_j-172,k_j-3] in extremely Whether small part data meet predetermined condition C₄, judge W_j5[k_j-173,k_j-4] at least portion in Whether divided data meets predetermined condition C₅, judge W_j6[k_j-174,k_j-5] at least partly count in According to whether meeting predetermined condition C₆, judge W_j7[k_j-175,k_j-6] in, at least part of data are No meet predetermined condition C₇, judge W_j8[k_j-176,k_j-7] in, at least part of data are the fullest Foot predetermined condition C₈, judge W_j9[k_j-177,k_j-8] in, whether at least part of data meet pre- Fixed condition C₉, judge W_j10[k_j-168,k_j+ 1] in, whether at least part of data meet predetermined bar Part C₁₀With judge W_j11[k_j-179,k_j+ 3] in, whether at least part of data meet predetermined condition C₁₁.The most in embodiments of the present invention, it is judged that potential cut-point k_aWhether it is data flow point Also in compliance with this principle during cutpoint, implement and no longer describe, be referred to judge potential point Cutpoint k_iDescription.When judging window W_j1In at least partly data meet predetermined condition C₁、 Window W_j2In at least partly data meet predetermined condition C₂, window W_j3In at least part of data Meet predetermined condition C₃, window W_j4In at least partly data meet predetermined condition C₄, window W_j5In at least partly data meet predetermined condition C₅, window W_j6In at least partly data meet Predetermined condition C₆, window W_j7In at least partly data meet predetermined condition C₇, window W_j8In At least partly data meet predetermined condition C₈, window W_j9In at least partly data meet predetermined Condition C₉, window W_j10In at least partly data meet predetermined condition C₁₀With window W_j11In extremely Small part data meet predetermined condition C₁₁Time, the most current potential cut-point k_jFor data flow point Cutpoint, k_jWith k_aBetween data constitute 1 data block, simultaneously according to k_aIdentical Mode skips minimum piecemeal size 4KB, it is thus achieved that next potential cut-point, and according to The rule preset on duplicate removal server 103, it is judged that whether next potential cut-point is several According to flow point cutpoint.When judging potential cut-point k_jWhen not being data flow point cutpoint, according to k_i Identical mode obtains next potential cut-point, and according on duplicate removal server 103 The rule preset and said method judge whether next potential cut-point is the segmentation of data stream Point.When the maximum data block exceeding setting does not the most find data flow point cutpoint, then From the end position of maximum data block as force-splitting point.

On the basis of the data flow point cutpoint shown in Fig. 3 is searched, the embodiment party shown in Figure 29 In formula, being preset with rule on duplicate removal server 103, described rule is: for potential cut-point k Determine 11 window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] corresponding making a reservation for Condition C_x, x is respectively 1 to 11 continuous print natural numbers, wherein, window W_x[p_x-A_x,p_x+B_x] In at least partly data meet the probability of predetermined condition is 1/2, A₁=169, B₁=0；A₂=171, B₂ =-2；A₃=173, B₃=-4；A₄=175, B₄=-6；A₅=177, B₅=-8；A₆=179, B₆=-10；A₇=181, B₇=-12；A₈=183, B₈=-14；A₉=185, B₉=-16；A₁₀=187, B₁₀=-18；A₁₁=189, B₁₁=-20； And C₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁, then 11 windows are respectively W₁ [k-169,k]、W₂[k-171,k-2]、W₃[k-173,k-4]、W₄[k-175,k-6]、W₅[k-177, k-8]、W₆[k-179,k-10]、W₇[k-181,k-12]、W₈[k-183,k-14]、W₉[k-185,k-16]、 W₁₀[k-187, k-18] and W₁₁[k-189,k-20]。k_aFor data flow point cutpoint, shown in Figure 29 Data flow point cutpoint search direction is from left to right, from data flow point cutpoint k_aSkip minimum data After block 4KB, at minimum data block 4KB end position as next potential cut-point k_i, for Potential cut-point k_iDetermine a p_ix, in the present embodiment, according to pre-on duplicate removal server 103 If rule, x is respectively 1 to 11 continuous print natural numbers.In the embodiment shown in Figure 29, According to pre-defined rule, for potential cut-point k_i11 windows determined are respectively W_i1[k_i-169, k_i]、W_i2[k_i-171,k_i-2]、W_i3[k_i-173,k_i-4]、W_i4[k_i-175,k_i-6]、W_i5[k_i-177, k_i-8]、W_i6[k_i-179,k_i-10]、W_i7[k_i-181,k_i-12]、W_i8[k_i-183,k_i-14]、W_i9 [k_i-185,k_i-16]、W_i10[k_i-187,k_i-18] and W_i11[k_i-189,k_i-20].Judge W_i1[k_i -169,k_iIn], whether at least part of data meet predetermined condition C₁, judge W_i2[k_i-171,k_i-2] In at least partly data whether meet predetermined condition C₂, judge W_i3[k_i-173,k_i-4] at least Whether part data meet predetermined condition C₃, judge W_i4[k_i-175,k_i-6] at least partly count in According to whether meeting predetermined condition C₄, judge W_i5[k_i-177,k_i-8] in, whether at least part of data Meet predetermined condition C₅, judge W_i6[k_i-179,k_i-10] in, whether at least part of data meet pre- Fixed condition C₆, judge W_i7[k_i-181,k_i-12] in, whether at least part of data meet predetermined condition C₇, judge W_i8[k_i-183,k_i-14] in, whether at least part of data meet predetermined condition C₈, sentence Disconnected W_i9[k_i-185,k_i-16] in, whether at least part of data meet predetermined condition C₉, judge W_i10 [k_i-187,k_i-18] in, whether at least part of data meet predetermined condition C₁₀With judge W_i11[k_i -189,k_i-20] in, whether at least part of data meet predetermined condition C₁₁.When judging window W_i1In At least partly data meet predetermined condition C₁, window W_i2In at least partly data meet predetermined bar Part C₂, window W_i3In at least partly data meet predetermined condition C₃, window W_i4In at least partly Data meet predetermined condition C₄, window W_i5In at least partly data meet predetermined condition C₅, window Mouth W_i6In at least partly data meet predetermined condition C₆, window W_i7In at least partly data meet Predetermined condition C₇, window W_i8In at least partly data meet predetermined condition C₈, window W_i9In extremely Small part data meet predetermined condition C₉, window W_i10In at least partly data meet predetermined condition C₁₀With window W_i11In at least partly data meet predetermined condition C₁₁Time, the most current potential cut-point k_iFor data flow point cutpoint.When data at least part of in any one window in 11 windows are unsatisfactory for During corresponding predetermined condition, as shown in figure 30, W_i4[k_i-175,k_i-6] at least part of data in It is unsatisfactory for predetermined condition C₄, then select next potential cut-point, for potential cut-point k_iDistrict Not, here shown as k_j, k_jIt is positioned at k_iThe right, and k_jWith k_i1 byte of spacing.Such as figure Shown in 30, according to the rule preset for duplicate removal server 103, for potential cut-point k_jDetermine 11 Window is respectively W_j1[k_j-169,k_j]、W_j2[k_j-171,k_j-2]、W_j3[k_j-173,k_j-4]、 W_j4[k_j-175,k_j-6]、W_j5[k_j-177,k_j-8]、W_j6[k_j-179,k_j-10]、W_j7[k_j-181, k_j-12]、W_j8[k_j-183,k_j-14]、W_j9[k_j-185,k_j-16]、W_j10[k_j-187,k_j-18] And W_j11[k_j-189,k_j, and C-20]₁=C₂=C₃=C₄=C₅=C₆=C₇=C₈=C₉=C₁₀=C₁₁。 Judge W_j1[k_j-169,k_jIn], whether at least part of data meet predetermined condition C₁, judge W_j2 [k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₂, judge W_j3[k_j-173, k_j-4] in, whether at least part of data meet predetermined condition C₃, judge W_j4[k_j-175,k_j-6] in At least partly whether data meet predetermined condition C₄, judge W_j5[k_j-177,k_j-8] at least portion in Whether divided data meets predetermined condition C₅, judge W_j6[k_j-179,k_j-10] at least part of data in Whether meet predetermined condition C₆, judge W_j7[k_j-181,k_j-12] in, at least part of data are the fullest Foot predetermined condition C₇, judge W_j8[k_j-183,k_j-14] in, whether at least part of data meet predetermined Condition C₈, judge W_j9[k_j-185,k_j-16] in, whether at least part of data meet predetermined condition C₉、 Judge W_j10[k_j-187,k_j-18] in, whether at least part of data meet predetermined condition C₁₀And judgement W_j11[k_j-189,k_j-20] in, whether at least part of data meet predetermined condition C₁₁.When judging window Mouth W_j1In at least partly data meet predetermined condition C₁, window W_j2In at least partly data meet Predetermined condition C₂, window W_j3In at least partly data meet predetermined condition C₃, window W_j4In extremely Small part data meet predetermined condition C₄, window W_j5In at least partly data meet predetermined condition C₅, window W_j6In at least partly data meet predetermined condition C₆, window W_j7In at least partly count According to meeting predetermined condition C₇, window W_j8In at least partly data meet predetermined condition C₈, window W_i9In at least partly data meet predetermined condition C₉, window W_j10In at least partly data meet Predetermined condition C₁₀With window W_j11In at least partly data meet predetermined condition C₁₁Time, the most currently Potential cut-point k_jFor data flow point cutpoint.When judging window W_j1、W_j2、W_j3、W_j4、W_j5、 W_j6、W_j7、W_j8、W_j9、W_j10And W_j11In in any one window at least partly data be discontented with During foot predetermined condition, as shown in figure 31, W_j3[k_j-173,k_j-4] in, at least part of data are discontented with Foot predetermined condition C₃Time, k_jIt is positioned at k_iThe right is from k_iJump along data flow point cutpoint search direction N number of byte, the most N number of byte is not more than ‖ B₄‖+max_x(‖A_x‖), shown in Figure 28 In embodiment, N number of byte is not more than 195 bytes, in the present embodiment, N=15, obtains Next potential cut-point, for potential cut-point k_i、k_jDistinguish, be expressed as k_l.Root According to Figure 29 institute embodiment being the default rule of duplicate removal server 103, for potential cut-point k_l Determine that 11 windows are respectively W_l1[k_l-169,k_l]、W_l2[k_l-171,k_l-2]、W_l3[k_l-173, k_l-4]、W_l4[k_l-175,k_l-6]、W_l5[k_l-177,k_l-8]、W_l6[k_l-179,k_l-10]、W_l7 [k_l-181,k_l-12]、W_l8[k_l-183,k_l-14]、W_l9[k_l-185,k_l-16]、W_l10[k_l-187, k_l-18] and W_l11[k_l-189,k_l-20].Judge W_l1[k_l-169,k_lIn], whether at least part of data Meet predetermined condition C₁, judge W_l2[k_l-171,k_l-2] in, whether at least part of data meet pre- Fixed condition C₂, judge W_l3[k_l-173,k_l-4] in, whether at least part of data meet predetermined condition C₃, judge W_l4[k_l-175,k_l-6] in, whether at least part of data meet predetermined condition C₄, sentence Disconnected W_l5[k_l-177,k_l-8] in, whether at least part of data meet predetermined condition C₅, judge W_l6[k_l -179,k_l-10] in, whether at least part of data meet predetermined condition C₆, judge W_l7[k_l-181, k_l-12] in, whether at least part of data meet predetermined condition C₇, judge W_l8[k_l-183,k_l-14] In at least partly data whether meet predetermined condition C₈, judge W_l9[k_l-185,k_l-16] at least Whether part data meet predetermined condition C₉, judge W_l10[k_l-187,k_l-18] at least partly count in According to whether meeting predetermined condition C₁₀With judge W_l11[k_l-189,k_l-20] in, at least part of data are No meet predetermined condition C₁₁.When judging window W_l1In at least partly data meet predetermined condition C₁、 Window W_l2In at least partly data meet predetermined condition C₂, window W_l3In at least partly data full Foot predetermined condition C₃, window W_l4In at least partly data meet predetermined condition C₄, window W_l5In At least partly data meet predetermined condition C₅, window W_l6In at least partly data meet predetermined bar Part C₆, window W_l7In at least partly data meet predetermined condition C₇, window W_l8In at least partly Data meet predetermined condition C₈, window W_l9In at least partly data meet predetermined condition C₉, window Mouth W_l10In at least partly data meet predetermined condition C₁₀With window W_l11In at least partly data full Foot predetermined condition C₁₁Time, the most current potential cut-point k_lFor data flow point cutpoint.As window W_l1、 W_l2、W_l3、W_l4、W_l5、W_l6、W_l7、W_l8、W_l9、W_l10And W_l11In middle either window When at least partly data are unsatisfactory for predetermined condition, select next potential cut-point, for potential Cut-point k_i、k_jAnd k_lDifference, is expressed as k_m, k_mIt is positioned at k_lThe right, and k_mWith k_lSpacing 1 byte.It is the rule that duplicate removal server 103 is preset according to embodiment illustrated in fig. 29, for potential Cut-point k_m11 windows determined are respectively W_m1[k_m-169,k_m]、W_m2[k_m-171,k_m-2]、 W_m3[k_m-173,k_m-4]、W_m4[k_m-175,k_m-6]、W_m5[k_m-177,k_m-8]、W_m6[k_m -179,k_m-10]、W_m7[k_m-181,k_m-12]、W_m8[k_m-183,k_m-14]、W_m9[k_m-185, k_m-16]、W_m10[k_m-187,k_m-18] and W_m11[k_m-189,k_m-20].Judge W_m1[k_m-169, k_mIn], whether at least part of data meet predetermined condition C₁, judge W_m2[k_m-171,k_m-2] in At least partly whether data meet predetermined condition C₂, judge W_m3[k_m-173,k_m-4] at least portion in Whether divided data meets predetermined condition C₃, judge W_m4[k_m-175,k_m-6] at least part of data in Whether meet predetermined condition C₄, judge W_m5[k_m-177,k_m-8] in, at least part of data are the fullest Foot predetermined condition C₅, judge W_m6[k_m-179,k_m-10] in, whether at least part of data meet pre- Fixed condition C₆, judge W_m7[k_m-181,k_m-12] in, whether at least part of data meet predetermined bar Part C₇, judge W_m8[k_m-183,k_m-14] in, whether at least part of data meet predetermined condition C₈、 Judge W_m9[k_m-185,k_m-16] in, whether at least part of data meet predetermined condition C₉, judge W_m10[k_m-187,k_m-18] in, whether at least part of data meet predetermined condition C₁₀And judgement W_m11[k_m-189,k_m-20] in, whether at least part of data meet predetermined condition C₁₁.When judging window Mouth W_m1In at least partly data meet predetermined condition C₁, window W_m2In at least partly data full Foot predetermined condition C₂, window W_m3In at least partly data meet predetermined condition C₃, window W_m4In At least partly data meet predetermined condition C₄, window W_m5In at least partly data meet predetermined bar Part C₅, window W_m6In at least partly data meet predetermined condition C₆, window W_m7In at least portion Divided data meets predetermined condition C₇, window W_m8In at least partly data meet predetermined condition C₈、 Window W_m9In at least partly data meet predetermined condition C₉, window W_m10In at least part of data Meet predetermined condition C₁₀With window W_m11In at least partly data meet predetermined condition C₁₁Time, then Current potential cut-point k_mFor data flow point cutpoint.When data at least part of in any one window not When meeting predetermined condition, scheme the most as described above performs jump, latent to obtain the next one At cut-point and determine whether data flow point cutpoint.

Embodiments provide one and judge window W_iz[k_i-A_z,k_i+B_zIn] at least Whether part data meet predetermined condition C_zMethod, in the present embodiment use random function Judge window W_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet predetermined condition C_z, As a example by the embodiment shown in Figure 21, according to the rule preset on duplicate removal server 103 Then, for potential cut-point k_iDetermine window W_i1[k_i-169,k_i], it is judged that W_i1[k_i-169,k_i] In at least partly data whether meet predetermined condition C₁, as shown in figure 32, W_i1Represent window Mouth W_i1[k_i-169,k_i], for judging W_i1[k_i-169,k_iIn], whether at least part of data meet Predetermined condition C₁, selecting 5 bytes, in Figure 32, " ■ " represents 1 byte selected, 42 bytes are differed between adjacent two bytes selected.By anti-for 5 byte datas of selection Utilize again 51 times, obtain 255 bytes altogether, to increase randomness.The most each byte by 8 compositions, are designated as a_m,1…a_m,8, represent in 255 bytes that the 1st of m-th byte the arrives 8th, therefore, position corresponding to 255 bytes can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n=1, work as a_m,nWhen=0, V_am,n=-1, Wherein a_m,nRepresent a_m,1…a_m,8In any one, position corresponding to 255 bytes is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, can be expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right).$ Choose a large amount of random number, form matrix, by random number According to composition matrix once form, keep constant, as from obey specific distribution (here with As a example by normal distribution) random number in select 255*8 random number to form matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_aM row and the m row of matrix R random Number is multiplied, and then summation obtains a value, is embodied as S_am=V_am,1*h_m,1+V_am,2*h_m,2 +…+V_am,8*h_m,8.According to the method, it is thus achieved that S_a1、S_a2... to S_a255, add up S_a1、S_a2… To S_a255In meet number K of value of specified conditions (here as a example by more than 0).Due to Matrix R Normal Distribution, then S_amAs matrix R, still Normal Distribution, According to theory of probability, the normal distribution random number probability more than 0 is 1/2, at S_a1、S_a2… To S_a255In, each value probability more than 0 is 1/2, so K meets binomial distribution: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistical result, it is judged that S_a1、S_a2… To S_a255Value more than 0 number K whether be even number, the random number of binomial distribution be idol The probability of number is 1/2, so K meets condition with the probability of 1/2.When K is even number, Show W_i1[k_i-169,k_iIn], at least part of data meet predetermined condition C₁；When K is odd number Time, show W₁[k_i-169,k_iIn], at least part of data are unsatisfactory for predetermined condition C₁, C here₁ I.e. refer to the S obtained according to aforesaid way_a1、S_a2... to S_a255Value more than 0 number K be Even number.In the embodiment shown in Figure 21, at W_i1[k_i-169,k_i]、W_i2[k_i-170,k_i -1]、W_i3[k_i-171,k_i-2]、W_i4[k_i-172,k_i-3]、W_i5[k_i-173,k_i-4]、W_i6[k_i-174, k_i-5]、W_i7[k_i-175,k_i-6]、W_i8[k_i-176,k_i-7]、W_i9[k_i-177,k_i-8]、W_i10[k_i -178,k_i-9] and W_i11[k_i-179,k_i-10] in, each window size is identical, i.e. window size is equal It is 169 bytes, judges in window, whether at least part of data meet predetermined condition simultaneously Mode is the most identical, is specifically shown in above-mentioned judgement W_i1[k_i-169,k_iIn], whether at least part of data Meet predetermined condition C₁Description.Therefore, as shown in figure 32,Represent and judge window W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂Time select 1 Individual byte, differs 42 bytes between adjacent two bytes selected.5 words that will select Joint number, according to recycling 51 times, obtains 255 bytes, altogether to increase randomness.The most every Individual byte is formed by 8, is designated as b_m,1…b_m,8, represent m-th byte in 255 bytes The the 1st to the 8th, therefore, position corresponding to 255 bytes can be expressed as: $\left(\begin{array}{cccc}{b}_{1,1}& b_{1,2}& \mathrm{...}& b_{1,8}\\ b_{2,1}& b_{2,2}& \mathrm{...}& b_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ b_{255,1}& b_{255,2}& \mathrm{...}& b_{255,8}\end{array}\right),$ Work as b_m,nWhen=1, V_bm,n=1, work as b_m,nWhen=0, V_bm,n=-1, Wherein b_m,nRepresent b_m,1…b_m,8In any one, position corresponding to 255 bytes is according to b_m,nWith V_bm,nTransformational relation obtain matrix V_b, can be expressed as: $\left(\begin{array}{cccc}{V}_{b1,1}& V_{b1,2}& \mathrm{...}& V_{b1,8}\\ V_{b2,1}& V_{b2,2}& \mathrm{...}& V_{b2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{b255,1}& V_{b255,2}& \mathrm{...}& V_{b255,8}\end{array}\right).$ Judge W_i1[k_i-169,k_iIn], whether at least part of data meet the mode of predetermined condition and sentence Disconnected window W_i2[k_i-170,k_i-1] in, whether at least part of data meet the mode of predetermined condition Identical, therefore use matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_bM row Being multiplied with the random number of the m row of matrix R, then summation obtains a value, concrete table It is shown as S_bm=V_bm,1*h_m,1+V_bm,2*h_m,2+…+V_bm,8*h_m,8.According to the method, it is thus achieved that S_b1、 S_b2... to S_b255, add up S_b1、S_b2... to S_b255In meet specified conditions (here to be more than As a example by 0) number K of value.Due to matrix R Normal Distribution, then S_bmWith matrix R Equally, still Normal Distribution, according to theory of probability, normal distribution random number is more than 0 Probability be 1/2, at S_b1、S_b2... to S_b255In, each value probability more than 0 is 1/2, So K meets binomial distribution: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistics Result, it is judged that S_b1、S_b2... to S_b255Value more than 0 number K whether be even number, two The probability that random number is even number of distribution for for 1/2, so K with 1/2 probability satisfied Condition.When K is even number, show W_i2[k_i-170,k_i-1] in, at least part of data meet Predetermined condition C₂；When K is odd number, show W_i2[k_i-170,k_i-1] at least partly count in According to being unsatisfactory for predetermined condition C₂, C here₂I.e. refer to the S obtained according to aforesaid way_b1、S_b2... arrive S_b255Value more than 0 number K be even number.In embodiment shown in Figure 21, W_i2[k_i -170,k_i-1] in, at least part of data meet predetermined condition C₂。

Therefore, as shown in figure 32,Represent and judge window W_i3[k_i-171,k_i-2] at least portion in Whether divided data meets predetermined condition C₃Time select 1 byte, adjacent two select bytes Between differ 42 bytes.5 byte datas selected are recycled 51 times, obtains 255 altogether Byte, to increase randomness.Then use and judge window W_i1[k_i-169,k_i] and W_i2[k_i-170,k_i -1] method that in, whether at least part of data meet predetermined condition, it is judged that W_i3[k_i-171,k_i-2] In at least data whether meet predetermined condition C₃.In embodiment shown in Figure 21, W_i3[k_i-171, k_i-2] in, at least part of data meet predetermined condition.As shown in figure 32,Represent and judge window Mouth W_i4[k_i-172,k_i-3] in, whether at least part of data meet predetermined condition C₄Time select 1 Byte, differs 42 bytes between adjacent two bytes selected.5 byte datas that will select Recycle 51 times, obtain 255 bytes altogether, to increase randomness.Then use and judge window W_i1 [k_i-169,k_i]、W_i2[k_i-170,k_i-1] and W_i3[k_i-171,k_i-2] in, at least part of data are the fullest The method of foot predetermined condition, it is judged that W_i4[k_i-172,k_i-3] in, whether at least part of data meet pre- Fixed condition C₄.In embodiment shown in Figure 21, W_i4[k_i-172,k_i-3] at least part of data in Meet predetermined condition C₄.As shown in figure 32,Represent and judge window W_i5[k_i-173,k_i-4] In at least partly data whether meet predetermined condition C₅Time select 1 byte, adjacent two choosings 42 bytes are differed between the byte selected.5 byte datas selected are recycled 51 times, altogether Obtain 255 bytes, to increase randomness.Then use and judge window W_i1[k_i-169,k_i]、W_i2 [k_i-170,k_i-1]、W_i3[k_i-171,k_i-2] and W_i4[k_i-172,k_i-3] in, whether at least part of data The method meeting predetermined condition, it is judged that W_i5[k_i-173,k_i-4] in, at least whether data meet predetermined Condition C₅.In embodiment shown in Figure 21, W_i5[k_i-173,k_i-4] in, at least part of data are not Meet predetermined condition C₅。

Work as W_i5[k_i-173,k_i-4] in, at least part of data are unsatisfactory for C during predetermined condition₅, from a p_i5 Along data flow point cutpoint search direction 7 bytes of jump, the end position the 7th byte obtains Obtain next potential cut-point k_j, as shown in figure 22, according to preset for duplicate removal server 103 Rule, for potential cut-point k_jDetermine window W_j1[k_j-169,k_j], it is judged that window W_j1[k_j-169, k_jIn], whether at least part of data meet predetermined condition C₁Mode with judge window W_i1[k_i -169,k_iIn], whether at least part of data meet predetermined condition C₁Mode identical, therefore as figure Shown in 33, W_j1Represent window, whether meet predetermined condition C for data at least part of in judging₁, Selecting 5 bytes, in Figure 33, " ■ " represents 1 byte selected, adjacent two bytes selected Between differ 42 bytes.5 byte datas selected are recycled 51 times, obtains 255 altogether Byte, to increase randomness.The most each byte is formed by 8, is designated as a_m,1'…a_m,8', table Show m-th byte in 255 bytes the 1st to the 8th, therefore, the position that 255 bytes are corresponding Can be expressed as: $\left(\begin{array}{cccc}{a_{1,1}}^{\′}& {a_{1,2}}^{\′}& \mathrm{...}& {a_{1,8}}^{\′}\\ {a_{2,1}}^{\′}& {a_{2,2}}^{\′}& \mathrm{...}& {a_{2,8}}^{\′}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ {a_{255,1}}^{\′}& {a_{255,2}}^{\′}& \mathrm{...}& {a_{255,8}}^{\′}\end{array}\right),$ Work as a_m,nDuring '=1, V_am,n'=1, works as a_m,n' When=0, V_am,n'=-1, wherein a_m,n' represent a_m,1'…a_m,8Any one in ', 255 bytes pair The position answered is according to a_m,n' and V_am,n' transformational relation obtain matrix V_a', can be expressed as: $\left(\begin{array}{cccc}{V_{a1,1}}^{\′}& {V_{a1,2}}^{\′}& \mathrm{...}& {V_{a1,8}}^{\′}\\ {V_{a2,1}}^{\′}& {V_{a2,2}}^{\′}& \mathrm{...}& {V_{a2,8}}^{\′}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ {V_{a255,1}}^{\′}& {V_{a255,2}}^{\′}& \mathrm{...}& {V_{a255,8}}^{\′}\end{array}\right).$ Judge in window, whether at least part of data meet predetermined Condition with judge window W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition Mode identical, therefore use matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_a' m Row is multiplied with the random number of the m row of matrix R, and then summation obtains a value, specifically represents For S_am'=V_am,1'*h_m,1+V_am,2'*h_m,2+…+V_am,8'*h_m,8.According to the method, it is thus achieved that S_a1'、 S_a2' ... to S_a255', add up S_a1'、S_a2' ... to S_a255Specified conditions are met (here with greatly in ' As a example by 0) number K of value.Due to matrix R Normal Distribution, then S_am' and matrix R Equally, still Normal Distribution, according to theory of probability, general more than 0 of normal distribution random number Rate is 1/2, at S_a1'、S_a2' ... to S_a255In ', each value probability more than 0 is 1/2, so K meets binomial distribution: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistical result, Judge S_a1'、S_a2' ... to S_a255' value more than 0 number K whether be even number, binomial distribution Random number be the probability of even number be 1/2, so K meets condition with the probability of 1/2.When K is even number Time, show W_j1[k_j-169,k_jIn], at least part of data meet predetermined condition C₁；When K is odd number Time, show W_j1[k_j-169,k_jIn], at least part of data are unsatisfactory for predetermined condition C₁。

Judge W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 33,Represent and judge window W_j2[k_j-170,k_j-1] in extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select 42 bytes are differed between byte.5 byte datas selected are recycled 51 times, obtains altogether 255 bytes, to increase randomness.The most each byte is formed by 8, is designated as b_m,1'…b_m,8', Representing in 255 bytes the 1st to the 8th of m-th byte, therefore, 255 bytes are corresponding Position can be expressed as: $\left(\begin{array}{cccc}{b_{1,1}}^{\′}& {b_{1,2}}^{\′}& \mathrm{...}& {b_{1,8}}^{\′}\\ {b_{2,1}}^{\′}& {b_{2,2}}^{\′}& \mathrm{...}& {b_{2,8}}^{\′}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ {b_{255,1}}^{\′}& {b_{255,2}}^{\′}& \mathrm{...}& {b_{255,8}}^{\′}\end{array}\right),$ Work as b_m,nDuring '=1, V_bm,n'=1, works as b_m,n' When=0, V_bm,n'=-1, wherein b_m,n' represent b_m,1'…b_m,8Any one in ', 255 byte correspondences Position according to b_m,n' and V_bm,n' transformational relation obtain matrix V_b', can be expressed as: $\left(\begin{array}{cccc}{V}_{b1,1}& V_{b1,2}& \mathrm{...}& V_{b1,8}\\ V_{b2,1}& V_{b2,2}& \mathrm{...}& V_{b2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{b255,1}& V_{b255,2}& \mathrm{...}& V_{b255,8}\end{array}\right).$ Judge window W_i2[k_i-170,k_i-1] in, at least part of data are No meet predetermined condition C₁And W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined Condition C₁Mode identical, the most still use matrix R: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,1}& h_{255,2}& \mathrm{...}& h_{255,8}\end{array}\right),$ By matrix V_b' m row be multiplied with the random number of the m row of matrix R, then summation obtain a value, It is embodied as S_bm'=V_bm,1'*h_m,1+V_bm,2'*h_m,2+…+V_bm,8'*h_m,8.According to the method, Obtain S_b1'、S_b2' ... to S_b255', add up S_b1'、S_b2' ... to S_b255Specified conditions (this is met in ' In as a example by more than 0) number K of value.Due to matrix R Normal Distribution, then S_bm' with Matrix R is the same, still Normal Distribution, and according to theory of probability, normal distribution random number is more than 0 Probability be 1/2, at S_b1'、S_b2' ... to S_b255In ', each value probability more than 0 is 1/2, So K meets binomial distribution: $P(k=n)=C_{255}^n{\left(\frac{1}{2}\right)}^n{\left(\frac{1}{2}\right)}^{255-n}=C_{255}^n{\left(\frac{1}{2}\right)}^{255}.$ According to statistics knot Really, it is judged that S_b1'、S_b2' ... to S_b255' value more than 0 number K whether be even number, binomial divides The random number of cloth be the probability of even number for for 1/2, so K meets condition with the probability of 1/2.Work as K During for even number, in showing, at least part of data meet predetermined condition C₂；When K is odd number, table Bright W_j2[k_j-170,k_j-1] in, at least part of data are unsatisfactory for predetermined condition C₂.In like manner, it is judged that W_i3 [k_i-171,k_i-2] in, whether at least part of data meet predetermined condition C₃Mode with judge W_j3 [k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₃Mode identical, in like manner, Judge W_j4[k_j-172,k_j-3] in, whether at least part of data meet predetermined condition C₄, judge W_j5 [k_j-173,k_j-4] in, whether at least part of data meet predetermined condition C₅, judge W_j6[k_j-174, k_j-5] in, whether at least part of data meet predetermined condition C₆, judge W_j7[k_j-175,k_j-6] in At least partly whether data meet predetermined condition C₇, judge W_j8[k_j-176,k_j-7] at least partly Whether data meet predetermined condition C₈, judge W_j9[k_j-177,k_j-8] in, whether at least part of data Meet predetermined condition C₉, judge W_j10[k_j-178,k_j-9] in, whether at least part of data meet pre- Fixed condition C₁₀With judge W_j11[k_j-179,k_j-10] in, whether at least part of data meet predetermined bar Part C₁₁, do not repeat them here.

The present embodiment use random function judge window W_iz[k_i-A_z,k_i+B_zAt least portion in] Whether divided data meets predetermined condition C_z, still as a example by Figure 21 illustrated embodiment, according to The rule preset on duplicate removal server 103, for potential cut-point k_iDetermine window W_i1[k_i-169, k_i], it is judged that W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁, as Shown in Figure 32, W_i1Represent window W_i1[k_i-169,k_i], for judging W_i1[k_i-169,k_iIn] at least Whether part data meet predetermined condition C₁, selecting 5 bytes, in Figure 32, " ■ " represents selection 1 byte, between the byte of adjacent two selections " ■ " differ 42 bytes.One of which Implementation is to use HASH function to calculate 5 bytes selected, and uses HASH function to calculate The numerical value obtained is one and fixing is uniformly distributed, if using the calculated number of HASH function Value is even number, then judge W_i1[k_i-169,k_iIn], at least part of data meet predetermined condition C₁, I.e. C₁Representing uses the calculated numerical value of HASH function to be even number according to aforesaid way.Therefore, W_i1[k_i-169,k_iThe probability that in], whether at least part of data meet predetermined condition is 1/2.At figure In embodiment shown in 21, Hash function is used to judge W_i2[k_i-170,k_i-1] at least partly Whether data meet predetermined condition C₂、W_i3[k_i-171,k_i-2] in, whether at least part of data meet Predetermined condition C₃、W_i4[k_i-172,k_i-3] in, whether at least part of data meet predetermined condition C₄With W_i5[k_i-173,k_i-4] in, whether at least part of data meet predetermined condition C₅, implement and can join Examining description Figure 21 illustrated embodiment uses Hash function to judge W_i1[k_i-169,k_iAt least portion in] Whether divided data meets mode C of predetermined condition₁, do not repeat them here.

Work as W_i5[k_i-173,k_i-4] in, at least part of data are unsatisfactory for predetermined condition C₅Time, from potential Cut-point k_iAlong data flow point cutpoint search direction 7 bytes of jump, at the knot of the 7th byte Bundle position obtains current potential cut-point k_j, as shown in figure 22, according to for duplicate removal server 103 The rule preset, for potential cut-point k_jDetermine window W_j1[k_j-169,k_j], it is judged that window W_j1 [k_j-169,k_jIn], whether at least part of data meet predetermined condition C₁Mode with judge window W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁Mode identical, therefore As shown in figure 33, W_j1Represent window W_j1[k_j-169,k_j], for judging W_j1[k_j-169,k_jIn] extremely Whether small part data meet predetermined condition C₁, selecting 5 bytes, in Figure 33, " ■ " represents choosing 1 byte selected, differs 42 bytes between adjacent two bytes " ■ " selected.Use Hash Function calculates from window W_j1[k_j-169,k_j5 bytes chosen in], if the numerical value obtained is Even number, then W_j1[k_j-169,k_jIn], at least part of data meet predetermined condition C₁.In Figure 33, sentence Disconnected W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂Mode and sentence Disconnected W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₂Mode identical, Therefore, as shown in figure 33,Represent and judge window W_j2[k_j-170,k_j-1] at least partly count in According to whether meeting predetermined condition C₂Time select 1 byte, adjacent two select bytes Between differ 42 bytes.Hash function is used to calculate 5 bytes selected, if obtain Numerical value is even number, then W_j2[k_j-170,k_j-1] in, at least part of data meet predetermined condition C₂.Figure In 33, it is judged that W_i3[k_i-171,k_i-2] in, whether at least part of data meet predetermined condition C₃Side Formula with judge W_j3[k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₃Side Formula is identical, therefore, as shown in figure 33,Represent and judge window W_j3[k_j-171,k_j-2] in extremely Whether small part data meet predetermined condition C₃Time select 1 byte, adjacent two select ByteBetween differ 42 bytes.Hash function is used to calculate 5 bytes selected, To numerical value be even number, then W_j3[k_j-171,k_j-2] in, at least part of data meet predetermined condition C₃。 In Figure 33, it is judged that W_j4[k_j-172,k_j-3] in, whether at least part of data meet predetermined condition C₄'s Mode and judge window W_i4[k_i-172,k_i-3] in, whether at least part of data meet predetermined condition C₄Mode, therefore, as shown in figure 33,Represent and judge window W_j4[k_j-172,k_j-3] In at least partly data whether meet predetermined condition C₄Time select 1 byte, adjacent two choosings The byte selectedBetween differ 42 bytes.Hash function is used to calculate 5 bytes selected, The numerical value obtained is even number, then W_j4[k_j-172,k_j-3] in, at least part of data meet predetermined condition C₄.According to said method, it is judged that W_j5[k_j-173,k_j-4] in, whether at least part of data meet pre- Fixed condition C₅, judge W_j6[k_j-174,k_j-5] in, whether at least part of data meet predetermined condition C₆、 Judge W_j7[k_j-175,k_j-6] in, whether at least part of data meet predetermined condition C₇, judge W_j8 [k_j-176,k_j-7] in, whether at least part of data meet predetermined condition C₈, judge W_j9[k_j-177, k_j-8] in, whether at least part of data meet predetermined condition C₉, judge W_j10[k_j-178,k_j-9] in At least partly whether data meet predetermined condition C₁₀With judge W_j11[k_j-179,k_j-10] at least Whether part data meet predetermined condition C₁₁, do not repeat them here.

The present embodiment use random function judge window W_iz[k_i-A_z,k_i+B_zAt least portion in] Whether divided data meets predetermined condition C_z, as a example by the embodiment shown in Figure 21, according to going The rule preset on weight server 103, for potential cut-point k_iDetermine window W_i1[k_i-169,k_i], Judge W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁, such as Figure 32 institute Show, W_i1Represent window W_i1[k_i-169,k_i], for judging W_i1[k_i-169,k_iAt least partly count in] According to whether meeting predetermined condition C₁, select 5 bytes, serial number 169 in Figure 32,127,85, The byte " ■ " of 43 and 1 represents 1 byte of selection respectively, adjacent two bytes selected it Between differ 42 bytes.The byte " ■ " of serial number 169,127,85,43 and 1 is turned respectively Change a decimal value into, be expressed as a₁、a₂、a₃、a₄And a₅.Because 1 Byte is formed by 8, so each byte " ■ " is as numerical value, then an a₁、a₂、a₃、 a₄And a₅In any one a_rIt is satisfied by 0≤a_r≤255。a₁、a₂、a₃、a₄And a₅Composition The matrix of 1*5.From the random number obeying binomial distribution, select 256*5 random number, form square Battle array R, is expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right),$

According to a₁Value and the row at place, search from matrix R correspondence value, such as a₁=36, a₁ It is positioned at the 1st row, then searches h_36,1Corresponding value；According to a₂Value and the row at place, from matrix R The middle value searching correspondence, such as a₂=48, a₂It is positioned at the 2nd row, then searches h_48,2Corresponding value；Root According to a₃Value and the row at place, search from matrix R correspondence value, such as a₃=26, a₃It is positioned at 3rd row, then search h_26,3Corresponding value；According to a₄Value and the row at place, look into from matrix R Look for the value of correspondence, such as a₄=26, a₄It is positioned at the 4th row, then searches h_26,4Corresponding value；According to a₅ Value and the row at place, search from matrix R correspondence value, such as a₅=88, a₅It is positioned at the 5th Row, then search h_88,5Corresponding value.S₁=h_36,1+h_48,2+h_26,3+h_26,4+h_88,5, because matrix R clothes From binomial distribution, therefore, S₁Also binomial distribution is obeyed.Work as S₁For even number, then W_i1[k_i-169, k_iIn], at least part of data meet predetermined condition C₁, work as S₁For odd number, then W_i1[k_i-169,k_i] In at least partly data be unsatisfactory for predetermined condition C₁, S₁Probability for even number is 1/2, C₁Represent Calculate S in a manner described₁For even number.In embodiment illustrated in fig. 21, W_i1[k_i-169,k_iIn] At least partly data meet predetermined condition C₁.As shown in figure 32,Represent and judge window W_i2[k_i -170,k_i-1] in, whether at least part of data meet predetermined condition C₂Time 1 byte selecting respectively, In Figure 32, represent by sequence number 170,128,86,44 and 2 respectively, adjacent two selections 42 bytes are differed between byte.Byte by sequence number 170,128,86,44 and 2Point It is not converted into a decimal value, is expressed as b₁、b₂、b₃、b₄And b₅.Because 1 Individual byte is formed by 8, so each byteAs numerical value, then a b₁、b₂、b₃、 b₄And b₅In any one b_rIt is satisfied by 0≤b_r≤255。b₁、b₂、b₃、b₄And b₅Composition 1*5 Matrix.In present embodiment, it is judged that W_i1And W_i2In at least partly data whether meet predetermined The mode of condition is identical, the most still uses matrix R, according to b₁Value and the row at place, from Matrix R searches the value of correspondence, such as b₁=66, b₁It is positioned at the 1st row, then searches h_66,1Corresponding Value；According to b₂Value and the row at place, search from matrix R correspondence value, such as b₂=48, b₂ It is positioned at the 2nd row, then searches h_48,2Corresponding value；According to b₃Value and the row at place, from matrix R The middle value searching correspondence, such as b₃=99, b₃It is positioned at the 3rd row, then searches h_99,3Corresponding value；Root According to b₄Value and the row at place, search from matrix R correspondence value, such as b₄=26, b₄It is positioned at 4th row, then search h_26,4Corresponding value；According to b₅Value and the row at place, look into from matrix R Look for the value of correspondence, such as b₅=90, b₅It is positioned at the 5th row, then searches h_90,5Corresponding value.S₂=h_66,1+ h_48,2+h_99,3+h_26,4+h_90,5, because matrix R obeys binomial distribution, therefore, S₂Also obey binomial to divide Cloth.Work as S₂For even number, then W_i2[k_i-170,k_i-1] in, at least part of data meet predetermined condition C₂, Work as S₂For odd number, then W_i2[k_i-170,k_i-1] in, at least part of data are unsatisfactory for predetermined condition C₂, S₂Probability for even number is 1/2.In embodiment illustrated in fig. 21, W_i2[k_i-170,k_i-1] in extremely Small part data meet predetermined condition C₂.Use same rule, judge W respectively_i3[k_i-171, k_i-2] in, whether at least part of data meet predetermined condition C₃, judge W_i4[k_i-172,k_i-3] in extremely Whether small part data meet predetermined condition C₄, judge W_i5[k_i-173,k_i-4] at least partly count in According to whether meeting predetermined condition C₅, judge W_i6[k_i-174,k_i-5] in, at least part of data are the fullest Foot predetermined condition C₆, judge W_i7[k_i-175,k_i-6] in, whether at least part of data meet predetermined bar Part C₇, judge W_i8[k_i-176,k_i-7] in, whether at least part of data meet predetermined condition C₈, sentence Disconnected W_i9[k_i-177,k_i-8] in, whether at least part of data meet predetermined condition C₉, judge W_i10[k_i -178,k_i-9] in, whether at least part of data meet predetermined condition C₁₀With judge W_i11[k_i-179,k_i -10] in, whether at least part of data meet predetermined condition C₁₁.In embodiment shown in Figure 21, W_i5[k_i-173,k_i-4] in, at least part of data are unsatisfactory for predetermined condition C₅, from potential cut-point k_i Along data flow point cutpoint search direction 7 bytes of jump, the end position the 7th byte obtains Obtain current potential cut-point k_j, as shown in figure 22, according to the rule preset for duplicate removal server 103 Then, for potential cut-point k_jDetermine window W_j1[k_j-169,k_j], it is judged that window W_j1[k_j-169,k_j] In at least partly data whether meet predetermined condition C₁Mode with judge window W_i1[k_i-169,k_i] In at least partly data whether meet predetermined condition C₁Mode identical, the most as shown in figure 33, W_j1Represent window W_j1[k_j-169,k_j], for judging W_j1[k_j-169,k_jIn], at least part of data are No meet predetermined condition C₁, the byte " ■ " of serial number 169,127,85,43 and 1 in Figure 33 Represent 1 byte of selection respectively, between adjacent two bytes selected, differ 42 bytes. The byte " ■ " of serial number 169,127,85,43 and 1 is converted into a decimal number respectively Value, is expressed as a₁'、a₂'、a₃'、a₄' and a₅'.Because 1 byte is formed by 8, So each byte " ■ " is as numerical value, then an a₁'、a₂'、a₃'、a₄' and a₅Appointing in ' One a_r' it is satisfied by 0≤a_r'≤255。a₁'、a₂'、a₃'、a₄' and a₅' composition 1*5 matrix. Judge window W_j1[k_j-169,k_jIn], whether at least part of data meet predetermined condition C₁Mode With judge window W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁Side Formula is identical, therefore, still uses matrix R, is expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right),$

According to a₁' value and the row at place, search from matrix R correspondence value, such as a₁'=16, a₁' be positioned at the 1st row, then search h_16,1Corresponding value；According to a₂' value and the row at place, from Matrix R searches the value of correspondence, such as a₂'=98, a₂' be positioned at the 2nd row, then search h_98,2Right The value answered；According to a₃' value and the row at place, search from matrix R correspondence value, as a₃'=56, a₃' be positioned at the 3rd row, then search h_56,3Corresponding value；According to a₄' value and place Row, search from matrix R correspondence value, such as a₄'=36, a₄' it is positioned at the 4th row, then Search h_36,4Corresponding value；According to a₅' value and the row at place, it is right to search from matrix R The value answered, such as a₅'=99, a₅' be positioned at the 5th row, then search h_99,5Corresponding value.S₁'=h_16,1 +h_98,2+h_56,3+h_36,4+h_99,5, because matrix R obeys binomial distribution, therefore, S₁' also obey two Item distribution.Work as S₁' for even number, then W_j1[k_j-169,k_jIn], at least part of data meet predetermined Condition C₁, work as S₁' for odd number, then W_j1[k_j-169,k_jIn], at least part of data are unsatisfactory for making a reservation for Condition C₁, S₁' it is 1/2 for the probability of even number.

Judge W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 33,Represent and judge window W_j2[k_j-170,k_j-1] in extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select Differ 42 bytes between byte, represent by sequence number 170,128,86,44 and 2 respectively, phase 42 bytes are differed between adjacent two bytes selected.By sequence number 170,128,86,44 and 2 ByteIt is converted into a decimal value respectively, is expressed as b₁'、b₂'、b₃'、 b₄' and b₅'.Because 1 byte is formed by 8, so each byteAs a numerical value, Then b₁'、b₂'、b₃'、b₄' and b₅Any one b in '_r' it is satisfied by 0≤b_r'≤255。b₁'、b₂'、b₃'、 b₄' and b₅' composition 1*5 matrix.With judge window W_i2[k_i-170,k_i-1] at least part of data in Whether meet predetermined condition C₂Use identical matrix R, according to b₁' value and the row at place, from Matrix R searches the value of correspondence, such as b₁'=210, b₁' be positioned at the 1st row, then search h_210,1Corresponding Value；According to b₂' value and the row at place, search from matrix R correspondence value, such as b₂'=156, b₂' be positioned at the 2nd row, then search h_156,2Corresponding value；According to b₃' value and the row at place, from square Battle array R searches the value of correspondence, such as b₃'=144, b₃' be positioned at the 3rd row, then search h_144,3Corresponding Value；According to b₄' value and the row at place, search from matrix R correspondence value, such as b₄'=60, b₄' It is positioned at the 4th row, then searches h_60,4Corresponding value；According to b₅' value and the row at place, from matrix R The middle value searching correspondence, such as b₅'=90, b₅' be positioned at the 5th row, then search h_90,5Corresponding value.S₂' =h_210,1+h_156,2+h_144,3+h_60,4+h_90,5, with S₂Rule of judgment identical, work as S₂' for even number, Then W_j2[k_j-170,k_j-1] in, at least part of data meet predetermined condition C₂, work as S₂' for odd number, Then W_j2[k_j-170,k_j-1] in, at least part of data are unsatisfactory for predetermined condition C₂, S₂' for even number Probability is 1/2.

In like manner, it is judged that W_i3[k_i-171,k_i-2] in, whether at least part of data meet predetermined condition C₃ Mode with judge W_j3[k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₃ Mode identical, in like manner, it is judged that W_j4[k_j-172,k_j-3] in, whether at least part of data meet pre- Fixed condition C₄, judge W_j5[k_j-173,k_j-4] in, whether at least part of data meet predetermined condition C₅、 Judge W_j6[k_j-174,k_j-5] in, whether at least part of data meet predetermined condition C₆, judge W_j7 [k_j-175,k_j-6] in, whether at least part of data meet predetermined condition C₇, judge W_j8[k_j-176, k_j-7] in, whether at least part of data meet predetermined condition C₈, judge W_j9[k_j-177,k_j-8] in At least partly whether data meet predetermined condition C₉, judge W_j10[k_j-178,k_j-9] at least portion in Whether divided data meets predetermined condition C₁₀With judge W_j11[k_j-179,k_j-10] at least partly count in According to whether meeting predetermined condition C₁₁, do not repeat them here.

The present embodiment use random function judge window W_iz[k_i-A_z,k_i+B_zAt least portion in] Whether divided data meets predetermined condition C_z, as a example by the embodiment shown in Figure 21, according to going The rule preset on weight server 103, for potential cut-point k_iDetermine window W_i1[k_i-169,k_i], Judge W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁, such as Figure 32 Shown in, W_i1Represent window W_i1[k_i-169,k_i], for judging W_i1[k_i-169,k_iIn] at least partly Whether data meet predetermined condition C₁, select 5 bytes, serial number 169 in Figure 32,127, 85, the byte " ■ " of 43 and 1 represents 1 byte of selection, adjacent two words selected respectively 42 bytes are differed between joint.By the byte " ■ " of serial number 169,127,85,43 and 1 point It is not converted into a decimal value, is expressed as a₁、a₂、a₃、a₄And a₅.Because 1 Individual byte is formed by 8, so each byte " ■ " is as numerical value, then an a₁、a₂、a₃、 a₄And a₅In any one a_sIt is satisfied by 0≤a_s≤255。a₁、a₂、a₃、a₄And a₅Composition 1*5 Matrix.From the random number obeying binomial distribution, select 256*5 random number, form matrix R, It is expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right),$ 256*5 is selected from the random number obeying binomial distribution Individual random number, forms matrix G, is expressed as: $\left(\begin{array}{cccc}{g}_{0,1}& g_{0,2}& \mathrm{...}& g_{0,5}\\ g_{1,1}& g_{1,2}& \mathrm{...}& g_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ g_{255,5}& g_{255,5}& \mathrm{...}& g_{255,5}\end{array}\right).$

According to a₁Value and the row at place, such as a₁=36, a₁It is positioned at the 1st row, then from matrix R Search h_36,1Corresponding value, searches g from matrix G_36,1Corresponding value；According to a₂Value and The row at place, such as a₂=48, a₂It is positioned at the 2nd row, then from matrix R, looks into h_48,2Corresponding value, G is searched from matrix G_48,2Corresponding value；According to a₃Value and the row at place, such as a₃=26, a₃ It is positioned at the 3rd row, then from matrix R, searches h_26,3Corresponding value, searches g from matrix G_26,3Right The value answered；According to a₄Value and the row at place, such as a₄=26, a₄It is positioned at the 4th row, then from matrix R searches h_26,4Corresponding value, searches g from matrix G_26,4Corresponding value；According to a₅Value and The row at place, such as a₅=88, a₅It is positioned at the 5th row, then from matrix R, searches h_88,5Corresponding value, G is searched from matrix G_88,5Corresponding value.S_1h=h_36,1+h_48,2+h_26,3+h_26,4+h_88,5, because matrix R obeys binomial distribution, therefore, S_1hAlso binomial distribution is obeyed；S_1g=g_36,1+g_48,2+g_26,3+g_26,4+ g_88,5, because matrix G obeys binomial distribution, therefore S_1gAlso binomial distribution is obeyed.Work as S_1hAnd S_1g In have 1 for even number, then W_i1[k_i-169,k_iIn], at least part of data meet predetermined condition C₁, Work as S_1hAnd S_1gIt is odd number, then W_i1[k_i-169,k_iIn], at least part of data are unsatisfactory for predetermined bar Part C₁, C₁The S that statement obtains according to the method described above_1hAnd S_1gIn have 1 for even number.Because S_1hWith S_1gAll obey binomial distribution, therefore S_1hProbability for even number is 1/2, S_1gProbability for even number It is 1/2, S_1hAnd S_1gIn to have 1 probability for even number be 1-1/4=3/4, therefore, W_i1[k_i-169, k_iIn], at least part of data meet predetermined condition C₁Probability be 3/4.In embodiment illustrated in fig. 21 In, W_i1[k_i-169,k_iIn], at least part of data meet predetermined condition C₁.Shown in Figure 21 In embodiment, at W_i1[k_i-169,k_i]、W_i2[k_i-170,k_i-1]、W_i3[k_i-171,k_i-2]、W_i4 [k_i-172,k_i-3]、W_i5[k_i-173,k_i-4]、W_i6[k_i-174,k_i-5]、W_i7[k_i-175,k_i-6]、W_i8 [k_i-176,k_i-7]、W_i9[k_i-177,k_i-8]、W_i10[k_i-178,k_i-9] and W_i11[k_i-179,k_i-10] In, each window size is identical, i.e. window size is 169 bytes, judges in window at least simultaneously The mode whether part data meet predetermined condition is the most identical, is specifically shown in above-mentioned judgement W_i1[k_i-169, k_iIn], whether at least part of data meet predetermined condition C₁Description.Therefore, as shown in figure 32,Represent and judge window W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined bar Part C₂Time 1 byte selecting respectively, in Figure 32, respectively with sequence number 170,128,86, 44 and 2 represent, differ 42 bytes between adjacent two bytes selected.By sequence number 170,128, 86, the byte of 44 and 2It is converted into a decimal value respectively, is expressed as b₁、 b₂、b₃、b₄And b₅.Because 1 byte is formed by 8, so each byteAs one Individual numerical value, then b₁、b₂、b₃、b₄And b₅In any one b_sIt is satisfied by 0≤b_s≤255。b₁、b₂、 b₃、b₄And b₅The matrix of composition 1*5.In present embodiment, it is judged that each window at least partly counts Identical according to the mode whether meeting predetermined condition, the most still use same matrix R and G.Root According to b₁Value and the row at place, such as b₁=66, b₁It is positioned at the 1st row, then from matrix R, searches h_66,1 Corresponding value, searches g from matrix G_66,1Corresponding value；According to b₂Value and the row at place, Such as b₂=48, b₂It is positioned at the 2nd row, then from matrix R, searches h_48,2Corresponding value, from matrix G Search g_48,2Corresponding value；According to b₃Value and the row at place, such as b₃=99, b₃It is positioned at the 3rd row, From matrix R, then search h_99,3Corresponding value, searches g from matrix G_99,3Corresponding value；According to b₄Value and the row at place, such as b₄=26, b₄It is positioned at the 4th row, then from matrix R, searches h_26,4Right The value answered, searches g from matrix G_26,4Corresponding value；According to b₅Value and the row at place, such as b₅ =90, b₅It is positioned at the 5th row, then from matrix R, searches h_90,5Corresponding value, searches from matrix G g_90,5Corresponding value.S_2h=h_66,1+h_48,2+h_99,3+h_26,4+h_90,5, divide because matrix R obeys binomial Cloth, therefore, S_2hAlso binomial distribution is obeyed.S_2g=g_66,1+g_48,2+g_99,3+g_26,4+g_90,5, because Matrix G obeys binomial distribution, therefore, S_2gAlso binomial distribution is obeyed.Work as S_2hAnd S_2gIn have 1 Individual for even number, then W_i2[k_i-170,k_i-1] in, at least part of data meet predetermined condition C₂, work as S_2h And S_2gIt is odd number, then W_i2[k_i-170,k_i-1] in, at least part of data are unsatisfactory for predetermined condition C₂, S_2hAnd S_2gIn to have 1 probability for even number be 3/4.In embodiment illustrated in fig. 21, W_i2 [k_i-170,k_i-1] in, at least part of data meet predetermined condition C₂.Use same rule, point Do not judge W_i3[k_i-171,k_i-2] in, whether at least part of data meet predetermined condition C₃, judge W_i4[k_i-172,k_i-3] in, whether at least part of data meet predetermined condition C₄, judge W_i5[k_i -173,k_i-4] in, whether at least part of data meet predetermined condition C₅, judge W_i6[k_i-174,k_i-5] In at least partly data whether meet predetermined condition C₆, judge W_i7[k_i-175,k_i-6] at least portion in Whether divided data meets predetermined condition C₇, judge W_i8[k_i-176,k_i-7] in, at least part of data are No meet predetermined condition C₈, judge W_i9[k_i-177,k_i-8] in, whether at least part of data meet pre- Fixed condition C₉, judge W_i10[k_i-178,k_i-9] in, whether at least part of data meet predetermined condition C₁₀With judge W_i11[k_i-179,k_i-10] in, whether at least part of data meet predetermined condition C₁₁.Figure In embodiment shown in 21, W_i5[k_i-173,k_i-4] in, at least part of data are unsatisfactory for predetermined bar Part C₅, from potential cut-point k_iAlong data flow point cutpoint search direction jump 7 bytes, The end position of the 7th byte obtains current potential cut-point k_j, as shown in figure 22, according to for The rule that duplicate removal server 103 is preset, for potential cut-point k_jDetermine window W_j1[k_j-169,k_j], Judge window W_j1[k_j-169,k_jIn], whether at least part of data meet predetermined condition C₁Mode With judge window W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁Side Formula is identical, the most as shown in figure 33, and W_j1Represent window W_j1[k_j-169,k_j], for judging W_j1[k_j -169,k_jIn], whether at least part of data meet predetermined condition C₁, serial number 169 in Figure 33,127, 85, the byte " ■ " of 43 and 1 represents 1 byte of selection, adjacent two words selected respectively 42 bytes are differed between joint.By the byte " ■ " of serial number 169,127,85,43 and 1 point It is not converted into a decimal value, is expressed as a₁'、a₂'、a₃'、a₄' and a₅'.Cause It is that 1 byte is formed by 8, so each byte " ■ " is as numerical value, then an a₁'、a₂'、 a₃'、a₄' and a₅Any one a in '_s' it is satisfied by 0≤a_s'≤255。a₁'、a₂'、a₃'、a₄' and a₅' composition 1*5 matrix.Use and judge window W_i1[k_i-169,k_iIn], at least part of data are No meet predetermined condition C₁Identical matrix R and G, is expressed as: $\left(\begin{array}{cccc}{h}_{0,1}& h_{0,2}& \mathrm{...}& h_{0,5}\\ h_{1,1}& h_{1,2}& \mathrm{...}& h_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{255,5}& h_{255,5}& \mathrm{...}& h_{255,5}\end{array}\right)$ With $\left(\begin{array}{cccc}{g}_{0,1}& g_{0,2}& \mathrm{...}& g_{0,5}\\ g_{1,1}& g_{1,2}& \mathrm{...}& g_{1,5}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ g_{255,5}& g_{255,5}& \mathrm{...}& g_{255,5}\end{array}\right).$

According to a₁' value and the row at place, such as a₁'=16, a₁' be positioned at the 1st row, then look into from matrix R Look for h_16,1Corresponding value, searches g from matrix G_16,1Corresponding value；According to a₂' value and place Row, such as a₂'=98, a₂' be positioned at the 2nd row, then from matrix R, search h_98,2Corresponding value, from square Battle array G searches g_98,2Corresponding value；According to a₃' value and the row at place, such as a₃'=56, a₃' position In the 3rd row, then from matrix R, search h_56,3Corresponding value, searches g from matrix G_56,3Corresponding Value；According to a₄' value and the row at place, such as a₄'=36, a₄' it is positioned at the 4th row, then from matrix R Middle lookup h_36,4Corresponding value, searches g from matrix G_36,4Corresponding value；According to a₅' value and The row at place, such as a₅'=99, a₅' be positioned at the 5th row, then from matrix R, search h_99,5Corresponding Value, searches g from matrix G_99,5Corresponding value.S_1h'=h_16,1+h_98,2+h_56,3+h_36,4+h_99,5, because of Binomial distribution, therefore, S is obeyed for matrix R_1h' also obey binomial distribution；S_1g'=g_16,1+g_98,2+ g_56,3+g_36,4+g_99,5, because matrix G obeys binomial distribution, therefore S_1g' also obey binomial distribution. Work as S_1h' and S_1g1 is had for even number, then W in '_j1[k_j-169,k_jIn], at least part of data meet pre- Fixed condition C₁, work as S_1h' and S_1g' be odd number, then W_j1[k_j-169,k_jAt least part of data in] It is unsatisfactory for predetermined condition C₁, S_1h' and S_1g' to have 1 probability for even number be 3/4.

Judge W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 33,Represent and judge window W_j2[k_j-170,k_j-1] in extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select 42 bytes are differed between byte.In fig. 33, respectively by sequence number 170,128,86,44 Represent with 2, between adjacent two bytes selected, differ 42 bytes.By sequence number 170,128, 86, the byte of 44 and 2It is converted into a decimal value respectively, is expressed as b₁'、 b₂'、b₃'、b₄' and b₅'.Because 1 byte is formed by 8, so each byteAs One numerical value, then b₁'、b₂'、b₃'、b₄' and b₅Any one b in '_s' it is satisfied by 0≤b_s'≤255。 b₁'、b₂'、b₃'、b₄' and b₅' composition 1*5 matrix.Use and judge window W_j2[k_j-170,k_j -1] in, whether at least part of data meet predetermined condition C₂Identical matrix R and G, according to b₁' Value and the row at place, such as b₁'=210, b₁' be positioned at the 1st row, then from matrix R, search h_210,1Right The value answered, searches g from matrix G_210,1Corresponding value；According to b₂' value and the row at place, as b₂'=156, b₂' be positioned at the 2nd row, then from matrix R, search h_156,2Corresponding value, from matrix G Search g_156,2Corresponding value；According to b₃' value and the row at place, such as b₃'=144, b₃' it is positioned at the 3rd Row, then search h from matrix R_144,3Corresponding value, searches g from matrix G_144,3Corresponding value； According to b₄' value and the row at place, such as b₄'=60, b₄' be positioned at the 4th row, then look into from matrix R Look for h_60,4Corresponding value, searches g from matrix G_60,4Corresponding value；According to b₅' value and place Row, such as b₅'=90, b₅' be positioned at the 5th row, then from matrix R, search h_90,5Corresponding value, from square Battle array G searches g_90,5Corresponding value.S_2h'=h_210,1+h_156,2+h_144,3+h_60,4+h_90,5,S_2g'=g_210,1+ g_156,2+g_144,3+g_60,4+g_90,5.Work as S_2h' and S_2g1 is had for even number, then W in '_j2[k_j-170,k_j -1] in, at least part of data meet predetermined condition C₂, work as S_2h' and S_2g' be odd number, then W_j2[k_j -170,k_j-1] in, at least part of data are unsatisfactory for predetermined condition C₂, S_2h' and S_2g1 is had for even in ' The probability of number is 3/4.

In like manner, it is judged that W_i3[k_i-171,k_i-2] in, whether at least part of data meet predetermined condition C₃ Mode with judge W_j3[k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₃ Mode identical, in like manner, it is judged that W_j4[k_j-172,k_j-3] in, whether at least part of data meet pre- Fixed condition C₄, judge W_j5[k_j-173,k_j-4] in, whether at least part of data meet predetermined condition C₅、 Judge W_j6[k_j-174,k_j-5] in, whether at least part of data meet predetermined condition C₆, judge W_j7 [k_j-175,k_j-6] in, whether at least part of data meet predetermined condition C₇, judge W_j8[k_j-176, k_j-7] in, whether at least part of data meet predetermined condition C₈, judge W_j9[k_j-177,k_j-8] in At least partly whether data meet predetermined condition C₉, judge W_j10[k_j-178,k_j-9] at least portion in Whether divided data meets predetermined condition C₁₀With judge W_j11[k_j-179,k_j-10] at least partly count in According to whether meeting predetermined condition C₁₁, do not repeat them here.

The present embodiment use random function judge window W_iz[k_i-A_z,k_i+B_zAt least portion in] Whether divided data meets predetermined condition C_z, as a example by the embodiment shown in Figure 21, according to going The rule preset on weight server 103, for potential cut-point k_iDetermine window W_i1[k_i-169,k_i], Judge W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁, such as Figure 32 Shown in, W_i1Represent window W_i1[k_i-169,k_i], for judging W_i1[k_i-169,k_iIn] at least partly Whether data meet predetermined condition C₁, select 5 bytes, serial number 169 in Figure 32,127, 85, the byte " ■ " of 43 and 1 represents 1 byte of selection, adjacent two words selected respectively 42 bytes are differed between joint.The byte " ■ " of serial number 169,127,85,43 and 1 is depended on Secondary regard 40 positions as, be expressed as a₁、a₂、a₃、a₄…a₄₀。a₁、a₂、a₃、a₄… a₄₀In arbitrary a_t, work as a_tWhen=0, V_at=-1, works as a_tWhen=1, V_at=1, according to a_tWith V_at Corresponding relation, generates V_a1、V_a2、V_a3、V_a4…V_a40.From the random number of Normal Distribution 40 randoms number of middle selection, are expressed as: h₁、h₂、h₃、h₄...h₄₀。S_a=V_a1*h₁+ V_a2*h₂+V_a3*h₃+V_a4*h₄+…+V_a40*h₄₀.Because h₁、h₂、h₃、h₄...h₄₀Just obey State is distributed, therefore, and S_aAlso Normal Distribution.Work as S_aFor positive number, then W_i1[k_i-169,k_i] In at least partly data meet predetermined condition C₁, work as S_aFor negative or 0, then W_i1[k_i-169,k_i] In at least partly data be unsatisfactory for predetermined condition C₁, S_aProbability for positive number is 1/2.At Figure 21 In illustrated embodiment, W_i1[k_i-169,k_iIn], at least part of data meet predetermined condition C₁.As Shown in Figure 32,Represent and judge window W_i2[k_i-170,k_i-1] in, whether at least part of data Meet predetermined condition C₂Time 1 byte selecting respectively, in Figure 32, respectively with sequence number 170, 128,86,44 and 2 represent, differ 42 bytes between adjacent two bytes selected.By sequence The byte of numbers 170,128,86,44 and 2Regard 40 positions successively as, be expressed as b₁、 b₂、b₃、b₄…b₄₀。b₁、b₂、b₃、b₄…b₄₀In arbitrary b_t, work as b_tWhen=0, V_bt=-1, Work as b_tWhen=1, V_bt=1, according to b_tWith V_btCorresponding relation, generates V_b1、V_b2、V_b3、V_b4…V_b40。 Judge window W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁Mode With judge window W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂'s Mode is identical, therefore, uses identical random number: h₁、h₂、h₃、h₄...h₄₀, S_b=V_b1 *h₁+V_b2*h₂+V_b3*h₃+V_b4*h₄+…+V_b40*h₄₀.Because h₁、h₂、h₃、h₄...h₄₀Clothes From normal distribution, therefore, S_bAlso Normal Distribution.Work as S_bFor positive number, then W_i2[k_i-170, k_i-1] in, at least part of data meet predetermined condition C₂, work as S_bFor negative or 0, then W_i2[k_i-170, k_i-1] in, at least part of data are unsatisfactory for predetermined condition C₂, S_bProbability for positive number is 1/2.? In embodiment illustrated in fig. 21, W_i2[k_i-170,k_i-1] in, at least part of data meet predetermined condition C₂.Use same rule, judge W respectively_i3[k_i-171,k_i-2] in, whether at least part of data Meet predetermined condition C₃, judge W_i4[k_i-172,k_i-3] in, whether at least part of data meet predetermined Condition C₄, judge W_i5[k_i-173,k_i-4] in, whether at least part of data meet predetermined condition C₅、 Judge W_i6[k_i-174,k_i-5] in, whether at least part of data meet predetermined condition C₆, judge W_i7 [k_i-175,k_i-6] in, whether at least part of data meet predetermined condition C₇, judge W_i8[k_i-176,k_i -7] in, whether at least part of data meet predetermined condition C₈, judge W_i9[k_i-177,k_i-8] at least Whether part data meet predetermined condition C₉, judge W_i10[k_i-178,k_i-9] at least part of data in Whether meet predetermined condition C₁₀With judge W_i11[k_i-179,k_i-10] in, whether at least part of data Meet predetermined condition C₁₁.In embodiment shown in Figure 21, W_i5[k_i-173,k_i-4] at least portion in Divided data is unsatisfactory for predetermined condition C₅, from potential cut-point k_iAlong data flow point cutpoint lookup side To 7 bytes of jumping, the end position the 7th byte obtains current potential cut-point k_j, as Shown in Figure 22, according to the rule preset for duplicate removal server 103, for potential cut-point k_jDetermine Window W_j1[k_j-169,k_j], it is judged that window W_j1[k_j-169,k_jIn], whether at least part of data meet Predetermined condition C₁Mode with judge window W_i1[k_i-169,k_iIn], at least part of data are the fullest Foot predetermined condition C₁Mode identical, the most as shown in figure 33, W_j1Represent window W_j1[k_j-169, k_j], for judging W_j1[k_j-169,k_jIn], whether at least part of data meet predetermined condition C₁, choosing Select 5 bytes, the byte " ■ " of serial number 169,127,85,43 and 1 table respectively in Figure 33 Show 1 byte of selection, between adjacent two bytes selected, differ 42 bytes.By sequence number Be 169,127,85,43 and 1 byte " ■ " regard 40 positions successively as, be expressed as a₁'、 a₂'、a₃'、a₄'…a₄₀'。a₁'、a₂'、a₃'、a₄'…a₄₀Arbitrary a in '_t', work as a_t'=0 Time, V_at'=-1, works as a_tDuring '=1, V_at'=1, according to a_t' and V_at' corresponding relation, generate V_a1'、 V_a2'、V_a3'、V_a4'…V_a40'.Judge window W_j1[k_j-169,k_jIn], whether at least part of data Meet predetermined condition C₁Mode with judge window W_i1[k_i-169,k_iIn], at least part of data are No meet predetermined condition C₁Mode identical, therefore use identical random number: h₁、h₂、h₃、 h₄...h₄₀。S_a'=V_a1'*h₁+V_a2'*h₂+V_a3'*h₃+V_a4'*h₄+…+V_a40'*h₄₀.Because h₁、 h₂、h₃、h₄...h₄₀Normal Distribution, therefore, S_a' also Normal Distribution.Work as S_a' it is Positive number, then W_j1[k_j-169,k_jIn], at least part of data meet predetermined condition C₁, work as S_a' it is negative Number or 0, then W_j1[k_j-169,k_jIn], at least part of data are unsatisfactory for predetermined condition C₁, S_a' just it is The probability of number is 1/2.

Judge W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂Side Formula and judge W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₂Side Formula is identical, therefore, as shown in figure 33,Represent and judge window W_j2[k_j-170,k_j-1] in extremely Whether small part data meet predetermined condition C₂Time select 1 byte, adjacent two select 42 bytes are differed between byte.In fig. 33, respectively by sequence number 170,128,86,44 Represent with 2, between adjacent two bytes selected, differ 42 bytes.By sequence number 170,128, 86, the byte of 44 and 2Regard 40 positions successively as, be expressed as b₁'、b₂'、b₃'、b₄'… b₄₀'。b₁'、b₂'、b₃'、b₄'…b₄₀Arbitrary b in '_t', work as b_tDuring '=0, V_bt'=-1, works as b_t'=1 Time, V_bt'=1, according to b_t' and V_bt' corresponding relation, generate V_b1'、V_b2'、V_b3'、V_b4'…V_b40'。 Judge W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂Mode and Judge W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₂Mode phase With, therefore, use identical random number: h₁、h₂、h₃、h₄...h₄₀, S_b'=V_b1'*h₁+V_b2' *h₂+V_b3'*h₃+V_b4'*h₄+…+V_b40'*h₄₀.Because h₁、h₂、h₃、h₄...h₄₀Just obey State is distributed, therefore, and S_b' also Normal Distribution.Work as S_b' for positive number, then W_j2[k_j-170,k_j-1] In at least partly data meet predetermined condition C₂, work as S_b' for negative or 0, then W_j2[k_j-170,k_j -1] in, at least part of data are unsatisfactory for predetermined condition C₂, S_b' it is 1/2 for the probability of positive number.

In like manner, it is judged that W_i3[k_i-171,k_i-2] in, whether at least part of data meet predetermined condition C₃Mode with judge W_j3[k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₃Mode identical, in like manner, it is judged that W_j4[k_j-172,k_j-3] in, whether at least part of data meet Predetermined condition C₄, judge W_j5[k_j-173,k_j-4] in, whether at least part of data meet predetermined condition C₅, judge W_j6[k_j-174,k_j-5] in, whether at least part of data meet predetermined condition C₆, judge W_j7[k_j-175,k_j-6] in, whether at least part of data meet predetermined condition C₇, judge W_j8[k_j -176,k_j-7] in, whether at least part of data meet predetermined condition C₈, judge W_j9[k_j-177,k_j-8] In at least partly data whether meet predetermined condition C₉, judge W_j10[k_j-178,k_j-9] at least Whether part data meet predetermined condition C₁₀With judge W_j11[k_j-179,k_j-10] at least partly Whether data meet predetermined condition C₁₁, do not repeat them here.

The present embodiment use random function judge window W_iz[k_i-A_z,k_i+B_zAt least portion in] Whether divided data meets predetermined condition C_z, still as a example by Figure 21 illustrated embodiment, according to The rule preset on duplicate removal server 103, for potential cut-point k_iDetermine window W_i1[k_i-169, k_i], it is judged that W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁, as Shown in Figure 32, W_i1Represent window W_i1[k_i-169,k_i], for judging W_i1[k_i-169,k_iIn] at least Whether part data meet predetermined condition C₁, select 5 bytes, serial number 169 in Figure 32,127, 85, the byte " ■ " of 43 and 1 represents 1 byte of selection, adjacent two words selected respectively 42 bytes are differed between joint.The byte " ■ " of serial number 169,127,85,43 and 1 is turned Changing 1 decimal number into, scope is 0-(2^40-1), uses uniform random number maker 1 designated value, record 0-(2^40-1) is generated for each decimal number in 0-(2^40-1) In each decimal number and designated value between corresponding relation R, once specify, this ten enters The designated value that number processed is corresponding is the most constant, and this designated value is obeyed and is uniformly distributed, if this designated value is Even number, then W_i1[k_i-169,k_iIn], at least part of data meet predetermined condition C₁If this refers to Definite value is odd number, then W_i1[k_i-169,k_iIn], at least part of data are unsatisfactory for predetermined condition C₁, C₁ Represent that the designated value obtained according to the method described above is even number.Because equally distributed random number is even The probability of number is 1/2, therefore, W_i1[k_i-169,k_iIn], at least part of data meet predetermined condition C₁ Probability be 1/2.In the embodiment shown in Figure 21, use same rule, sentence respectively Disconnected W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂, it is judged that W_i3[k_i -171,k_i-2] in, whether at least part of data meet predetermined condition C₃, judge W_i4[k_i-172,k_i-3] In at least partly data whether meet predetermined condition C₄, judge W_i5[k_i-173,k_i-4] at least portion in Whether divided data meets predetermined condition C₅, do not repeat them here.

Work as W_i5[k_i-173,k_i-4] in, at least part of data are unsatisfactory for predetermined condition C₅, from potential point Cutpoint k_iAlong data flow point cutpoint search direction 7 bytes of jump, in the end of the 7th byte Position obtains current potential cut-point k_j, as shown in figure 22, according to pre-for duplicate removal server 103 If rule, for potential cut-point k_jDetermine window W_j1[k_j-169,k_j], it is judged that window W_j1[k_j -169,k_jIn], whether at least part of data meet predetermined condition C₁Mode with judge window W_i1 [k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁Mode identical, therefore, Use the corresponding pass between each decimal number with the designated value in identical 0-(2^40-1) It is R, as shown in figure 33, W_j1Represent window, for judging W_j1[k_j-169,k_jIn] at least partly Whether data meet predetermined condition C₁, selecting 5 bytes, in Figure 33, " ■ " represents 1 selected Individual byte, differs 42 bytes between adjacent two bytes " ■ " selected.By serial number 169, 127, the byte " ■ " of 85,43 and 1 is converted into 1 decimal number, searches this decimal scale at R The designated value that number is corresponding, if this designated value is even number, then W_j1[k_j-169,k_jIn] at least partly Data meet predetermined condition C₁If this designated value is odd number, then W_j1[k_j-169,k_jIn] at least Part data are unsatisfactory for predetermined condition C₁, because the probability that equally distributed random number is even number is 1/2, therefore, W_j1[k_j-169,k_jIn], at least part of data meet predetermined condition C₁Probability be 1/2.In like manner, it is judged that W_i2[k_i-170,k_i-1] in, whether at least part of data meet predetermined condition C₂ Mode and judge W_j2[k_j-170,k_j-1] in, whether at least part of data meet predetermined condition C₂ Mode identical, it is judged that W_i3[k_i-171,k_i-2] in, whether at least part of data meet predetermined condition C₃Mode with judge W_j3[k_j-171,k_j-2] in, whether at least part of data meet predetermined condition C₃Mode identical, in like manner, it is judged that W_j4[k_j-172,k_j-3] in, whether at least part of data meet Predetermined condition C₄, judge W_j5[k_j-173,k_j-4] in, whether at least part of data meet predetermined condition C₅, judge W_j6[k_j-174,k_j-5] in, whether at least part of data meet predetermined condition C₆, judge W_j7[k_j-175,k_j-6] in, whether at least part of data meet predetermined condition C₇, judge W_j8[k_j -176,k_j-7] in, whether at least part of data meet predetermined condition C₈, judge W_j9[k_j-177,k_j-8] In at least partly data whether meet predetermined condition C₉, judge W_j10[k_j-178,k_j-9] at least Whether part data meet predetermined condition C₁₀With judge W_j11[k_j-179,k_j-10] at least partly Whether data meet predetermined condition C₁₁, do not repeat them here.

Duplicate removal server 103 in the embodiment of the present invention shown in Fig. 1, refers to realize this The device of the technical scheme described by bright embodiment, as shown in figure 18, generally includes central authorities' process Unit, main storage and input/output interface.CPU, main storage and input The intercommunication of output interface, main memory store executable instruction, CPU is held The executable instruction of storage in row main storage, thus perform specific function, make duplicate removal service Device 103 possesses specific function, the lookup data as described by embodiment of the present invention Figure 20 to Figure 33 Flow point cutpoint.Therefore, as shown in figure 19, according to the embodiment of the present invention shown in 20 to Figure 33, Duplicate removal server 103, is preset with rule on duplicate removal server 103, and described rule is: be latent M window W is determined at cut-point k_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] right Predetermined condition C answered_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor Integer；

Duplicate removal server 103 includes determining unit 1901 and judging processing unit 1902.Wherein, Determine that unit 1901 is for performing step a):

A) it is current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz[k_i-A_z, k_i+B_z], i and z is integer, and 1≤z≤M；

Judge processing unit 1902, be used for judging described window W_iz[k_i-A_z,k_i+B_zIn] at least Whether part data meet predetermined condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described current potential cut-point k_iAlong described data flow point cutpoint search direction jump N Individual data flow point cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖), Obtain new potential cut-point, the most described determine that unit 1901 is that described new potential cut-point is held Row step a)；

As described current potential cut-point k_iM window in each window W_ix[k_i-A_x,k_i +B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor number According to flow point cutpoint.

Further, described rule also includes: at least two window W_ie[k_i-A_e,k_i+B_e] and W_if [k_i-A_f,k_i+B_f], meet condition: | A_e+B_e|=| A_f+B_f|, C_e=C_f.Further, Described rule also includes: A_eAnd A_fFor positive integer.Further, described rule also includes: A_e -1=A_f, B_e+ 1=B_f。

Further, it is judged that processing unit 1902 judges window specifically for using random function W_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet predetermined condition C_z.Further, Judge that processing unit 1902 specifically used hash function judges window W_iz[k_i-A_z,k_i+B_zIn] extremely Whether small part data meet predetermined condition C_z。

Further, it is judged that processing unit 1902 is for as described window W_iz[k_i-A_z,k_i+B_z] In at least partly data be unsatisfactory for described predetermined condition C_z, from described current potential cut-point k_iEdge The described data flow point cutpoint search direction N number of data flow point cutpoint minimum of jump searches unit U, obtains Described new potential cut-point, described determine that unit 1901 be that described potential cut-point newly is held Row step a), according to described rule, the window W determined for described new potential cut-point_ic[k_i -A_c,k_i+B_c] left margin and described window W_iz[k_i-A_z,k_i+B_z] right margin overlap or The described window W determined for described new potential cut-point_ic[k_i-A_c,k_i+B_c] left margin position In described window W_iz[k_i-A_z,k_i+B_zWithin the scope of]；Wherein, for described new potential segmentation The described window W that point determines_ic[k_i-A_c,k_i+B_c] it is according to described rule, for described new diving Sequence the in the sequence that M the window determined at cut-point obtains according to data stream search direction The window of one.

Further, it is judged that processing unit 1902 uses random function to judge described window W_iz[k_i -A_z,k_i+B_zIn], whether at least part of data meet described predetermined condition C_z, specifically include:

At described window W_iz[k_i-A_z,k_i+B_zF byte is selected, by anti-for described F byte in] Utilizing H time again, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& \mathrm{...}& a_{1,8}\\ a_{2,1}& a_{2,2}& \mathrm{...}& a_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ a_{F*H,1}& a_{F*H,2}& \mathrm{...}& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& \mathrm{...}& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& \mathrm{...}& V_{a2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ V_{aF*H,1}& V_{aF*H,2}& \mathrm{...}& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& \mathrm{...}& h_{1,8}\\ h_{2,1}& h_{2,2}& \mathrm{...}& h_{2,8}\\ .& .& .& .\\ .& .& .& .\\ .& .& .& .\\ h_{F*H,1}& h_{F*H,2}& \mathrm{...}& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[k_i-A_z, k_i+B_zIn], at least part of data meet described predetermined condition C_z。

According to shown in 20 to Figure 33 the embodiment of the present invention provide based on whois lookup data In the method for flow point cutpoint, for potential cut-point k_iDetermine window W_ix[k_i-A_x, k_i+B_x], its In, x is respectively 1 and arrives M continuous print natural number, M >=2, can judge in M window every parallel In one window, whether at least part of data meet predetermined condition C_x, or judge successively in window At least partly whether data meet predetermined condition, it is also possible to window W successively_i1[k_i-A₁, k_i+B₁], In at least partly data meet predetermined condition C₁Time, then judge W_i2[k_i-A₂, k_i+B₂In] at least Part data meet predetermined condition C₂Time, until judging W_im[k_i-A_m, k_i+B_mAt least portion in] Divided data meets predetermined condition C_m.In embodiment, the judgement of other windows is identical with this, no longer Repeat.

It addition, according to the embodiment of the present invention shown in 20 to Figure 33, on duplicate removal server 103 It is preset with rule, described rule: determine M window W for potential cut-point k_x[k-A_x,k+B_x] With window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, x is respectively 1 to M continuous print certainly So number, M >=2, in this preset rules, A₁、A₂、A₃…A_mCan not be the most equal, B₁、 B₂、B₃…B_mCan not be the most equal, C₁、C₂、C₃…C_MCan not also be the most identical. In the embodiment shown in Figure 21, at W_i1[k_i-169,k_i]、W_i2[k_i-170,k_i-1]、W_i3[k_i -171,k_i-2]、W_i4[k_i-172,k_i-3]、W_i5[k_i-173,k_i-4]、W_i6[k_i-174,k_i-5]、W_i7 [k_i-175,k_i-6]、W_i8[k_i-176,k_i-7]、W_i9[k_i-177,k_i-8]、W_i10[k_i-178,k_i-9] and W_i11[k_i-179,k_i-10] in, each window size is identical, i.e. window size is 169 bytes, simultaneously Judge that the mode that in window, whether at least part of data meet predetermined condition is the most identical, be specifically shown in State and judge W_i1[k_i-169,k_iIn], whether at least part of data meet predetermined condition C₁Description, but In the embodiment shown in Figure 11, W_i1[k_i-169,k_i]、W_i2[k_i-170,k_i-1]、W_i3[k_i -171,k_i-2]、W_i4[k_i-172,k_i-3]、W_i5[k_i-173,k_i-4]、W_i6[k_i-174,k_i-5]、 W_i7[k_i-175,k_i-6]、W_i8[k_i-176,k_i-7]、W_i9[k_i-177,k_i-8]、W_i10[k_i-168, k_i+ 1] and W_i11[k_i-179,k_i+ 3] window size can differ, and judges in window at least simultaneously Whether part data meet the mode of predetermined condition can also differ.In all embodiments, According to the rule preset for duplicate removal server 103, it is judged that window W_i1In at least partly whether data Meet predetermined condition C₁Mode with judge window W_j1In at least partly data whether meet predetermined Condition C₁Mode inevitable the most identical, it is judged that W_i2In at least partly data whether meet predetermined condition C₂Mode with judge W_j2In at least partly data whether meet predetermined condition C₂Mode inevitable Identical ... to judge window W_iMIn at least partly data whether meet predetermined condition C_MMode with Judge window W_jMIn at least partly data whether meet predetermined condition C_MMode inevitable the most identical. Do not repeat them here.

According to the embodiment of the present invention shown in 20 to Figure 33, duplicate removal server 103 is preset with Rule, k_a、k_i、k_j、k_lAnd k_mFor searching cut-point along data flow point cutpoint search direction Time obtain potential cut-point, k_a、k_i、k_j、k_lAnd k_mAll according to this rule.The present invention is real Execute the window W in example_x[k-A_x,k+B_x] represent a particular range, select at this particular range Data are to judge whether these data meet predetermined condition C_x, specifically, can be at this specific model Enclose interior selection part data, it is also possible to select total data pre-to judge whether these data meet Fixed condition C_x.Window concept specifically used in the embodiment of the present invention can refer to window W_x[k-A_x, k+B_x], do not repeat them here.

Window W_x[k-A_x,k+B_xIn], (k-A_x) and (k+B_x) represent this window W_x[k- A_x,k+B_x] two borders, wherein (k-A_x) represent window W_x[k-A_x,k+B_x] relatively It is positioned at data flow point cutpoint in potential cut-point k and searches reciprocal border, (k+B_x) table Show window W_x[k-A_x,k+B_x] it is positioned at the lookup of data flow point cutpoint relative to potential cut-point k The border in direction.Specifically, in embodiments of the present invention, in the data shown in Figure 20 to Figure 33 Flow point cutpoint search direction is from left to right, wherein (k-A_x) represent window W_x[k-A_x,k+ B_x] it is positioned at data flow point cutpoint lookup (the i.e. left side, reciprocal border relative to potential cut-point k Boundary), (k+B_x) represent window W_x[k-A_x,k+B_x] it is positioned at number relative to potential cut-point k Border (i.e. right margin) according to flow point cutpoint search direction.If shown in Figure 20 to Figure 33 Data flow point cutpoint search direction is from right to left, wherein (k-A_x) represent window W_x[k-A_x, k+B_x] it is positioned at the reciprocal border of data flow point cutpoint lookup (i.e. relative to potential cut-point k Right margin), (k+B_x) represent window W_x[k-A_x,k+B_x] relative to potential cut-point k position Border (i.e. left margin) in data flow point cutpoint search direction.

Those of ordinary skill in the art are it is to be appreciated that combine embodiment of the present invention Figure 20 to Figure 33 The unit of each example described and algorithm steps, the key feature of the embodiment of the present invention can be with it He combines at technology, presents with increasingly complex form, but still can comprise the crucial special of the present invention Levy.May use standby cut-point in true environment, such as one embodiment is, according to The rule preset for duplicate removal server 103, for potential cut-point k_iDetermine 11 window W_x[k-A_x, k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, x be 1 to 11 continuous print from So number, as each window W in 11 windows_x[k-A_x,k+B_xIn], at least part of data are the fullest Foot predetermined condition C_x, the most potential cut-point k_iFor data flow point cutpoint, when the maximum exceeding setting During data block, do not find cut-point yet, at this moment may use standby preset rules, standby Preset rules is similar with the rule preset on duplicate removal server 103, and standby preset rules is: The most potential cut-point k_iDetermine 10 window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k +B_x] corresponding predetermined condition C_x, x is 1 to 10 continuous print natural numbers, determines when in 10 windows Each window W_x[k-A_x,k+B_xIn], at least part of data are satisfied by predetermined condition C_x, then dive At cut-point k_iFor data flow point cutpoint, when exceeding the maximum data block of setting, search not yet During to data flow point cutpoint, from the end position of maximum data block as force-splitting point.

According to the embodiment of the present invention shown in 20 to Figure 33, on duplicate removal server 103 It is preset with rule, described rule determines M window for potential cut-point k, and differs Provisioning request first has a potential cut-point k, and M the window that can be determined by judges Potential cut-point k.

Those of ordinary skill in the art are it is to be appreciated that combine enforcement disclosed herein The unit of each example that example describes and algorithm steps, it is possible to electronic hardware or calculating Being implemented in combination in of machine software and electronic hardware.These functions are actually with hardware or software Mode performs, and depends on application-specific and the design constraint of technical scheme.Specialty Technical staff can to each specifically should be used for using different methods to realize described Function, but this realization is it is not considered that beyond the scope of this invention.

Those skilled in the art is it can be understood that arrive, for the convenience described and letter Clean, the specific works process of the system of foregoing description, device and unit, before being referred to State the corresponding process in embodiment of the method, do not repeat them here.

In the several embodiments provided, it should be understood that disclosed system, method, Can realize by another way.Such as, device embodiment described above is only It is schematic, such as, the division of described unit, it is only a kind of logic function and divides, Actual can have other dividing mode, the most multiple unit or assembly to tie when realizing Close or be desirably integrated into another system, or some features can be ignored, or not performing. Another point, shown or discussed coupling each other or direct-coupling or communication connection Can be the INDIRECT COUPLING by some interfaces, device or unit or communication connection, permissible It is electrical, machinery or other form.

The described unit illustrated as separating component can be or may not be physically Separate, the parts shown as unit can be or may not be physical location, I.e. may be located at a place, or can also be distributed on multiple NE.Permissible Select some or all of unit therein to realize the present embodiment side according to the actual needs The purpose of case.

It addition, each functional unit in each embodiment of the present invention can be integrated in one In processing unit, it is also possible to be that unit is individually physically present, it is also possible to two or two Individual above unit is integrated in a unit.

If described function realizes and as independent product using the form of SFU software functional unit When selling or use, an embodied on computer readable non-volatile memory medium can be stored in In.Based on such understanding, technical scheme is the most in other words to existing skill Part or the part of this technical scheme that art contributes can be with the forms of software product Embodying, this computer software product is stored in a non-volatile memory medium, Including some instructions with so that a computer equipment (can be personal computer, take Business device, or the network equipment etc.) perform the whole of method described in each embodiment of the present invention Or part steps.And aforesaid non-volatile memory medium includes: USB flash disk, portable hard drive, Read only memory (Read-Only Memory, ROM), magnetic disc or CD etc. are various The medium of program code can be stored.

The above, the only detailed description of the invention of the present invention, but the protection model of the present invention Enclosing and be not limited thereto, any those familiar with the art the invention discloses Technical scope in, can readily occur in change or replace, all should contain the guarantor in the present invention Within the scope of protecting.Therefore, protection scope of the present invention answers the described protection with claim Scope is as the criterion.

Claims (48)

1. a method based on whois lookup data flow point cutpoint, it is characterised in that:

Being preset with rule on described server, described rule is: determine for potential cut-point k M some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x, p_x+B_x] corresponding predetermined condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_x And B_xFor integer；

Described method includes:

A) it is current potential cut-point k according to described rule_iDetermine a p_izAnd described some p_izCorresponding Window W_iz[p_iz-A_z,p_iz+B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet Predetermined condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined Condition C_z, from described some p_izAlong the described data flow point cutpoint search direction N number of data flow point of jump Cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i-p_ix) ‖), Obtain new potential cut-point, perform step a)；

C) as described current potential cut-point k_iM window in each window W_ix[p_ix- A_x,p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential segmentation Point k_iFor data flow point cutpoint.

Method the most according to claim 1, it is characterised in that described rule also includes: At least two point p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f。

Method the most according to claim 2, it is characterised in that described rule also includes: Described at least two point p_eAnd p_f, relative to described potential cut-point k, at described data flow point Cutpoint searches in the reverse direction.

The most according to the method in claim 2 or 3, it is characterised in that described rule is also wrapped Include: described at least two point p_eAnd p_fBetween distance be 1 U.

5. according to the arbitrary described method of claims 1 to 3, it is characterised in that judge described Window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z, Specifically include:

Random function is used to judge described window W_iz[p_iz-A_z,p_iz+B_zAt least part of data in] Whether meet described predetermined condition C_z。

Method the most according to claim 5, it is characterised in that described use random function Judge described window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined Condition C_z, it is specially and uses hash function to judge described window W_iz[p_iz-A_z,p_iz+B_zIn] extremely Whether small part data meet described predetermined condition C_z。

7. according to the arbitrary described method of claims 1 to 3, it is characterised in that when described window Mouth W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined condition C_z, from described Point p_izSearch along the described data flow point cutpoint search direction N number of data flow point cutpoint minimum of jump Unit U, it is thus achieved that described new potential cut-point, according to described rule, for described new potential The point p that cut-point determines_icCorresponding window W_ic[p_ic-A_c,p_ic+B_c] left margin and described window Mouth W_iz[p_iz-A_z,p_iz+B_z] right margin overlap or determine for described new potential cut-point Described some p_icCorresponding described window W_ic[p_ic-A_c,p_ic+B_c] left margin be positioned at described window W_iz[p_iz-A_z,p_iz+B_zWithin the scope of]；Wherein, the institute determined for described new potential cut-point State a p_icIt is according to described rule, puts according to number for M determined for described new potential cut-point According to the point of sequence first in the sequence that stream search direction obtains.

Method the most according to claim 5, it is characterised in that use random function to judge Described window W_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet described predetermined condition C_z, specifically include:

At described window W_iz[p_iz-A_z,p_iz+B_zF byte is selected, by described F byte in] Recycling H time, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& ...& a_{1,8}\\ a_{2,1}& a_{2,2}& ...& a_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ a_{F*H,1}& a_{F*H,2}& ...& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& ...& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& ...& V_{a2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ V_{aF*H,1}& V_{aF*H,2}& ...& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& ...& h_{1,8}\\ h_{2,1}& h_{2,2}& ...& h_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ h_{F*H,1}& h_{F*H,2}& ...& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[p_iz-A_z, p_iz+B_zIn], at least part of data meet described predetermined condition C_z。

9. a method based on whois lookup data flow point cutpoint, it is characterised in that

Being preset with rule on described server, described rule is: determine for potential cut-point k M window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；

Described method includes:

A) it is current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz[k_i-A_z, k_i+B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet pre- Fixed condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described pre- Fixed condition C_z, from described current potential cut-point k_iLook into along described data flow point cutpoint Looking for the direction N number of data flow point cutpoint minimum of jump to search unit U, N*U is not more than ‖ B_z‖+max_x(‖A_x‖), it is thus achieved that new potential cut-point, step a) is performed；

C) as described current potential cut-point k_iM window in each window W_ix[k_i-A_x, k_i+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

Method the most according to claim 9, it is characterised in that described rule also includes: At least two window W_ie[k_i-A_e,k_i+B_e] and W_if[k_i-A_f,k_i+B_f], meet condition: | A_e+B_e|=| A_f+B_f|, C_e=C_f。

11. methods according to claim 10, it is characterised in that described rule also includes: A_eAnd A_fFor positive integer.

12. according to the method described in claim 10 or 11, it is characterised in that described rule is also Including: A_e-1=A_f, B_e+ 1=B_f。

13. according to the arbitrary described method of claim 9 to 11, it is characterised in that judge institute State window W_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet described predetermined condition C_z, Specifically include:

Random function is used to judge described window W_iz[k_i-A_z,k_i+B_zAt least part of data in] Whether meet described predetermined condition C_z。

14. methods according to claim 13, it is characterised in that the random letter of described use Number judges described W_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet described predetermined bar Part C_z, it is specially and uses hash function to judge described W_iz[k_i-A_z,k_i+B_zAt least partly count in] According to whether meeting described predetermined condition C_z。

15. according to the arbitrary described method of claim 9 to 11, it is characterised in that when described Window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined condition C_z, from institute State current potential cut-point k_iAlong the described data flow point cutpoint search direction N number of data flow point of jump Cutpoint minimum searches unit U, it is thus achieved that described new potential cut-point, according to described rule, for The window W that described new potential cut-point determines_ic[k_i-A_c,k_i+B_c] left margin and described window Mouth W_iz[k_i-A_z,k_i+B_z] right margin overlap or determine for described new potential cut-point Described window W_ic[k_i-A_c,k_i+B_c] left margin be positioned at described window W_iz[k_i-A_z,k_i+ B_zWithin the scope of]；Wherein, the described window W determined for described new potential cut-point_ic[k_i- A_c,k_i+B_c] be according to described rule, M the window determined for described new potential cut-point by According to the window of sequence first in the sequence that data stream search direction obtains.

16. methods according to claim 13, it is characterised in that use random function to sentence Disconnected described window W_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet described predetermined bar Part C_z, specifically include:

At described window W_iz[k_i-A_z,k_i+B_zF byte is selected, by anti-for described F byte in] Utilizing H time again, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& ...& a_{1,8}\\ a_{2,1}& a_{2,2}& ...& a_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ a_{F*H,1}& a_{F*H,2}& ...& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& ...& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& ...& V_{a2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ V_{aF*H,1}& V_{aF*H,2}& ...& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as:

$\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& ...& h_{1,8}\\ h_{2,1}& h_{2,2}& ...& h_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ h_{F*H,1}& h_{F*H,2}& ...& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[k_i-A_z, k_i+B_zIn], at least part of data meet described predetermined condition C_z。

17. 1 kinds for searching the server of data flow point cutpoint, it is characterised in that described clothes Business device includes CPU and main storage, described CPU and described primary storage Device communicates, and is preset with rule on described server, and described rule is: for potential cut-point k Determine M some p_x, some p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x, p_x+B_x] corresponding predetermined condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_x And B_xFor integer；

Described main storage is used for storing executable instruction, and described CPU performs described Executable instruction, to perform following steps:

A) it is current potential cut-point k according to described rule_iDetermine a p_izAnd described some p_izCorresponding Window W_iz[p_iz-A_z,p_iz+B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[p_iz-A_z,p_iz+B_zIn], whether at least part of data meet Predetermined condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined Condition C_z, from described some p_izAlong the described data flow point cutpoint search direction N number of data flow point of jump Cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i- p_ix) ‖), it is thus achieved that new potential cut-point, perform step a)；

C) as described current potential cut-point k_iM window in each window W_ix[p_ix- A_x,p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential segmentation Point k_iFor data flow point cutpoint.

18. according to server described in claim 17, it is characterised in that described rule also includes: At least two point p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f。

19. according to server described in claim 18, it is characterised in that described rule also includes: Described at least two point p_eAnd p_f, relative to described potential cut-point k, at described data flow point Cutpoint searches in the reverse direction.

20. according to the server described in claim 18 or 19, it is characterised in that described rule Also include: described at least two point p_eAnd p_fBetween distance be 1 U.

21. according to the arbitrary described server of claim 17 to 19, it is characterised in that described CPU judges described window W specifically for using random function_iz[p_iz-A_z,p_iz+ B_zIn], whether at least part of data meet described predetermined condition C_z。

22. servers according to claim 21, it is characterised in that described central authorities process Unit judges described window W specifically for using hash function_iz[p_iz-A_z,p_iz+B_zIn] at least Whether part data meet described predetermined condition C_z。

23. according to the arbitrary described server of claim 17 to 19, it is characterised in that work as institute State window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined condition C_z, from Described some p_izMinimum along the described data flow point cutpoint search direction N number of data flow point cutpoint of jump Search unit U, it is thus achieved that described new potential cut-point, according to described rule, for described new The point p that potential cut-point determines_icCorresponding window W_ic[p_ic-A_c,p_ic+B_c] left margin and institute State window W_iz[p_iz-A_z,p_iz+B_z] right margin overlap or be that described potential cut-point newly is true Described fixed p_icCorresponding described window W_ic[p_ic-A_c,p_ic+B_c] left margin be positioned at described Window W_iz[p_iz-A_z,p_iz+B_zWithin the scope of]；Wherein, determine for described new potential cut-point Described some p_icIt is according to described rule, presses for M determined for described new potential cut-point According to the point of sequence first in the sequence that data stream search direction obtains.

24. servers according to claim 21, it is characterised in that described central authorities process Unit uses random function to judge described window W_iz[p_iz-A_z,p_iz+B_zAt least part of data in] Whether meet described predetermined condition C_z, specifically include:

At described window W_iz[p_iz-A_z,p_iz+B_zF byte is selected, by described F byte in] Recycling H time, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& ...& a_{1,8}\\ a_{2,1}& a_{2,2}& ...& a_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ a_{F*H,1}& a_{F*H,2}& ...& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& ...& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& ...& V_{a2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ V_{aF*H,1}& V_{aF*H,2}& ...& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& ...& h_{1,8}\\ h_{2,1}& h_{2,2}& ...& h_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ h_{F*H,1}& h_{F*H,2}& ...& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[p_iz-A_z, p_iz+B_zIn], at least part of data meet described predetermined condition C_z。

25. 1 kinds for searching the server of data flow point cutpoint, it is characterised in that described clothes Business device includes CPU and main storage, described CPU and described primary storage Device communicates, and is preset with rule on described server, and described rule is: for potential cut-point k Determine M window W_x[k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding making a reservation for Condition C_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；

Described main storage is used for storing executable instruction, and described CPU performs described Executable instruction, to perform following steps:

A) it is current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz[k_i-A_z,

k_i+B_z], i and z is integer, and 1≤z≤M；

B) described window W is judged_iz[k_i-A_z,k_i+B_zIn], whether at least part of data meet pre- Fixed condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described current potential cut-point k_iAlong described data flow point cutpoint search direction jump N Individual data flow point cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖), Obtain new potential cut-point, perform step a)；

C) as described current potential cut-point k_iM window in each window W_ix[k_i-A_x, k_i+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

26. servers according to claim 25, it is characterised in that described rule is also wrapped Include: at least two window W_ie[k_i-A_e,k_i+B_e] and W_if[k_i-A_f,k_i+B_f], meet condition: |A_e+B_e|=| A_f+B_f|, C_e=C_f。

27. servers according to claim 26, it is characterised in that for described server Preset rules, described rule also includes: A_eAnd A_fFor positive integer.

28. according to the server described in claim 26 or 27, it is characterised in that described rule Also include: A_e-1=A_f, B_e+ 1=B_f。

29. according to the arbitrary described server of claim 25 to 27, it is characterised in that described CPU judges described window W specifically for using random function_iz[k_i-A_z,k_i+B_z] In at least partly data whether meet described predetermined condition C_z。

30. servers according to claim 29, it is characterised in that described central authorities process Unit judges described window W specifically for using hash function_iz[k_i-A_z,k_i+B_zIn] at least Whether part data meet described predetermined condition C_z。

31. according to the arbitrary described server of claim 25 to 27, it is characterised in that work as institute State window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined condition C_z, from Described current potential cut-point k_iAlong the described data flow point cutpoint search direction N number of data stream of jump Cut-point minimum searches unit U, it is thus achieved that described new potential cut-point, according to described rule, The window W determined for described new potential cut-point_ic[k_i-A_c,k_i+B_c] left margin with described Window W_iz[k_i-A_z,k_i+B_z] right margin overlap or be that described potential cut-point newly is true Fixed described window W_ic[k_i-A_c,k_i+B_c] left margin be positioned at described window W_iz[k_i-A_z,k_i+ B_zWithin the scope of]；Wherein, the described window W determined for described new potential cut-point_ic[k_i- A_c,k_i+B_c] be according to described rule, M the window determined for described new potential cut-point by According to the window of sequence first in the sequence that data stream search direction obtains.

32. servers according to claim 29, it is characterised in that described central authorities process Unit uses random function to judge described window W_iz[k_i-A_z,k_i+B_zAt least part of data in] Whether meet described predetermined condition C_z, specifically include:

At described window W_iz[k_i-A_z,k_i+B_zF byte is selected, by anti-for described F byte in] Utilizing H time again, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& ...& a_{1,8}\\ a_{2,1}& a_{2,2}& ...& a_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ a_{F*H,1}& a_{F*H,2}& ...& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& ...& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& ...& V_{a2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ V_{aF*H,1}& V_{aF*H,2}& ...& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& ...& h_{1,8}\\ h_{2,1}& h_{2,2}& ...& h_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ h_{F*H,1}& h_{F*H,2}& ...& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[k_i-A_z, k_i+B_zIn], at least part of data meet described predetermined condition C_z。

33. 1 kinds for searching the server of data flow point cutpoint, it is characterised in that described Being preset with rule on server, described rule is: determine M some p for potential cut-point k_x、 Point p_xCorresponding window W_x[p_x-A_x,p_x+B_x] and window W_x[p_x-A_x,p_x+B_x] right Predetermined condition C answered_x, wherein, x is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor Integer；

Described server comprises determining that unit, is used for performing step a): a) according to described rule For current potential cut-point k_iDetermine a p_izAnd described some p_izCorresponding window W_iz[p_iz-A_z, p_iz+B_z], i and z is integer, and 1≤z≤M；

Judge processing unit, be used for judging described window W_iz[p_iz-A_z,p_iz+B_zAt least portion in] Whether divided data meets predetermined condition C_z；

As described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are unsatisfactory for described predetermined Condition C_z, from described some p_izAlong the described data flow point cutpoint search direction N number of data flow point of jump Cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖+‖(k_i- p_ix) ‖), it is thus achieved that new potential cut-point, the most described determine that unit is described new potential point Cutpoint performs step a)；

As described current potential cut-point k_iM window in each window W_ix[p_ix-A_x, p_ix+B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_i For data flow point cutpoint.

34. servers according to claim 33, it is characterised in that described rule is also wrapped Include: at least two point p_eAnd p_f, meet condition A_e=A_f, B_e=B_f, C_e=C_f。

35. servers according to claim 34, it is characterised in that described rule is also wrapped Include: described at least two point p_eAnd p_f, relative to described potential cut-point k, in described data Flow point cutpoint searches in the reverse direction.

36. according to the server described in claim 34 or 35, it is characterised in that described rule Also include: described at least two point p_eAnd p_fBetween distance be 1 U.

37. according to the arbitrary described server of claim 33 to 35, it is characterised in that described Judge that processing unit is specifically for using random function to judge described window W_iz[p_iz-A_z,p_iz+ B_zIn], whether at least part of data meet described predetermined condition C_z。

38. according to the server described in claim 37, it is characterised in that described judgement processes Unit judges described window W specifically for using hash function_iz[p_iz-A_z,p_iz+B_zIn] at least Whether part data meet described predetermined condition C_z。

39. according to the arbitrary described server of claim 33 to 35, it is characterised in that described Judge that processing unit is for as described window W_iz[p_iz-A_z,p_iz+B_zIn], at least part of data are discontented with Described predetermined condition C of foot_z, from described some p_izAlong described data flow point cutpoint search direction jump N Individual data flow point cutpoint minimum searches unit U, it is thus achieved that described new potential cut-point, described really Cell is that described new potential cut-point performs step a), according to described rule, for described The point p that new potential cut-point determines_icCorresponding window W_ic[p_ic-A_c,p_ic+B_c] left margin With described window W_iz[p_iz-A_z,p_iz+B_z] right margin overlap or be described potential segmentation newly The described window W that point determines_ic[p_ic-A_c,p_ic+B_c] left margin be positioned at described window W_iz[p_iz- A_z,p_iz+B_zWithin the scope of]；Wherein, the described window determined for described new potential cut-point W_ic[p_ic-A_c,p_ic+B_c] it is according to described rule, the M determined for described new potential cut-point The sequence that individual point obtains according to data stream search direction sorts first point.

40. according to the server described in claim 37, it is characterised in that described judgement processes Unit judges described window W specifically for using random function_iz[p_iz-A_z,p_iz+B_zIn] at least Whether part data meet described predetermined condition C_z, specifically include:

At described window W_iz[p_iz-A_z,p_iz+B_zF byte is selected, by described F byte in] Recycling H time, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& ...& a_{1,8}\\ a_{2,1}& a_{2,2}& ...& a_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ a_{F*H,1}& a_{F*H,2}& ...& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& ...& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& ...& V_{a2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ V_{aF*H,1}& V_{aF*H,2}& ...& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& ...& h_{1,8}\\ h_{2,1}& h_{2,2}& ...& h_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ h_{F*H,1}& h_{F*H,2}& ...& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[p_iz-A_z, p_iz+B_zIn], at least part of data meet described predetermined condition C_z。

41. 1 kinds for searching the server of data flow point cutpoint, it is characterised in that described Being preset with rule on server, described rule is: determine M window W for potential cut-point k_x [k-A_x,k+B_x] and window W_x[k-A_x,k+B_x] corresponding predetermined condition C_x, wherein, x It is 1 to M continuous print natural number, M >=2, A_xAnd B_xFor integer；

Described server comprises determining that unit, is used for performing step a):

A) it is current potential cut-point k according to described rule_iDetermine the window W of correspondence_iz[k_i-A_z,

k_i+B_z], i and z is integer, and 1≤z≤M；

Judge processing unit, be used for judging described window W_iz[k_i-A_z,k_i+B_zIn] at least partly Whether data meet predetermined condition C_z；

As described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are unsatisfactory for described predetermined bar Part C_z, from described current potential cut-point k_iAlong described data flow point cutpoint search direction jump N Individual data flow point cutpoint minimum searches unit U, and N*U is not more than ‖ B_z‖+max_x(‖A_x‖), Obtain new potential cut-point, the most described determine that unit is that described new potential cut-point performs step A)；

As described current potential cut-point k_iM window in each window W_ix[k_i-A_x,k_i +B_xIn], at least part of data meet predetermined condition C_x, the most described current potential cut-point k_iFor Data flow point cutpoint.

42. servers according to claim 41, it is characterised in that described rule is also wrapped Include: at least two window W_ie[k_i-A_e,k_i+B_e] and W_if[k_i-A_f,k_i+B_f], meet condition: |A_e+B_e|=| A_f+B_f|, C_e=C_f。

43. servers according to claim 42, it is characterised in that described rule is also wrapped Include: A_eAnd A_fFor positive integer.

44. according to the server described in claim 42 or 43, it is characterised in that described rule Also include: A_e-1=A_f, B_e+ 1=B_f。

45. according to the arbitrary described server of claim 41 to 43, it is characterised in that described Judge processing unit specifically for

Random function is used to judge described window W_iz[k_i-A_z,k_i+B_zAt least part of data in] Whether meet described predetermined condition C_z。

46. servers according to claim 45, it is characterised in that described judgement processes Unit specifically used hash function judges described window W_iz[k_i-A_z,k_i+B_zIn] at least partly Whether data meet described predetermined condition C_z。

47. according to the arbitrary described server of claim 41 to 43, it is characterised in that described Judge that processing unit is for as described window W_iz[k_i-A_z,k_i+B_zIn], at least part of data are not Meet described predetermined condition C_z, from described current potential cut-point k_iAlong described data flow point cutpoint Search direction N number of data flow point cutpoint minimum of jumping searches unit U, it is thus achieved that described new potential point Cutpoint, described determines that unit is that described new potential cut-point performs step a), according to described Rule, the window W determined for described new potential cut-point_ic[k_i-A_c,k_i+B_c] left margin With described window W_iz[k_i-A_z,k_i+B_z] right margin overlap or be described newly potential point The described window W that cutpoint determines_ic[k_i-A_c,k_i+B_c] left margin be positioned at described window W_iz[k_i- A_z,k_i+B_zWithin the scope of]；Wherein, the described window determined for described new potential cut-point W_ic[k_i-A_c,k_i+B_c] it is according to described rule, the M determined for described new potential cut-point The sequence that individual window obtains according to data stream search direction sorts first window.

48. servers according to claim 46, it is characterised in that described judgement processes Unit uses random function to judge described window W_iz[k_i-A_z,k_i+B_zAt least part of data in] Whether meet described predetermined condition C_z, specifically include:

At described window W_iz[k_i-A_z,k_i+B_zF byte is selected, by anti-for described F byte in] Utilizing H time again, obtain F*H byte altogether, the most each byte is formed by 8, is designated as a_m,1… a_m,8, represent the 1st to the 8th of m-th byte in described F*H byte, described F*H word The position that joint is corresponding can be expressed as: $\left(\begin{array}{cccc}{a}_{1,1}& a_{1,2}& ...& a_{1,8}\\ a_{2,1}& a_{2,2}& ...& a_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ a_{F*H,1}& a_{F*H,2}& ...& a_{F*H,8}\end{array}\right),$ Work as a_m,nWhen=1, V_am,n =1, work as a_m,nWhen=0, V_am,n=-1, wherein a_m,nRepresent a_m,1…a_m,8In any one, described Position corresponding to F*H byte is according to a_m,nWith V_am,nTransformational relation obtain matrix V_a, described matrix V_aIt is expressed as: $\left(\begin{array}{cccc}{V}_{a1,1}& V_{a1,2}& ...& V_{a1,8}\\ V_{a2,1}& V_{a2,2}& ...& V_{a2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ V_{aF*H,1}& V_{aF*H,2}& ...& V_{aF*H,8}\end{array}\right),$ Select from the random number of service normal distribution Select F*H*8 random number composition matrix R, described matrix R to be expressed as: $\left(\begin{array}{cccc}{h}_{1,1}& h_{1,2}& ...& h_{1,8}\\ h_{2,1}& h_{2,2}& ...& h_{2,8}\\ \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}& \begin{array}{c}.\\ .\\ .\end{array}\\ h_{F*H,1}& h_{F*H,2}& ...& h_{F*H,8}\end{array}\right),$ By described matrix V_aThe m row of m row and described matrix R Random number be multiplied, then summation obtain a value, be embodied as S_am=V_am,1*h_m,1+V_am,2 *h_m,2+…+V_am,8*h_m,8, in like manner, it is thus achieved that S_a1、S_a2... to S_aF*H, add up S_a1、S_a2… To S_aF*HIn meet number K of value more than 0, when K is even number, the most described window W_iz[k_i-A_z, k_i+B_zIn], at least part of data meet described predetermined condition C_z。