CN102508879A - Wavelet transform-based method for detecting copying of semi-structured text structure - Google Patents

Abstract

The invention provides a method for detecting duplication of semi-structured text structure based on wavelet transform. Its purpose is to detect text structure duplication, plagiarism, etc., to help inspectors quickly and correctly complete the entire inspection process, reduce the risk of false detection and missed detection, and shorten the detection time. The method of the invention at least comprises the steps of cleaning the semi-structured text, encoding the semi-structured text, obtaining the structural features of the semi-structured text through wavelet transform, calculating the similarity of the structure, judging whether the structure is identical, and the like. This method also converts semi-structured text into time series, but uses wavelet transform method to obtain structural features. In this way, better local structural features and details can be obtained, thereby improving the detection accuracy of local similar structures.

Description

A method of detecting duplication of semi-structured text structure based on wavelet transform

Technical field:

The invention belongs to the field of intelligent information processing and computer technology, and specifically relates to an accurate and effective detection method for duplication and plagiarism of semi-structured text structures.

Background technique:

The current era of rapid development of the Internet, a large number of web pages will emerge every day. Most of these web pages and the information hidden behind them are stored in the format of HyperText Markup Language (HyperText Markup Language, HTML) text or Extensible Markup Language (eXtensible Markup Language, XML) text. Both HTML text and XML text are semi-structured text. Protecting the intellectual property rights of semi-structured electronic texts in the network environment, especially combating illegal copying, plagiarism and other misconduct, has become the consensus of all circles at home and abroad, and it is also a hot issue that needs to be resolved urgently. In the face of complex and diverse copy plagiarism, only focusing on its global structural similarity often reduces the accuracy of copy detection. Because many copied texts are not all copied from the original text, but partial copy and splicing and so on. This requires us to compare local information, reduce the missed detection rate of local similar structures, and improve the accuracy of similar detection.

At present, there are three main categories of detection methods for semi-structured text duplication: methods based on kernel matrix, methods based on tree edit distance, and time series methods based on Fourier transform. The method based on the kernel matrix uses the matrix M describing the correlation between text structural units and their contribution to the text similarity measurement, and the matrix of the text in the SLVM-based model space to perform similarity calculations and measure text similarity. The tree edit distance method refers to transforming a text into another text, and measures its similarity with the minimum cost spent in the conversion process. The obvious defect of this method is that the calculation is expensive, and its time complexity is O(N ² ), where N is the number of elements in the text, that is, the number of tags. So the tree edit distance method is not suitable for large texts. The time series method based on Fourier transform converts semi-structured text into time series, then obtains time series features through Fourier transform, and finally performs copy detection according to the similarity of feature vectors. But the Fourier transform reflects the overall frequency characteristics of the signal at all times, and cannot provide the frequency characteristics at local time. Therefore, the local features and details of the semi-structured text cannot be observed by the method of Fourier transform, and the investigation of the similarity is not detailed enough.

Invention content:

In view of the above problems, the present invention provides a method for detecting duplication of semi-structured text structures based on wavelet transform. This method also converts semi-structured text into time series, but uses wavelet transform method to obtain structural features. In this way, better local structural features and details can be obtained, thereby improving the detection accuracy of local similar structures.

The invention provides a method for detecting duplication of semi-structured text structure based on wavelet transform. Its purpose is to detect text structure duplication, plagiarism, etc., to help inspectors quickly and correctly complete the entire inspection process, reduce the risk of false detection and missed detection, and shorten the entire detection time.

To achieve the above purpose, the method of the present invention at least includes the steps of cleaning the semi-structured text, encoding the semi-structured text, obtaining structural features of the semi-structured text through wavelet transform, calculating structural similarity, and judging whether the structure is similar. Clean semi-structured text to remove non-standard characters in semi-structured text, modify unmatched tags and invalid string formats, and convert original semi-structured text into semi-structured text that conforms to specifications; semi-structured text encoding Serialize the structure of the semi-structured text to obtain a structural code sequence; obtain the structural features of the semi-structured text through wavelet transform means to perform wavelet transform on the structural code sequence to obtain the structural feature vector of the semi-structured text; calculate the structural similarity Calculate the structural distance between the semi-structured text feature vectors, so as to obtain the similarity between the semi-structured text structures; to determine whether the structures are similar, judge two semi-structured texts according to the structural distance between the semi-structured text structures Whether the structure is the same, if the structure distance is less than the given threshold, it is the same, otherwise it is not the same.

The described cleaning of semi-structured text is to remove the non-standard characters (such as &, <, >) in the original semi-structured text, and modify the unmatched tags (such as the tag <img>, which has no termination tag, and the tag should be changed to is <img/>) and an invalid string format (such as: a=0, 0 is an invalid string, it should be changed to a="0"), and finally the original semi-structured text becomes a semi-structured one that conforms to the specification text.

In the semi-structured text encoding, each tag in the semi-structured text is a pair, consisting of a start tag and an end tag; all start tags are encoded as 1, and the end tag is encoded as -1 , and finally get a text structure encoding sequence according to the order of appearance of the tags in the text, and this sequence represents the structural features of the text. During the encoding process, the content of the semi-structured text is removed, and only the corresponding tags are kept.

The described semi-structured text structural features obtained by wavelet transform are obtained according to the following process:

(1) represent time coordinates with label occurrence position, then get the square wave signal representation of text structure coding sequence;

(2) Use Haar (Haar) wavelet to carry out wavelet transformation on the square wave signal of the text structure coding sequence, and obtain the corresponding wavelet coefficient vector;

(3) Compress the wavelet coefficient vector, that is, compare the absolute value of the wavelet coefficient sequence with the specified threshold value, and the point less than or equal to the threshold value becomes 0; the point greater than the threshold value becomes the difference between the point value and the threshold value. Sparse coefficient sequence filled with values;

(4) Square the non-zero coefficients and sort them from large to small, take the first m maximum values, and record the position information corresponding to the coefficients to obtain a two-dimensional structural feature vector, that is, semi-structured text structural features. As follows:

表示半结构化文本(a)经过小波变换之后第m个系数的平方，

表示该系数对应的位置。where F ^a represents the structural feature vector of the semi-structured text (a),

Represents the square of the mth coefficient of semi-structured text (a) after wavelet transformation,

Indicates the position corresponding to the coefficient.

The method for calculating structural similarity is to first normalize the structural feature vector, and then calculate the structural distance of two semi-structured texts according to the following formula:

$\mathrm{<span\; class="notranslate">\; Dist</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\sqrt{{<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; e</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; e</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\sqrt{{<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; no</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; no</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; ;</span>$

表示半结构化文本(a)经过归一化处理之后的第i个小波系数平方值，

表示该系数对应位置经过归一化处理之后的值，

表示半结构化文本(b)经过归一化处理之后的第i个小波系数平方值，

表示该系数对应位置经过归一化处理之后的值，m表示小波系数的个数。where Dist(F ^a , F ^b ) represents the structural distance between semi-structured text (a) and semi-structured text (b),

Indicates the square value of the ith wavelet coefficient of the semi-structured text (a) after normalization,

Indicates the value of the corresponding position of the coefficient after normalization processing,

Indicates the square value of the ith wavelet coefficient of the semi-structured text (b) after normalization processing,

Indicates the value of the corresponding position of the coefficient after normalization processing, and m indicates the number of wavelet coefficients.

The method for judging whether the structures are similar is to judge whether the two semi-structured texts are similar in structure according to the structural distance between the semi-structured text structures. If the structural distance is less than a given threshold, they are similar, otherwise they are not.

The distance value describes the similarity between text structures. The larger the distance value is, the less similar the text structure is, and the smaller the distance value is, the more similar the text structure is. If the distance value is less than a given threshold, the structural similarity of the two semi-structured texts exceeds the standard, which can be determined as structural similarity. Thereby, it helps the inspectors to realize the detection of similar text structures.

The semi-structured text described in the method of the present invention includes Extensible Markup Language (eXtensibleMarkup Language, XML) text and HyperText Markup Language (HyperText Markup Language, HTML) text.

Description of drawings:

FIG. 1 is a flow chart of a method for detecting duplication of an XML text structure.

Figure 2 is a schematic diagram of the XML structure (a).

Fig. 3 is a schematic diagram of XML structure (b).

Detailed ways:

The present invention will be further described below in conjunction with accompanying drawing.

The invention provides a method for detecting duplication of a semi-structured text structure based on wavelet transform, which can help inspectors quickly and accurately detect semi-structured texts with the same structure, reduce missed detection and false detection, and shorten the detection time. To achieve the purpose of combating illegal copying, plagiarism and other acts. The basic idea of the present invention is: firstly, we remove all the content in the semi-structured text conforming to the specification, and only keep the structural framework composed of tags. The tags in the semi-structured text are encoded by the tag encoding method, and an ordered encoding sequence is obtained according to the nesting and sequence of the tags. Think of this sequence as a time series composed of equally spaced points on the time axis, so that the wavelet transform method can be used to extract the characteristics of this time series signal, then the characteristics of the time series signal are the semi-structured text structure features. Finally, the similarity between semi-structured text structures is measured by calculating the similarity between feature vectors, and then whether the text structure is similar is detected.

According to the idea of the method of the present invention, as referring to the detection process shown in Figure 1, the method at least includes cleaning the semi-structured text (01), encoding the semi-structured text (02), obtaining the structural features of the semi-structured text by wavelet transform (03), calculating structural similarity (04), judging whether the structure is identical (05) and other steps.

After the original semi-structured text is cleaned in step 01, the standardized semi-structured text is obtained, and then the text structure is encoded in step 02, and a coding sequence that fully represents the structure of the semi-structured text is obtained, and the structure is extracted by wavelet transform in step 03 The features are compressed to obtain the structural feature vector, and the structural distance between the structural feature vectors is calculated after normalizing the structural feature vectors in step 04 to describe the structural similarity. Finally, in step 05, compare the result obtained in step 04 with a given threshold to determine whether the structures of the two semi-structured texts are similar (that is, whether they are identical). If the structure is similar, it is judged as identical (step 07), otherwise it is judged as not identical (step 06).

The following are preferred embodiments given by the inventor.

Referring to FIG. 2, it is a pure structure diagram of an original XML text (a) after removing text content and ignoring tag value information. The original tag sequence for this text is:

<xml>, <book>, <title>, </title>, <author>, </author>, </book>, <book>, <title>, </title>, <author>, </ author>, </book>, </xml>

Referring to FIG. 3 is another original XML text (b) a pure structure diagram of the XML text retained after removing the text content and ignoring the tag value information. The original tag sequence for this text is:

<xml>, <book>, <title>, </title>, <author>, </author>, </book>, <book>, <title>, </title>, <author>, </ author>, </book>, <book>, <title>, </title>, <author>, </author>, </book>, </xml>

According to step 02 in Figure 1, encode the XML text structure, encode all start tags (like "<>") to 1, and all end tags (like "</>") to -1. Then the corresponding encoding sequence of the XML text structure shown in Figure 2 is:

Enc(a)={1, 1, 1, -1, 1, -1, -1, 1, 1, -1, 1, -1, -1, -1}

The corresponding encoding sequence of the XML text structure shown in Figure 3 is:

Enc(b)={1, 1, 1, -1, 1, -1, -1, 1, 1, -1, 1, -1, -1, 1, 1, -1, 1, -1, -1,-1}

According to step 03 in Figure 1, when performing wavelet transform on XML text to extract structural features, the lengths of the two sequences to be compared are first changed to be the same. Complement the length of sequence Enc(a) and sequence Enc(b), and add 0 to the end of the shorter sequence to get:

Enc(a)={1, 1, 1, -1, 1, -1, -1, 1, 1, -1, 1, -1, -1, -1, 0, 0, 0, 0, 0 ,0}

Enc(b)={1, 1, 1, -1, 1, -1, -1, 1, 1, -1, 1, -1, -1, 1, 1, -1, 1, -1, -1,-1}

Then, Haar (Haar) wavelet is used to carry out the maximum scale wavelet transform on the above signal to obtain its coefficient sequence.

Coef(a)={0, 0, 1, 0, 0.7071, 0.7071, 0, 1, 0, 0, -1, 0, 0, 1.4142, 1.4142, -1.4142, 1.4142, 1.4142, 0, 0, 0, 0}

Coef(b)={0.5, -2, 0.5, 0, 0.7071, 0, 0, 1, 0, 0, 0, 1, 0, 1.4142, 1.4142, -1.4142, 1.4142, 1.4142, -1.4142, 1.4142, 1.4142 ,0}

Next, after compressing the wavelet coefficients, the structural feature vectors of the two XML texts are obtained as follows:

F ^a = [(1.7272, 14), (1.7272, 15), (1.7272, 16), (1.7272, 17), (1.7272, 18), (0.81, 3) (0.81, 8), (0.81, 11) , (0.3686, 5), (0.3686, 6)]

F ^b = [(1.6716, 2), (0.5, 14), (0.5, 15), (0.5, 16), (0.5, 17), (0.5, 18), (0.5, 19), (0.5, 20 ), (0.5, 21), (0.0858, 8)]

Finally, the structural distance is calculated as:

Dist(F ^a , F ^b )=3.9103

Assume the given threshold is: 4. Then, since Dist(F ^a , F ^b )=3.9103<4, it is determined that text (a) and text (b) have the same structure.

Claims (5)

1. A semi-structured text structure duplication detection method based on wavelet transform, it is characterized in that, comprises the following steps: cleaning semi-structured text, coding semi-structured text, obtaining semi-structured text structure feature by wavelet transform, calculating Structural similarity, judging whether the structure is similar; Clean semi-structured text to remove non-standard characters in semi-structured text, modify unmatched tags and invalid string formats, and convert original semi-structured text into semi-structured text that conforms to specifications; Encoding the semi-structured text Serialize the structure of the semi-structured text to obtain a structure encoding sequence; Obtaining the structural features of semi-structured text by wavelet transform refers to performing wavelet transform on the structural coding sequence to obtain the structural feature vector of semi-structured text; calculating the structural similarity is to calculate the structural distance between the semi-structured text feature vectors, so as to obtain the semi-structured text similarity between structured text structures; To determine whether the structure is similar, it is determined whether the structure of the two semi-structured texts is similar according to the structural distance between the semi-structured text structures. If the structural distance is less than a given threshold, they are similar, otherwise they are not. 2. duplication detection method according to claim 1 is characterized in that: when semi-structured text is encoded, each label in semi-structured text is all paired, is made of start label and end label; The start tag is encoded as 1, the end tag is encoded as -1, and finally a text structure encoding sequence is obtained according to the order of appearance of the tags in the text, and this sequence represents the structural characteristics of the text; semi-structured text is eliminated during the encoding process , keeping only the corresponding tags. 3. copy detection method according to claim 1, is characterized in that: when obtaining semi-structured text structure feature by wavelet transform, (1) represent time coordinates with label occurrence position, then get the square wave signal representation of text structure coding sequence; (2) Carry out wavelet transform to the square wave signal of the text structure coding sequence with Haar wavelet, and obtain the corresponding wavelet coefficient vector; (3) Compress the wavelet coefficient vector, that is, compare the absolute value of the wavelet coefficient sequence with the specified threshold value, and the point less than or equal to the threshold value becomes 0; the point greater than the threshold value becomes the difference between the point value and the threshold value. Sparse coefficient sequence filled with values; (4) Square the non-zero coefficients and sort them from large to small, take the first m maximum values, and record the position information corresponding to the coefficients to obtain a two-dimensional structural feature vector, that is, semi-structured text structural features. 4. duplication detection method according to claim 1, is characterized in that: when calculating structural similarity, at first structural feature vector is carried out normalization process, then calculates the structural distance of two semi-structured texts according to following formula: $\mathrm{<span\; class="notranslate">\; Dist</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; f</span>}}^{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\sqrt{{<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; e</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; e</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; +</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}}\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; m</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\sqrt{{<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; no</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; a</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; ~</span>}{\mathrm{<span\; class="notranslate">\; no</span>}}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; b</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$ where Dist(F ^a , F ^b ) represents the structural distance between semi-structured text a and semi-structured text b, Indicates the square value of the ith wavelet coefficient of semi-structured text a after normalization processing,

Indicates the value of the corresponding position of the coefficient after normalization processing, Indicates the square value of the ith wavelet coefficient of the semi-structured text b after normalization,

Indicates the value of the corresponding position of the coefficient after normalization processing, and m indicates the number of wavelet coefficients. 5. The copy detection method according to claim 1, characterized in that: the semi-structured text includes Extensible Markup Language XML text and Hypertext Markup Language HTML text.

Images