CN103793491B - Chinese news story segmentation method based on flexible semantic similarity measurement - Google Patents

Abstract

The invention discloses a method for segmenting Chinese news stories based on flexible semantic similarity measurement. The method comprises the following steps: inputting a target corpus, performing word segmentation on each news story script T _i in the corpus; establishing a context graph; The context graph and the quick sorting algorithm iteratively propagate the context correlation between words to obtain a flexible semantic correlation matrix; the flexible semantic similarity between sentences is defined by the flexible semantic correlation matrix; using the flexible Semantic Similarity Segmentation of Chinese News Stories. The flexibility measurement method proposed by the present invention can more reasonably represent the semantic similarity between words and word sets. The experimental results show that in the Chinese news story segmentation technology, based on the same segmentation criteria, compared with the traditional similarity measurement method, using this flexible semantic similarity measurement method can improve the segmentation accuracy to 3%‑10%.

Description

A Chinese News Story Segmentation Method Based on Flexible Semantic Similarity Measure

technical field

The invention relates to the field of Chinese news story segmentation, in particular to a Chinese news story segmentation method based on flexible semantic similarity measurement.

Background technique

With the popularization and development of the network, for example: multimedia content such as broadcast news, meeting minutes, and online open courses is increasing rapidly, and now there is an urgent need for an effective method to automatically organize such multimedia data for topic-based text retrieval and analysis. A multimedia document, such as a one-hour broadcast news program, usually consists of multiple stories. In order to perform efficient semantic retrieval, it is very important to guide users to find the beginning and end of the topics they are interested in. At the same time, , a well-segmented multimedia document is an important prerequisite for high-level semantic browsing such as topic tracking ^[1] , classification and summarization ^[2] . The purpose of news story segmentation technology is to segment news story scripts into stories with consistent themes. Technically speaking, the efficiency of news story segmentation technology is related to two factors: one is the similarity between words and the measurement method of the similarity between this sentence set; the other is the criteria for segmenting news story scripts.

Many previous works have focused on designing reasonable segmentation criteria, such as: TextTiling ^{[3] [4]} Minimum Normalized Segmentation Criterion (Minimum NCuts) ^{[5] [6]} , Maximum Lexical Connection Criterion ^[7] , etc. Compared with widely studied segmentation criteria, most works at this stage use a simple repetition-based hard similarity measure, that is, the similarity between the same words is 1, and the similarity between different words is 0. Obviously, this repetition-based rigid similarity measurement method ignores the potential semantic correlation between different words, which makes the semantic relationship measurement inaccurate, and the obtained Chinese news story segmentation results are inaccurate. Therefore, it is necessary to propose a more reasonable semantic similarity measurement method to help improve the efficiency and accuracy of segmentation.

Contents of the invention

The present invention provides a Chinese news story segmentation method based on flexible semantic similarity measurement. The present invention can reasonably represent the semantic similarity between words, and can significantly improve the accuracy of Chinese news story segmentation technology. See the following description for details:

A method for segmenting Chinese news stories based on flexible semantic similarity measurement, said method comprising the following steps:

(1) Input the target corpus Carry out word segmentation for each news story script T _i in the corpus;

(2) Establish a context diagram;

(3) Iteratively propagating the context correlation between words through the context graph and the quick sorting algorithm to obtain a flexible semantic correlation matrix;

(4) Define the flexible semantic similarity between sentences through the flexible semantic correlation matrix;

(5) Segment Chinese news stories using the flexible semantic similarity.

The steps of establishing the context graph are specifically:

1) Read in each news story script in turn, and perform word frequency statistics on the included words;

2) Delete high-frequency words and low-frequency words according to the defined word frequency threshold;

3) Take the reserved words as the nodes in the context graph, and its set is V;

4) Determine whether any two words in the set appear in a news story script at the same time, and the distance between the two words is less than or equal to the distance threshold, and if so, establish an edge between the two words, and the edge The set of is E; if no, re-judgment any other two words, until the words in the entire set are traversed;

5) The weight S _C of the edge is represented by the weight sim _C (a,b) between words and the weight sim _C (a,a) of the word itself;

6) The context graph is expressed as G=V,E,S _C .

The weight sim _C (a, b) between the words is specifically:

${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; freq</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}$

Among them, freq(a,b) indicates the number of times that word a and word b appear at the same time, freq _max =max( _i,j ){freq(i,j)} indicates the maximum frequency of word pair (i,j), ε is a constant to ensure that 0≤sim _C (a,b)≤1.

The weight sim _C (a, a)=1 of the word itself.

The step of obtaining the flexible semantic correlation matrix by iteratively propagating the context correlation between words through the context graph and the quick sorting algorithm is specifically:

1) Define the semantic similarity between words in the context graph as sim _S (a,b), which satisfies the following three criteria:

The similarity between a word and itself is 1, that is, sim _S (a, a)=1; sim _S (a, b) is positively correlated with sim _C (a, b); the relationship between sim _S (a, b) and their neighbors proportional to the similarity between

2) Define the iterative propagation process of semantic similarity:

${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$ ${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; c</span>}}{\mathrm{<span\; class="notranslate">\; Z</span>}}\underset{\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ~</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; ~</span>\mathrm{<span\; class="notranslate">\; b</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; the\; si</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$ ${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span><span\; class="notranslate">\; \&infin;</span>}{\mathrm{<span\; class="notranslate">\; lim</span>}}{\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

Among them, u～a, v～b indicate that u and v are the neighbor nodes of word a and word b respectively in the context diagram, Z is the normalization factor, c is the control factor, Indicates the semantic similarity between word a and word b at the tth iteration, Indicates the semantic similarity between word a and word b at the t-1th iteration, Indicates initialization;

3) Use the quick sorting algorithm to solve the relational expression defined in 2), obtain the semantic correlation, and obtain the semantic correlation for each two words, and several semantic correlations form a flexible semantic correlation matrix, which records for S _S .

The step of defining the flexible semantic similarity between sentences through the flexible semantic correlation matrix is specifically:

$\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}^{\mathrm{<span\; class="notranslate">\; T</span>}}\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}$

Among them, s _i and s _j represent sentences respectively, ||f _i || and ||f _j || represent the two norms of word frequency vectors of two sentences respectively, and T is the transposition.

The beneficial effect of the technical solution provided by the present invention is: the present invention proposes a non-supervised semantic similarity measurement method through the quick sorting algorithm, and improves the traditional cosine similarity so that it can be incorporated into the potential semantic relationship between words , and use this flexible semantic similarity to improve Chinese news story segmentation technology. The flexibility measurement method proposed by the present invention can more reasonably represent the semantic similarity between words and word sets. The experimental results show that in the segmentation technology of Chinese news stories, based on the same segmentation criterion, compared with the traditional similarity measurement method, using this flexible semantic similarity measurement method can improve the segmentation accuracy to 3%-10%.

Description of drawings

Figure 1 is a flowchart of Chinese news story segmentation technology based on flexible semantic similarity;

Fig. 2 is a schematic diagram of a context diagram;

Figure 3 is a comparison chart of the sentence similarity ratio between stories and within stories on the standard dataset CCTV and TDT2;

Figure 4 is a comparison of the results of the Chinese news story segmentation algorithm using three different similarity measures on 100 sets of random parameters on the standard data set CCTV-75-s.

detailed description

In order to make the object, technical solution and advantages of the present invention clearer, the implementation manner of the present invention will be further described in detail below in conjunction with the accompanying drawings.

The measurement of semantic similarity is a very challenging research topic in natural language processing. Existing methods mainly fall into two categories: supervised and unsupervised. Supervised methods mainly include WordNet ^[8][9] and DISCO. WordNet is used to measure the similarity between any two English words. WordNet relies on the well-marked corpus to divide the rankings, verbs, adjectives and adverbs into levels, and the division is based on the semantic definitions of these words by language experts. Due to its simplicity and effectiveness, WordNet has been widely used in natural language processing tasks. Similar to WordNet, DISCO is another commonly used supervised method for retrieving the similarity between any given two words. Compared with WordNet, DISCO supports more languages, such as: English, German, French, Spanish, etc. The supervised method can be directly used in the language space defined in advance without any additional calculations. At the same time, the supervised method also covers almost all common words. However, supervised methods rely on the knowledge of linguists, and the measure of similarity between words is usually defined by subjective consciousness. At the same time, supervised methods are not suitable for applications based on specific corpora. Unsupervised methods mainly include PMI, LSA and pLSA. PMI counts the number of times two words appear on the same webpage at the same time by querying the statistical data obtained by the website search engine. The more times, the higher the PMI score of these two words. LSA is also an unsupervised semantic similarity measurement method, which incorporates the mechanism of human learning knowledge to obtain the similarity between words or text paragraphs. The key step of LSA is dimensionality reduction through singular value decomposition. At the same time, LSA can also solve the problem of synonymity in natural language processing. pLSA is an improved algorithm to LSA. Different from the LSA algorithm derived from linear algebra, on the basis of inheriting the advantages of LSA, pLSA uses the method of probability theory to analyze the correlation between pairs of words, and can handle synonymy and polysemy very well The problem. Compared with LSA, pLSA is more common.

In recent years, the development of graph theory has attracted the attention of natural language processing scientists. Widdows et al. proposed an unsupervised method based on graph models to obtain semantic similarity. In this graph model, nodes represent words, and edges are the relationships between words. In addition, the graph model is based on a specific corpus, which can handle word ambiguity. Ambwani et al. proposed another graph model to measure the semantic similarity of words. Each word is represented as a series of nodes, each node corresponds to a sentence within the scope of influence of the word, and the weight on the edge represents correlation between words. The model incorporates the mutual influence between words, and determines the relevance of words according to the context. The unsupervised methods for computing semantic similarity discussed above are all based on specific corpora and are more suitable for specific applications than supervised methods. Among these unsupervised methods, due to the simplicity and efficiency of the graphical model, the semantic similarity calculation method based on the graphical model has attracted the attention of more and more natural language processing research experts.

Semantic similarity measurement between word collections (such as paragraphs, texts, etc.) is also an urgent problem to be solved. The traditional method of measuring semantic similarity of word sets is cosine similarity. Based on the bag-of-words assumption, each word set is represented as a word frequency vector, and the cosine similarity is used to measure the angle between the word frequency vectors. The larger the angle, the smaller the similarity; otherwise, the larger it is. Because the use of cosine similarity is very simple and effective, cosine similarity is widely used to measure the semantic similarity of word sets, but cosine similarity only considers the relationship between the same words, ignoring the relationship between words in the word set Correlations, which make the measure of word set similarity inaccurate. In order to make the measurement of the semantic similarity of word sets more accurate and meaningful, it is more meaningful to take the correlation between words into account when measuring the similarity between word sets. Therefore, it is urgent to propose a method that integrates the mutual relationship between words into the similarity measurement between word sets.

In order to reasonably represent the semantic similarity between words and significantly improve the accuracy of the Chinese news story segmentation technology, an embodiment of the present invention provides a Chinese news story segmentation method based on a flexible semantic similarity measure, see Figure 1, This method is aimed at a specific data set in the process of semantic similarity calculation and news story segmentation. At the same time, in order to reflect the rationality of the flexible semantic similarity measurement method, the verification criteria are redesigned to verify the rationality, see the following description for details:

101: Input target corpus Segment each news story script Ti in the corpus;

That is, through this step, each sentence in the news story script is divided into several words. This step is well known to those skilled in the art, and will not be described in detail in this embodiment of the present invention.

102: Establish a context diagram;

1) Read in the news story scripts one by one, and count the word frequency of the words contained in it;

2) According to the defined word frequency threshold, delete high-frequency words and low-frequency words.

3) Take the reserved words as the nodes in the context graph, and its set is V.

4) Determine whether any two words in the set appear in a news story script at the same time, and the distance between the two words is less than or equal to the distance threshold, and if so, establish an edge between the two words, and the edge The set of is E; if no, re-evaluate any other two words until the words in the entire set are traversed.

5) The weight S _C of the edge is represented by the weight sim _C (a,b) between words and the weight sim _C (a,a) of the word itself;

The weight sim _C (a,b) between words is defined by the following formula:

${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; freq</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; freq</span>}}_{\mathrm{<span\; class="notranslate">\; max</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}}$

Among them, freq(a,b) indicates the number of times that word a and word b appear at the same time, freq _max =max( _i,j ){freq(i,j)} indicates the maximum frequency of word pair (i,j), ε is A constant to ensure that 0≤sim _C (a,b)≤1 (a≠b).

At the same time, the weight sim _C (a,a)=1 of the word itself.

6) Therefore, the context graph can be expressed as G=V,E,S _C . Figure 2 is a schematic diagram of constructing a context graph of a document in a data set, where w _u represents the uth word in T _i , and the line represents relationship between words.

103: Obtain a flexible semantic correlation matrix by iteratively propagating the contextual correlation between words through the context graph and the quick sorting algorithm ^[10][11] ;

1) Define the semantic similarity between words in the context graph as sim _S (a,b), which satisfies the following three criteria:

The similarity between the word and itself is 1, that is, sim _S (a,a)=1;

sim _S (a,b) is positively correlated with sim _C (a,b);

sim _S (a,b) is proportional to the similarity between their neighbors.

2) The iterative propagation process that defines semantic similarity is defined by the following relation as follows:

${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 0</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; C</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; c</span>}}{\mathrm{<span\; class="notranslate">\; Z</span>}}\underset{\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ~</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; ~</span>\mathrm{<span\; class="notranslate">\; b</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; the\; si</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; ;</span>$

${\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; \&Right\; Arrow;</span><span\; class="notranslate">\; \&infin;</span>}{\mathrm{<span\; class="notranslate">\; lim</span>}}{\mathrm{<span\; class="notranslate">\; sim</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}^{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; a</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; b</span>}<span\; class="notranslate">\; )</span>$

Among them, u～a, v～b indicate that u and v are the neighbor nodes of word a and word b respectively in the context diagram, Z is the normalization factor, c is the control factor, Indicates the semantic similarity between word a and word b at the tth iteration, Indicates the semantic similarity between word a and word b at the t-1th iteration, Indicates initialization.

3) Use the quick sorting algorithm to solve the relational expression defined in 2), obtain the semantic correlation, and obtain the semantic correlation for each two words, and several semantic correlations form a flexible semantic correlation matrix, which records for S _S . Similarly, the traditional hard semantic similarity is defined as _SH = I, where I represents the identity matrix.

The assumption based on the quick sort algorithm: if the neighbor words of two words are similar (correlation greater than or equal to 0.5), then the two words are also similar;

The quick sort algorithm takes the context graph as input; the algorithm complexity is O(k|V| ² ), where k is the average degree in G, |V| represents the number of nodes in the context graph, and O is the algorithm complexity .

In this invention, a GPU-based full-point pair parallel algorithm of the quick sorting algorithm is implemented. Through experimental observation, it is found that with the same context graph as input, the speed of the quick sort algorithm based on GPU is about 1000 times faster than that of the traditional quick sort algorithm based on CPU.

The output of the quick sort algorithm is the flexible semantic correlation matrix S _S ={sim _S (a,b)} _a,b∈C after iterative propagation;

104: Define the flexible semantic similarity between sentences (that is, a collection of words that appear consecutively in a news story script) through a flexible semantic correlation matrix;

In the news story segmentation technology, in addition to the semantic similarity of words, it is also necessary to measure the similarity between word sets, that is, sentences. Each sentence in the story can be identified as a word frequency vector, which is used to record the number of times each word appears in the sentence. For a given flexible semantic relevance matrix, the flexible semantic similarity between sentences is defined as follows:

$\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{{{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}^{\mathrm{<span\; class="notranslate">\; T</span>}}{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}$

Among them, s _i and s _j represent sentences respectively, ||f _i || and ||f _j || represent the two norms of word frequency vectors of two sentences respectively, and T is the transposition. This definition is an improvement on the traditional cosine similarity, which takes into account the potential semantic correlation between different words, so it can more reasonably represent the semantic similarity between sentences.

105: Segmenting Chinese News Stories Using Flexible Semantic Similarity.

1) The criterion based on Chinese news story segmentation technology is the normalization criterion ^[5][6] ;

Among them, the normalization criterion is specifically: the criterion is based on a graph model; sentences are identified as nodes in the graph model, and the relationship between sentences is represented as an edge in the graph model; the similarity between sentences is represented as weights; the news story segmentation problem is transformed into a graphical model segmentation problem.

2) Segment Chinese news stories on news story scripts contained in the input dataset using flexible semantic similarity between sentences.

The feasibility of a kind of Chinese news story segmentation method based on flexible semantic similarity measure provided by the present invention is verified below with specific test:

Experiment on a standard dataset:

In order to verify the effectiveness of this method, this method is tested on two standard datasets CCTV and TDT2. The CCTV data set contains a total of 71 Chinese news story scripts. According to the length of news stories and the recognition error rate, the CCTV data set can be divided into 8 subsets, which are recorded as CCTV_59_f/s, CCTV_66_f/s, CCTV_75_f/s and CCTV_ref_f/s , where f represents the long story set, s represents the short story set, and ref represents the reference dataset. The TDT2 data set contains 177 scripts of Chinese news stories. According to the recognition error rate, these 177 scripts can be divided into two subsets, which are respectively recorded as TDT2_ref and TDT2_rcg. News stories in CCTV and TDT2 datasets are segmented using hard semantic similarity _SH , contextual semantic similarity SC and soft semantic similarity _{S S respectively} , and their segmentation accuracy is compared. The segmentation accuracy is represented by the F1 score. Table 1 lists the segmentation accuracy on CCTV and TDT2 datasets using different similarity measures.

Table 1

It can be observed from Table 1 that compared with the traditional hard semantic similarity, the use of soft semantic similarity can significantly improve the segmentation accuracy, and the improvement range is about 3% to 10%. At the same time, it can also be found that the contextual semantic similarity is better than the rigid semantic similarity, and the contextual semantic similarity can be improved through the quick sort algorithm. In order to show the robustness of this method, this method implements another more rigorous experiment on the CCTV_75_s dataset, by comparing the segmentation accuracy of different methods using 100 sets of random parameters. Figure 4 shows the results of this experiment. In the news story segmentation technology, the similarity between sentences can be measured by the ratio between the similarity between stories and the similarity inside the story. This ratio corresponds to the distinguishability of sentences. The smaller the ratio, the better the The better the similarity, the ratio is defined by the following formula:

$\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; mean</span>}}_{\mathrm{<span\; class="notranslate">\; lab</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&NotEqual;</span>\mathrm{<span\; class="notranslate">\; lab</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; exp</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; mean</span>}}_{\mathrm{<span\; class="notranslate">\; lab</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; lan</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}\mathrm{<span\; class="notranslate">\; Sim</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; S</span>}}_{\mathrm{<span\; class="notranslate">\; S</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}$

Among them, lab(s _i ) represents the label of the story to which sentence s _i belongs, lab(s _j ) represents the label of the story to which sentence s _j belongs, and mean represents the average value.

This method compares the hard semantic similarity _SH , the contextual semantic similarity S _C and the soft semantic similarity S _S on the standard data set, and the comparison results are shown in Figure 4. It is found through experiments that the R ratio obtained by using the soft semantic similarity S _S is lower than the other two similarities, and the context semantic similarity S _C is lower than the hard semantic similarity _SH . The experiment shows that the flexible semantic similarity S _S is more reasonable than the traditional hard semantic similarity _SH , and the flexible semantic similarity (that is, the semantic similarity after iterative propagation) solved by the quick sort algorithm is more reasonable. Applying this method to Chinese news story segmentation technology can significantly improve the segmentation accuracy.

references:

[1]. J. Allan, Ed., Topic Detection and Tracking: Event-based Information Organization, Kluwer Academic Publishers, 2002.

[2].L.-S.Lee and B.Chen, "Spoken document understanding and organization," vol.22, no.5, pp.42–60, 2005.

[3].S.Banerjee and I.A.Rudnicky, "A TextTiling based approach to topicboundary detection in meetings," in INTERSPEECH, 2006.

[4].L.Xie, J.Zeng, and W.Feng, “Multi-scale TextTiling for automatic story segmentation in Chinese broadcast news,” in AIRS, 2008.

[5].I.Malioutov and R.Barzilay,"Minimum cut model for spoken lecturesegmentation,"in ACL,2006.

[6]. J. Zhang, L. Xie, W. Feng, and Y. Zhang, “A subword normalized cut approach to automatic story segmentation of Chinese broadcast news,” in AIRS, 2009.

[7]. Z.Liu, L.Xie, and W.Feng, “Maximum lexical cohesion for fine-grained news story segmentation,” in INTERSPEECH, 2010.

[8].T.Pedersen, S.Patwardhan, and J.Michelizzi, "Wordnet::similarity-measuring the relatedness of concepts," in AAAI (Intelligent Systems Demonstration), 2004.

[9]. Christiane Fellbaum, Ed., WordNet: An Electronic Lexical Database, MIT Press, 1998.

[10].G.Jeh and J.Widom, "SimRank: A measure of structural-context similarity," in ACM SIGKDD, 2002.

[11].G.He,H.Feng,C.Li,and H.Chen,"Parallel SimRank computation on large graphs with iterative aggregation,"in ACM SIGKDD,2010.

Those skilled in the art can understand that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the above-mentioned embodiments of the present invention are for description only, and do not represent the advantages and disadvantages of the embodiments.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (5)

1. a kind of Chinese News Stories dividing method based on flexible Semantic Similarity tolerance is it is characterised in that methods described bag Include following steps:

(1) target collected works are inputtedTo each News Stories script t in collected works_iCarry out participle；

(2) set up context relation figure；

(3) context dependence between word is iterated propagate by described context relation figure and quick sorting algorithm Obtain flexible semantic dependency matrix；

(4) it is defined by described flexibility flexible Semantic Similarity between sentence for the semantic dependency matrix；

(5) using described flexibility Semantic Similarity, Chinese News Stories are split；

The described step setting up context relation figure particularly as follows:

1) read in each News Stories script successively, word frequency statisticses are carried out to the word being comprised；

2) according to the word frequency threshold value defining, frequent words and low frequency word are deleted；

3) using the word retaining as context relation in figure node, its set be v；

4) judge gather in any two word whether simultaneously appear in a certain News Stories script, and this two words it Between distance be less than or equal to distance threshold, if it is between this two words, set up side, the set on side is e；If No rejudge other any two words, until whole gather in word be all traversed；

5) the weights s on side_cBy the weights sim between word_cThe weights sim of (a, b), word itself_c(a a) represents；

6) described context diagram is shown as g=< v, e, s_c>.

2. method according to claim 1 is it is characterised in that weights sim between described word_c(a, b) particularly as follows:

${\mathrm{sim}}_c(a,b)=\frac{freq(a,b)}{{\mathrm{freq}}_{max}+\mathrm{\&epsiv;}}$

Wherein, freq (a, b) represents the number of times that word a and word b occurs simultaneously, freq_max=max_(i,j){ freq (i, j) } table Show the frequency maxima to (i, j) for the word, ε is a constant in order to guarantee 0≤sim_c(a,b)≤1.

3. method according to claim 1 is it is characterised in that the weights sim of described word itself_c(a, a)=1.

4. method according to claim 1 is it is characterised in that described calculated by described context relation figure and quicksort Method the context dependence between word is iterated propagate the step obtaining flexible semantic dependency matrix particularly as follows:

1) defining the Semantic Similarity between context relation in figure word is sim_s(a, b), following three criterions of satisfaction:

Word is 1 with the similitude of itself, that is,

sim_s(a, a)=1；sim_s(a, b) and sim_c(a, b) positive correlation；sim_sSimilitude between (a, b) and their neighbours just becomes Than；

2) the iterative diffusion process of definition Semantic Similarity:

${\mathrm{sim}}_s^{\left(0\right)}(a,b)={\mathrm{sim}}_c(a,b);{\mathrm{sim}}_s^{\left(t\right)}(a,b)=\frac{c}{z}\underset{u~a,v~b}{\mathrm{\&sigma;}}{\mathrm{sim}}_s^{(t-1)}(a,b);{\mathrm{sim}}_s(a,b)=\underset{t\&rightarrow;\mathrm{\&infin;}}{\lim }{\mathrm{sim}}_s^{\left(t\right)}(a,b);$

Wherein, u～a, v～b represent that u and v is word a and the neighbor node of word b in context relation in figure respectively, and z is to return The one change factor, c is controlling elements,Represent the Semantic Similarity of word a and word b during the t time iteration,Represent the Semantic Similarity of word a and word b during the t-1 time iteration,Represent initialization；

3) use quick sorting algorithm solve 2) defined in relational expression, obtain semantic dependency, each two word is all asked for Semantic dependency, several semantic dependencies constitute flexible semantic dependency matrix, and this correlation matrix is designated as s_s.

5. method according to claim 1 it is characterised in that described by described flexibility semantic dependency matrix to sentence Between the step that is defined of flexible Semantic Similarity particularly as follows:

$sim(s_i,s_j|s_s)=\frac{f_i^t{s}_s{f}_j}{\left|\right|f_i\left|\right|\left|\right|f_j\left|\right|}$

Wherein s_iAnd s_jRepresent sentence respectively, | | f_i| | and | | f_j| | represent two norms of two sentence word frequency vectors respectively, t is to turn Put.