CN110096705A - A kind of unsupervised english sentence simplifies algorithm automatically - Google Patents

Abstract

The invention discloses an unsupervised automatic simplification algorithm for English sentences in the Internet field, which is carried out according to the following steps: step 1, vector representation of training words; step 2, obtaining the frequency of words; step 3, obtaining the simplified sentence set and A collection of complex sentences; step 4, filling the phrase table; step 5, training the simplified sentence language model and the complex sentence language model respectively; step 6, building a simplified sentence model based on phrases; step 7, iteratively executing the back-translation strategy for better training The sentence simplification model of the present invention effectively improves the accuracy of English sentence simplification by making full use of the English Wikipedia corpus without using any marked parallel corpus.

Description

An Unsupervised Algorithm for Automatic Simplification of English Sentences

technical field

The invention relates to an Internet text algorithm, in particular to an unsupervised automatic English sentence simplification algorithm.

Background technique

In recent years, textual materials on the Internet have provided a lot of useful knowledge and information to a wider range of users. Then, the way texts are written online, such as lexical and syntactic results, can be difficult to read and understand for many people, especially those with low literacy, cognitive or language disabilities, or limited knowledge of the language of the text. Texts that contain very wordy words or long and complex sentences are not only difficult to read and understand by humans, but also difficult for machines to analyze. Automatic text simplification is to simplify the content of the original text as much as possible while retaining the original text information, so as to make it easier to be read and understood by a wider audience.

Existing text simplification algorithms use machine translation algorithms to learn simplified sentences from parallel corpus pairs of complex sentences and simplified sentences in one language. This text reduction algorithm is a supervised learning task, and its effectiveness relies heavily on a large number of parallel reduced corpora. However, the existing English parallel simplified corpus is mainly obtained from the general English Wikipedia and the English Wikipedia for children, and sentences are selected from two different Wikipedias as parallel sentence pairs through matching algorithms. The parallel simplified corpus currently available is not only small in number, but also contains many non-simplified sentence pairs and wrong sentence pairs, mainly because the children's version of Wikipedia is written by non-professionals and does not correspond to ordinary Wikipedia one-to-one. It makes it difficult to choose a suitable sentence matching algorithm. Because of the problem of simplifying the parallel corpus, the effect of the existing text simplification algorithm is not very ideal.

Contents of the invention

The purpose of the present invention is to provide an unsupervised automatic simplification algorithm for English sentences. Without the need for any parallel simplification corpus, only the publicly downloaded Wikipedia corpus is used to realize the automatic simplification of English sentences, thereby allowing users to read and read more easily. Comprehend English sentences, especially for those with cognitive or language disabilities.

The purpose of the present invention is achieved in that a kind of unsupervised English sentence automatic simplification algorithm, carries out as follows:

Step 1. Use the public English Wikipedia corpus D as the training corpus, and use the word embedding algorithm Word2vec to obtain the vector representation v _t of the word t; the word vector representation obtained by the Word2vec algorithm can well capture the semantic features of the word; use Skip -Gram model learning word embedding algorithm Word2vec; given corpus D and word t, consider a sliding window centered on t, and use W _t to represent the set of words appearing in the context window of t; observe the logarithmic probability definition of the context word set as follows:

In Equation (1), v'w is the context vector representation of word _w , and V is the vocabulary of D; then, the overall objective function of Skig-Gram is defined as follows:

In formula (2), the word vector representation can be learned by maximizing the objective function;

Step 2, using Wikipedia corpus D, count the frequency f(t) of each word t, f(t) represents the number of occurrences of word t in D;

Step 3, using the Wikipedia corpus D to obtain a simplified sentence set S and a complex sentence set C;

Step 4, using the vector representation of the word and the frequency of the word, populate the phrase table PT (Phrase Table) _representing the probability of word translation into another word; in PT, the translation probability p( _{t j} | The calculation formula of t _i ) is as follows:

In formula (4), cos represents the calculation formula of cosine similarity;

Step 5. For the simplified sentence set _S and the complex sentence set _C , the language model _KenLM algorithm is used for training respectively, and the simplified language model LMS and the complex language model LMC are obtained; LMS and LMC remain unchanged in the subsequent iterative learning process _. Change;

Step 6. Using the phrase table PT, the simplified language model _LMS and the complex language model LMC, the phrase _- based machine translation algorithm PBMT (Phrased-based Machine Translation) is used to construct a simplified algorithm from complex sentences to simplified sentences Given a complex sentence c, The algorithm uses formula (5) to calculate the scores of the sentence s composed of different word combinations, and finally select the sentence s' with the highest score as the simplified sentence:

s'=argmax _s p(c|s)p(s) (5)

In formula (5), the PBMT algorithm decomposes p(c|s) as the inner product of the phrase table PT, and p(s) is the probability of the sentence _s , which is obtained from the language model LMS;

Step 7. Use the initial PBMT algorithm Iteratively execute the Back-translation strategy to generate a better text simplification algorithm.

As a further limitation of the present invention, step 3 specifically includes:

Step 3.1, for each sentence s in the Wikipedia corpus D, use the Flesch Reading Ease (FRE) algorithm to score, such as formula (3), and sort by the score from high to low;

In formula (3), FRE(s) represents the FRE score of sentence s, tw(s) represents the number of all words in sentence s, and ts(s) represents the number of all syllables in sentence s;

Step 3.2, remove the sentence set with a score of more than 100, remove the sentence set with a score lower than 20, and remove the sentence set with an intermediate score; finally, select the sentence set with a high score as the simplified sentence set S and the sentence set with a low score as a complex sentence collection C.

As a further limitation of the present invention, said step 7 specifically includes:

Step 7.1, first use Algorithm, translate the complex sentence set C to obtain a newly synthesized simplified sentence set S ₀ , and then perform steps 7.2 to 7.5 in a loop, and the number of iterations i is from 1 to N;

Step 7.2. Use the synthesized parallel corpus ( _{S i-1} , _C ), simplified language model LMS and complex language model LMC to train a new PBMT algorithm from simplified sentences to complex sentences

Step 7.3, use Translate the simplified sentence set S to obtain a new synthetic complex sentence set C _i ;

Step 7.4, use the synthesized parallel corpus ( _{C i} , _S ), simplified language model LMC and complex language model LMS to train a new PBMT algorithm from complex sentences to simplified sentences

Step 7.5, use Translate the complex sentence set C to obtain a new synthesized simplified sentence set S _i ; go back to step 7.2 and repeat until N iterations.

Compared with prior art, the beneficial effect of the present invention is:

1. In the process of filling the phrase table, the present invention combines the word vector representation and word frequency obtained from the Wikipedia corpus, can capture the semantic information of words and the frequency of use of words, and overcome the traditional machine translation based on phrases The PBMT algorithm needs to use parallel corpus to fill the phrase table;

2. The present invention uses the Wikipedia corpus as a knowledge base, and uses the Flesch Reading Ease (FRE) algorithm to score sentences, thereby obtaining simplified sentence sets and complex sentence sets, thereby enabling more accurate training of complex sentence language models and simplified sentence languages Model;

3. The present invention uses the obtained phrase table, complex sentence language model and simplified sentence language model to construct an initial unsupervised text simplification algorithm based on the PBMT algorithm; this text simplification algorithm is not only an unsupervised algorithm, but also simple and easy Explain and be able to train quickly;

4. After constructing the initial simplification algorithm, the present invention uses the simplification algorithm to generate parallel corpus, thereby adopting the strategy of back-translation to optimize the existing text simplification model, correcting possible wrong entries in the initial phrase table, and further improving the algorithm type performance.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments.

An unsupervised automatic simplification algorithm for English sentences is carried out as follows:

Step 1. Use the public English Wikipedia corpus D as the training corpus, which can be downloaded from " https://dumps.wikimedia.org/enwiki/ ", and use the word embedding algorithm Word2vec to obtain the vector representation v _{t of the word t} ; through the Word2vec algorithm The obtained word vector representation can well capture the semantic features of the word; after obtaining the word vector representation, the similarity of the word can be obtained to help find the highly similar word set of each word; in this example, each vector’s The dimension is set to 300, and the Skip-Gram model is used to learn the word embedding algorithm Word2vec; given a corpus D and a word t, consider a sliding window centered on t, and use W _t to represent the set of words appearing in the context window of t; sliding The window is set to 5 words before and 5 words after t; the logarithmic probability of observing the context word set is defined as follows:

In Equation (1), v'w is the context vector representation of word _w , and V is the vocabulary of D; then, the overall objective function of Skig-Gram is defined as follows:

In Equation (2), the word vector representation can be learned by maximizing the objective function by using stochastic gradient descent algorithm and negative sampling.

Step 2. Use the Wikipedia corpus D to count the frequency f(t) of each word t, f(t) represents the number of occurrences of the word t in D; in the field of text simplification, the complexity measurement of words will take into account words Generally speaking, the higher the frequency of a word, the easier it is to understand the word; therefore, word frequency can be used to find the most understandable word from the highly similar word set of word t.

Step 3, a very large corpus in the Wikipedia corpus D contains a large number of complex sentence sets and simple sentence sets; using the Wikipedia corpus D, obtain the simplified sentence set S and the complex sentence set C;

Step 3.1, for each sentence s in the Wikipedia corpus D, use the FRE (Flesch Reading Ease) algorithm to score, such as formula (3), and sort according to the score from high to low; the higher the score, the sentence The simpler, the lower the score means the more difficult the sentence;

In formula (3), FRE(s) represents the FRE score of sentence s, tw(s) represents the number of all words in sentence s, ts(s) represents the number of all syllables in sentence s; FRE algorithm is usually used to evaluate The quality of the final simplification result of the text simplification model;

Step 3.2, remove the set of sentences with a score of more than 100, remove the set of sentences with a score of less than 20, and remove the set of sentences with an intermediate score; remove sentences with high and low scores to remove particularly extreme sentences; remove sentences with an intermediate score In order to establish a clear boundary between S and C; finally, select the high-scoring sentence set as the simplified sentence set S and the low-scoring sentence set as the complex sentence set C; in this example, S and C are both selected 1 Thousands of sentences.

Step 4, using the vector representation of the word and the frequency of the word to fill the phrase table PT (Phrase Table) representing the probability of the word being translated into another word. In PT, the calculation formula of the translation probability p(t _j |t _i ) from word t _i to word t _j is as follows:

In formula (4), cos represents the cosine similarity calculation formula; considering that it is not feasible to learn the probability conversion of all words, in this example, the most frequent 300,000 words are selected, and only the most similar 200 words are calculated. The probability of each word; for the proper nouns in the words, only the probability of itself is calculated.

Step 5. For the simplified sentence set _S and the complex sentence set _C obtained in step 3, the language model _KenLM algorithm is used to train respectively, and the simplified language model LMS and the complex language model LMC are obtained; LMS and _LMC are iteratively learned later The process remains unchanged; the language model is used to calculate the probability given to the word sequence specified in the corpus; the simplified language model and the complex language model help to improve the quality of the simplified model by calculating the probability of the word sequence by performing local replacement and word order rearrangement.

Step 6. Using the phrase table PT, the simplified language model _LMS and the complex language model LMC, the phrase _- based machine translation algorithm PBMT (Phrased-based Machine Translation) is used to construct a simplified algorithm from complex sentences to simplified sentences The PBMT algorithm was first proposed in "Statistical phrase-based translation" in 2007, and is used for machine translation with dual languages; given a complex sentence c, The algorithm uses formula (5) to calculate the scores of the sentence s composed of different word combinations, and finally select the sentence s' with the highest score as the simplified sentence:

s'=argmax _s p(c|s)p(s) (5)

In formula (5), the PBMT algorithm decomposes p(c|s) as the inner product of the phrase table PT, and p(s) is the probability of the sentence _s , which is obtained from the language model LMS.

Step 7. Since only non-parallel corpus can be obtained, use the initial PBMT algorithm Iterative implementation of the Back-translation strategy can transform very difficult unsupervised learning problems into supervised learning tasks, thereby generating better text simplification algorithms;

Step 7.1, first use Algorithm, translate the complex sentence set C to obtain a newly synthesized simplified sentence set S ₀ ; then, perform steps 7.2 to 7.5 in a loop, and the number of iterations i is from 1 to N;

Step 7.2, use the synthesized parallel corpus ( _{S i-1} , _C ), simplified language model LMC and complex language model LMS to train a new PBMT algorithm from simplified sentences to complex sentences

Step 7.3, use Translate the simplified sentence set S to obtain a new synthetic complex sentence set C _i ;

Step 7.4, use the synthesized parallel corpus ( _{C i} , _S ), simplified language model LMC and complex language model LMS to train a new PBMT algorithm from complex sentences to simplified sentences

Step 7.5, use Translate the complex sentence set C to obtain a new synthesized simplified sentence set S _i ; go back to step 7 and 2 and repeat until N iterations; in this example, N is set to 3.

Intuitively, since the input of the PBMT algorithm contains noise, many entries in the phrase table are incorrect; however, in the process of generating simplified sentences, the language model can help correct some errors; as long as this happens , as the iteration continues, the phrase table and the translation algorithm will be improved accordingly; as more entries in the phrase table will be corrected, the PBMT algorithm will become more and more powerful.

The present invention is not limited to the above-mentioned embodiments. On the basis of the technical solutions disclosed in the present invention, those skilled in the art can make some replacements and modifications to some of the technical features according to the disclosed technical content without creative work. Deformation, these replacements and deformations are all within the protection scope of the present invention.

Claims (3)

1. a kind of unsupervised english sentence simplifies algorithm automatically, which is characterized in that carry out as follows:

Step 1, using disclosed English wikipedia corpus D as training corpus, obtained using word embedded mobile GIS Word2vec The vector of word t indicates v_t；The term vector obtained by Word2vec algorithm indicates to can be good at catching the semanteme of word special Sign；Using Skip-Gram model learning word embedded mobile GIS Word2vec；Given corpus D and word t considers that one with t is The sliding window of the heart, uses W_tIt indicates to appear in the set of words in t contextual window；The logarithm for observing context words set is general Rate is defined as follows:

In formula (1), v'_wIt is the context vector expression of word w, V is the vocabulary of D；Then, the overall goals letter of Skig-Gram Number is defined as foloows:

In formula (2), the vector expression of word can be learnt by maximizing the objective function；

Step 2, using wikipedia corpus D, count the frequency f (t) of each word t, f (t) indicates appearance of the word t in D Number；

Step 3 utilizes wikipedia corpus D, the simplified sentence set S and complex sentence subclass C of acquisition；

Step 4 is indicated and the frequency of word, filling indicate that word is translated as the phrase table PT of another word probability using the vector of word (Phrase Table)；In PT, word t_iTo word t_jTranslation probability p (t_j|t_i) calculation formula it is as follows:

In formula (4), cos indicates cosine similarity calculation formula；

Step 5 is directed to simplified sentence set S and complex sentence subclass C, and language model KenLM algorithm is respectively adopted and is trained, Obtain reduction language model LM_SWith complex language model LM_C；LM_SAnd LM_CIt is remained unchanged in iterative learning procedure below；

Step 6 utilizes phrase table PT, reduction language model LM_SWith complex language model LM_C, using phrase-based machine translation Algorithm PBMT (Phrased-based Machine Translation) constructs complicated sentence to the simplification algorithm for simplifying sentenceGiven complexity sentence c,Algorithm utilizes formula (5), calculates separately the score of the sentence s of different contamination compositions, Finally selecting score to be high sentence s ' will be as simplified sentence:

S'=argmax_sp(c|s)p(s) (5)

In formula (5), PBMT algorithm decomposes inner product of the p (c | s) as phrase table PT, and p (s) is the probability of sentence s, is from language mould Type LM_SIt obtains；

Step 7 utilizes initial PBMT algorithmIteration executes the strategy of retroversion (Back-translation), generates more Excellent text simplifies algorithm.

2. the unsupervised english sentence of one kind according to claim 1 simplifies algorithm automatically, which is characterized in that step 3 tool Body includes:

Step 3.1, for each sentence s in wikipedia corpus D, using Flesch Reading Ease (FRE) algorithm into Row marking, such as formula (3), and is ranked up from high to low by score value；

In formula (3), FRE (s) indicates the FRE score of sentence s, and tw (s) indicates the number of all words in sentence s, and ts (s) indicates sentence The articulatory number of institute in sub- s；

Step 3.2, removal score are more than 100 sentence set, and removal obtains the sentence set lower than 20 points, removes intermediate comparison scores Sentence set；Finally, it is multiple to select the sentence set of high score to be used as the sentence set for simplifying sentence set S and low score Miscellaneous sentence set C.

3. the unsupervised english sentence of one kind according to claim 1 simplifies algorithm automatically, which is characterized in that the step 7 specifically include:

Step 7.1, first withAlgorithm translates complex sentence subclass C, obtains the simplification sentence set S of new synthesis₀, Then, circulation executes step 7.2 to 7.5, and the number of iterations i is from 1 to N；

Step 7.2, the parallel corpus (S using synthesis_i-1, C), reduction language model LM_SWith complex language model LM_C, training is newly From simplify sentence to complicated sentence PBMT algorithm

Step 7.3 utilizesIt translates and simplifies sentence set S, obtain the complex sentence subclass C of new synthesis_i；

Step 7.4, the parallel corpus (C using synthesis_i, S), reduction language model LM_CWith complex language model LM_S, train newly From complicated sentence to the PBMT algorithm for simplifying sentence

Step 7.5 utilizesComplex sentence subclass C is translated, the simplification sentence set S of new synthesis is obtained_i；It comes back to Step 7.2 repeats, until iteration n times.