TWI288335B - Method to automatically summarize Chinese digital documents - Google Patents

Abstract

A method is developed to automatically summarize Chinese digital documents. It evaluates the importance of each sentence in the digital document, finds out the similarity between the document title and each of its sentences, and combines the title with the reduced sentence to become a summary candidate with its number of characters below an assigned limit. Finally, according to the ratio of the number of characters and the similarity, it sorts those summary candidates and provides the result for the user to select. This invention simultaneously considers factors including the number of words after summarization, the importance covered by the content, the readability, and the coherence of the meaning. It effectively represents the key points of original document within limited number of words and greatly reduces the human labor burden.

Description

!288335 15843twf.doc/g IX. Description of the invention: [Technical field to which the invention pertains] = „ There is a summary method for a file, and in particular, an automatic summary method for a Chinese digital file. [Prior Art] / , The widespread use of digital technology, in recent years, the production of digital documents is about 7_ to 5 digits: and Lu's single news website can produce a new door every day, plus right every tens of minutes. As for the update-time instant 2, it is quite large, and it is easy for an individual to cause the information to become useless or I. 'Automatically handle the development and operation of digital files in an automated way--a way to quickly get information, and Second, mobile news can provide people with instant convenience. Secondly, different pages of mobile news and online news must be deleted, and the secondary and repeated content must be deleted. The machine type is ^8 ϊζ ^_69 Full-size characters are limited to half-words. Sub-transmissions, and some deleted mobile phones are limited to 45 full-form two-two sentences. Figure 1, only &quot;, almost only the conventional method of the temple news sample 'its title Only two sentences Words, nouns, and connections' discard each word with a part-of-speech tag (such as a link), and then cut the long sentence into a 1288335 15843twf.doc/g clause (Meaningful Unit, MU). : "fragments" separated by commas. If the starting words are "and" and "and", the "fragments" are merged forward; if the starting vocabulary is a verb, the subject should be preceded by " Fragment, so this "fragment" is also merged forward. Taking the second sentence of the example as an example, according to the above method, the last piece must be linked to the "Saga beef sukiyaki" which is top-notch. But the previous episode, "Residing in a restaurant with a maximum of 60,000 yen a night" can't be used as a starting point for a clause, and must be linked forward. However, the "Interview" of "The Interview with the Professional Stars Wang Zhizhi" is a verb. It is necessary to continue to link to "except for seeing the Japanese professional baseball team". In this way, I can finally get "Yellow Zi to visit Wang Haozhi to visit the Fukuoka Dome to enjoy the Japanese professional baseball game" (23 words), "Although the two-day trip is tight, but this trip can be said to be "top luxury tour" "The word", and "In addition to being able to witness the Japanese professional baseball team, interviewing the professional baseball star Wang Zhizhi, living in a restaurant with a maximum of 60,000 yen in the evening, and wishing to eat the top Saga beef sukiyaki" ( 53 words) three clauses. After splitting the file into clauses, combine the predetermined lengths. Since the predetermined length is 69 words or 45 words, it can only accommodate one or two clauses. The general meaning of the document, so that the abstract extracted from the 昼 has the finishing touches of the two fruits, can choose the title as one of them - the rest of the words and then the other clauses. The remaining clauses should be selected with the longest <[over the total length of the clause to match the given word limit as much as possible. In the above example, the last sentence is combined with the title in the original order, and you can get a summary of 67 words. Suwa Wang Zhizhi, a luxury Japanese trip, except to see Japan in person.

First, the correct word segmentation, part of speech mark and segmentation clause is not easy. Because there are many unknown words and unpredictable grammars, if there is no prior analysis, the segmented clauses are difficult to guarantee their semantic integrity. Second, the selected clause is not optimal. For example, in the 45-word abstract of the previous example, you can choose the 45-word abstract that makes the title and the last snippet of the file better. "Su Shi Wang Zhizhi, a luxurious Japanese trip, still live in a restaurant with a maximum of 60,000 yen a night. And if you want to eat the top Saga beef sukiyaki. 』 (because the length is closer to 45 words and the content is more complementary). Third, it is difficult to handle long files. When the length of the file is longer and the length of the abstract is still fixed so short, the difficulty of the abstract is higher. Different people do 1288335 15843twf.doc/g professional baseball's interview with professional baseball star Wang Zhizhi, still living in - night up to 60,000 In the Japanese currency restaurant, I would like to eat the top Saga beef Shouxi welding. , η 69 words, can be said to be the best summary. If you want to be a student, you can combine the title with the second long sentence to get 37? "Make 贞" = 'Luxury Japan trip'. Although the 2 day trip is tight, you can see the top level. "The trip to China", this sentence is a 35-word abstract than the other one: "Wu visit Wang Zhizhi, the luxury sputum behavior visit Wang Yu to Fuxian eggs to appreciate the Japanese professional baseball is better than (because the length is closer to 45 words) And the content is more complementary) &gt;. The above methods can be summarized as follows: first break the file, make a part-of-speech tag, and divide it into clauses according to some rules; then select the longest sentence that is combined with the title and not longer than the predetermined length, and the title is combined with the title. A summary of the fetch. However, the above approach raises several questions: The abstract is also different. 1288335 15843twf.doc/g [Summary of the Invention] The purpose of the present invention is to provide an automatic method for digital files, and to find a sentence similar to a title in the file according to the title of the file 'combining these similar sentences with the title into no more than Refers to the candidate sentences of the text, which are sorted and used for selection. The invention of the number 1 proposes that the number of files in the pattern of the towel domain includes a plurality of sentences. The automatic summary method first identifies the importance of each sentence in the number file, and takes the most important first η sentences. _ is = select sentence, then in each candidate sentence, find the appropriate sentence point virtual U to generate the summary candidate sentence, and finally according to the summary candidate: proportional and similarity will sort the summary candidate sentence, and from high to low According to the Chinese digital file method according to the preferred embodiment of the present invention, the above method evaluates the longest of the sentences in the sentence, or the highest frequency of occurrence = the large repeated substring. The receivers use the maximum repeated string and any library to deal with the number of mosquitoes, and stop the repeated words of the _ more than two, (that is, appear more than - times) as the keyword of the digital file, then # The importance of each eye (four), according to the number of words in each sentence 1: and the importance of these traits, from the big, ', take the first η sentences as a candidate. The method for generating a digest candidate sentence according to the preferred embodiment of the present invention is as follows: firstly, the editing distance between the title and each candidate sentence is compared in a dynamic manner to find out句 The smallest sentence (that is, the most similar to the title) and the punctuation position. 1288335 15843twf.doc/g Then replace the sentence segment with the title, and use the punctuation position as the sentence to combine the remaining segments of each candidate sentence to get the summary candidate sentence. If the number of words of the summary candidate sentence after the sentence exceeds a predetermined limited number of words, the clause of the sentence is concatenated, and the maximum number of clauses not exceeding the number of the limited number of words is connected to the title; After the sentence, the summary candidate sentence has fewer words than the qualified summary sub-number, then try to increase the clause of the clause and connect to the title with the most clauses that do not exceed the limit of the limited number of words.

According to the automatic summary method of the Chinese digital file according to the preferred embodiment of the present invention, the step of sorting the summary candidate sentences is to first convert the edit distance of each summary candidate sentence into a similarity. If the content of the original digit file exceeds a certain number, the similarity of each of the second sentence money is multiplied by a special factor as the last similarity of the sentence. The number of words combined is then divided by the specified number of digest words to convert it to a ratio of words between 0 and i. Finally, the final ordering of these summary candidates is determined by a preset rule. According to the automatic summary method of the Chinese digital file according to the preferred embodiment of the present invention, the upper _ is set to three steps. First, the first-sum summary candidate with the most similarity and the second summary candidate with the highest number of words_ are compared. If the first-sum candidate sentence is the second summary candidate sentence, the first summary candidate sentence is output, and is deleted from the _chouse sentence collection cap-summary candidate sentence. Secondly, if the similarity of the first abstract candidate sentence is greater than a specific multiple of the phase of the dichosum candidate sentence: and the number of words of the first-digest candidate sentence is outputted by the first extract Weiler, if not, the second is the second Summary candidate 9

1288335 15843twf.doc/g ^Original Abstract (4) (4) The towel will be picked up by Wei (the fourth step) until the original summary candidate set is taken right and the candidate is left. And any abstract according to the preferred embodiment of the present invention, the Chinese digital file, the method, the above-mentioned decision to withdraw the money, the last phase of the method package = as defined by the rules and machine | | learning to find the most Good rules. According to the preferred embodiment of the preferred embodiment of the present invention, the method for selecting the Chinese digital file is given first. If not, the above-mentioned method is used to capture the key of the digital file. The word dish, after which the keywords are extracted according to the importance of these key words, and finally the title is generated according to the order in which they appear in the digital file. In summary, the automatic digest method of the Chinese digital file of the present invention is suitable for general Chinese digital files, and is particularly suitable for the automatic, mobile news newsletter. It is characterized in that the abstract generated by it can take into account the number of words after the summary, the content of the content, the readability, and the coherence of the semantics, etc., which can effectively express the key content of the original document in the specified number of words. And greatly reduce the human burden. The above and other objects, features, and advantages of the present invention will become more apparent <RTIgt; [Embodiment] FIG. 2 is a flow chart of an automatic digest method for a Chinese digital file according to a preferred embodiment of the present invention. In this embodiment, a 1288335 15843 twf.doc/g Chinese digit file is first obtained, and it is determined whether or not to limit the number of digest words (step S210). For example, if the abstract of the Chinese digital file is used for the newsletter of the mobile phone, the summary word count may be limited to 45 words or 69 words within the range. Next, it is judged whether or not the Chinese digital file contains a set of titles (step S220), and if not, the title is obtained from the digital file (step S230). Wherein, the method for determining the title includes, for example, the automatic extraction method of the key features of the digital document of the prior invention (the Republic of China Announcement No. 153789 patent), extracting the keyword of the digital file, and then according to the importance of the vocabulary The six successors are discarded and the title is generated in the order in which they appear in the digital file. However, in this case, only the keyword method is obtained by a specific method, and the method for finding a keyword pool is not limited. Those skilled in the art can select different key word screening methods to obtain keyword sinks. After obtaining the title, the importance of each sentence in the Chinese digital file can be evaluated, and each sentence is ranked from highest to lowest according to importance, and the most important first η sentences are taken as candidate sentences (step S24). . And the candidate sentence is called to select an appropriate sentence to be combined with the title to form a summary candidate sentence (step S250). Finally, the word proportion and similarity of the summary candidate sentence are calculated, and the summary candidate sentences are sorted according to the order of high-to-low output, and the word ratio and similarity information are provided to facilitate user selection (step S26). . FIG. 3 is a flow chart showing the generation of a sentence according to a preferred embodiment of the present invention. First, the maximum repeated string in the Chinese digital file is obtained by the automatic extraction method of the key features of the digital file previously invented (step S3i). Among them, because the subject vocabulary of the file will be repeated, but not all of the weights 1288335 15843twf.doc / g complex word ^ is a useful keyword, therefore, only the longest, or the most frequently occurring repeated string is called The maximum repeating string is the keyword that is filtered by the stop word and the largest repeated string is regarded as the keyword of the file. Since the title is quite important in the news, the title is treated as a word breaker with a 120,000-word lexicon. After the filter is removed from the stoppage, it is also considered a keyword for the document. Thus, the keyword of the file is obtained from both the file title and the document context (step S320). Then, the above-mentioned keywords are summarized according to the following formula: the order of the 'fity' is sorted from large to small, and the first n sentences are taken as the follow-up kings (step S330) 'where 匕 is the keyword sink Ice is in the digit frequency, and is the shed where the most frequent keyword appears in the file, η can be regarded as the number of candidate sentences specified by the user.

After the computer is finished, the number of summary candidates available for the user to select. The automatic feature collection method and the 120,000 word vocabulary with I Ι ίί are only applied to the embodiment. In the actual operation, the specific method and the importance formula are not limited, and the user can select different 10,000 according to their needs. / Go to the summary candidate for the desired sentence. = Σ (〇·5 + 0.5 X tfw/max^tf) In order to get the abstract from the ,, it has content consistency and consistency 'so the sentence combined with the title 'preferably with the title on the content 'two ΐ ' That is, there is enough similarity. Of course, when i is the highest, the cloth binary is the same, which is not appropriate. Under the headline of the general news editor, the sentence of Jiejie J itself is completely copied, but it is more concentrated and simpler. The result is a sentence similar to the title, with the title ..., and the complement of the content. In addition to this, you also need to know from that fragment.

12 1288335 15843twf.doc/g Start working with the title. So not only find the most similar places. For the same time;; this is like === planning method between the title and the sentence: the series of students > 4: Yes:: Ben, the example of the summary of the candidate sentence produced by the example of the secret _ heading and secret candidate sentence 'find the editing distance is the smallest (that is, the second with the title): ; symbol position (step S, and replace the sentence piece &amp; with the title, and use the punctuation position as the point combination

The remaining segments are obtained to obtain a summary candidate sentence (step S420). However, the summary of the W-bit file is used as a new source for mobile phones, and the number of digested words is given to the number of digested words. Common examples include 45, words and ^ words. After the limited number of digest words is given, it can be determined whether the number of words of the digest candidate sentence after the sentence exceeds the limit digest word number (step S43〇), and if it is 1, the sentence point is moved backward to shorten the sub-sentence The sentence (step S45〇) 1 attempts to move the sentence point forward to increase the sentence of the sentence (step S440). The combined sentence is based on the principle that the number of words in the summary is not exceeded. FIG. 5 is a %IL private diagram of a summary candidate sentence sequence according to a preferred embodiment of the present invention. In order to facilitate comparison of similarities of sentences of different lengths, the edit distances are converted into similarity illusions by the following formula (step S51〇), where exp is a natural index, an edit distance, w is the number of words of the title, π The number of words in the candidate sentence. However, the above is only an example of this specific formula, and the user can use different formulas to obtain the desired similarity as needed. . / edit , edit — sim = exp(- 13 1288335 15843twf.doc/g Since the way the news is written is in a pyramidal way, the description of the details is more detailed, relative, and the more the front text, the more Therefore, in order to enhance the similarity of the preceding sentences, the first few sentences need to be considered first. Therefore, it is then determined whether the context of the original digit file exceeds three sentences (step S520). If so, the second sentence is followed by the sentence. The similarity is multiplied by a specific multiple such as 〇85, as the final similarity (step S530). After the similarity is determined, the number of combined words is further divided by the specified number of digest words to be converted into The ratio of the number of words between 1 (step S540). Finally, all the summary candidate sentences are arranged in a preset rule.

The sequence is for the user to select (step S550). The method includes selecting the summary candidate sentence A with the highest similarity and the summary candidate sentence B having the largest proportion of words from the summary candidate sentence (step S551). And it is judged whether or not the digest candidate sentence a is equal to the digest candidate sentence B (step S552). If yes, the summary candidate sentence A output is appended to the sorted result, and is deleted from the original summary candidate sentence set (step B is not ^3), and then directly proceeds to step S556; if not, then the determined The similarity of the sentence A is greater than the similarity of the abstract candidate sentence B, and the number of words of the summary sentence A is greater than the number of words of the candidate number of the candidate c (Ϊ). For example, it is judged whether or not the abstract is ίί2_1 abstract candidate (four) similarity (four) times, and if so, the proportion of words is large (four) than the number of words _ 〇. 75 times. Excerpts = ^ Γ 53' whose implementation content is as above; ^ No, then, zhongzhong delete == fruit 2' = summary candidate sentence not rounded one). If: == extract 1288335 15843twf.doc/g continue to repeat the sorting action; if not, the sorting program is terminated S557), and the sorting result of the summary candidate sentence is obtained by rotation. In the above steps, in the step of sorting the candidate sentences, the specific number and the specific multiple are given as an embodiment of the present invention, and the size is not limited, and the user needs to increase or decrease the visibility to achieve the effect of the present invention. Figure 6 is an example of calculating the edit distance in a dynamic plan in accordance with a preferred embodiment of the present invention. In dynamic programming, the so-called edit distance refers to the use of "insert", "delete", and "substitution" actions to convert a string into another string "required minimum steps" (or "required" Minimum calculation cost"). The dynamic programming method used in this embodiment is as follows: Suppose there are two strings A and B, each having a length of 11 and 111. The two words are compared from the beginning, and the edit distance of the ith word (indicated by 八) to A and the jth word of B (in B[j]) is: d[i, j] = min( d[il,j] + w(A[i],0), d[i_l,j-1] + w(A[i],B[j]), d[i,j_l] +w(0,B[j])) where min(X,Y,Z) represents the smallest of the three numbers X, Y, and Z, and its initial value is ··d[0? 0] = 〇d[i,0] = d[il,0] + w(A[i],0),l&lt;=i&lt;=nd[0,j] = 〇,l&lt;=j&lt;=m ～ The meaning of the function w(X, Y) is: w(A[i], B[j]) ·· represents the computational cost of substituting A[i] into B[j] (A[i], 〇) : indicates the calculation cost of inserting A[i] 15 1288335 15843twf.doc/gw(0, B[j]): indicates the calculation cost of deleting B[j] 乂 "4" A - acjc '"candidate sentence" b decadecf is For example, suppose that the calculation cost of substitution and deletion is 丨, then 丨, 』] can be expressed as the edited distance calculated by the above formula from the _th row of the i-th column of the matrix, and the result is as о. From the last scan of the matrix - the last scan, the place where the first edit distance is found is the most similar part of B and A. When you accept the sentence, replace the word in front of B with the _ of A, and make the summary zero. The above dynamic programming comparison is completed. After scanning from the last column, find the position with the smallest editing distance, because The header with the title and the candidate must be connected to the punctuation mark to maintain readability, so the nearest punctuation must be scanned left and right by a position with the smallest edit distance to find the punctuation position with the smallest edit distance. Taking this as a result, the title is combined with the candidate sentence to generate a summary candidate sentence. However, the method of dynamic programming used in this embodiment is only for calculating the example of the editing, and does not limit the detailed specifications and parameters therein. Familiarity: skill = away • When using a similar formula and Parameters, borrowed from the factory distance. &quot;In addition to the above, the self-method of the Chinese digital file provided by the present invention can find the sentence most similar to the title, and the title is required to be selected, and the user selection is provided after sorting. The selected raw extracts can effectively express the key content of the original document in a specified number of words, and the abstract is lightly burdened. 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 。 The scope of protection of the present invention is defined by the scope of the appended claims. [Simple Description of the Drawings] Figure 1 shows an example of instant news. 02 is a flow chart of an automatic digest method for a Chinese digital file according to a preferred embodiment of the present invention.

FIG. 3 is a flow chart showing the generation of candidate sentences according to a preferred embodiment of the present invention. FIG. 4 is a flow chart showing a summary generation according to a preferred embodiment of the present invention. Figure 5 is a flow diagram of a summary ordering of greens in accordance with the preferred embodiment of the present invention. Shown by Dynamic Programming FIG. 6 is an example of calculating an edit distance according to a preferred embodiment of the present invention. [Main component symbol description] S210: Given Chinese digital file and limited summary word count S220: Determine whether there are S230 in this Chinese several file: Find the title non-importance from the digital file, take the previous S240: Evaluate this Chinese digital file The sentences in the middle sentence are selected as candidate sentences = select the selected sentence to find the appropriate sentence to combine with the title, do 17 1288335 15843twf.doc / g will summarize the candidate = order to calculate the word count ratio of the summary sentence _ S310. The automatic feature extraction method of the key feature finds the maximum repeated string S320: breaks, processes and disables the touch filter substring S330 to calculate the importance, takes the first n sentences as the candidate sentence S410: finds the sentence segment with the smallest edit distance and Punctuation mark position S420: replacing the sentence segment with the title, and determining whether the number of words of the summary candidate sentence exceeds the limit digest word number S440 by combining the candidate sentence S43〇··: moving the sentence point backward to shorten the clause of the sentence S450: Move the sentence forward to increase the sentence of the sentence S510: Calculate the similarity S520: Determine whether the original number file exceeds three sentences S530 · The sentence after the first sentence Multiplied by the similarity billion% S540: calculate the ratio of the number of words · 5550: in a default rule, all of the digest candidate ranker

5551 - Selecting the summary candidate sentence a with the highest similarity and the summary candidate sentence B 5552 having the largest proportion of words - determining whether the summary candidate sentence a is equal to the summary candidate sentence b 5553: outputting the summary candidate sentence A, and extracting from the original summary candidate sentence Delete i S554 · Determine whether the similarity of the summary candidate sentence a is greater than 1.25 times the similarity of the summary candidate sentence B, and the ratio of the number of words of the summary candidate sentence a is greater than 0.75 times the ratio of the number of words of B. 1288335 15843twf.doc/g 5555 · Output the summary candidate sentence B, and delete 5556 from the original summary candidate sentence set: whether the remaining summary candidate sentence 5557: end sorting

Claims (1)

1288335 X. Application for patents: The automatic summarization method of the scoop cup 1^pin digital file, the digital file card includes a sentence, the automatic digest method includes ·· fiti the longest number of documents, or a speech The highest number of maximum repeated strings; the maximum word of the _ and any existing vocabulary to perform word segmentation on the digital file, and after the stop word filtering, to obtain the repeated vocabulary (that is, more than one occurrence) As the multiple key words of the digital file; the importance of counting these key words; according to the number of the riding money and the corresponding importance of the good towels, sorting (four), sorted by big to small After that, the first n sentences are taken as a plurality of candidate sentences; in the candidate sentences, an appropriate sentence is found and combined with a standard to generate a plurality of summary candidate sentences; and, according to the summary candidate sentences The word size ratio and similarity rank the summary candidate sentences and output them in order from high to low. ~ 2· As in the automatic digest method of the Chinese digital file described in the first paragraph of the patent application, the steps of generating the digest candidate sentences include: comparing the title with the candidate sentences by a sorrow pattern Edit the distance, find the sentence segment with the smallest edit distance and a punctuation mark position; and 20 !288335 fr replace the w segment with the title, and mix the period with the period to combine the remaining segments of the candidate sentences. Abstract candidate summary == The number of words of the abstract candidate sentence after the automatic sentence of the Chinese digital file mentioned in item 2 exceeds the limit of one sentence, and the clause of the sentence is shortened, so as not to exceed the limited summary. The word count clause is connected to the title, and the number of words of the summary candidate sentence is smaller than the limited summary word count, and the clause of the sentence is increased, and the number of words of the sentence is not exceeded. The most clauses are connected to the title. 4. The automatic digest method of the Chinese digital file as described in claim 1, wherein the step of sorting the digest candidate sentences comprises: converting the edit distance into a similarity; if the original digital file is in the text When the sentence exceeds a certain number, the similarity of each sentence after the number of the special number is multiplied by a specific system $' as the final similarity of the sentences; the number of combined words is divided by the specified abstract The number of words is converted into a ratio of the number of words between 〇 and 1; and a final rule is used to determine the final ordering of the summary candidates. 21 1288335 5. The Chinese digital file I self-digest method according to item 4 of the patent application scope, wherein the preset rule comprises: comparing the highest similarity-first-sum summary candidate sentence with the highest number of words-second The abstract waiter, if the first-thinking phrase is the second abstract candidate sentence, 'mixes the first-summary candidate sentence, and deletes the first-sum summary candidate sentence from the selected sentence; - the similarity degree of the abstract candidate sentence is greater than the - first specific multiple of the similarity of the second abstract candidate sentence, and the ratio of the number of words of the first abstract candidate sentence is greater than the second abstract ship (four) word number comparator - second special a number, then outputting the first-summary candidate sentence, if not, outputting the second summary candidate sentence, and extracting the output from the summary candidate sentences; and 4 repeating the step of 'decoding until there is no domain waiting Le Remaining until 6. Please refer to the automatic summary method for the digital file of the towel mentioned in item 4, which is the final method of the four-dimensional summary of the ordering method, including the preset rule and machine learning to find out the most Good rules. 7. If you apply for a patent range! The automatic summarization method of the Chinese digital file described in the item is given in advance. If not, the job takes the multi-side key words in the digital file, and the New Zealand depends on the importance of the vocabulary. The removal of the 'most money' combines in the order in which they appear in the mosquitoes to produce the desired title. twenty two