JP3017098B2 - Natural language processing apparatus and method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a natural language processing apparatus and method, and more particularly to a natural language processing apparatus and method for automatically creating a document summary or abstract using a computer.

[0002]

2. Description of the Related Art Heretofore, summarizing a document using a computer (to summarize the main points of the content briefly.
There are the following three types of natural language processing apparatuses that automatically perform abstracting and abstracting (to write out and to write out) and the following (a), (b), and (c). Hereinafter, the abstract and the abstract are collectively referred to as an abstract.

(A) The first type of device is to understand the meaning of an input original sentence using a computer and generate a summary sentence from the result. In this method, since the understanding of the meaning by the computer is a complicated process, the target is limited to short sentences. Therefore, in the case of a document including a long sentence, there is a problem that the meaning cannot be understood or misunderstood, and the summarization cannot be performed, or even if the summarization can be performed, the summarization accuracy is deteriorated.

[0004] (b) The second type of device utilizes the connection relationship between sentences, and has the advantage that summarization can be performed by simple processing compared to the device of (a). But,
Since the connection relation between sentences is used, if the number of sentences in the entire document including the long sentence is small, the extraction of the summary sentence is too rough, and the summarization accuracy also deteriorates in this case.

(C) The third type of device extracts words (keywords) that play an important role in a document, and extracts a portion containing many keywords as a summary.
However, in a long sentence, a phenomenon may occur in which keywords are concentrated only in a part of the sentence. If this type of device is applied to a long sentence and such a sentence is cut out as a summary sentence, there is a problem that important parts and non-important parts are mixed and the summarization accuracy is deteriorated.

[0006] Here, the summarization accuracy refers to the degree of perfection of the summaries when subjectively evaluated.

[0007]

As described above, in the various conventional natural language processing devices as described above, (a),
As described in (b) and (c), when summarizing or abstracting a document including a long sentence, there is a common problem that the summarization accuracy is deteriorated.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and has as its object to provide a natural language processing apparatus and method capable of creating a summary with high accuracy even for a document including a long sentence.

[0009]

According to a first aspect of the present invention, there is provided a natural language processing apparatus for automatically generating a summary document according to the contents from an input document. Determining a division point by obtaining an information element sequence from the input document and comparing it with a predetermined division pattern,
Dividing means for dividing the input document into a plurality of short sentences; generating a summary sentence employing a predetermined number of short sentences based on a predetermined parameter from the plurality of divided short sentences; and other short sentences continuous with the adopted short sentences. And a summary generating unit that outputs the summary document as a sentence in a completed form by adjusting words in short sentences before and after the division point according to whether or not is adopted.

[0010]

[0011]

In the apparatus according to the second aspect of the present invention, the summary generating means changes, adds, or deletes a word at a connection point of the summary sentence by using a word change rule, whereby the word of the word is changed. After the adjustment, the summary document is output.

In the apparatus according to the third aspect of the present invention, both of the input document and the summary sentence are used to adjust the word at the connection point, and are included in the divided portion of the input document. At least one of the cohesiveness of the sentence, tense, and legal information is included in the summary document.

According to a fourth aspect of the present invention, there is provided a natural language processing method for automatically generating a summary document according to the content from an input document. Determining a division point by obtaining a sequence and comparing it with a predetermined division pattern; dividing the input document into a plurality of short sentences; and a predetermined number of divisions based on a predetermined parameter from the plurality of divided short sentences. A summary sentence that employs a short sentence is generated, and the word is adjusted in the short sentences before and after the division point in accordance with whether or not another short sentence that is continuous with the adopted short sentence is adopted, so that the summary document is sentenced. And a summary generation step of outputting in a completed format.

[0015]

[0016]

In the method according to the present invention, in the summary generation step, a phrase at a connection point of the summary sentence is changed or added or deleted by using a phrase change rule, thereby forming the phrase. After the adjustment, the summary document is output.

Further, in the method according to the present invention, the phrase at the connection point is adjusted using both the input document and the summary sentence, and the sentence included in the divided portion of the input document is adjusted. At least one of the cohesiveness, tense, and legal information is included in the summary document.

[0019]

Embodiments of the present invention will be described below in detail with reference to the drawings.

(First Embodiment) FIG. 1 is a flow chart showing a basic processing procedure of a first embodiment of a natural language processing apparatus to which the present invention is applied. The process of creating a summary document Sab from Sin is shown.

The device of the present invention comprises an interface,
It can be configured by a computer device having a general configuration including an arithmetic device, a storage device, and the like. This storage device is provided with a divided pattern file, which will be described later, along with a word dictionary and the like, and the programs shown in FIGS. The device of the present invention that operates based on this program is functionally
Document input unit 10, automatic short sentence division unit 12, and automatic summarization unit 14
, And a connection point (division point) adjustment unit 16 and a summary document output unit 18. The document input unit 10 inputs and stores an input document Sin for which a summary is to be made, since the XX country has rejected the special inspection, the IAEA has been decided. ,
It is composed of a voice input device and a RAM.

Before summarizing, the automatic short sentence segmentation unit 12
A document Ssh obtained by automatically dividing an input document Sin including a long sentence into a plurality of short sentences is output. Automatic summarization unit 14
Is a summary sentence Ssa that summarizes the short sentence-divided document Ssh.
b is output. The connection point (division point) adjustment unit 16 adjusts the connection point of the abstract sentence Ssab, which is the division point of the input document Sin. The connection point (division point) adjustment unit 16 includes, in addition to the summary sentence Ssab output from the automatic summarization unit 14, the original sentence before summarization (input document Sin) output from the document input unit 10.
Is also input, and the connection point of the summary sentence Ssab is adjusted using the information of both. Automatic short sentence division unit 12 and automatic summarization unit 14
And the connection point (division point) adjustment unit 16 controls the CPU and the like.
This can be realized by an arithmetic unit reading out a program stored in the program ROM and performing control and arithmetic using the RAM as a work area.

In FIG. 1, the automatic short sentence division unit 12 converts the input document Sin into three short sentences S1 "special inspection is performed.", S2 "IAEA is obtained...
The document Ssh divided into S3 "violation of the nuclear inspection agreement ... it has been decided" is output. The automatic summarizing unit 14 generates a short sentence S1,
The input document Sin made up of S2 and S3 is summarized into two short sentences S1 "I am doing a special inspection ..." and S2 "IAEA."
Is gaining ... I can't do that. , And the short sentence S
The summary sentence Ssab composed of S1 and S2 is automatically output.

The connection point (division point) adjustment unit 16
By deleting the period (.) From the connection point of the abstract sentence Ssab composed of 1, 1 and S2 and inserting the word “or more,”
After the adjustment, the summary sentence Sab is output. The summary document output unit 18 outputs the summary document Sa whose connection points (division points) have been adjusted.
b, for example, a display device, a printer,
It is composed of an audio output device and the like.

FIG. 2 is a flowchart showing a detailed processing procedure by the automatic short sentence division unit 12.

In this example, morphological analysis and syntax analysis are used for the short sentence division processing. The input document Sin “IAEA... Has been decided because the XX country has rejected the special inspection” is subjected to morphological analysis in step S20, word recognition is performed, and the expression form, standard Information on the form, part of speech, and semantic classification number is added. Morphological analysis is a process in which a Japanese document is divided into words and the attributes of each word are assigned, and the technical methods are well known.
The detailed description is omitted in Document 1 ("New Language Information Processing Handbook, 14th Edition," Natural Language Processing "": Information Processing Society of Japan, Ohmsha, 1995).

The morphological analysis will be described briefly.
Its main purpose is to recognize the breaks of morphemes that are considered to have the smallest meaning unit among the elements that make up a solid written Japanese sentence, and to recognize the part of speech of morphemes. Although there are several methods for morphological analysis, an example using a school grammar method is shown here. For example, morpheme breaks are recognized as shown in S4 in the figure. For example, [adv] as a part of speech for "special", [noun] as a part of speech for "inspection", and a case particle as a part of speech for "wo" [C
ase], and “verb” is given as a part of speech to “reject”.

Next, in step S22, the syntax within the clause is analyzed to determine the internal structure of the clause. The intra-clause parsing is a process of determining a bunsetsu boundary for a morpheme sequence and assigning attributes of the bunsetsu, and its technical method is well known. Automatic Japanese Sentence Short Text Segmentation and Subject Completion for Strategy ": K. Kanabuchi, H. Ehara: Transactions of Information Processing Society of Japan, Vol. 35, No. 6,
pp. 1018-1028, June, 1994), which is described in detail as the extraction of the information element sequence, and a detailed description thereof will be omitted.

Briefly describing the syntax analysis within a clause, four types of information element strings (surface element strings, standard element strings, symbol element strings, and short element strings) from morphological information obtained by morphological analysis, that is, attribute information A sequence corresponding to the input document Sin to which the input document Sin has been added and the information sequence corresponding to the input document Sin and the grammar in the phrase are recognized to identify a phrase boundary, thereby obtaining a phrase boundary as shown in S5 in FIG. b1, b2, b
3, b4 and the like are determined to obtain the combined information of the information element sequence.

Next, in step S24, a simple inter-segment parsing is performed to combine a plurality of clauses and determine the range of the clause. The phrase connection (simplified inter-phrase parsing) is a process of determining whether or not a phrase and a phrase should be combined or separated by using connection information obtained by morphological analysis and intra-phrase parsing. Since it is a well-known technical method as described in detail above, a detailed description thereof will be omitted here.

Briefly describing the simple inter-clause parsing, a group is formed by moving forward from the beginning of the sentence to the end of the sentence on the symbol obtained by the intra-clause parsing, and ends with a continuous phrase in this group. A plurality of clauses up to the group are combined to determine the range of the clause, and a division point candidate is determined for each of the consecutive clauses. In FIG. 2, as in S6, "since the XX country refuses the special inspection,""IA"
The data obtained by the EA cannot be used to further verify the purpose of the use of nuclear material, "and" it has adopted a resolution violating the Nuclear Inspection Agreement. " Shown as a candidate. “/” Indicates the position of the division point.

Next, in step S26, the information element sequence of the input document Sin is subjected to division point determination processing. This division point determination processing is a process of comparing the information element sequence of the input document Sin with a plurality of division patterns set and stored in the division pattern file Fdv in advance from the beginning of the sentence to the end of the sentence, and obtaining a point to be divided. It is. The division pattern expresses an information element sequence pattern of a point to be divided or a point not to be divided in the same format as the information element sequence of the input document Sin.

For example, in the case where the pattern “verb conjunctive form + conjunctive particle + reading point” is to be divided in the divided pattern file Fdv, and a similar pattern exists in a part of the information element sequence of the input document Sin. Determines the division point after the "reading point". Also, if the pattern “matching + te + reading point” should not be divided in the divided pattern file Fdv, and there is a similar pattern in a part of the information element sequence of the input document Sin, the “reading point”
Is not determined after. The technical method for determining the division point is well known, and is described in detail in Document 2 as pattern matching, and therefore detailed description is omitted here.

In the present embodiment, an example has been described in which morphological analysis and syntactic analysis are used for automatic short sentence segmentation. In addition to these analyses, character analysis, semantic analysis, context analysis, or any one of The automatic short sentence segmentation process may be performed using any analysis of. These analysis methods are well-known, and are described in References 1 and 3 (“Characteristic analysis method and its evaluation in Japanese text revision support tool“ Revision ”): Akira Suganuma et al .:
IPSJ Natural Language Processing Workshop Material, 68-8, 19
88), so a detailed description is omitted here.

Subsequently, the input document Sin is stored in step S26.
Is divided in step S28 in accordance with the result of the division point determination processing of step S28, and is output as a document Ssh divided into short sentences S1, S2, and S3.

When outputting the document Ssh, step S
A process of closing the sentence end sentence of each clause obtained in step 24 is performed, and a process of temporarily adjusting the form as a sentence is performed.
If the conjunctive conjunctive particle is included in the division point, a process such as inserting a conjunctive conjunctive into the sentence after the division point is performed, and the connection information is included in the document Ssh as much as possible. As described above, the process performed when the document Ssh is output is only the process regarding the cohesiveness of the sentence, and as a result, the document XX rejects the special inspection shown in the illustrated document Ssh.
The data obtained by the IAEA cannot be used to further verify the intended use of nuclear material. It has adopted a resolution violating the nuclear inspection agreement. Is output.

FIG. 3 is a flowchart showing a detailed processing procedure by the automatic summarizing section 14.

The automatic summarizing section 14 may use any existing summarizing method. FIG. 3 shows an example in which the method (C) is used among the conventional methods.

First, in step S32, keywords are extracted as parameters from the short sentence-divided document Ssh, and in step S34, a predetermined number of short sentences are selected from the short sentences in which the number of keywords included in the document Ssh is the largest. The summary sentence Ssab is output in step S36. Here, the keyword may be a keyword in a preset keyword list,
Alternatively, it may be dynamically determined each time using the frequency of use of words in the document Ssh. Here, the former method extracts "XX country" and "IAEA" as keywords, selects short sentences S1 and S2 suitable as a summary, and selects other short sentences S3 that do not include the keyword "Resolution of violation of nuclear inspection agreement" Is not adopted in the summary sentence Ssab.

When a long sentence is automatically divided into short sentences as described above, since the division point is in the middle of the input document Sin, the divided expression does not satisfy the sentence form. Therefore, it is necessary to adjust the words at the division points when creating the summary.

FIG. 4 is a flow chart showing a detailed processing procedure by the connection point (division point) adjusting section 16, in which the word at the division point is adjusted.

For example, when a short sentence consisting of a part before a division point is adopted as a summary sentence, the adjustment of the phrase is performed by deleting the connecting particle attached to the division point from the short sentence, and If a short sentence consisting of the part after the dividing point is adopted as a summary sentence, or if a short sentence consisting of the part after the division point is adopted as a summary sentence, add a connective equivalent to the meaning of the deleted connective particle to the short sentence. In this process, the meaning of the connecting particle in the original sentence is included in the divided short sentence.

More specifically, referring to FIG. 4, first, in step S40, the input document Sin is input,
In step S41, the summary sentence Ssab output from the automatic summarizing unit 14 is input. In the processing of the connection point (division point) adjustment unit 16, in the following step S42, the two sentences before and after the connection point to be adjusted are input document Si.
It is determined whether n is continuous. In other words, a plurality of divided short sentences S1, S2 adopted in the summary sentence Ssab
Determines whether two short sentences that were continuous in the input document Sin have been adopted as the summary sentence Ssab, and the subsequent processing system branches according to the determination result.

When the subsequent short sentence S2 is adopted as in the short sentence S1, the adjustment using the divided portion in step S44 is performed. That is, among the plurality of short sentences S1 and S2 adopted in the summary sentence Ssab, the adjacent short sentences S1 and S2 that have been continuous in the input document Sin are set to the dividing points of the short sentences S1 and S2 adopted as the summary sentence Ssab. On the other hand, an adjustment is made so as to return to the connected form of the input document Sin without changing the phrase “does more than or equal to” at the connection point (division point in terms of the input sentence Sin) of the input document Sin.

On the other hand, if the subsequent short sentence S3 is not adopted as in the short sentence S2, the word / phrase is adjusted using the end of the sentence in step S46. In other words, the short sentence S2 is subjected to a process of adding information on the tense or the law at the end of the sentence Sin of the input document Sin, and the result is output as a final summary document Sab. Such a process can be performed by using the information of the input document Sin to adjust the word at the division point. By doing so, it is possible to accurately maintain information on the cohesiveness, tense and law of the sentence of the input document Sin in the summary document Sab. Here, the cohesiveness of a sentence is the meaning of a phrase that expresses the relation between sentences, such as connecting particles, and the tense is a concept that is included in a sentence, such as the past, present, or future. Is the author's subjective meaning in a sentence, such as a desire or question.

When steps S44 and S46 are completed, the flow advances to step S48 to output a summary document Sab with connection points adjusted.

As described above, the connection point (division point) adjustment unit 16
Then, the phrase at the connection point of the abstract sentence Ssab is changed, added, or deleted by using the above-described phrase change rule in consideration of the cohesiveness of the sentence, tense and legal information, and the completed form as a sentence The summary document Sab is output as a combination of the short sentences.

As described above, according to the present embodiment, before summarizing the original sentence (input document Sin), the input document Sin is subjected to the automatic short sentence division processing shown in FIG. 2 to combine a long sentence summarization with a plurality of short sentences. By substituting summaries of things, the accuracy of summarization can be improved. Furthermore, by adjusting the beginning and end expressions of a short sentence, it is possible to maintain the sentence form. Furthermore, by using the information on the segmentation point of the original sentence before the division and adjusting the words at the beginning and end of the sentence of the short sentence segment, information on the cohesiveness of the sentence, the tense and the law of the original sentence is summarized Can be included. Therefore, it is possible to realize the information on the cohesion, tense, and law of the input document Sin in the summary document Sab with high accuracy.

[0049]

As described above, according to the first or fourth aspect of the present invention, the input document is divided into a plurality of short sentences by determining an information element sequence from the input document and collating it with a predetermined division pattern to determine a division point. When generating a summary document based on the plurality of divided short sentences and then, from the plurality of divided short sentences, generates a summary sentence employing a predetermined number of short sentences based on predetermined parameters, and the adopted short sentence By adjusting the words to the short sentences before and after the division point according to whether or not another continuous short sentence is adopted, the summary document is output as a completed sentence, so the sentence form is maintained. A suitable summary document can be obtained, and the accuracy of the summary can be improved.

[0050]

[0051]

According to the second or fifth aspect of the present invention, the word is adjusted by changing or adding or deleting the word at the connection point of the summary using the word change rule, and the summary document is output. In addition, it is possible to output a summary document in the form of a sentence.

According to the third or sixth aspect of the present invention, the phrase at the connection point is adjusted by using both the input document and the abstract sentence, so that the coherence, tense,
At least one piece of information of law can be included in the summary document.

[Brief description of the drawings]

FIG. 1 is a flowchart showing a basic processing procedure by a natural language processing apparatus to which the present invention is applied.

FIG. 2 is a flowchart showing a detailed processing procedure by an automatic short sentence division unit 12;

FIG. 3 is a flowchart showing a detailed processing procedure by an automatic summarizing unit 14;

FIG. 4 is a flowchart showing a detailed processing procedure by a connection point (division point) adjustment unit 16;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Document input part 12 Automatic short sentence division part 14 Automatic summarization part 16 Connection point (division point) adjustment part 18 Abstract sentence output part Fdv division pattern file

Continuation of the front page (56) References JP-A-7-192000 (JP, A) JP-A-61-117658 (JP, A) JP-A-2-44462 (JP, A) JP-A-3-138756 (JP, A) , A) JP-A-2-81175 (JP, A) Kanabuchi, M. and Ehara, T., "Japanese single sentence segmentation and subject complementation for Japanese-to-English machine translation," IPSJ Transactions, vol. 35, no. 1018-1028 (published June 15, 1994) Takeshi Quartz, Yoshihiko Hayashi, "Division of Japanese compound sentences based on connection structure analysis", Transactions of Information Processing Society of Japan, vol. 33, no. 5, pp. 652-662 (issued on May 15, 1992) Hiromi Abe, Toshiyuki Okunishi et al., "A Method of Sentence Segmentation Focusing on Conjunctive Particles," The 42nd Information Processing Society of Japan (Early 1991) ) National Convention Lecture Papers, pp. 3-13 to 3-14 (published February 25, Heisei 19) Edited by Makoto Nagao et al., Dictionary of Iwanami Information Science, pp. 54-55 (Issue 1st May 25, 1990) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06F 17/27 G06F 17/30

Claims (6)

(57) [Claims] 1. A natural language processing apparatus for automatically generating a summary document corresponding to the content from an input document, wherein a division point is determined by obtaining an information element sequence from the input document and collating it with a predetermined division pattern. A dividing means for dividing the input document into a plurality of short sentences; generating a summary sentence employing a predetermined number of short sentences based on a predetermined parameter from the divided short sentences; Summary generating means for outputting the summary document as a completed sentence by performing word / phrase adjustment on the short sentences before and after the division point according to whether or not the short sentence is adopted. Natural language processor. 2. The method according to claim 1, wherein the summarizing means adjusts the word by changing or adding or deleting a word at a connection point of the summarizing sentence using a word changing rule, and outputs the summarized document. The natural language processing device according to claim 1, wherein: 3. The method according to claim 1, wherein both of the input document and the summary sentence are used to adjust the phrase at the connection point, and at least information on the cohesiveness, tense, and law of the sentence included in the divided portion of the input document 3. The natural language processing apparatus according to claim 2, wherein any one of the information is included in the summary document. 4. A natural language processing method for automatically generating a summary document according to the contents from an input document, wherein a division point is determined by obtaining an information element sequence from the input document and collating with a predetermined division pattern. A dividing step of dividing the input document into a plurality of short sentences; generating a summary sentence employing a predetermined number of short sentences based on a predetermined parameter from the divided short sentences; A summary generating step of outputting the summary document as a completed sentence by performing word / phrase adjustment on the short sentences before and after the division point according to whether or not the short sentence is adopted. Natural language processing method. 5. The method according to claim 1, wherein in the summarizing step, the word is adjusted by changing or adding or deleting a word at a connection point of the summarizing sentence using a word change rule, and the summarizing document is output. Claim 4
Natural language processing method described in 1. 6. A method of adjusting a word at the connection point using both the input document and the summary sentence, and at least one of information on cohesiveness, tense, and law of a sentence included in a divided portion of the input document. 6. The natural language processing method according to claim 5, wherein at least one piece of information is included in the summary document.