JPH07219951A - Specific range extracting device and sentence extracting device - Google Patents

Abstract

PURPOSE:To highly accurately determi.ne a specific range consisting of quotation parentheses etc., by referring to a specific range stored in a specific range storing part and recurringly setting up a specific range. CONSTITUTION:A specific range setting part 2 repeatedly sets up a range satisfying a specific condition in an input text inputted from an input part 1 based upon definition stored in a specific range definition storing part 7 until an area satisfying the specific range disappears. A set range storing part 5 registers a set range in each setting, and at the time of judging whether the condition is satisfied or not in succeeding processing, the set area is referred to and judged. When the area satisfying the condition disappears, the setting part 2 outputs areas set up in the past to an output part 3. A sentence delimination setting part 6 sets up sentence deliminations in the text based upon the specific range extracted from a specific range extracting part and information stored in a sentence end information storing part 8.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a special range extraction device for setting and extracting a special range consisting of citations, parentheses, dialogues, etc. in document processing and natural language processing, and electronic digitized text data containing a plurality of sentences. The present invention relates to a sentence extraction device that specifies a sentence break.

[0002]

2. Description of the Related Art Conventionally, citation of natural language text data, determination of special areas such as parentheses, dialogue, etc. has been performed by observing from the beginning of the text and switching between quotation and non-quotation each time a quotation mark appears. there were.

Further, in the case where the text is divided into sentences and the natural language processing is performed, a method of unconditionally separating places where terminal symbols such as periods and question marks appear is used. It was As a related technique, a language compiler or the like has a method of determining a correspondence between parentheses and quotation marks by applying a rule described in a format such as BNF notation to text data.

[0004]

When the method of switching the inside and outside of the quotation range by the appearance of the quotation mark is used, the possibility that the quotation mark itself is enclosed in another quotation mark or parentheses is ignored. The wrong range was sometimes extracted. Therefore, it was not possible to accurately extract many types of quotations and parentheses that appear in natural language. In addition, when extracting a quoted part surrounded by quotation marks, if a mark is erroneously recognized as a start symbol or an end symbol, it is often erroneous to judge other quoted parts due to the error.

On the other hand, the method of determining correspondences such as parentheses according to grammatical rules, which is used in a computer language compiler or the like, can be applied to arbitrary natural language texts in order to limit the input format. There wasn't.

Further, in the sentence extraction process, if quoting is not taken into consideration, there is a problem in that when a sentence including a sentence surrounded by quotation marks appears, it is divided into a plurality of sentences. Especially when there are multiple sentences in the quotation area, the position of the delimiter between these sentences is recognized as a delimiter because the surrounding situation does not change from the ordinary sentence delimiter, and the entire sentence including the citation is recognized. Was not recognized as one sentence.

In order to avoid this, even if a method of recognizing the range of quotation or parenthesis by the conventional method is adopted, the range of the quotation is often wrong due to the problem in the above-mentioned quotation range extraction. I couldn't cut out such a sentence.

FIG. 9 shows an example of sentence segmentation by a conventional method for a gathered sentence. here,""."
Or "?" Or "!" Occurs, and sets the sentence delimiter only if the next character with a space is an uppercase letter or quotation mark. " The sentence cut out as a result is shown in FIG.

Of these, sentence 1, sentence 2, sentence 3, sentence 5, sentence 7, and sentence 8 are correctly cut out, but sentence 4, sentence 6,
Sentence 11 is cut out in an unnatural form as a sentence with only one of the quotation marks. Moreover, the sentence 9 is divided in the middle of the quotation. An unnatural "sentence" such as sentence 4, sentence 6, sentence 9, and sentence 11 becomes a serious impediment factor when attempting to perform syntax analysis and computer processing such as information extraction.

[0010]

A special range extraction device according to the present invention for solving the above-mentioned problems, regardless of the type when quoting or bracketing, determines the range of quotes or brackets. Only those that do not exist are set as the quotation / parenthesis range, and thereafter the quotation / parenthesis range is determined with high accuracy by determining the outer quotation / parenthesis so as not to conflict with the already set quotation / parenthesis range. can do. This can be realized by storing the range each time the range is set and configuring it so that it can be accessed in the subsequent processing.

Further, the sentence extraction device uses the range of quotation marks and parentheses extracted by the above means, and prohibits the middle of this range from being used as a sentence delimiter, so that the quotation marks such as dialogues can be obtained. Extracts a sentence that contains another sentence that is sandwiched with high accuracy without being cut off.

[0012]

The special range setting section repeatedly sets a range satisfying a specific condition in the input text until no region satisfying the condition exists. Then, it is possible to write in the condition whether such a range has already been set at a certain position in the text. The special range setting section registers the setting range in the setting range storage section every time the range is set, and refers to the set area when determining whether or not the condition is satisfied in the subsequent processing and makes a determination. The special range setting unit outputs all the areas set up to that point when the area satisfying the conditions no longer exists.

Further, in the sentence extracting device, the output information of the special range setting section is sent to the sentence division setting section. In the sentence delimiter setting part, the position in the middle of the range set by the bracket delimiter range setting part is excluded, only the terminating character string of the remaining range is made a candidate for sentence demarcation, and other conditions are checked and the sentence delimiter is set. To do.

[0014]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an embodiment of the present invention. In this embodiment, the target language is English. In FIG. 1, 1 is an input unit for inputting text, 7 is a special range definition storage unit in which a special range is recursively defined by a condition using a specific symbol, 2 is a quotation or parentheses in the input text, The special range setting unit that estimates the special range based on the definition of 7, the output unit 3 that outputs the range and the sentence such as the extracted quotation or parenthesis, and 4 is "it '
An SQ exclusion list that remembers patterns in which single quotes are used for purposes other than enclosing quotes, such as "s".
Reference numeral 5 denotes a set range storage unit that stores the set quotation and bracket range.

Reference numeral 8 denotes a sentence end information storage unit that stores character or character string information that may be the end of a sentence, and reference numeral 6 denotes a sentence delimiter based on the set quotation or bracket range and the sentence end information stored in the sentence end information storage unit 8. The operation using this will be described later, and in this embodiment, the output of the special range estimation unit is directly output and used as a special range extraction device.

The input is general English text. The special range setting unit 2 estimates and sets the range of parentheses and citations for the input English text using an algorithm described later. Hereinafter, the range set in this way will be referred to as a special range. The final result of the set range is represented by a set of pairs of numerical values representing the position from the beginning of the file, and is sent to the output unit 3 together with the input text. The output unit 3 displays the quoted portion cut out based on the obtained information.

For convenience of the following description, some character sets are defined. This is shown in (Table 1). Further, the special range determination conditions stored in the special range definition storage unit 7 are shown in (Table 2).

[0018]

[Table 1]

[0019]

[Table 2]

In (Table 2), it is "blank" that any of space character, tab character, and line feed character is
If not, it is called "not blank". FIG. 2 shows a flowchart showing the operation of the special range setting section. This is an algorithm for setting only the special range setting that satisfies the condition of (Table 2).

The operation of the parenthesis / quotation range estimation unit will be described with reference to FIG. Basically, the operation is defined in the special range definition storage unit 7 with respect to the inputted electronic text file.
The process of searching for a range that matches the six determination conditions stored as definition 6 and registering that range as a special range is repeated until there is no range that satisfies the conditions.

The array variable stat [] is a variable indicating whether each position in the file is within the special range. If position i is within the special range then stat [i] = 1, else stat [i] = 0. Note that stat [] is a storage area corresponding to the setting range storage unit 5 in FIG. First, all 0s are assigned to the elements of stat []. The two-dimensional array variable ans [] [] is for registering a special range.
The start position of each X-th special range
It will be registered in ans [X] [1] and ans [X] [2] at the end position. If not registered, it is defined as -1. First, assign -1 to all elements of ans.
In addition, 0 is assigned to the pointer ans_max for ans [] [].

The following "initial setting" subroutine is
The variables used in the subroutine for setting various special ranges are initialized, and the processing related to """will be described later as an example.

The variable change is a flag indicating whether or not the setting is made in the process of the loop in FIG. It is initialized to 0 (no setting has been made) at the beginning of processing. The order does not matter, but here, starting from the processing of the part sandwiched by """, various quotations and parentheses are set in order.

Specific operations in these subroutines will be described later, but these are subroutines for extracting a range that satisfies each condition of Table 2 at that time. As a result, it may or may not be set. In each subroutine, the variable chan is set only when the setting is made.
Set 1 to ge.

After all types of special ranges have been set, if change is 0, the special range cannot be set any more, and the process ends. change is 1
If so, it means that the setting has been performed at least once, so the change is reset to 0 and the processing is started again. The above is repeated until the state where change remains 0 even after the loop ends.

Since the judgment conditions 1 to 6 are described recursively, there is a possibility that a region that has not satisfied the condition will satisfy the condition as the setting is performed.

As an example, "" Did you say, "What a pit
Consider condition 1 (definition 1) for input of text that includes the part "y"? ", the man said. Since no special range has been set when the text is entered, the part "" Did you say, "Whata pity"? "" Does not meet condition 1 [4] of definition 1, so condition 1 Not even satisfy the whole. However, since the part "" What a pity "" satisfies the condition 1, the range is set as the special range. As a result, after setting this, "" Did you sa
y, "What a pity"? "", """between both ends of""are all included in" Other special range "of" What a pity ", condition 1 [4] of is satisfied this time, and "" Did you say, "What a pity"? "" Is set as a special range.

Next, the operation of each subroutine in FIG. 2 will be described. FIG. 3 is a flow chart of "initial setting" for the processing of "setting of special range sandwiched between""", and FIG. 4 is a flow chart of a subroutine body of "setting of special range sandwiched between"". It is a flowchart. The operation will be described using these as examples. Other subroutines and initialization can be realized in almost the same manner.

In FIG. 3, as an initial setting, all the positions where ““ ”appear in the text are represented by the number of characters from the beginning position of the text, and are sequentially substituted into the array dq [] from the beginning. Substitute the number of """that appears for the variable dq_max.

Next, the operation of the "set special range sandwiched by""subroutine will be described with reference to FIG. 4-1,4-
In the 2,4-3,4-4 part, we search for two pairs of """such that there is no""" symbol in a position that is not within the special range. If such pairs exist, their positions are assigned to dq [i] and dq [j] at the time when processing 4-5 starts.

In processing 4-5, it is checked whether or not the pair of """satisfies definition 1 in Table 1 (<= definition 1?). This can be easily accomplished by examining the characters before and after it in the input text.

In the case of "", it does not occur, but when setting a special range surrounded by "'" and "` ", here,
It is necessary to perform a process of determining whether or not the situation around the text matches the pattern registered in the SQ exclusion list 4 of FIG.

In the SQ exclusion list, a pattern is described by a regular expression or the like, and in order to match it with a surrounding character string, a conventional technique relating to character string pattern matching may be used.

As a result of processing 4-5, if the condition is satisfied, the dq [i]
And dq [j] as special ranges in ans, and dq of stat []
Set all the elements from the [i] th element to the dq [j] th element to 1 ("is within the special range"). After that, it searches for other pairs of """and continues the process, so that there is no""" symbol "that is not in the special range between them.
And the subroutine is finished. Each subroutine is realized as described above.

Looking at the processing of FIG. 2 as a whole, at first, only the range without the same kind of quotation / bracket symbols is registered as a special range, and when the quotation and the parentheses are duplicated, the outside The quotation / bracket will be set as a special range after the inner one is set as a special range. In this way, even when many kinds of citations and parentheses appear at the same time and they are multiplexed, the outermost range can be set without fail.

The data of the setting range determined as described above is sent to the output unit in the form of the position in the file. The output section displays each quoted part set from this numerical value.

Next, the above-mentioned "" Did you say, "What a pi
The whole operation is explained using an example of "ty"? ", the man said.".

The only special range that can be set when no special range is set is the "" What apity "" range. "" Did you say, "" is the definition 1 in (Table 2) [6]
In addition, ""? "Is defined in [Table 2] definition 1 [5] and""Did you sa
y, "What a pity" or "What a pity"? "or" Did y
ou say, "What a pity"? "" is defined in [Table 2] definition 1 [4],
Because they conflict with each other. (Note that the last three do not conflict with the definition, but are not selected as a pair of dq [i] and dq [j] in the first place when using the algorithm of Fig. 4) However, it is called "What a pity" After the range is set as a special range, "Did you say," What a pi
ty "?""no longer conflicts with Definition 1 [4] in (Table 2).

Therefore, "" Did you say, "What a pity"
? "" Is set as a special range. In addition, "" Did you s
“Ay,“ What a pity ”” and ““ What a pity ”?” ”now violate the definition 1 [2] in (Table 2), so they are not cut out. (The "pity" part is already at the end position of the other special range, and the "" What "part is already at the start position of the other special range. However, if the algorithm of FIG. 4 is used,
Not selected as a pair of dq [i], dq [j]).

As described above, "" Did you say, "",
Wrong citations such as ""? "And" What a pity "?" Are not cut out, but "" What a pity "" and "Did you say," What a pity "?"" Processing is realized so that the two ranges are cut out as intended by the writer.

In addition, "" Did you say, 'What a pity'? ",
Expressions that combine two or more types of quotations and parentheses, such as "the 'Visitor'said.", are determined from the internal ones, so it can be set without any problems.

Next, the operation of the sentence extracting device will be described. In FIG. 1, 6 is a sentence break setting unit that sets sentence breaks. 7, it is assumed that <Compound sentence end character string> and <Sentence end character string> shown in (Table 1) are stored as sentence end expressions. In this embodiment, the information on the special range estimated by the special range setting unit is sent to the sentence break setting unit 6.

FIG. 5 shows a flowchart for explaining the operation of the sentence break setting unit. In FIG. 5, "position" is a numerical value indicating the number of characters from the beginning of the input file, and the initial value is 1 (the beginning of the file). First, check whether that position is the end of the "end-of-sentence character string" in Table 1 (5-2 in the figure). If not, proceed to the next position without a break. If so, it is further determined whether or not the position is within the special range. If it is not within the special range, the process proceeds to processing 5-5.

If it is a special range, it advances to the next position without a break, but in English, when a quotation or parenthesis is at the end of a sentence, a period is written inside the quotation range like "....". Therefore, in that case, the process proceeds exceptionally to process 5-5 (5-4 in the figure). In 5-5, check whether the first alphabetic character after that position is uppercase or lowercase. If it is a lowercase letter, it is considered to be an abbreviated expression such as "US" that appears in the sentence, so proceed to the processing at the next position without separating it. If it is an uppercase letter, that point (end of sentence end character string) is set as a delimiter between sentences, and the process proceeds to the next position.

As described above, even a sentence including a dialogue composed of a plurality of sentences can be extracted as a whole without being divided in the middle of the dialogue.

Next, the operation of sentence extraction will be described using a simple example sentence. The above "" Did you say, "What a pity"
"It was stran," including the example "?", the man said.
ge. "Did you say," What a pity "?", the man said. Sh
Suppose there is a part "e couldn't answer." in the text. In general sentence segmentation processing, "?" In "pity"? "Is a symbol that represents the end of the sentence, so sentences are separated here, and" Did you say, "What a pity"? "",
"Sentences" such as "the man said." are extracted.
On the other hand, if this embodiment is used, "" Did you say, "W
Since the part "hat a pity"? "" is set as a special range, even if "?" is a symbol indicating the end of a sentence, it is not separated. (It is excluded because it is not "only" in the part 5-4 in Fig. 5.) Therefore, "It was strange.""" Did yo
u say, "What a pity"? ", the man said.""Sh
The three sentences "e couldn't answer." can be correctly extracted.

Moreover, "He said," It is very easy. Com
Even when a quotation or dialogue is composed of multiple sentences, such as "e on!"", the entire sentence including the quotation can be extracted without being divided because the quoted portion is a special range. .

Further, if quotation marks and parentheses are set by the above method, the range is not set when there is only one of the start symbol and the end symbol due to the writer's error, etc. You can prevent the extraction range of sentences from being greatly disturbed by symbols and exceptional symbols,
There is also the effect. For example, if there is a wrongly entered """symbol but the corresponding""" symbol does not appear, the sentence is cut out normally thereafter. "T
his is a "pen. ... (long text consisting of many sentences) .... He said," I want to go there. "" appears and the """symbol in front of the pen is incorrect It was input. In the past, when trying to prevent sentences from being separated in quotes, all parts like "" pen .... (long text consisting of many sentences) .... He said, "" were spliced together. There was a problem of being lost.

On the other hand, when the present invention is used, the """symbol corresponding to the""in front of the pen does not appear, so the special range starting from here is not set. "" Pen .... (long text consisting of many sentences) .... He said, "" violates definition 1 [6] of (Table 2). Special range "I wan
Only "t to go there." will be set.

Therefore, the "(long text consisting of a large number of sentences)" portion is normally subjected to sentence cutting processing to prevent erroneous extraction.

The operation for the last collected sentence will be described. FIG. 9 shows an example of setting a delimiter position of a series of sentences by the conventional method as described above, and FIG. 10 shows a sentence cut out by using the delimiter position. In FIG. 10, only a part of the sentence including the quotation is cut out like sentence 4, sentence 6, sentence 9, and sentence 11. FIG. 6 shows the result of setting the special range for the same example sentence. In FIG. 6, the range of quotation marks is correctly cut out by taking into consideration blanks before and after the quotation. FIG. 7 shows how sentence breaks are set using the results. FIG. 7 is different from FIG. 9 in that a delimiter inside the special range is not set. FIG. 8 shows a sentence cut out by using the delimiter position of FIG. 6 as a delimiter. In FIG. 8, all the sentences are extracted without being cut off during the citation.

[0053]

As described above, the present invention provides means for accurately extracting multiple citations, dialogues, and parentheses even if they are mixed. It also provides a way to accurately separate documents containing these expressions sentence by sentence. Moreover, these processes can be realized at high speed without using a heavy process such as looking up a dictionary to know the meaning of a word. Extraction of quoted parts and sentences can be utilized for document editing and as preprocessing for practical natural language processing by a computer.

[Brief description of drawings]

FIG. 1 is a conceptual diagram showing the configuration of a sentence extraction device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of a special range setting unit of the present invention.

FIG. 3 is a flowchart showing an initialization routine of special range setting according to definition 1 in an embodiment of the present invention.

FIG. 4 is a flowchart showing the operation of special range setting processing according to definition 1 in an embodiment of the present invention.

FIG. 5 is a flowchart showing the operation of a sentence break setting unit according to an embodiment of the present invention.

FIG. 6 is a special range setting result diagram according to an embodiment of the present invention.

FIG. 7 is a sentence extraction processing result diagram according to an embodiment of the present invention.

FIG. 8 is a diagram showing a result of sentence extraction according to an embodiment of the present invention.

[Figure 9] Input digitized text data diagram

FIG. 10 is a processing result diagram of FIG. 9 according to the related art.

[Explanation of symbols]

 1 Input Section 2 Special Range Setting Section 3 Output Section 4 SQ Exclusion List 5 Setting Range Storage Section 6 Sentence Separation Setting Section 7 Special Range Definition Storage Section 8 Sentence Information Storage Section

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Masanori Takahashi 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd.

Claims (2)

[Claims] 1. A special range definition storage unit that recursively defines a special range, and a special range setting unit that sets a special range in computerized text data based on the definition stored in the special range definition storage unit. A special range storage unit that stores the special range set by the special range setting unit, wherein the special range setting unit is stored in the special range storage unit based on the definition stored in the special range definition storage unit. A special range extraction device characterized in that the special range is recursively set with reference to the special range. 2. The special range extraction device according to claim 1, a sentence end information storage unit that stores a character or a character string that can represent the end of a sentence, a special range extracted by the special range extraction device, and the sentence end information. A sentence extraction device comprising a sentence division setting unit for setting sentence divisions in electronic text data based on information stored in a storage unit.