JP6589704B2 - Sentence boundary estimation apparatus, method and program - Google Patents

Description

  The present invention relates to a technique for estimating sentence boundaries in natural language processing.

  When analyzing a natural language in sentence units, first, processing for estimating sentence boundaries is required. Hereinafter, sentence boundaries are also simply referred to as sentence boundaries. The sentence boundary generally exists immediately after the sentence end symbol representing the end of the sentence, but the sentence end symbol may appear at a location other than the end of the sentence. For example, a period character in English may appear at the end of a sentence, but may appear in the middle of a sentence by being included in a word abbreviation. Therefore, in the process of estimating the sentence boundary, how to determine the sentence boundary becomes a problem.

  An example of a technique for estimating sentence boundaries related to such a problem is described in Patent Document 1. In the related technology described in Patent Document 1, words such as abbreviations including periods are registered in a dictionary. Then, in the related technology, when a character string delimited by a space in a sentence includes a period and is registered in the dictionary, it is determined that a sentence boundary does not exist immediately after the period. Further, in this related technology, when a character string delimited by a space is not registered in the dictionary and ends with a period, it is determined that a sentence boundary exists immediately after the period.

  However, the word including the sentence end symbol includes a word that does not appear at the end of the sentence and a word that can appear at the end of the sentence or at the end of the sentence. For example, the word “Mr.” representing the abbreviation of “Mister” always appears in the middle of a sentence and never appears at the end of the sentence. On the other hand, a word representing an abbreviation of “United States”, “U.S.”, may appear in the middle or end of a sentence. Therefore, there are cases where sentence boundaries cannot be estimated correctly only by registering words including sentence end symbols in the dictionary.

  Therefore, a method of applying a related technique described in Patent Document 1 and registering a compound word including a sentence ending symbol in a dictionary can be considered (see paragraph 0006 of Patent Document 2). In this technique, for a word that does not appear at the end of the sentence, such as “Mr.”, the word itself is registered as a type 1 in the dictionary. For words that contain a sentence ending symbol such as “US”, a compound word such as “US President” that contains the word is registered in the dictionary as type 2. The

In this case, it is possible to determine whether or not a sentence boundary exists immediately after the sentence end symbol appearing in the sentence, according to the following determination criteria.
Criteria 1: If a sentence ending symbol appearing in a sentence is included in a range matching a word registered in the dictionary as type 1, it is determined that there is no sentence boundary immediately after that.
-Criterion 2: If the end-of-sentence symbol that appears in the sentence is included in a range that matches a word registered in the dictionary as type 2, and is not the last character in the range, the sentence boundary immediately follows Judged not to exist.
-Standard 3: When the criteria 1 and 2 are not met, general criteria for sentence boundaries are applied.

General judgment criteria for sentence boundaries will be described. According to a general criterion for determining a sentence boundary, for example, when all of the following conditions 1 and 2 are satisfied for a sentence end symbol appearing in a sentence, it is determined that a sentence boundary exists immediately after the sentence end symbol. If at least one of the conditions 1 and 2 is not satisfied, it is determined that there is no sentence boundary immediately after the sentence end symbol.
Condition 1: If a character exists immediately after the end-of-sentence symbol, the character is a blank character.
• Condition 2: If a word exists immediately after the end of the sentence, the word starts with a capital letter.

  A specific example of estimating a sentence boundary using such a method of registering a compound word including a sentence end symbol in a dictionary will be described. Here, it is assumed that “Mr.” is registered as a type 1 word and “U.S. President” and “U.S. Senate” are registered as type 2 words in the dictionary.

  It is assumed that a sentence “A former U.S. President gave a speech at the U.S. Senate yesterday” is input. In this case, each of the two periods included in the first “US” is included in a range that matches the type 2 entry “US President” registered in the dictionary, and is not the last character of the range. . For this reason, it is determined that there is no sentence boundary immediately after these periods according to the above-mentioned criterion 2. Similarly, the two periods included in the second “US” are each included in a range that matches the type 2 entry “US Senate” registered in the dictionary, and are not the last character of the range. . For this reason, it is determined that there is no sentence boundary immediately after these periods according to the above-mentioned criterion 2.

  Further, it is assumed that a sentence “Canada is located next to the U.S. Japan is located next to China” is input. In this case, the period immediately after “U” of “U.S.” is not included in the range matching the entry registered in the dictionary, and does not correspond to the above criteria 1 and 2. Therefore, a general criterion for sentence boundaries is applied to this period. Since this period does not satisfy the above condition 1 “the character immediately after is a blank character”, it is determined that there is no sentence boundary immediately after this. Similarly, a general sentence boundary criterion is applied to a period immediately after “S” in “U.S.”. Since this period satisfies the above condition 1 “the character immediately after is a blank character” and condition 2 “the word Japan immediately after begins with a capital letter”, it is determined that there is a sentence boundary immediately after this.

  Another example of a technique for estimating sentence boundaries is described in Patent Document 2. In the related technique described in Patent Document 2, a sentence is first cut out using a sentence ending symbol not included in a word. And this related technique estimates a part of speech about each word which comprises the sentence when the sentence end symbol exists in the middle of the cut-out sentence. In this related technique, when a main verb appears in front of and behind the end-of-sentence symbol in the extracted sentence, it is estimated that a sentence boundary exists immediately after the end-of-sentence symbol. Further, in this related technique, when a main verb appears only in one of the front and rear, it is estimated that there is no sentence boundary immediately after the sentence end symbol.

Japanese Patent Application Laid-Open No. 11-282841 JP 2003-108550 A

  However, the related technology described above has the following problems.

  The related technique described in Patent Document 1 does not consider a word that can appear at the middle or end of a sentence among words including a sentence end symbol. Therefore, the related technique described in Patent Document 1 determines that there is no sentence boundary immediately after a word including a sentence end symbol at the end appears.

  Moreover, the related technique described in Patent Document 2 requires a process of estimating the part of speech for each word constituting the extracted sentence when a sentence ending symbol appears in the middle of the extracted sentence. As the part of speech estimation processing, for example, it is shown that a probabilistic method using a corpus to which part of speech information is added is used. Such a part-of-speech estimation process has a problem of increasing the load on a device that estimates sentence boundaries.

  Further, the method of registering a compound word including a sentence ending symbol in the dictionary has a problem that it takes time to register as many compound words including a sentence ending symbol as possible in the dictionary.

  This problem will be described using a specific example. For example, there are many compound words including “United States” which is the full spelling of “U.S.”. For example, “United States President”, “United States Senate”, “United States Department of State”, “United States Department of Defense”, and the like. For all of these compound words, it takes time to register compound words in which “United States” is replaced with “U.S.”, which is an abbreviation including a sentence ending symbol, in the dictionary. This problem becomes particularly noticeable when the full spelling of a word including a sentence ending symbol has a high word-making ability.

  In addition, in the method of registering a compound word including a sentence ending symbol in a dictionary, in order to avoid presuming that a sentence boundary exists at a place where a sentence boundary should not exist originally, a compound word including a sentence ending symbol is used. It is necessary to register more words in the dictionary. However, as more compound words are registered in the dictionary, this method increases the possibility of estimating that a sentence boundary does not exist at a place where a sentence boundary should exist.

  For example, assume that a compound word “U.S. Japan” is registered in the dictionary as a compound word expressing “Universal Studios (registered trademark) Japan”. At this time, for example, it is assumed that a sentence “We enjoyed ourselves a lot yesterday at the U.S. Japan.” Is input.

  In this case, this method determines that there is no sentence boundary between “US” and “Japan” because the compound word “US Japan” is registered in the dictionary, and corrects the whole as one sentence correctly. Can be certified.

  However, the character string “U.S. Japan” does not necessarily mean “Universal Studios Japan”. For example, it is assumed that a sentence “Canada is located next to the U.S. Japan is located next to China” is input. In this sentence, “U.S.” means “United States”, and there is a sentence boundary between “U.S.” and “Japan”. However, in this method, since the compound word “U.S. Japan” is registered in the dictionary, it is determined that there is no sentence boundary between “U.S.” and “Japan”.

  As described above, the method of registering a compound word including a sentence ending symbol in the dictionary not only takes time to register the dictionary, but also has a problem that the estimation accuracy of the sentence boundary decreases as the dictionary in which the compound word is registered is strengthened. ing.

  The present invention has been made to solve the above-described problems. That is, an object of the present invention is to provide a technique for estimating a sentence boundary more accurately while suppressing an increase in processing load and an increase in time required for dictionary registration.

  The sentence boundary estimation apparatus of the present invention includes a first dictionary for storing a word string, a replacement source word string that is a word string including a sentence end symbol, and the sentence end symbol to represent the same meaning as the replacement source word string. A second dictionary that stores a replacement destination word string that is a word string that is represented in a correlated manner, and the replacement source word string in the original text based on the second dictionary, the replacement destination word string A text expansion unit that generates a post-replacement text that is replaced with, and a sentence boundary in the original text when the sentence boundary in the original text is estimated based on the post-replacement text and the first dictionary. If the text is within the replacement source word string replaced with the replacement destination word string included in the range matching the word string registered in the first dictionary, the end of the sentence And a sentence boundary estimation unit for estimating a non-sentence boundary immediately after the issue.

  Further, the method of the present invention does not include the first dictionary for storing the word string, the replacement source word string that is a word string including a sentence end symbol, and the sentence end symbol so as to represent the same meaning as the replacement source word string. The replacement source word string in the original text is associated with the replacement destination based on the second dictionary using the second dictionary that is stored in association with the replacement destination word string that is the word string represented in When generating a replacement text replaced with a word string and estimating a sentence boundary in the original text based on the replacement text and the first dictionary, the sentence ending symbol in the original text is If it is inside the replacement source word string replaced with the replacement destination word string included in the range that matches the word string registered in the first dictionary, the sentence is immediately followed by the sentence It is estimated that not the world.

  Further, the program of the present invention does not include the first dictionary for storing the word string, the replacement source word string that is a word string including a sentence end symbol, and the sentence end symbol so as to represent the same meaning as the replacement source word string. The replacement source word string in the original text is associated with the replacement destination based on the second dictionary using the second dictionary that is stored in association with the replacement destination word string that is the word string represented in A text expansion step for generating a post-substitution text replaced with a word string; and when estimating a sentence boundary in the original text based on the post-substitution text and the first dictionary, In the post-replacement text, the replacement source word string replaced by the replacement destination word string included in the range matching the word string registered in the first dictionary is included. If, to execute the sentence boundary detection step of estimating the non-text boundary immediately following the end of the sentence symbols, the computer device.

  The present invention can provide a technique for estimating a sentence boundary more accurately while suppressing an increase in processing load and an increase in time required for dictionary registration.

It is a block diagram which shows the structure of the sentence boundary estimation apparatus as the 1st Embodiment of this invention. It is a figure which shows an example of the hardware constitutions of the sentence boundary estimation apparatus as the 1st Embodiment of this invention. It is a figure which shows an example of the information stored in a 1st dictionary in the 1st Embodiment of this invention. It is a figure which shows an example of the information stored in a 2nd dictionary in the 1st Embodiment of this invention. It is a flowchart explaining operation | movement of the sentence boundary estimation apparatus as the 1st Embodiment of this invention. It is a figure explaining the specific example of the operation | movement by the sentence boundary estimation apparatus as the 1st Embodiment of this invention. It is a block diagram which shows the structure of the sentence boundary estimation apparatus as the 2nd Embodiment of this invention. It is a flowchart explaining operation | movement of the sentence boundary estimation apparatus as the 2nd Embodiment of this invention. It is a figure which shows an example of the information stored in a 1st dictionary in the 2nd Embodiment of this invention. It is a figure which shows an example of the information stored in a 2nd dictionary in the 2nd Embodiment of this invention. It is a figure explaining the specific example of the operation | movement by the sentence boundary estimation apparatus as the 2nd Embodiment of this invention. It is a figure explaining the other specific example of the operation | movement by the sentence boundary estimation apparatus as the 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 shows a functional block configuration of a sentence boundary estimation apparatus 1 as a first embodiment of the present invention. In FIG. 1, the sentence boundary estimation device 1 includes a first dictionary 11, a second dictionary 12, a text development unit 13, and a sentence boundary estimation unit 14.

  Here, the sentence boundary estimation apparatus 1 can be configured by hardware elements as shown in FIG. In FIG. 2, the sentence boundary estimation device 1 includes a CPU (Central Processing Unit) 1001, a memory 1002, an output device 1003, and an input device 1004. The memory 1002 includes a RAM (Random Access Memory), a ROM (Read Only Memory), an auxiliary storage device (such as a hard disk), and the like. The output device 1003 is configured by a device that outputs information, such as a display device or a printer. The input device 1004 is configured by a device that accepts user operations and information input, such as a keyboard, a mouse, an OCR (Optical Character Reader) device, and the like. In this case, each functional block of the sentence boundary estimation device 1 is configured by a CPU 1001 that reads and executes a computer program stored in the memory 1002 and controls each unit of the output device 1003 and the input device 1004. Note that the hardware configuration of the sentence boundary estimation device 1 and each functional block thereof is not limited to the above-described configuration.

  Next, details of each functional block will be described.

  The first dictionary 11 stores word strings. A word string is a meaningful character string consisting of one or more words. A word string made up of a plurality of words is a so-called compound word. The first dictionary 11 stores word strings that are generally used in the target field, such as single words and compound words. In particular, the first dictionary 11 stores a word string itself that does not include a sentence ending symbol so as to represent the same meaning as a word string that includes a sentence ending symbol, or a compound word that further includes such a word string. It is desirable that However, the first dictionary 11 may store not only such a word string but also other word strings. As the first dictionary 11, a general dictionary that includes not only a single word in the target field but also a compound word as a headword can be applied.

  FIG. 3 is an example of information stored in the first dictionary 11 when an English sentence is assumed as a target for estimating a sentence boundary. Here, word strings such as “United States”, “United States President”, “United States Senate”, “Post Office”, “Post Office Box” are registered. These are the word sequences "United States" and "Post Office" that are expressed without the end-of-word symbol to indicate the same meaning as the word sequences "US" and "PO" that include the end-of-word symbol, or such word strings. Is a compound word containing In English, words including a period, which is a sentence ending symbol, are often used, which are called acronyms, abbreviations or abbreviations. Thus, in this way, the first dictionary 11 includes an acronym including a period, a full spelling word string that can be expressed as an abbreviation or abbreviation, or a compound word further including such a full spelling word string. Stored.

  In FIG. 3, an ID is assigned to each entry of the first dictionary 11 for the sake of explanation. Hereinafter, an entry whose ID is X is also referred to as an ID: X entry.

  The second dictionary 12 stores the replacement source word string and the replacement destination word string in association with each other. The replacement source word string is a meaningful character string composed of one or more words in which at least one word includes a sentence end symbol. The replacement-destination word string is a meaningful character string composed of one or more words expressed without including a sentence ending symbol so as to represent the same meaning as the replacement-source word string.

  FIG. 4 is an example of information stored in the second dictionary 12 when an English sentence is assumed as a target for estimating a sentence boundary. Here, a replacement-destination word string “United States”, which is a full spelling expression in which the abbreviation is spelled out, is stored in association with an abbreviated word including a period “U.S.” as a replacement-source word string. Further, a replacement word string “Post Office” which is a full spelling of the abbreviation is stored in association with an abbreviated word including a period “P.O.” as a replacement word string.

  As in this example, the second dictionary 12 only needs to store at least an entry having a word string (for example, “U.S.”) in a minimum unit expressed including a sentence end symbol as a replacement source word string. That is, the second dictionary 12 may not store an entry having a compound word (for example, “U.S. President”) further including such a replacement source word string as a replacement source word string.

  In FIG. 4, an ID is assigned to each entry of the second dictionary 12 for explanation. Hereinafter, an entry whose ID is X is also referred to as an ID: X entry.

  Based on the second dictionary 12, the text expansion unit 13 generates post-substitution text in which the replacement source word string in the original text is replaced with the associated replacement destination word string.

  Note that the text development unit 13 may acquire the text input via the input device 1004 as the original text. Alternatively, the text development unit 13 may acquire the text stored in the memory 1002 as an original text. In this case, the text development unit 13 may acquire information indicating the storage position of the target text in the memory 1002 via the input device 1004. In addition, the text development unit 13 may acquire text input from a device connected via a network as original text. In addition, the text development unit 13 may acquire the text stored in the portable storage medium as the original text.

  The sentence boundary estimation unit 14 estimates a sentence boundary in the original text based on the replaced text and the first dictionary 11. At this time, the sentence boundary estimation unit 14 replaces the replacement source word string with the replacement word string included in the range in which the sentence end symbol of the original text matches the word string registered in the first dictionary 11 in the replacement text. If it is inside the, it is presumed that it is not a sentence boundary immediately after the sentence ending symbol.

Specifically, for example, the sentence boundary estimation unit 14 detects a range that matches a word string registered in the first dictionary 11 in the replaced text. And the sentence boundary estimation part 14 should just determine that the next is not a sentence boundary, when both the following conditions 3 and 4 are satisfy | filled.
Condition 3: The sentence end symbol in the original text is included in the replaced source word string.
Condition 4: The replacement destination word string replaced from the replacement source word string in the post-substitution text is included in a range that matches the word string registered in the first dictionary 11 previously detected.

  Note that the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing when the sentence end symbol in the original text does not satisfy any one of the above conditions 3 and 4. A known technique may be applied to normal sentence boundary estimation processing.

However, the sentence boundary estimation unit 14 is located inside the replacement source word string in which the sentence end symbol of the original text is replaced with the replacement destination word string included in the range matching the word string in the first dictionary 11 in the replacement text. However, in the following case, the determination that the immediately following is not a sentence boundary is suspended. In that case, the following conditions 5 and 6 are both satisfied.
Condition 5: The corresponding sentence ending symbol is the last character of the replacement source word string replaced in the original text.
Condition 6: In the post-substitution text, the last word in the replacement destination word string replaced from the replacement source word string and the last word in the range matching the word string in the first dictionary 11 including the replacement destination word string Matches.

  In this case, the sentence boundary estimation part 14 should just estimate the presence or absence of the sentence boundary immediately after the sentence end symbol by the normal sentence boundary estimation process.

  The operation of the sentence boundary estimation apparatus 1 configured as described above will be described with reference to FIG.

  In FIG. 5, first, the text development unit 13 acquires the original text (step S1).

  Next, the text expansion unit 13 refers to the second dictionary 12 and searches for a character string that matches the replacement source word string in the original text. Then, the text expansion unit 13 generates the replaced text by replacing the character string that matches the searched replacement source word string with the associated replacement destination word string (step S2).

  Here, it is assumed that there are entries of a plurality of applicable replacement source word strings in the second dictionary 12 for a certain character string in the original text. In this case, the text expansion unit 13 may generate a plurality of post-substitution texts by replacing the places with the respective replacement destination word strings. As a case where there are a plurality of applicable replacement source word strings, for example, there are several cases as follows. The first is a case where there are a plurality of entries associated with different replacement destination word strings for the same replacement source word string. For example, there may be an entry in which “United States” is associated with the replacement source word string “U.S.” as a replacement destination word string and an entry in which “Universal Studios” is associated as the replacement destination word string. The second case is a case where there are a plurality of entries relating to different replacement source word strings that are in an inclusive relationship as character strings. For example, there is an entry of a replacement source word string “U.S.” and an entry of a replacement source word string “U.S. Japan”.

  Next, the sentence boundary estimation unit 14 detects a character string that matches the word string registered in the first dictionary 11 in the replaced text, and records the range (step S3).

  Next, the sentence boundary estimation part 14 performs the process of step S4-S6 which judges the presence or absence of a sentence boundary immediately after that for each sentence end symbol in an original text.

  Here, the sentence boundary estimation unit 14 further performs the following determination when the sentence end symbol is a part of the replacement source word string replaced in the original text. In this case, the sentence boundary estimation unit 14 is a range in which the replacement destination word string replaced from the replacement source word string in the post-substitution text matches the word string already registered in the first dictionary 11 detected previously. (Step S4).

  However, even when the corresponding replacement destination word string is included in the range matching the word string already registered in the first dictionary 11, the sentence boundary estimation unit 14 sets the determination result in step S4 as No. To do. In such a case, this sentence ending symbol is the last character of the replacement source word string, and the last word of the replacement destination word string and the word registered in the first dictionary 11 including the replacement destination word string This is the case when the last word in the range that matches the column matches.

  When a plurality of post-substitution texts are generated in step S2, the sentence boundary estimation unit 14 may perform the determination process in step S4 on each post-substitution text. In this case, the sentence boundary estimation unit 14 may set the determination result as the entire step S4 to Yes when the determination result in the at least one post-substitution text is Yes. On the other hand, the sentence boundary estimation part 14 is good also as No as the determination result as the whole step S4, when the determination result in all the texts after substitution becomes No.

  Here, the case where it becomes Yes at step S4 is demonstrated. In this case, the sentence boundary estimation unit 14 determines that there is no sentence boundary immediately after the sentence end symbol in the original text (step S5).

  On the other hand, the case where it becomes No at step S4 is demonstrated. In this case, the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing on the sentence end symbol to estimate the presence or absence of a sentence boundary immediately after that (step S6).

  As described above, a known technique may be applied to normal sentence boundary estimation processing. For example, as a normal sentence boundary estimation process, when both condition 1 and condition 2 described in the background art are satisfied, a process of determining that there is a sentence boundary immediately after that may be applied.

  When the processes in steps S4 to S6 are completed for all the end-of-sentence symbols in the original text, the sentence boundary estimation unit 14 outputs information representing the estimated sentence boundary position (step S7).

  Thus, the sentence boundary estimation unit 14 ends the operation.

  Next, the operation of the sentence boundary estimation apparatus 1 will be shown as a specific example. Here, the original text is English, and the sentence ending symbol is a period. In addition, it is assumed that the information shown in FIGS. 3 and 4 is stored in the first dictionary 11 and the second dictionary 12, respectively. Each functional block recognizes a blank in the text as a word boundary.

  First, it is assumed that the text development unit 13 acquires the original text shown in FIG. 6 (step S1).

  Next, the text expansion unit 13 refers to the second dictionary 12 to generate a replacement text from the original text as shown in FIG. 6 (step S2).

  Here, in the original text of FIG. 6, the underlined portion indicates a replacement source word string portion. That is, in this original text, the replacement source word string “U.S.” of ID: 1 in the second dictionary 12 shown in FIG. 4 appears in two places. The appearance position of this replacement source word string is the third word and the tenth word of the original text. The text expansion unit 13 replaces these two “U.S.” with “United States”, which is a replacement destination word string associated in the second dictionary 12.

  In the post-replacement text of FIG. 6, the double underline indicates the part of the replacement destination word string that is replaced from the replacement source word string of the original text. The first “United States” is located at the 3rd to 4th words of the replacement text, and the second “United States” is located at the 11th to 12th words of the replacement text.

  Next, the sentence boundary estimation unit 14 detects a range that matches the word string registered in the first dictionary 11 in the replaced text (step S3). In FIG. 6, a portion surrounded by a broken-line rectangle indicates a range that matches a word string registered in the first dictionary 11. The first third to fifth words are ranges that match the word string “United States President” with ID: 2 in the first dictionary 11. The second 11th to 13th words are a range that matches the word string “United States Senate” with ID: 3 in the first dictionary 11.

  Next, the sentence boundary estimation unit 14 repeats the determination as to whether or not a sentence boundary immediately follows the sentence end symbol for each sentence end symbol of the original text.

  First, the sentence boundary estimation unit 14 pays attention to the period immediately after U of the third word “U.S.” as the first period in the original text. This period is included in the replacement word string “U.S.” of the third word replaced by the text expansion unit 13. Further, this replacement source word string is replaced with the replacement destination word string “United States” of the third to fourth words in the replaced text. Furthermore, in the post-replacement text, the third to fifth words match the word string “United States President” with ID: 2 registered in the first dictionary 11.

  That is, the period of interest exists in “the inside of the replacement source word string replaced with the replacement destination word string included in the range matching the word string registered in the first dictionary 11 in the replaced text”. The period is not the last character of the replacement source word string. Thereby, about this period, the judgment result of step S4 by the sentence boundary estimation part 14 will be Yes.

  Therefore, the sentence boundary estimation unit 14 determines that it is not a sentence boundary immediately after the first period (step S5).

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after S of the third word “U.S.” as the second period in the original text. Similarly to the first period, this period is also replaced with the replacement destination word string “United States” included in the range “United States President” that matches the word string registered in the first dictionary 11 in the replacement text. Exists within the replacement source word string “US”. In addition, this period is the last character of the replacement source word string “US”, but the last word “States” of the replacement destination word string is the last character in the range matching the word string registered in the first dictionary 11. Is different from the word “President”. Thereby, about this period, the judgment result of step S4 by the sentence boundary estimation part 14 will be Yes.

  Therefore, the sentence boundary estimation unit 14 determines that it is not a sentence boundary immediately after the second period (step S5).

  Similarly, the sentence boundary estimation unit 14 pays attention to each of two periods in the tenth word “U.S.” as the third to fourth periods in the original text. These periods are replaced by the replacement source word string “US” replaced by the replacement destination word string “United States” included in the range “United States Senate” that matches the word string registered in the first dictionary 11 in the replaced text. Inside. " Each of these periods is not the last character of the replacement source word string, or a range in which the last word “States” of the replacement destination word string matches the word string registered in the first dictionary 11. Is different from the last word "Senate". Therefore, for these periods, the determination result in step S4 by the sentence boundary estimation unit 14 is Yes.

  Therefore, the sentence boundary estimation unit 14 determines that it is not a sentence boundary immediately after these third to fourth periods.

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after “yesterday” as the fifth period in the original text. This period is not included in the replacement source word string replaced by the text expansion unit 13. Therefore, the determination result of step S4 by the sentence boundary estimation part 14 becomes No about this period.

  Therefore, the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing for this period (step S6).

  Here, the sentence boundary estimation unit 14 determines that there is a sentence boundary immediately after the corresponding period satisfies both the conditions 1 and 2 described in the background art as a normal sentence boundary estimation process. And The period immediately after “yesterday” satisfies the condition 1 because the period immediately after it is a blank character. Also, this period satisfies condition 2 because the word “The” immediately after it begins with a capital letter. Therefore, the sentence boundary estimation unit 14 determines that there is a sentence boundary immediately after the fifth period.

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after “everyone” as the sixth period in the original text. This period is not included in the replacement source word string replaced by the text expansion unit 13. Therefore, the determination result of step S4 by the sentence boundary estimation part 14 becomes No about this period.

  Therefore, the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing for this period (step S6). The period immediately after “everyone” satisfies both condition 1 and condition 2 because there is no character or word immediately after that. Therefore, the sentence boundary estimation unit 14 determines that a sentence boundary exists immediately after the sixth period.

  And the sentence boundary estimation part 14 outputs the information showing an estimation result, as shown in FIG. In FIG. 6, the text in which the character string “{sentence boundary}” is inserted at a position where it is determined that there is a sentence boundary in the original text is output as information representing the estimation result. That is, in the original text, it is presumed that there is a sentence boundary at a position immediately after the next period after “yesterday” and at a position immediately after the next period after “everyone”.

  Next, effects of the first exemplary embodiment of the present invention will be described.

  The sentence boundary estimation apparatus according to the first exemplary embodiment of the present invention can estimate sentence boundaries more accurately while suppressing an increase in processing load and an increase in labor for dictionary registration.

  The reason will be described. In the present embodiment, the first dictionary stores word strings. Further, the second dictionary is a replacement source word string that is a word string including a sentence ending symbol, and a replacement destination word string that is a word string that is expressed without including a sentence ending symbol so as to represent the same meaning as the replacement source word string. Are stored in association with each other. Then, the text expansion unit expands the replacement source word string in the input original text into a post-replacement text that is replaced with the associated replacement destination word string. Then, the sentence boundary estimation unit estimates the sentence boundary in the original text based on the replaced text and the first dictionary. At this time, when the sentence boundary estimation unit is within the replacement source word string replaced with the replacement destination word string included in the range matching the word string of the first dictionary in the replacement text, the sentence ending symbol of the original text This is because it is estimated that the sentence boundary is not immediately after the sentence end symbol.

  As described above, in this embodiment, when it is not desired to estimate that there is a sentence boundary inside or immediately after a word string including a sentence end symbol, various compound words further including the word string including the sentence end symbol are stored in the dictionary. There is no need to register. In the present embodiment, the word string including the sentence ending symbol and the word string not including the sentence ending symbol representing the equivalent meaning are associated and registered in the second dictionary, and a compound word having a notation not including the sentence ending symbol is registered. It only has to be registered in the first dictionary. Therefore, the present embodiment greatly reduces the trouble of dictionary registration. Moreover, since this embodiment estimates sentence boundaries based on such first and second dictionaries, high-load processing such as part-of-speech estimation processing is not required for each word.

  The effect of this embodiment will be specifically described using the above-described specific example.

  In the specific example described above, the period immediately after S in “U.S.” appearing in two places in the original text is immediately followed by a blank character, and the immediately following words (“President” and “Senate”) begin with a capital letter. That is, the conditions 1 and 2 are satisfied only by the normal sentence boundary estimation process to which a general technique is applied. For this reason, when this embodiment is not used, these periods are recognized as being present immediately after the sentence boundary originally does not exist.

  Here, in order to prevent this in a normal sentence boundary estimation process to which a general technique is applied, it is necessary to register “U.S. President” and “U.S. Senate” in the dictionary as word strings. Also, if you have another compound word that includes “United States” (for example, “United States Ministry of Defense”) and you do not want to infer that there is a sentence boundary, of Defense "must also be registered in the dictionary. As described above, there are many enormous numbers of compound words that further include a word string including a sentence ending symbol, and it is very troublesome to register them all in the dictionary.

  On the other hand, in the specific example of the present embodiment, a compound word that further includes a word including a sentence ending symbol such as “U.S. President” and “U.S. Senate” may not be registered in any dictionary. In the present embodiment, the word strings “United States President” and “United States Senate” spelled out may be registered in the first dictionary. Further, in the present embodiment, the word “U.S.” including the sentence ending symbol and the word string “United States” spelled out thereof may be associated and registered in the second dictionary. That is, the present embodiment also includes the second dictionary even when there is another compound word including “United States” (for example, “United States Ministry of Defense”) and it is not desired to estimate that there is a sentence boundary inside. There is no need to increase registrations. In this case, in the present embodiment, the word string “United States Ministry of Defense”, which is a spelled out spelling method, may be registered in the first dictionary.

  As described above, in this embodiment, it is not necessary to register all notations including sentence ending symbols in a dictionary for a compound word that it is desired to avoid having a sentence boundary inside. In this embodiment, if one association between an acronym including a sentence ending symbol and its spell-out notation is registered in the second dictionary, only the spelled-out compound word is the first. Just register it in the dictionary.

  As a result, this embodiment can estimate sentence boundaries more accurately without requiring high-load processing such as part-of-speech estimation processing and reducing the effort required for dictionary registration.

(Second Embodiment)
Next, a second embodiment of the present invention will be described in detail with reference to the drawings. Note that, in each drawing referred to in the description of the present embodiment, the same reference numerals are given to the same configuration and steps that operate in the same manner as in the first embodiment of the present invention, and the detailed description in the present embodiment. Description is omitted.

  First, the structure of the sentence boundary estimation apparatus 2 as the 2nd Embodiment of this invention is shown in FIG. In FIG. 7, the sentence boundary estimation device 2 is replaced with a second dictionary 22 and a text development unit 13 instead of the second dictionary 12 with respect to the sentence boundary estimation device 1 as the first embodiment of the present invention. And a text expansion unit 23, and further includes a replacement destination word string detection unit 25. The sentence boundary estimation device 2 and each functional block thereof can be configured by the same hardware elements as those in the first embodiment of the present invention described with reference to FIG. However, the hardware configuration of the sentence boundary estimation device 2 and each functional block thereof is not limited to the above-described configuration.

  The second dictionary 22 stores information indicating the presence / absence of appearance restrictions in association with the replacement source word string in addition to the replacement destination word string. That is, each entry in the second dictionary 22 includes a replacement source word string, a replacement destination word string, and information indicating the presence / absence of an appearance constraint. Hereinafter, an entry including information indicating that there is (or is not) an appearance constraint, or a replacement source word string or a replacement destination word string included in the entry is simply an entry that has (or is not) an appearance constraint. Also described as a replacement source word string or a replacement destination word string.

  The replacement-destination word string detection unit 25 determines whether or not each replacement-destination word string for which there is an appearance constraint in the second dictionary 22 is included in the original text.

  The text expansion unit 23, for a replacement source word string that appears in the original text, has an appearance restriction, and when the associated replacement destination word string is included in the original text, to the replacement destination word string Perform the replacement process. Further, the text expansion unit 23, when there is an appearance restriction among the replacement source word strings appearing in the original text, if the associated replacement destination word string is not included in the original text, the replacement destination word Do not perform column replacement processing. In addition, the text development unit 23 determines whether a replacement source word string that appears in the original text has no appearance restriction regardless of whether or not the associated replacement destination word string appears in the original text. A replacement process for the replacement word string is executed.

  In addition, there is no restriction | limiting in particular about the position where a replacement destination word sequence appears in an original text in the determination process by the replacement destination word sequence detection part 25. FIG. For example, the appearance position of the replacement destination word string may appear in the same sentence or the same paragraph as the appearance position of the replacement source word string associated with the replacement destination word string, or in a different sentence or a different paragraph. May appear.

  In English, etc., abbreviations such as acronyms are generally ambiguous, and there are often a plurality of compound words (full spelling word strings) corresponding to the abbreviations. For this reason, abbreviations such as acronyms are often written together with full spellings that are not abbreviations at the first occurrence, except for some abbreviations that are frequently used. For example, a full spell word is described in the first appearance, and an abbreviation is written immediately after that in parentheses. As described above, in English, it is recommended by various style guides and the like to indicate what word string the abbreviation corresponds to. By following this approach, it is easy to understand what abbreviations mean when humans read text. In view of such circumstances, the text development unit 23 determines the result of determination by the replacement-destination word string detection unit 25 when replacing a replacement-destination word string as an abbreviation such as an acronym with a replacement-source word string that is a full spelling. Based on the above, a replacement process is performed.

  That is, when the original text is English, an abbreviation such as an acronym is stored in the second dictionary 22 as a replacement source word sequence, and a full spelling word sequence is stored as a replacement destination word sequence. In this case, the text development unit 23 replaces the acronym in the original text with the full spelling word string only when the full spelling word string appears somewhere in the original text. On the other hand, the text development unit 23 does not replace an acronym in which the full spelling word string appears anywhere in the original text with the full spelling.

  The operation of the sentence boundary estimation apparatus 2 configured as described above will be described with reference to FIG.

  In FIG. 8, first, the text development unit 23 executes step S1 in the same manner as in the first embodiment of the present invention, and acquires the original text.

  Next, the replacement-destination word string detection unit 25 determines whether or not each replacement-destination word string that has an appearance constraint in the second dictionary 22 appears in the original text (step S21).

  Next, the text expansion unit 23 refers to the second dictionary 22 and searches for a character string that matches the replacement source word string in the original text. Then, the text expansion unit 23 generates the replaced text by replacing the searched character string that matches the replacement source word string with the replacement destination word string. However, at this time, the text expansion unit 23 replaces the replacement source word string for which the appearance restriction is present in the second dictionary 22 when the associated replacement destination word string is detected in step S21. Performs replacement processing for the previous word string. In this case, the text development unit 23 does not perform the replacement process when the associated replacement destination word string is not detected in step S21 (step S22).

  Thereafter, the sentence boundary estimation device 2 operates in the same manner as in the first embodiment of the present invention up to steps S3 to S7.

  Thus, the sentence boundary estimation device 2 ends the operation.

  Next, the operation of the second exemplary embodiment of the present invention will be shown as a specific example. Here, the original text is English, and the sentence ending symbol is a period. Each functional block recognizes a blank in the text as a word boundary. Further, it is assumed that the information shown in FIG. 9 is stored in the first dictionary 11. Further, it is assumed that the information shown in FIG. 10 is stored in the second dictionary 22.

  As shown in FIG. 10, the second dictionary 22 stores information indicating the presence / absence of the appearance restriction in addition to the replacement source word string and the replacement destination word string. For example, an entry (ID: 3) having “U.S.” as a replacement source word string and “Universal Studios” as a replacement destination word string includes information indicating “present” as an appearance constraint. As described above, for the entry having the appearance restriction, the replacement processing by the text development unit 23 has detected that the replacement destination word string has appeared in the original text by the replacement destination word string detection unit 25. It must be a condition. Further, the entry (ID: 1) having “U.S.” as the replacement source word string and “United States” as the replacement destination word string includes information indicating “none” as the appearance constraint. As described above, for the entry for which there is no appearance restriction, the replacement processing by the text development unit 23 is performed regardless of the presence or absence of the replacement destination word string in the original text.

  For example, “US” as an abbreviation for “United States” is very frequently used and should not be used alone without explicitly indicating that “US” is an abbreviation for “United States”. There are many. For this reason, in the entry related to this abbreviation in the second dictionary 22, there is no appearance restriction. Similarly, “P.O.” as an abbreviation of “Post Office” is often used without being clearly indicated. For this reason, even the entry related to this abbreviation in the second dictionary 22 has no appearance restriction.

  Based on this assumption, two specific examples will be described.

<Specific Example 1>
In this specific example, first, it is assumed that the text development unit 23 acquires the original text 1 shown in FIG. 11 (step S1).

  Next, the replacement-destination word string detection unit 25 determines whether or not each replacement-destination word string that has an appearance constraint in the second dictionary 22 appears in the original text 1 (step S21).

  Here, of the entries shown in FIG. 10, the entry of ID: 3 has an appearance restriction. Here, the replacement destination word string “Universal Studios” of the entry of ID: 3 does not appear in the original text 1. Therefore, the replacement-destination word string detection unit 25 determines that the replacement-destination word string of the entry with ID: 3 does not appear in the original text 1.

  Next, the text expansion unit 23 refers to the second dictionary 22 to generate the replaced text 1 from the original text 1 as shown in FIG. 11 (step S2).

  Here, the replacement source word string “U.S.” of the entry of ID: 1 in the second dictionary 22 matches the seventh word of the original text 1. In the original text 1 of FIG. 11, the underlined portion represents the replacement source word string portion. In addition, this entry has an appearance constraint “none”. Therefore, the text development unit 23 performs a process of replacing the seventh word of the original text 1 with the replacement destination word string “United States”. Further, the replacement source word string “U.S.” of the entry of ID: 3 in the second dictionary 22 also matches the seventh word of the original text 1. However, this entry has an appearance constraint “Yes”, and the determination result by the replacement destination word string detection unit 25 is “Not Appear”. Therefore, the text expansion unit 23 does not perform a replacement process based on this entry. As a result, the post-replacement text 1 shown in FIG. 11 is generated. In the post-replacement text 1 of FIG. 11, the double underlined portion indicates the part of the replacement destination word string that is replaced from the replacement source word string of the original text 1.

  Next, the sentence boundary estimation unit 14 detects a range that matches the word string registered in the first dictionary 11 in the replacement text 1 (step S3). In the post-replacement text 1 in FIG. 11, a portion surrounded by a broken-line rectangle indicates a range that matches the word string registered in the first dictionary 11 in FIG. 9. Here, the seventh to eighth words are ranges that match the word string “United States” with ID: 1 in the first dictionary 11.

  Next, the sentence boundary estimation unit 14 repeatedly determines whether there is a sentence boundary immediately after each sentence end symbol of the original text 1.

  First, the sentence boundary estimation unit 14 pays attention to the period immediately after U of the word “U.S.” as the first period in the original text 1. This period is included in the replacement source word string “U.S.” replaced with “United States”. Also, the replacement source word string “U.S.” is replaced by the replacement destination word string “United States” of the seventh to eighth words in the replaced text 1 by the text development unit 23. Furthermore, in the text 1 after replacement, the seventh to eighth words are also a range that matches the word string “United States” of ID: 1 registered in the first dictionary 11. That is, the replaced replacement destination word string is included in a range that matches the word string in the first dictionary 11. The period of interest is not the last character of the replacement source word string “U.S.”. Thereby, about this period, the judgment result of step S4 by the sentence boundary estimation part 14 will be Yes.

  Therefore, the sentence boundary estimation unit 14 determines that it is not a sentence boundary immediately after the first period (step S5).

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after S of the word “U.S.” as the second period in the original text 1. This period is included in the replacement source word string “U.S.” replaced with “United States”. Also, the replacement source word string “U.S.” is replaced by the replacement destination word string “United States” of the seventh to eighth words in the replaced text 1 by the text development unit 23. Furthermore, in the text 1 after replacement, the seventh to eighth words are also a range that matches the word string “United States” of ID: 1 registered in the first dictionary 11. That is, the replaced replacement destination word string is included in a range that matches the word string in the first dictionary 11. However, this period is the last character of the replacement source word string “U.S.”. Also, the last word (eighth word) of the replacement destination word string “United States” in the post-replacement text 1 and the last word (eight words) in the range matching the entry in the first dictionary 11 including this replacement destination word string Eye) matches. Thereby, about this period, the judgment result of step S4 by the sentence boundary estimation part 14 becomes No.

  Therefore, the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing for this period (step S6). Here, as in the specific example in the first embodiment of the present invention, the sentence boundary estimation unit 14 performs conditions 1 and 2 described in the background art as the corresponding period as the normal sentence boundary estimation processing. When both are satisfied, it is determined that there is a sentence boundary immediately after that. Here, since this period is a space character immediately after that, the condition 1 is satisfied. Also, this period satisfies condition 2 because the word “Japan” immediately after it begins with a capital letter. Therefore, the sentence boundary estimation unit 14 determines that a sentence boundary exists immediately after the second period.

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after “China” as the third period in the original text 1. This period is not included in the replacement source word string replaced by the text expansion unit 23. Therefore, the determination result of step S4 by the sentence boundary estimation part 14 becomes No about this period.

  Therefore, the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing for this period (step S6). Here, this period satisfies both Condition 1 and Condition 2 because there is no character or word immediately after that. Therefore, the sentence boundary estimation unit 14 determines that there is a sentence boundary immediately after the third period.

  Then, as shown in FIG. 11, the sentence boundary estimation unit 14 outputs information representing the sentence boundary estimation result 1 regarding the original text 1. In FIG. 11, the text in which the character string “{sentence boundary}” is inserted at a position where it is determined that there is a sentence boundary in the original text 1 is output as information representing the estimation result 1. That is, in the original text 1, it is estimated that there is a sentence boundary at a position immediately after the next period of “U.S.” and a position immediately after the next period of “China”.

  Thus, the sentence boundary estimation device 2 ends the sentence boundary estimation operation for the original text 1.

<Specific example 2>
In this specific example, first, it is assumed that the text development unit 23 acquires the original text 2 shown in FIG. 12 (step S1).

  Next, the replacement-destination word string detection unit 25 determines whether or not each replacement-destination word string that has an appearance constraint in the second dictionary 22 appears in the original text 2 (step S21).

  Here, the replacement destination word string “Universal Studios” of the entry of ID: 3 whose application restriction is “Yes” appears in the second to third words of the original text 2. In the original text 2 of FIG. 12, the part enclosed by the dashed-dotted line rectangle indicates the replacement destination word string with appearance restrictions. Therefore, the replacement destination word string detection unit 25 determines that the replacement destination word string of the entry of ID: 3 “appears in the original text 2”.

  Next, the text expansion unit 23 generates a replacement text from the original text 2 by referring to the second dictionary 22 (step S2).

  Here, the replacement source word string “U.S.” of the entry of ID: 1 in the second dictionary 22 matches the 20th word of the original text 2. In the original text 2 of FIG. 12, the underlined portion indicates the part of the replacement source word string. In addition, this entry has an appearance constraint “None”. Therefore, the text development unit 23 performs a process of replacing the 20th word of the original text 2 with the replacement destination word string “United States”. Thereby, the post-replacement text 2-1 in FIG. 12 is generated.

  Further, the replacement source word string “U.S.” of the entry of ID: 3 in the second dictionary 22 matches the 20th word of the original text 2. In addition, this entry has an appearance constraint “Yes”. Further, the replacement destination word string “Universal Studios” of this entry appears in the original text 2. Therefore, the text development unit 23 performs a process of replacing the 20th word of the original text 2 based on this entry. However, the corresponding replacement source word string “U.S.” is in the same position as the replacement source word string that was replaced when the post-replacement text 2-1 was generated earlier. Therefore, the text development unit 23 performs processing to replace the 20th word of the original text 2 with the replacement destination word string “Universal Studios” of this entry, and replaces the post-substitution text 2-2 different from the post-substitution text 2-1. Generate. Thereby, the post-replacement text 2-2 in FIG. 12 is generated.

  Note that, in the post-replacement texts 2-1 and 2-2 shown in FIG.

  Next, the sentence boundary estimation unit 14 detects a range that matches the word string registered in the first dictionary 11 in the replaced texts 2-1 and 2-2 (step S3). In the post-replacement texts 2-1 and 2-2 in FIG. 12, a portion surrounded by a broken-line rectangle indicates a range that matches the word string registered in the first dictionary 11.

  Here, in the post-replacement text 2-1, the second to third words are ranges that match the word string “Universal Studios” with ID: 6 in the first dictionary 11. Also, the 20th to 21st words are a range that matches the word string “United States” of ID: 1 in the first dictionary 11.

  Further, in the post-replacement text 2-2, the second to third words are ranges that match the word string “Universal Studios” with ID: 6 in the first dictionary 11. Further, the 20th to 22nd words are a range that matches the word string “Universal Studios Japan” with ID: 7.

  Next, the sentence boundary estimation unit 14 repeatedly determines whether there is a sentence boundary immediately after each sentence end symbol of the original text 2.

  First, as the first period in the original text 2, pay attention to the period immediately after “yesterday”. This period is not included in the replacement source word string replaced by the text expansion unit 23. Therefore, the determination result of step S4 by the sentence boundary estimation part 14 becomes No about this period.

  Therefore, the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing for this period (step S6).

  Here, since the period immediately after “yesterday” is a space character immediately after that, the condition 1 is satisfied. Also, this period satisfies condition 2 because the word “If” immediately after it begins with a capital letter. Therefore, the sentence boundary estimation unit 14 determines that a sentence boundary exists immediately after the first period.

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after U of “U.S.” as the second period in the original text 2. In this example, post-substitution texts 2-1 and 2-2 with different replacement destination word strings are generated for this location. Therefore, the sentence boundary estimation unit 14 tries the determination of step S4 sequentially based on the replaced texts 2-1 and 2-2 for this period.

  First, in the post-substitution text 2-1, this period is included in the replacement source word string “U.S.” replaced with “United States”. The replacement source word string “U.S.” is replaced with the replacement destination word string “United States” of the 20th to 21st words in the post-replacement text 2-1. Furthermore, in the post-replacement text 2-1, the 20th to 21st words are also a range that matches the word string “United States” of ID: 1 registered in the first dictionary 11. That is, the replaced replacement destination word string is included in a range that matches the word string in the first dictionary 11. The period of interest is not the last character of the replacement source word string “U.S.”. Thereby, about this period, the judgment result of step S4 based on the post-replacement text 2-1 by the sentence boundary estimation part 14 becomes Yes.

  Here, as described above, when a plurality of post-replacement texts are generated, the sentence boundary estimation unit 14 determines that the determination result of Step S4 is Yes in at least one post-replacement text as the entire Step S4. The determination result is assumed to be Yes. That is, since the determination result of step S4 based on the replaced text 2-1 is Yes, the sentence boundary estimation unit 14 does not perform the determination process of step S4 based on the replaced text 2-2 for this period. The determination result for step S4 is “Yes”.

  Therefore, the sentence boundary estimation unit 14 determines that it is not a sentence boundary immediately after the second period (step S5).

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after S of “U.S.” as the third period in the original text 2. In this example, post-substitution texts 2-1 and 2-2 with different replacement destination word strings are generated for this location. Therefore, the sentence boundary estimation unit 14 tries the determination of step S4 sequentially based on the replaced texts 2-1 and 2-2 for this period.

  First, in the post-substitution text 2-1, this period is included in the replacement source word string “U.S.” replaced with “United States”. The replacement source word string “U.S.” is replaced with the replacement destination word string “United States” of the 20th to 21st words in the post-replacement text 2-1. Furthermore, in the post-replacement text 2-1, the 20th to 21st words are also a range that matches the word string “United States” of ID: 1 registered in the first dictionary 11. That is, the replaced replacement destination word string is included in a range that matches the word string in the first dictionary 11. However, the period of interest is the last character of the replacement source word string “U.S.”. Further, the last word (21st word) of the replacement destination word string “United States” in the post-replacement text 2-1 and the last word in a range matching the entry of the first dictionary 11 including this replacement destination word string ( Matches the 21st word). Thereby, about this period, the judgment result of step S4 based on the text 2-1 after substitution by the sentence boundary estimation part 14 becomes No.

  In the post-replacement text 2-2, this period is included in the replacement source word string “U.S.” replaced by “Universal Studios”. The replacement source word string “U.S.” is replaced with the replacement destination word string “Universal Studios” of the 20th to 21st words in the post-replacement text 2-2. Furthermore, in the post-replacement text 2-2, the 20th to 22nd words are ranges that match the word string “Universal Studios Japan” with ID: 7 registered in the first dictionary 11. That is, the replaced replacement destination word string is included in a range that matches the word string in the first dictionary 11. The period of interest is the last character of the replacement source word string “U.S.”. However, the range “Universal Studios Japan” that matches the last word (21st word) of the replacement destination word string “Universal Studios” in the post-replacement text 2-2 and the entry in the first dictionary 11 that includes this replacement destination word string. Is different from the last word (22nd word). Thereby, about this period, the judgment result of step S4 based on the post-replacement text 2-2 by the sentence boundary estimation part 14 will be Yes.

  Here, for this period, since the determination result in step S4 is Yes in at least one post-substitution text, the sentence boundary estimation unit 14 sets the determination result as the entire step S4 to Yes.

  Therefore, the sentence boundary estimation unit 14 determines that it is not a sentence boundary immediately after the third period (step S5).

  Next, the sentence boundary estimation unit 14 pays attention to the period immediately after “Japan” as the fourth period in the original text 2. This period is not included in the replacement source word string replaced by the text expansion unit 23. Therefore, the determination result of step S4 by the sentence boundary estimation part 14 becomes No about this period.

  Therefore, the sentence boundary estimation unit 14 performs normal sentence boundary estimation processing for this period (step S6). Here, this period satisfies both conditions 1 and 2 because there is no character or word immediately after that. Therefore, the sentence boundary estimation unit 14 determines that a sentence boundary exists immediately after the fourth period.

  Then, the sentence boundary estimation unit 14 outputs information representing the sentence boundary estimation result 2 regarding the original text 2 as shown in FIG. In FIG. 12, the text in which the character string “{sentence boundary}” is inserted at a position where it is determined that there is a sentence boundary in the original text 2 is output as information representing the estimation result 2. That is, in the original text 2, it is estimated that there is a sentence boundary between the position immediately after the next period of “yesterday” and the position immediately after the next period of “Japan”.

  Thus, the sentence boundary estimation device 2 ends the sentence boundary estimation operation for the original text 2.

  Next, the effect of the second exemplary embodiment of the present invention will be described.

  The sentence boundary estimation device according to the second exemplary embodiment of the present invention prevents a sentence boundary estimation accuracy from being lowered due to a word string including a sentence ending symbol being interpreted in a meaning different from the original meaning.

  The reason will be described. In the present embodiment, in addition to the same configuration as that of the first embodiment of the present invention, the second dictionary stores information indicating the presence / absence of occurrence restrictions for the entries of the replacement source word string and replacement destination word string Is stored. Then, the replacement-destination word string detection unit determines whether or not the replacement-destination word string of the entry having the appearance restriction appears in the original text. Then, the text expansion unit replaces the replacement source word string of the entry with the appearance restriction by replacing it with the replacement destination word string when the replacement destination word string appears in the original text. This is because text is generated.

  Thereby, this Embodiment can replace the replacement source word string including the sentence end symbol in the original text with a more appropriate replacement destination word string. As a result, this embodiment unintentionally matches the compound word including the replacement destination word string by replacing the replacement source word string including the sentence end symbol with the unintended replacement destination word string. Can avoid a situation where it is estimated that there is no sentence boundary at a position where there should be a sentence boundary.

  The effect of this embodiment will be specifically described using the above-described specific example.

  Assume that sentence boundary estimation processing is performed on the original texts 1 and 2 of the above-described specific example using a sentence boundary estimation method in which a word including a sentence end symbol is registered in a dictionary. Here, the sentence boundary estimation method for registering words including sentence endings in the dictionary assumes that there is no sentence boundary in the dictionary if the word sequence “US Japan” is registered in the dictionary. For words not registered in, normal sentence boundary estimation processing is performed. As described above, the sentence boundary estimation method for registering a word including a sentence ending symbol in a dictionary can obtain only a uniform estimation result for the same word string. In other words, if “US Japan” is not registered, it is correctly estimated that there is a sentence boundary immediately after “US” in original text 1, but it is assumed that there is a sentence boundary immediately after “US” in original text 2. Therefore, a correct estimation result cannot be obtained. On the other hand, if "US Japan" is registered, it is correctly estimated that there is no sentence boundary immediately after "US" in Original Text 2, but it is assumed that there is no sentence boundary immediately after "US" in Original Text 1. As a result, a correct estimation result cannot be obtained.

  On the other hand, in the present embodiment, whether or not the word string “U.S. Japan” in each original text may be interpreted as an abbreviation of “Universal Studios Japan” is determined as follows. In other words, the present embodiment determines whether or not to interpret as an abbreviation of “Universal Studios Japan” based on whether or not the corresponding original text includes a spelled-out replacement word string “Universal Studios”. To do. Thereby, this Embodiment can obtain the correct estimation result of sentence boundaries for both the original text 1 and the original text 2.

  As described above, in this embodiment, a word string such as an acronym including a sentence ending symbol is replaced with a word string that does not include a ending symbol different from the original meaning. You can avoid getting it.

  In each of the embodiments of the present invention described above, the example in which the original text is English is mainly described, but the language of the original text is not limited to this.

  Further, in each of the embodiments of the present invention described above, an example in which a period is applied as a sentence end symbol has been mainly described. However, the sentence end symbol may be a semicolon, other characters, or a character string. In addition, a plurality of types of characters or character strings may be assumed as sentence ending symbols.

  Further, in each of the embodiments of the present invention described above, the first dictionary not only stores a word string expressed without including a sentence ending symbol so as to represent the same meaning as a word including a sentence ending symbol, A word including a sentence ending symbol or a word string including such a word may be stored. This corresponds to a combination of each embodiment of the present invention and the method of registering a word or compound word including a sentence ending symbol in the dictionary as described in the background art. For example, the first dictionary may store the word alone as type 1 if the word including the sentence end symbol does not appear at the end of the sentence. In addition, the first dictionary may store a word string (compound word) including the word as type 2 if the word including the sentence end symbol is a word that can appear in the middle of the sentence or at the end of the sentence. In this case, the sentence boundary estimation unit according to each embodiment performs the type 1 or type 2 registered in the first dictionary even when the sentence end symbol of the original text is not included in the replaced source word string. If it is included in the word string, estimation processing may be performed as described in the background art.

  As described above, the above-described embodiments of the present invention can be implemented in combination with other known sentence boundary estimation techniques.

  Furthermore, in the machine learning-based sentence boundary estimation apparatus, the sentence boundary estimation result according to each embodiment of the present invention described above may be input as one of the features for the estimation apparatus. Thereby, each embodiment can improve the estimation accuracy of the machine learning-based sentence boundary estimation device.

  Moreover, in each embodiment of the present invention described above, an example of information stored in the first dictionary is shown in FIGS. 3 and 9, but the contents and format of the information stored in the first dictionary are as follows. It is not limited.

  In each of the embodiments of the present invention described above, examples of information stored in the second dictionary are shown in FIGS. 4 and 10, but the contents and format of the information stored in the second dictionary are as follows. It is not limited.

  Moreover, in each embodiment of the present invention described above, examples of information output as sentence boundary estimation results are shown in FIGS. 6, 11, and 12. It is not limited to this.

  Further, in each of the above-described embodiments of the present invention, an example has been described in which each functional block of the sentence boundary estimation device is realized by a CPU that executes a computer program stored in a memory. However, the present invention is not limited to this, and some, all, or a combination of each functional block may be realized by dedicated hardware.

  Moreover, in each embodiment of this invention mentioned above, the functional block of a sentence boundary estimation apparatus may be distributed and implement | achieved by several apparatuses.

  In each embodiment of the present invention described above, the operation of the sentence boundary estimation device described with reference to each flowchart is stored in a storage device (storage medium) of the computer device as a computer program of the present invention. . Then, the computer program may be read and executed by the CPU. In such a case, the present invention is constituted by the code of the computer program or a storage medium.

  Moreover, each embodiment mentioned above can be implemented in combination as appropriate.

  The present invention is not limited to the above-described embodiments, and can be implemented in various modes.

  Examples of the use of the present invention include an information extraction device and a machine translation device. By applying the sentence boundary estimation technique of the present invention to these, information extraction or machine translation can be performed in a more appropriate unit as one sentence. Thereby, a user of an information extraction device or a machine translation device to which the sentence boundary estimation technique of the present invention is applied can obtain an information extraction result or a translation result with higher accuracy and more easily understood.

DESCRIPTION OF SYMBOLS 1, 2 Sentence boundary estimation apparatus 11 1st dictionary 12, 22 2nd dictionary 13, 23 Text expansion part 14 Sentence boundary estimation part 25 Replacement destination word sequence detection part 1001 CPU
1002 Memory 1003 Output device 1004 Input device

Claims (6)

A first dictionary for storing word strings;
A replacement source word string that is a word string including a sentence ending symbol and a replacement destination word string that is a word string expressed without including the sentence ending symbol so as to represent the same meaning as the replacement source word string are stored in association with each other. A second dictionary,
A text expansion unit that generates a post-substitution text based on the second dictionary by replacing the replacement source word string in the original text with the associated replacement destination word string;
When estimating a sentence boundary in the original text based on the replaced text and the first dictionary, the sentence end symbol in the original text matches a word string registered in the first dictionary in the replaced text A sentence boundary estimator that estimates that a sentence boundary is not immediately after the sentence end symbol when the replacement source word string is replaced with the replacement destination word string included in the range to be replaced;
A sentence boundary estimation apparatus comprising:   The sentence boundary estimation unit includes a replacement source word replaced with the replacement destination word string included in a range in which the sentence end symbol in the original text matches the word string registered in the first dictionary in the replacement text Even within the column, the sentence end symbol is the last character of the replacement source word sequence, and the last word of the replacement destination word sequence replaced from the replacement source word sequence in the replacement text And a word at the end of the range that matches the word string in the first dictionary including the replacement destination word string, the estimation that the next word is not a sentence boundary is suspended. Item 1. The sentence boundary estimation device according to Item 1. A replacement destination word string detection unit that determines whether or not the replacement destination word string registered in the second dictionary is included in the original text;
The text expansion unit is associated with the replacement source word string determined to be included in the original text by the replacement destination word string detection unit for the replacement source word string in the original text. 3. The sentence boundary estimation apparatus according to claim 1, wherein a replacement process for the replacement destination word string is executed. The second dictionary further stores, in association with the replacement source word string, information indicating the presence / absence of occurrence restrictions,
The text development unit relates to a determination result by the replacement-destination word string detection unit for a replacement-source word string associated with information indicating that there is no appearance restriction among the replacement-source word strings in the original text. First, replacement with the replacement destination word string is performed, and the replacement source word string associated with the information indicating that there is an appearance constraint is included in the original text by the replacement destination word string detection unit. 4. The sentence boundary estimation apparatus according to claim 3, wherein, when associated with the determined replacement destination word string, a replacement process for the replacement destination word string is executed. A first dictionary that stores a word string, a replacement source word string that is a word string including a sentence ending symbol, and a word string that is expressed without including the sentence ending symbol so as to represent the same meaning as the replacement source word string Using the second dictionary that stores the replacement destination word string in association with each other,
Based on the second dictionary, generate a replacement text in which the replacement source word string in the original text is replaced with the associated replacement destination word string;
When estimating a sentence boundary in the original text based on the replaced text and the first dictionary, the sentence end symbol in the original text matches a word string registered in the first dictionary in the replaced text A method of estimating that a sentence boundary is not immediately after the sentence end symbol when the word is inside the replacement source word string replaced with the replacement destination word string included in the range to be replaced. A first dictionary that stores a word string, a replacement source word string that is a word string including a sentence ending symbol, and a word string that is expressed without including the sentence ending symbol so as to represent the same meaning as the replacement source word string Using the second dictionary that stores the replacement destination word string in association with each other,
A text expansion step of generating a post-substitution text based on the second dictionary by replacing the replacement source word string in the original text with the associated replacement destination word string;
When estimating a sentence boundary in the original text based on the replaced text and the first dictionary, the sentence end symbol in the original text matches a word string registered in the first dictionary in the replaced text A sentence boundary estimating step that estimates that the sentence immediately after the sentence ending symbol is not a sentence boundary if it is inside the replacement source word string replaced with the replacement destination word string included in the range to be
That causes a computer device to execute the program.

Images