JP2006277674A - Sentence division computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a computer program for making a computer accurately and quickly divide various sentences without having to perform pre-processing or editing data. <P>SOLUTION: This sentence division program 86 is used with a database 88 including three or more pairs constituted of a sentence and divided character strings obtained by dividing the sentence. The sentence division program 86 is provided with a division program section 90 for, when given a sentence to be processed, selecting arbitrary two sentences from the database 88, and for generating an analogical sentence from the two sentences and the sentence to be processed, and for, when any analogical sentence exists in the database 88, reading divided character strings making a pair with the analogical sentence and the two sentences in the database 88, and for generating an analogical divided character strings by using the read three divided character strings. This sentence division program 86 is also provided with an input/output program section 92 for giving an input sentence 82 to the division program section 90, and for outputting the analogical divided character strings generated by the division program section 90 from the input sentence 82 as an output sentence 84. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to natural language processing, and more particularly to a technique for dividing a character string into predetermined units each including a word or the like.

  In order to process a natural language sentence with a computer, first, it is necessary to divide the sentence to be processed into units such as words or morphemes and to identify what individual elements are. For example, if the boundary between words is represented by the symbol “|”, the Japanese sentence “I want to leave next week” is divided into words, then “Next week | "There is." Especially, sentences in languages such as Japanese, Chinese, Thai, etc. are written in a single line without being divided into words as in languages such as English, so the process of dividing the sentence into units such as words Is an essential process.

  As methods for dividing a sentence into words, the following two types of methods are known. That is, one is a technique using a dictionary and the other is a statistical technique.

  When division is performed by a method using a dictionary, generally a dictionary configured as a finite state automaton is first prepared. Then, the characters in the sentence and the characters in the dictionary are collated in order from the beginning of the sentence, and the boundary between the last character constituting the preceding word and the first character constituting the next word is determined. This process is sequentially performed toward the end of the sentence to determine the boundary between words.

  When the division is performed by a statistical method, first, a so-called learning stage process is performed in which learning data prepared in advance is processed to obtain statistical information for inferring a probability model. There are various probabilistic models, but a model that depends on the N-gram frequency is usually used. The probability model is smoothed and evaluated using, for example, the Good-Turing evaluation method. As a result of such learning, information in the form of state transition is obtained. This state transition gives knowledge about the character being processed or to be processed, so whether or not the character being processed is the last character of a word, i.e. whether a blank should be placed at that position. Can be determined. The process in the learning stage is executed until the learning data is appropriately processed and knowledge is obtained from the probability model. After that, the boundary between words is determined using the knowledge obtained by learning.

Japanese Patent No. 3172511

  In the method using a dictionary, the processing is nondeterministic. That is, the same sentence may be divided in different ways. Further, in order to perform division by this method, it is necessary to edit a large-scale and appropriate dictionary in advance based on grammatical knowledge of the language to be processed.

  In the technique based on the statistical method, the prepared data must be preprocessed. That is, it is necessary to perform learning using a huge amount of learning data prior to processing. Further, when the N-gram model is used as the probability model, the context to be processed for division is fixed to a predetermined length, which is not suitable for processing in a long context.

  Therefore, an object of the present invention is to provide a computer program that allows a computer to divide various sentences accurately and quickly without requiring an enormous amount of preprocessing and data editing.

  The sentence division computer program according to the first aspect of the present invention is a computer program that is used together with a computer-readable database that includes three or more pairs of a sentence and a divided character string obtained by dividing the sentence by a predetermined unit. is there. When a sentence to be processed is given, the sentence dividing computer program uses a database to generate a divided character string part for generating a divided character string for the sentence to be processed, and an input sentence given from the outside as a sentence dividing program part. And an input / output program part for outputting a divided character string output by the sentence dividing program part for the input sentence as a divided character string for the input sentence. When a sentence to be processed is given, the sentence division program part selects arbitrary two sentences from the database, generates a predetermined first analogy from the two sentences and the sentence to be processed, A first analogy program part for generating an analogy sentence by solving the analogy formula, a determination program part for determining whether the analogy sentence exists in the database, and an analogy sentence in the database by the determination program part In response to being determined to exist, it reads from the database the split character string that is paired with the analogy sentence in the database and the two split character strings that are paired with two sentences in the database. A second analogical equation having a predetermined relationship with the first analogical equation is generated using the read program portion for the output and the three divided character strings read by the read program portion; And generating an analogy divided character string by solving the analogy equation and a second analogy program part for outputting as the return value of the sentence divided program parts.

  Preferably, when a sentence to be processed is given, the first analogizing program part includes a program part for selecting all order pairs (A, B) of sentences in the database, and each of the selected order pairs. On the other hand, the first analogical expression “A: B :: x: D” is generated with the sentence D to be processed, and the first analogical expression is solved to generate a solution x = C. And an analog reasoning solution program part for outputting all the solutions C.

  More preferably, sentences A, B, and C in the database are paired with divided character strings A ′, B ′, and C ′, respectively, in the database, and the second analogy program part is the read program part. Using the three divided character strings A ′, B ′, and C ′ read out by the second analogy formula “A ′: B ′: : C ′: y ”is generated, a program part for solving the second analogy formula and generating an analogy split character string y = D ′ and outputting it as a return value of the sentence split program part is included.

  The sentence split program part is a recursive call for recursively calling the sentence split program part using the analogy sentence as a processing target sentence in response to the judgment program part determining that the analog reason sentence does not exist in the database. The program part may further include a program part for adding to the database a pair of the program part, the analogy sentence, and the split character string output by the sentence split program part called by the recursive call program part.

  A recording medium according to a second aspect of the present invention is a computer-readable recording medium on which any sentence division computer program according to the first aspect of the present invention is recorded.

[Overview]
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this specification, Japanese is given as an example of a language written without being divided in units of words. As is clear from the following description, the present invention relates to a language, a character, a grammatical structure, and the like. It is applicable. In describing this embodiment, the boundary between words in Japanese is represented by the symbol “|” in the drawings and specification. Moreover, the boundary between words in English is represented by a blank as usual.

  The sentence division system according to this embodiment includes a sentence composed of a character string and a symbol string in which boundary symbols representing the boundaries are inserted at the boundaries between words in the sentence (hereinafter, this symbol string is referred to as a “divided character string”). The word unit is divided by symbol string analogy using a number of examples consisting of pairs. Refer to Patent Document 1 for symbol string analogy.

  Japanese Patent Application Laid-Open No. 2004-151561 discloses a method of generating another symbol string by analogy from three symbol strings given in a predetermined order. In this method, first, an analogy relation established between two symbol strings constituting three given symbol strings is specified. Subsequently, another symbol string is generated in which the same relationship as the analogy relationship is established with the remaining one symbol string constituting the three symbol strings. An analogy relationship similar to the analogy relationship established between two symbol strings is established between the remaining one symbol string and the generated symbol string. Here, the generation of the symbol string D from the symbol string A, the symbol string B, and the symbol string C by analogy is expressed as “A: B :: C: D”.

  In this embodiment, first, when a character string to be divided is a sentence D, sentences A, B, and C that satisfy A: B :: C: D are searched. When the sentences A, B, and C are searched, a divided character string A ′, a divided character string B ′, and a divided character string C ′ corresponding to the sentences A, B, and C are prepared using an example. The divided character strings A ′, B ′, and C ′ are also symbol strings and can be used for symbol string analogy. Therefore, a symbol string D ′ that satisfies A ′: B ′ :: C ′: D ′ is generated from the divided character strings A ′, B ′, and C ′ by symbol string analogy.

  Since the divided character strings A ′, B ′, and C ′ are divided character strings, the symbol string D ′ is similarly divided character strings. Since the divided character strings A ′, B ′, and C ′ correspond to the sentences A, B, and C, respectively, the symbol string D ′ also corresponds to the sentence D. Therefore, the symbol string D ′ is a divided character string corresponding to the sentence D.

  The sentence division system of this embodiment further replenishes the example by recursive processing for sentences generated in the process of searching for sentences A, B, and C.

[Constitution]
The sentence division system of this embodiment is realized by computer hardware, a program executed by the computer hardware, and data stored in the computer hardware. FIG. 1 shows the external appearance of the computer system 30, and FIG. 2 shows the internal configuration of the computer system 30.

  Referring to FIG. 1, a computer system 30 includes a computer 40 having a CD-ROM (Compact Disc Read Only Memory) drive 50 and an FD (Flexible Disc) drive 52, a monitor 42, a keyboard 46, and a mouse 48. including.

  2, in addition to the FD drive 52 and the CD-ROM drive 50, the computer 40 includes a hard disk 54, a CPU (central processing unit) 56, a CPU 56, a hard disk 54, an FD drive 52, and a CD-ROM. A bus 66 connected to the drive 50, a read-only memory (ROM) 58 connected to the bus 66 for storing a boot-up program and the like, and a bus 66 for storing a program command, a system program, work data, and the like. Random access memory (RAM) 60. The computer system 30 further includes a printer 44.

  Although not shown here, the computer 40 may further include a network adapter board that provides a connection to a local area network (LAN).

  A computer program for causing the computer system 30 to operate as the sentence division system of this embodiment is stored in the CD-ROM 62 or FD 64 inserted into the CD-ROM drive 50 or FD drive 52 and further stored in the hard disk 54. Transferred. Alternatively, the program may be transmitted to the computer 40 through a network (not shown) and stored in the hard disk 54. The program is loaded into the RAM 60 when executed. The program may be loaded directly into the RAM 60 from the CD-ROM 62, from the FD 64, or via a network.

  This program includes a plurality of instructions that cause the computer 40 to operate as the sentence division system of this embodiment. Some of the basic functions required to perform this operation are provided by operating system (OS) or third party programs running on the computer 40 or modules of various toolkits installed on the computer 40. Therefore, this program does not necessarily include all functions necessary for realizing the sentence division system of this embodiment. This program may include only instructions that execute the above-described operation as a sentence splitting system by calling an appropriate function or “tool” in a controlled manner so as to obtain a desired result. That's fine. The operation of computer system 30 is well known and will not be repeated here.

  FIG. 3 shows a functional configuration of the sentence division system of this embodiment in a block diagram form. Referring to FIG. 3, the sentence division system 80 of this embodiment receives an input sentence 82 composed of character strings, and generates a divided character string in which a symbol representing a boundary between words is inserted into the input sentence 82. A word dividing device 86 for outputting as an output sentence 84, and a database 88 for storing a large number of pairs of sentences and divided character strings corresponding to the sentence as data used for processing of the word dividing device 86; including.

  FIG. 4 shows the structure of data stored in the database 88. Referring to FIG. 4, a database 88 includes a first file 100 that includes a large number of sentences composed of character strings, and a second file 102 that includes a large number of divided character strings respectively corresponding to sentences included in the first file. including. The first file 100 shown in FIG. 4 is data in a list format and includes a large number of entries. Each entry of the first file 100 includes an identification number of the entry and an undivided sentence. The second file 102 is also data in a list format similar to the first file 100, and includes a large number of entries. Each entry includes an identification number of the entry and a divided character string. Each entry having the same identification number in the first file 100 and the second file 102 stores a sentence and a divided character string corresponding to the sentence.

  Referring to FIG. 3 again, when the word segmentation device 86 is called with a sentence as an argument, it uses the argument and the example in the database 88 to generate a segmented character string corresponding to the argument and return it as a return value. In addition, a memory for temporarily holding a split program 90 for executing a process for supplementing the example in the database 88, a split character string generated by the process by the split program 90, and an argument corresponding to the process 94, in response to the input sentence 82 being input, the division program 90 is called with the input sentence 82 as an argument, and as a result, the input / output for outputting the division character string returned from the division program 90 as the output sentence 84 Part 92.

  FIG. 5 shows a flowchart of the division program 90. Referring to FIG. 5, the division program 90 is called with a sentence as an argument D. When the dividing program 90 is started, the processing of Step 122, Step 124, and Step 126, which is surrounded by Step 120A and Step 120B, is performed on all the different sentences in the first file that can be considered including the order. Repeat until the processing is completed for the pair (ordered pair) (A, B).

  In step 122, symbol string analogy is performed based on three sentences consisting of two sentences A and B and an argument D constituting a sentence pair to be processed by the method described in Patent Document 1. That is, it tries to generate a sentence x = C in which the analogy relation A: B :: x: D is established (hereinafter, this sentence is referred to as a “first analogy sentence”). In step 124, it is determined whether or not the first kind inference C is generated in step 122. If generated, go to step 126. Otherwise, go to step 120B.

  In step 126, a process using the first analogy sentence C generated in step 122 is executed. That is, the search for the first kind inference C in the database 88 (see FIG. 4), the first kind inference, the sentence pair (A, B), and the second file 102 of the database 88 are used as the argument D. A process for generating a second class inference D ′ to be a corresponding divided character string is performed. When the process of step 126 ends, the process proceeds to step 120B. The processing in step 126 will be described later with reference to FIG.

  When a series of processes surrounded by step 120A and step 120B is completed for all sentence pairs, the process proceeds to step 128. In step 128, a return value is set. That is, if a divided character string D ′ corresponding to the argument D is generated by a series of processes and a pair of the argument D and the divided character string D ′ is stored in the memory 94 (see FIG. 3), it indicates a normal end. A value (for example, “0”) and an address indicating a storage position of a pair of a sentence and a divided character string on the memory 94 are set as a return value. If the divided character string D ′ corresponding to the argument D is not generated, a predetermined value (for example, “1”) indicating an error is set as a return value. Then, the division program 90 is terminated.

  FIG. 6 shows a flowchart of processing executed in step 126 (see FIG. 5). Referring to FIG. 6, when the process of step 126 is started, steps 142 to 142 surrounded by steps 140 </ b> A and 140 </ b> B are performed for all first type inference C generated in step 122 shown in FIG. 5. The process of step 158 is executed.

  In step 142, the first file 100 (see FIG. 4) is searched for a sentence that matches the first analogy sentence C. In step 144, the result of the search in step 142 is determined. That is, if a matching sentence is obtained from the first file 100 by the search, the process proceeds to step 146. Otherwise, go to step 154.

  In step 146, three sentences including a pair of sentences to be processed (A, B) and a first analogy sentence C to be processed, a first file 100 (see FIG. 4), and a second file 102 (see FIG. 4). And the three divided character strings A ′, B ′, and C ′ corresponding to the three sentences are read from the second file 102. In step 148, symbol string analogy similar to that in step 122 (see FIG. 5) is performed based on the three divided character strings A ′, B ′, and C ′ read in step 146. That is, the generation of the second analogy sentence y = D ′ in which the analogy formula A ′: B ′ :: C ′: y is established is attempted. The second type inference D ′ generated here is a divided character string corresponding to the argument D. In step 150, it is determined whether or not the second kind inference D 'has been generated in step 148. If generated, the process proceeds to step 152. Otherwise, go to step 140B. In step 152, the pair of the argument D and the second analogy inference D ′ is stored in the memory 94 (see FIG. 3), and the process proceeds to step 140B.

  When the process proceeds from step 144 to step 154, in step 154, the division program 90 is recursively called with the first type inference as an argument. By this recursive call, a divided character string C ′ corresponding to the first class reasoning C is generated, and as a return value, a value (0) indicating normal termination and the divided character string C ′ corresponding to the first class reasoning C are used. The address on the pair of memories 94 is returned. Alternatively, the divided character string C ′ corresponding to the first kind inference C is not generated, and a value (1) representing an error is returned as a return value. If a return value is returned, the process proceeds to step 156. In step 156, the return value in step 154 is determined. If the return value is a value (1) indicating an error, the process proceeds to step 140B. Otherwise, go to step 158. In step 158, the database 88 (see FIG. 4) is replenished with pairs of all sentences and divided character strings stored in the memory 94 at that time. That is, the first type reasoning is stored in the first file 100 (see FIG. 4), and the divided character string for the first type reasoning is stored in the second file. When the replenishment is completed, the process proceeds to step 140B.

  When the above series of processing is completed for all the first kind inferences, the processing of step 126 shown in FIG. 5 is completed. The process proceeds to step 120B in FIG. 5 as described above.

[Operation]
The sentence division system 80 of this embodiment operates as follows.

-Generation and retrieval of first kind reasoning-
First, an operation in which the sentence division system 80 generates a first type inference and searches for the first type inference in the database 88 will be described. FIG. 7 schematically shows an operation of generating a first type inference and searching for the first type inference in the database 88. Referring to FIG. 7, it is assumed that an input sentence 82 “Is there anything more light?” Is given to word segmentation device 86 (see FIG. 3). The input / output unit 92 (see FIG. 3) calls the division program 90 using the input sentence 82 as an argument D200.

  When the dividing program 90 is called, in step 120A shown in FIG. 5, two arbitrary sentences A and B are selected from the first file 100 (see FIG. 4) of the database 88 (see FIGS. 3 and 4). A sentence pair 202 consisting of sentences A and B is formed. Here, from the first file 100 shown in FIG. 4, “Are there more dark colors?” With identification number 2 is selected as sentence A, and the sentence “Is there a lighter color” with identification number 4 as sentence B? Is chosen.

  When the sentence pair 202 is selected, in a subsequent step 122, a three-sentence group 204 composed of an argument D200 and sentences A and B constituting the sentence pair 202 is formed. Then, by symbol string analogy, an attempt is made to generate a first analogy C206 in which A: B :: C: D is established from these three sentences. For example, in the three-sentence group 204 shown in FIG. 7, the sentence B includes the character “dark” in the sentence A as “light” and the character string “color” as “color”. This is the replaced sentence. The character string of the part excluding these two character strings in the sentence B is common to the character string of the part in the sentence A. The argument D includes a character “light” and a character string “Iro”, and the character string of the portion excluding these two character strings is common to the character strings of the corresponding portions in the sentence A and the sentence B. In the method described in Patent Document 1, a part common to sentence A, sentence B, and sentence D, a character “dark” that has a replacement relationship with the character “light” in sentence D, and a character “Iro” in sentence D A new sentence is generated from the character “color” in the relationship of replacement. The generated sentence is a sentence "Is there a darker color?" In this embodiment, this sentence is the first kind inference C206.

  In the method described in Patent Document 1, there is a case in which an analogy relationship is not established in the set of three sentences 204, and the first class sentence C cannot be generated. In this case, a sentence is newly selected from the first file 100 (see FIG. 4), and a new sentence pair 202 is formed. In the method described in Patent Document 1, a plurality of first type inferences may be generated. In this case, the process described below is performed for each of the generated first kind inferences.

  When the first analog reasoning C206 is generated, step 126 is started. When step 126 is started, a sentence that matches the first analogy sentence C206 is searched in the first file 100 of the database 88 (see FIG. 4). In the database 88 shown in FIG. 4, the sentence with the identification number 3 in the first file 100 matches the first analogy sentence 206.

  When a sentence that matches the first kind inference 206 exists in the first file 100, the sentences A and B and the first kind inference C206 that constitute the sentence pair 202 (see FIG. 7) are all in the database 88 ( It exists in the first file 100 of FIG. Therefore, a set of three sentences 220 consisting of three sentences of sentence A, sentence B, and first kind inference C206 is taken as a search result.

  If there is no sentence in the first file 100 (see FIG. 4) that matches the first kind inference, the divided character string corresponding to the first kind inference C206 also does not exist in the database 88. In this case, the division program 90 is recursively called with the first kind inference C206 as an argument, and the database 88 is supplemented with examples. This operation will be described later.

-Generation of second analogy-
FIG. 8 schematically shows an operation of generating a second analog reasoning using the searched three sentence set 220 and an example. Referring to FIG. 8, between the sentence A, sentence B, and first class reasoning sentence C206 (see FIG. 7) constituting the searched three sentence group 220, and the argument D200, the reasoning relation is as described above. A: B :: C: D is established. That is, the argument D200 can be inferred from the set of three sentences 220. Each of the sentence A, the sentence B, and the first analogy sentence C constituting the three sentence set 220 exists in the first file 100 (see FIG. 4). That is, the divided character strings corresponding to these sentences can be obtained from the second file 102 (see FIG. 4). In step 146 (see FIG. 6), further, the divided character strings A ′, B ′, and C ′ corresponding to the sentences A, B, and C constituting the set of three sentences 220 are read from the second file 102, A divided character string set 224 including three divided character strings A ′, B ′, and C ′ is formed. In this example, the sentences of the identification numbers 2, 4, and 3 are read from the second file 102 of the database 88 shown in FIG.

  When the divided character string set 224 is formed, in step 148, a second class inference D'226 is generated from the divided character string set 224 by symbol string analogy. For example, from the divided character string set 224 shown in FIG. 8, a divided character string “more | pale | The divided character strings A ′, B ′, and C ′ correspond to the sentences A, B, and C, respectively. Therefore, the second kind inference D'226 is a divided character string corresponding to the argument D200. When the generation of the second class inference D ′ 226 is successful, the pair 228 of the argument D 200 and the second class inference D ′ 226 is stored in the memory 94 (see FIG. 3). If the analog generation of the second analogy inference D ′ 226 fails, nothing is stored in the memory 94 and the series of processes for the first analogistic inference C 206 is completed.

  When the above processing is completed for all the first kind inferences generated in step 122 shown in FIG. 5, in step 120A shown in FIG. 5, a sentence is selected again from the first file 100 (see FIG. 4) and a new one is newly created. A pair of sentences is formed, and processing for a new pair of sentences is started.

  When the above processing is completed for all sentence pairs, the return value of the division program 90 is set. That is, if the sentence 228 and the divided character string pair 228 are stored in the memory 94 (see FIG. 3), the value (0) indicating normal termination and the sentence 94 and the divided character string pair 228 stored in the memory 94 are stored. The address indicating the position is set as the return value of the division program 90 (FIG. 3). If a pair of a sentence and a divided character string is not stored in the memory 94, a value (1) indicating an error is set as a return value.

  Referring to FIG. 3, when a value (0) indicating normal termination and an address indicating the storage position of a pair 228 of a sentence and a divided character string are given as return values to the input / output unit 92, The input / output unit 92 (see FIG. 3) reads a pair 228 of the sentence and the divided character string stored in the memory based on the return value, and outputs a divided character string 226 constituting the pair 228 of the sentence and the divided character string. Output as sentence 84.

-Supplementation of examples-
Hereinafter, an operation in a case where a sentence that matches the first analogy sentence C206 is not in the first file 100 (see FIG. 4) will be described. If there is no sentence in the first file 100 (see FIG. 4) that matches the first kind inference, the pair of the first kind inference C206 and its divided character string is not stored in the database 88. .

  If such a divided character string corresponding to the first type inference C206 is obtained, the first type inference C206 and the divided character string are added to the first file 100 and the second file 102 of the database 88, respectively. The database 88 can be enriched. Therefore, in step 154 (see FIG. 6), a recursive call of the division program 90 is performed using the first kind inference C206 as a new argument, and an attempt is made to generate a divided character string corresponding to the first kind inference. If a second kind of reasoning for a new argument is generated by recursive call, that is, a new divided character string of the argument is generated, the divided character string is obtained by a return value. If no divided character string is generated, an error is indicated. The value (1) is returned. The sentence corresponding to the generated divided character string is stored in the first file 100 of the database 88 (see FIG. 4), and the generated divided character string is stored in the second file 102.

[Experiment]
-Japanese word segmentation experiment-
Using the method of this embodiment, an experiment was conducted to divide a Japanese sentence into words. During the experiment, a corpus of 15373 sentences was prepared. FIG. 9 shows an example of the prepared sentence. Referring to FIG. 9, the first file 250 lists sentences that are not divided. The second file 252 lists correct answers of the divided character strings obtained when the sentences listed in the first file 250 are divided. In the second file 252, a symbol “|” is inserted between words in the sentence. In the 15373 sentences prepared, the average number of words constituting one sentence was 7.60 and the standard deviation was 3.34.

  In this experiment, each sentence listed in the first file 250 was used as the input sentence 82 (see FIG. 3). When an output sentence 84 is obtained for this input, the output sentence is collated with the second file 252.

  As a result of dividing the prepared 15373 sentences by the method of this embodiment, an output sentence 84 could be obtained for 15370 sentences. As a result of collating the obtained output sentence 84 with the correct answer, it was found that both were correctly divided.

-English word segmentation experiment-
In addition, using the method of this embodiment, an experiment was performed to divide an English sentence into words. During the experiment, a 52848 sentence corpus was prepared. FIG. 10 shows an example of the prepared sentence. Referring to FIG. 10, since the English sentence is divided for each word, the prepared sentence becomes the correct answer in the division. A list of prepared sentences is the second file 262. The first file 260 is a character string that becomes the input sentence 82 in this experiment, and is a list of character strings from which blank characters are removed from the sentences listed in the second file 262. In the prepared 52848 sentences, the number of words constituting one sentence was 5.69 on average, standard deviation 2.12 and 23 at maximum.

  In this experiment, the character strings listed in the first file 260 were used as input sentences. In other words, the division process performed in this experiment is a process of reinserting a blank character in the input sentence. When the output sentence 84 is obtained by this processing, the obtained output sentence 84 is collated with the second file 262.

  As a result of dividing the prepared 52848 sentences in units of words by the method of this embodiment, an output sentence 84 could be obtained for 50897 sentences. As a result of collating the obtained output sentence and the correct answer, out of 50897 output sentences 84, only four sentences were incorrect.

  As described above, the sentence division method in this embodiment is merely to prepare an example consisting of a pair of an arbitrary character string that is not divided and the divided character string, regardless of language, characters, grammatical rules, and the like. The sentence can be divided. Further, since the database is divided while automatically replenishing the database, it is possible to perform the division without preparing a large amount of data in advance or performing the learning stage process.

  Further, unlike the statistical method using the N-gram model, the size of the context to be processed is not fixed, and the input sentence can be processed in a long context of a sentence unit.

  In the above embodiment, the division is performed using only the example as shown in FIG. 4, but the present invention is not limited to such an embodiment. For example, a division method using a dictionary can be used in combination.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the external appearance of the computer system for implement | achieving the sentence division | segmentation system which concerns on one embodiment of this invention. It is a block diagram of the computer system shown in FIG. It is a block diagram which shows the functional structure of the sentence division | segmentation system 80 of one embodiment of this invention. It is a figure which shows the structure of the database 88. FIG. It is a flowchart of the division | segmentation program 90 shown in FIG. It is a flowchart of the process performed in step 126 of FIG. 5 of the search of a 1st kind reason sentence, the production | generation of a 2nd kind reason sentence, and an example. It is a figure which shows schematically the operation | movement which produces | generates a 1st kind inference and searches a 1st kind inference in the database 88. FIG. It is a figure which shows roughly the operation | movement which produces | generates the output sentence 84 from the searched three character strings. It is a figure which shows the outline | summary of the Japanese sentence used for the experiment conducted about one embodiment of this invention. It is a figure which shows the outline | summary of the sentence of English used for the experiment conducted about one embodiment of this invention.

Explanation of symbols

80 sentence division system 82 input sentence 84 output sentence 86 word division device 88 database 90 division program 92 input / output unit 100 first file 102 second file

Claims (3)

A sentence division computer program used with a computer-readable database, including three or more pairs of sentences and divided character strings obtained by dividing the sentence by a predetermined unit,
When a sentence to be processed is given, a sentence division program part for generating a divided character string for the sentence to be processed using the database;
Input / output program part for giving an input sentence given from the outside to the sentence division program part and outputting a divided character string output by the sentence division program part for the input sentence as a divided character string for the input sentence Including
The sentence division program part is:
When the processing target sentence is given, arbitrary two sentences are selected from the database, a predetermined first analogy is generated from the two sentences and the processing target sentence, and the first analogy is generated. A first analogy program part for solving the formula and generating an analogy,
A determination program part for determining whether or not the analogy sentence exists in the database;
In response to determining that the analogy is present in the database by the determination program part, a divided character string paired with the analogy in the database, and the two sentences in the database A read program portion for reading from the database two divided character strings each paired;
Using the three divided character strings read out by the read program part, a second analogical expression having a predetermined relationship with the first analogical expression is generated, and the second analogical expression is solved to generate an analogy divided character A sentence division computer program including a second analogy program part for generating a sequence and outputting it as a return value of the sentence division program part. The first analogy program part is:
Given the sentence to be processed, a program portion for selecting all the ordered pairs (A, B) of sentences in the database;
For each of the selected ordered pairs, a first analogical expression “A: B :: x: D” is generated with the sentence D to be processed, and the first analogical expression is solved to solve The sentence division computer program according to claim 1, further comprising: an analogy solution program part for trying to generate x = C and outputting all the generated solutions C. The sentence division program part is:
A recursive call program for recursively calling the sentence division program portion using the analogy sentence as a processing target sentence in response to the determination program portion determining that the analogy sentence does not exist in the database Part,
The program part for adding the said analogy sentence and the division | segmentation character string output by the said sentence division | segmentation program part called by the said recursive calling program part as a pair to the said database is further included. The sentence division computer program according to claim 2.

Images