JP2020166501A - Segmentation model generation system, text segmentation device and segmentation model generation method - Google Patents

Abstract

To provide a segmentation model generation system and a segmentation model generation method capable of generating a segmentation model for segmenting a sentence, and a text segmentation device capable of generating a sentences with a clear meaning.SOLUTION: A segmentation model generation system 10 comprises an input unit 11 to which first structured document data including a plurality of first sentences and second structured document data including a plurality of second sentences are inputted, a learning data generation unit 12 for generating from among the plurality of first sentences and the plurality of second sentences a combination of the first sentence and the second sentence that are in a predetermined correspondence relation as learning data, a learning data storage unit 13 for storing the generated learning data, and a segmentation model generation unit 14 for learning based on the learning data stored in the learning data storage unit and generating a segmentation model 144 that segments a sentence.SELECTED DRAWING: Figure 1

Description

The present invention relates to a split model generation system, a text segmentation device, and a split model generation method.

Conventionally, there is known a sentence conversion device that generates a sentence that can be easily understood by a user by converting a predetermined sentence into another sentence. For example, a machine translation device uses a computer to automatically translate a given natural language into another natural language. As the dependency structure becomes more complicated, mistranslations and the like are more likely to occur, so that the conversion accuracy decreases as the sentence becomes longer. Therefore, in the sentence conversion device, it is required to shorten the sentence and improve the conversion accuracy by dividing the sentence of the conversion source.

In the technique of Patent Document 1, the division position of the sentence is determined so as to optimize a predetermined objective function. In this case, the machine translation system calculates the likelihood of each of the split sentence candidates according to a predetermined likelihood calculation formula. The machine translation system calculates the similarity between the split sentence candidate and the predetermined language by a predetermined similarity calculation formula. The machine translation system divides the input sentence by selecting the split sentence candidate having the highest score defined by the likelihood and similarity of the sentence. This makes it possible to determine the division position of a long sentence with high reliability.

In the technique of Patent Document 2, the division position is detected from the delimiter such as punctuation marks included in the text and the text format. In this case, the text segmentation device can correctly detect sentence breaks that cannot be represented only by the delimiter and cut out the text one sentence at a time.

Japanese Unexamined Patent Publication No. 2006-18354 Japanese Patent Application Laid-Open No. 6-19662

In Patent Document 1, the input sentence is divided based on the score defined by the likelihood and similarity of the sentence. However, the machine translation system uses the word before the split position to generate the split sentence. The machine translation system uses the words after the split position to generate a split sentence. If the split source sentence has a nested structure, a statement that does not make sense even if split is generated.

In Patent Document 2, the division position is detected from the delimiter included in the text and the format of the text. However, there is no mention of using sentence dependency to detect split positions.

Therefore, the present invention has been made to solve the above-mentioned problems, and is meaningful by using a division model generation system and a division model generation method capable of generating a division model for dividing a sentence, and a division model generation system. Provided is a text segmentation device capable of generating a clear sentence.

The split model generation system is an input unit in which a first document data structured including a plurality of first sentences and a second document data structured including a plurality of second sentences are input. A learning data generation unit that generates training data by combining a combination of the first sentence and the second sentence having a predetermined correspondence relationship among the plurality of first sentences and the plurality of second sentences, and the generated learning data. It has a learning data storage unit that stores data, and a division model generation unit that learns based on the learning data stored in the learning data storage unit and generates a division model that divides a sentence.

According to the present invention, it is possible to generate a sentence having a clear meaning.

Schematic diagram of a text segmentation device. The schematic diagram of the text segmentation apparatus in 1st Example. Hardware configuration diagram of the text segmentation device. Process flow diagram of the split model generation system. Explanatory diagram of the input interface. Explanatory drawing of the learning data selection part. Explanatory drawing of the split model generation system. Explanatory drawing of the learning data division part. Explanatory drawing of evaluation presentation part 26. Explanatory drawing of the output interface. Process flow diagram of the editorial department. The schematic diagram of the text segmentation apparatus in 2nd Example. Processing flow chart of the dependency model. Process flow diagram of the editorial department.

The present embodiment relates to a division model generation system applicable to preprocessing for accurately converting sentences, and a text segmentation device that converts a predetermined sentence into another sentence.

Hereinafter, one embodiment of the present invention will be described with reference to FIG. 1, but the present invention is not limited to the embodiment shown in FIG. This embodiment can be suitably used for structured documents such as patent documents and science and technology documents. The present embodiment can also be applied to structured documents other than patent documents or scientific and technological documents.

FIG. 1 is a schematic view of a text segmentation device 1. The text segmentation device 1 includes, for example, a division model generation system 10, an editing unit 20, a conversion processing unit 30, a conversion evaluation processing unit 40, and a language server 50.

In the figure, the "part" may be omitted. For example, the editorial unit 20 may abbreviate as "edit" in the figure.

The division model generation system 10 is a function of generating a division model 144 that divides a sentence. The divided model generation system 10 includes, for example, an input unit 11, a learning data generation unit 12, a learning data storage unit 13, and a divided model generation unit 14.

The input unit 11 is, for example, a function of receiving two corresponding corpora. In the two corresponding corpora, one corpus is composed of a plurality of converted sentences by organizing the sentences of the other corpus by meaning. The conversion source corpus is an example of "first document data" and includes a plurality of first sentences. The converted corpus is an example of "second document data", and the first sentence includes a plurality of converted second sentences.

The first sentence and the second sentence include a plurality of semantic expressions of a group. A group of semantic expressions is a group of two or more words that express one meaning. For example, the sentence "a library published on a network is called a digital library, and the content is enriched by uploading materials" is called "a library published on a network is called a digital library." Includes the meaning expression "The content of the digital library is enriched by uploading the material." A group of semantic expressions is indicated by a "phrase" in a sentence or a "clause" in a sentence. Hereinafter, a group of semantic expressions may be abbreviated as semantic expressions.

The learning data generation unit 12 is a function of extracting learning data used for learning from each input corpus. The learning data is the data of the first sentence and the data of the second sentence having a predetermined correspondence relationship. The learning data storage unit 13 is a database that stores the extracted learning data. The division model generation unit 14 is a function of learning based on the learning data and generating the division model 144.

The division model generation unit 14, for example, includes a first target data conversion unit 141, a first source data conversion unit 142, an aggregation unit 143, a learning data division unit 144, a second target language conversion unit 145, and a second. It has a source language conversion unit 146 and.

The first target data conversion unit 141 is a function of converting the data of the second sentence into one or a plurality of semantic expression data. The first source data conversion unit 142 is a function of converting the data of the first sentence into one or a plurality of semantic expression data. The aggregation unit 143 is a function of converting the semantic expression data converted by the first target data conversion unit 141 into one semantic expression data shown as a compound sentence or a compound sentence.

The learning data division unit 144, which is an example of the “division model 144”, divides one or more semantic expression data of the first sentence and one semantic expression data of the second sentence into a plurality of semantic expression data. It is a function to do. The second target language conversion unit 145 is a function of converting the semantic expression data of the second sentence divided by the learning data division unit 144 into document format data. The second source language conversion unit 146 is a function of converting the semantic expression data of the first sentence divided by the learning data division unit 144 into document format data.

The editorial unit 20 is a function of dividing a predetermined sentence into a plurality of sentences based on the division model 144. The conversion processing unit 30 is a function of converting a predetermined divided sentence into another sentence. The conversion evaluation processing unit 40 is a function of evaluating the conversion accuracy of the conversion processing unit 30. The language server 50 is a database in which word data is stored.

The division model generation system 10 shown in the present embodiment divides a sentence for each semantic expression by having an input unit 11, a learning data generation unit 12, a learning data storage unit 13, and a division model generation unit 14. It is possible to generate a division model 144 to be used. As a result, the division model generation system 10 can be adopted as a preprocessing for accurately converting a predetermined sentence.

The text segmentation device 1 includes, for example, a division model generation system 10, an editing unit 20, a conversion processing unit 30, a conversion evaluation processing unit 40, and a language server 50, and thus a predetermined division model 144 is used. Sentences can be converted. As a result, the text segmentation device 1 can generate a sentence having a clear meaning.

An embodiment of the text segmentation device 1 shown in FIG. 1 will be described with reference to each drawing.

FIG. 2 is a schematic diagram showing a functional configuration of the text segmentation device 1. The text segmentation device 1 divides the input predetermined sentence according to the meaning, and converts each divided sentence into another sentence. The text segmentation device 1 includes, for example, a division model generation system 10, an editing unit 20, a conversion processing unit 30, a conversion evaluation processing unit 40, and a language server 50.

The division model generation system 10 is a function of generating a division model 144 that divides a sentence. The divided model generation system 10 includes, for example, an input unit 11, a learning data generation unit 12, a learning data storage unit 13, and a divided model generation unit 14.

The input unit 11 is a function of receiving two corresponding corpora. The two corresponding corpora include a source language corpus and a target language corpus, which is a transformation of the source language corpus sentences organized by meaning.

The source language corpus includes the first document data 71 (see FIG. 6). The target language corpus includes the second document data 72. The first document data 71 is structured to include a plurality of first sentences. The second document data 72 is structured to include a plurality of second sentences. The input unit 11 transmits the first document data 71 and the second document data 72 to the learning data generation unit 12. The input unit 11 will be described later with reference to FIG.

The first sentence and the second sentence are either one sentence containing a plurality of semantic expressions or a plurality of simple sentences. A sentence containing a plurality of semantic expressions is, for example, a compound sentence or a compound sentence. Hereinafter, a sentence containing a plurality of semantic expressions may be abbreviated as "multiple semantic sentences". A simple sentence is a sentence composed of one semantic expression.

The learning data generation unit 12 is a function of generating as learning data a combination of the first sentence and the second sentence having a predetermined correspondence relationship among the plurality of first sentences and the plurality of second sentences. The learning data generation unit 12 transmits the learning data to the learning data storage unit 13. The learning data generation unit 12 will be described later with reference to FIG.

In the predetermined correspondence relationship, the combination of the first sentence and the second sentence includes at least one of a one-to-many relationship, a many-to-one relationship, and a many-to-many relationship. The relationship between the first sentence and the second sentence is a case where the semantic expression included in the first sentence and the semantic expression included in the second sentence have the same meaning. The one-to-many relationship is a relationship in which the first sentence is one plural meaning sentence and the second sentence is a plurality of simple sentences. The many-to-one relationship is a relationship in which the first sentence is a plurality of simple sentences and the second sentence is a single plural meaning sentence. The many-to-many relationship is a relationship in which the first sentence and the second sentence are each a plurality of simple sentences.

The learning data storage unit 13 is a database for recording learning data. The learning data storage unit 13 supplies the learning data to the division model generation unit 14.

The division model generation unit 14 is a function of learning based on the learning data and generating the division model 144. The division model generation unit 14 divides the first sentence and the second sentence for each semantic expression, and generates the division model 144 based on the divided first sentence and the divided second sentence. The division model generation unit 14 transmits the division model to the division model storage unit 23 described later. The division model generation unit 14 will be described later with reference to FIG. One example of the division model 144 is the learning data division unit 144 described in FIG. Therefore, the reference numeral 144 can be given to the division model 144.

The editorial unit 20 is a function of dividing a predetermined sentence for each semantic expression based on the division model 144 generated by the division model generation system 10. The editing unit 20 converts, for example, the input unit 21, the division target selection unit 22, the division model storage unit 23, the division unit 24, the division evaluation unit 25, the evaluation presentation unit 26, and the division result extraction unit 27. It has a data storage unit 28.

The input unit 21 is a function of receiving predetermined document data 331 (see FIG. 11). The predetermined document data 331 is data structured including a plurality of sentences. The input unit 21 outputs data to the division target selection unit 22.

The division target selection unit 22 is a function of selecting a sentence to be divided from the predetermined document data 331. The division target selection unit 22 transmits the data of the sentence to be divided to the division unit 24. The division target selection unit 22 may be operated by the user to extract a set of sentences to be divided from the predetermined document data 331.

The division model storage unit 23 is a database that stores the division model 144. The division model storage unit 23 transmits the division model 144 to the division unit 24. The division unit 24 is a function of dividing one sentence to be divided into a plurality of sentences based on the division model 144. The division unit 24 transmits the data of the divided sentence to the division evaluation unit 25.

The division evaluation unit 25 is a function of evaluating the division result of the division unit 24 based on a predetermined evaluation index. The predetermined evaluation index is, for example, the fluency of the sentence of the division result. The divided evaluation unit 25 transmits the evaluation result to the evaluation presentation unit 26.

The evaluation presentation unit 26 has a function of presenting an evaluation value to the user and a function of accepting a threshold value set by the user. The user selects a sentence suitable for the conversion process from the division result by setting the threshold value based on the evaluation value presented by the evaluation presentation unit 26. The evaluation presentation unit 26 transmits the evaluation value and the threshold value to the division result extraction unit 27. The evaluation presentation unit 26 will be described later with reference to FIG.

The division result extraction unit 27 is a function of extracting a sentence to be converted by the conversion processing unit 30 from a plurality of sentences divided by the division unit 24 based on the evaluation of the division evaluation unit 25. The division result extraction unit 27 transmits the data of the sentence to be converted to the conversion data storage unit 28. The conversion data storage unit 28 is a database that stores the data of the sentence to be converted. The conversion data storage unit 28 transmits the data of the sentence to be converted to the conversion unit 31.

The conversion processing unit 30 is a function of converting a plurality of divided sentences into other sentences. The conversion processing unit 30 includes, for example, a conversion unit 31, a conversion software storage unit 32, and an output unit 33.

The conversion unit 31 is a function of performing conversion processing on the text data stored in the conversion data storage unit 28. The conversion unit 31 converts a sentence using the conversion software from the conversion software storage unit 32 and the text data acquired from the conversion data storage unit 28, and outputs the result to the output unit 33. The conversion software storage unit 32 is a database that stores the conversion software used by the conversion unit 31. The conversion software storage unit 32 transmits the data of the conversion software to the conversion unit 31. The output unit 33 is a function of outputting the conversion result. The output unit 33 will be described later with reference to FIG.

The conversion evaluation processing unit 40 is a function of evaluating the conversion accuracy of the conversion processing unit 30. The conversion evaluation processing unit 40 includes, for example, a conversion evaluation unit 41 and an output unit 42.

The conversion evaluation unit 41 is a function of evaluating the conversion accuracy of the conversion unit 31. The conversion evaluation unit 41 transmits the result of the conversion evaluation to the output unit 42. The output unit 42 is a function of outputting an evaluation value of conversion accuracy. The output unit 42 will be described later with reference to FIG.

The language server 50 is a database that stores word data used by the division unit 24 and the conversion unit 31. The language server 50 transmits word data to the division unit 24 and the conversion unit 31.

FIG. 3 is a hardware configuration diagram of the text segmentation device. The text dividing device 1 includes, for example, an input / output device 61, a storage unit 62, a CPU (Central Processing Unit) 63, a memory 64, and a communication interface (in the figure, it may be referred to as a communication I / F (InterFace)). It has 65, an input / output circuit 66, and a data transmission line 67 for bidirectionally connecting the functions 61 to 66.

The input / output device 61 has an input device and an output device (neither of them is shown). The input device is, for example, a voice input device such as a pointing device microphone such as a keyboard or a mouse. The output device is, for example, a display, a printer, a voice synthesizer, or the like. Further, the input device and the output device may be integrated like a tablet or an AR (Augmented Reality) display. The input / output device 61 may be provided in a computer different from the computer in which the text segmentation device 1 is provided. For example, a personal computer, a mobile phone (including a so-called smartphone), a personal digital assistant, or the like may be used as the input / output device 61 to the text segmentation device 1.

The storage unit 62 is, for example, a hard disk or a non-volatile storage device such as an SSD (Solid State Drive). The type of storage medium does not matter. The storage unit 62 stores a computer program (hereinafter referred to as a program) for realizing the division model generation system 10, the editing unit 20, the conversion processing unit 30, and the conversion evaluation processing unit 40, respectively. The storage unit 62 also stores a database such as the language server 50.

The CPU 63 reads and executes each program from the storage unit 62 via the memory 64. The memory 64 is, for example, a volatile storage device such as "RAM (Random Access Memory)".

The communication interface 65 is a device that communicates with an external device via a communication network such as a LAN (Local Area Network) or the Internet.

The input / output circuit 66 is a terminal to which a storage medium 68 such as a USB (Universal Serial Bus) memory or an optical disk is connected. The input / output circuit 66 is, for example, a USB port, a connector, or the like.

The storage medium 68 is a portable storage medium. The storage medium 68 may store a program that realizes the divided model generation system 10. The user may provide the division model generation system 10 on the computer by installing the program of the division model generation system 10 stored in the storage medium 68 on the computer. ..

The program that realizes the split model generation system 10 may be stored in a predetermined server on the cloud. In this case, the user may realize the divided model generation system 10 on the computer by installing the program of the divided model generation system 10 on the computer via the communication interface 65 from the cloud.

Hereinafter, the division model generation system 10 is based on the example that the first document data 71 is a patent specification described in Japanese and the second document data 72 is a patent specification described in English. The process will be described. The first document data 71 and the second document data 72 are not limited to patent specifications, but may be structured documents such as research papers.

FIG. 4 is a processing flow chart of the split model generation system 10. The division model generation system 10 is executed by the processing of the input unit 11 (S11), the processing of the learning data generation unit 12 (S12, S13), and the processing of the division model generation unit 14 (S14, S15).

The split model generation system 10 is started by, for example, a user operation. The process of the split model generation system 10 moves to the process (S11) of the input unit 11. The input unit 11 causes the input / output device 61 to display input fields for the source language corpus and the target language corpus.

FIG. 5 is an explanatory diagram of the input unit 11. The input unit 11 causes the input / output device 61 to display, for example, the source language file input field 111, the target language file input field 112, the execute button 113, and the cancel button 114.

The source language file input field 111 is a function of accepting a source language corpus as a source language file. The target language file input field 112 is a function of accepting the target language corpus as the target language file. The execution button 113 is a button for executing the division model generation processing (S12 to S15). The cancel button 114 is a button for canceling the display of the input unit 11.

The user may enter the directory of the source language corpus file in the source language file input field 111. The user may enter the directory of the target language corpus file in the target language file input field 112. Alternatively, a pull-down menu may be provided in columns 111 and 112, and a file may be specified from the pull-down menu. Further, the file may be input by dropping the file icon in the fields 111 and 112.

Returning to FIG. 4, the learning data generation unit 12 receives the data of the source language corpus and the data of the target language corpus from the input unit 11. The learning data generation unit 12 extracts learning data from the first document data 71 and the second document data 72 (S12).

FIG. 6 is an explanatory diagram of the learning data generation unit 12. The first document data 71 and the second document data 72 have a plurality of items arranged in a predetermined order.

The first document data 71 has a plurality of items c1 to c7 indicated in Japanese. Item c1 indicates, for example, a “document name”. Item c2 indicates, for example, "the name of the invention". Item c3 indicates, for example, "technical field". Item c4 indicates, for example, "background technology". Item c5 indicates, for example, "Prior Art Document". Item c6 indicates, for example, "outline of the invention". Item c7 indicates, for example, "the problem to be solved by the invention".

The second document data 72 has a plurality of items c8 to c14 shown in English. Item c8 indicates, for example, "DESCPRIPTION". Item c9 indicates, for example, "Title of Invention". For example, item c10 indicates "Technical Field". Item c11 indicates, for example, "Background Art". Item c12 indicates, for example, "Citation List". Item c13 indicates, for example, "Summery of Invention". Item c14 indicates, for example, "Technical Problem".

Item c1 corresponds to item c8. Item c2 corresponds to item c9. Item c3 corresponds to item c10. Item c4 corresponds to item c11. Item c5 corresponds to item c12. Item c6 corresponds to item c13. Item c7 corresponds to item c14.

The learning data generation unit 12 sets the first sentence and the second sentence between the corresponding items of the items c1 to c7 of the first document data 71 and the items c8 to c14 of the second document data 72. Extract. For example, it is assumed that the learning data generation unit 12 selects the item c4 of the first document data 71. In this case, the learning data generation unit 12 extracts "JP: Sen (Sentence)" as the first sentence from the plurality of sentences shown in the item c4. Here, "JP: Sen" is, for example, a plural semantic sentence including a plurality of (for example, three) semantic expressions. The first sentence as the learning data is not limited to the plural semantic sentence including three semantic expressions, and may be a plural semantic sentence including two or four or more semantic expressions.

As an example of "JP: Sen", "a tire contains polypropylene having a melting point of 160 ° C. or higher, and the content of polypropylene having a melting point of 160 ° C. or higher is 60% by mass or less of the entire resin composition. Including the composition. " Therefore, "JP: Sen" includes a first semantic expression of "a tire containing a resin composition" and a second semantic expression of "the resin composition contains polypropylene having a melting point of 160 ° C. or higher". , "The content of polypropylene contained in the resin composition having a melting point of 160 ° C. or higher is 60% by mass or less of the entire resin composition", and three semantic expressions are included.

The learning data generation unit 12 selects "EN: Seg1", "EN: Seg2", and "EN: Seg3" as the second sentence from the plurality of sentences shown in the item c11 of the second document data 72. Extract. "EN: Seg1", "EN: Seg2" and "EN: Seg3" are simple sentences written in English.

"EN: Seg1" is, for example, the sentence "A tire which compactions a resin composition." "EN: Seg2" is, for example, the sentence "The resin composition contours a polypropylene having having a melting point of 160 ° C or more." "EN: Seg3" is, for example, "The content of the polypropylene having a melting point of 160 ° C or more is 60% by mass or less of the statement".

"JP: Sen" and "EN: Seg1", "EN: Seg2" and "EN: Seg3" have a one-to-many relationship. That is, "JP: Sen" has the semantic expressions of the three sentences "EN: Seg1", "EN: Seg2" and "EN: Seg3".

The learning data generation unit 12 extracts learning data from a combination having a predetermined correspondence between the first sentence and the second sentence based on at least one of the number of words and the translation accuracy. May be good. The learning data generation unit 12 may extract learning data based on other criteria, not limited to at least one of the number of words and the translation accuracy.

The learning data generation unit 12 may extract, for example, a combination of a first sentence having 10 or more words and a second sentence having 80 or less words. The learning data generation unit 12 is not limited to the combination of the first sentence having 10 or more words and the second sentence having 80 words or less, and uses the learning data based on the number of other words. It may be extracted.

The learning data generation unit 12 extracts, for example, a combination of the first sentence and the second sentence in which the alignment accuracy between the first sentence and the second sentence satisfies "0.1" or more. The alignment accuracy between the first sentence and the second sentence may be calculated using an index such as BLEU (BiLingual Evaluation Environment) and RIBES (Rankbased Intuitive Bilingual Score). The learning data generation unit 12 is not limited to the alignment accuracy of "0.1" or more, and may extract learning data based on other values.

Returning to FIG. 4, the learning data generation unit 12 records the learning data in the learning data storage unit 13 (S13). The learning data generation unit 12 records, for example, the data of “JP: Sen” and the data of “EN: Seg1”, “EN: Seg2” and “EN: Seg3” in the learning data storage unit 13.

The division model generation unit 14 acquires learning data from the learning data storage unit 13 and learns the division model 144 (S14). FIG. 7 is an explanatory diagram of the divided model generation unit 14. The division model generation unit 14 includes, for example, encoders 141 and 142, aggregation unit 143, learning data division unit 144, and decoders 145 and 146.

The encoder 141 corresponds to the first target data conversion unit 141 shown in FIG. The encoder 142 corresponds to the first source data conversion unit 142 shown in FIG. The decoder 145 corresponds to the second target language conversion unit 145 shown in FIG. The decoder 146 corresponds to the second source language conversion unit 146 shown in FIG.

The encoder 141 is a function of converting the data of the second sentence into semantic expression data. Semantic expression data is data indicating the semantic expression of a sentence. The semantic expression data is, for example, an n-dimensional vector. n is a predetermined constant. The encoder 141 converts, for example, the data of "EN: Seg1", the data of "EN: Seg2", and the data of "EN: Seg3" into "EN: h1", "EN: h2", and "EN: h3", respectively. Convert. “EN: h1”, “EN: h2”, and “EN: h3” are semantic expression data having one semantic expression, respectively.

The encoder 142 is a function of converting the data of the first sentence into semantic expression data. The encoder 142 converts, for example, the data of “JP: Sen” into “JP: h”. "JP: h" is semantic expression data including three semantic expressions.

The aggregation unit 143 is a function of calculating the semantic expression data as a plurality of semantic sentences by aggregating a plurality of semantic expression data. The aggregation unit 143 aggregates, for example, a plurality of semantic expression data of the second sentence.

The aggregation unit 143 acquires, for example, the data of “EN: h1”, “EN: h2”, and “EN: h3” from the encoder 141. The aggregation unit 143 aggregates the data of "EN: h1", "EN: h2" and "EN: h3", and acquires "EN: h". “EN: h” is semantic expression data when the second sentence is shown as a plural semantic sentence.

The aggregation unit 143 may aggregate a plurality of semantic expression data by a method of adding a plurality of semantic expression data, a method of setting a time-series dependency between the plurality of semantic expression data, and the like. The aggregation unit 143 is not limited to the method of adding a plurality of semantic expression data and the method of setting a time-series dependency between the plurality of semantic expression data, and aggregates the plurality of semantic expression data by another method. May be good.

The learning data division unit 144 is a function of dividing one semantic expression data into a plurality of semantic expression data by using the learning function. The learning data division unit 144 divides, for example, "JP: h" and "EN: h".

The learning data division unit 144 acquires "JP: h" from the encoder 142, for example. The learning data division unit 144 calculates "JP: hs1", "JP: hs2", and "JP: hs3" by, for example, dividing "JP: h" using the learned division method. To do. “JP: hs1”, “JP: hs2”, and “JP: hs3” are semantic expression data indicating one semantic expression in the first sentence.

The learning data division unit 144 acquires "EN: h" from, for example, the aggregation unit 143. The learning data division unit 144 calculates "EN: hs1", "EN: hs2", and "EN: hs3" by, for example, dividing "EN: h". “EN: hs1”, “EN: hs2”, and “EN: hs3” are semantic expression data indicating one semantic expression of the second sentence.

The learning data division unit 144 learns the division method so as to restore a plurality of semantic expression data before aggregation from the semantic expression data after aggregation, for example. The learning data division unit 144 learns the division method so as to restore "EN: h1", "EN: h2", and "EN: h3" from "EN: h1", for example.

The decoder 145 is a function of converting data indicating the meaning of the divided second sentence into a document format. The decoder 145 converts, for example, "EN: hs1", "EN: hs2" and "EN: hs3" into "EN: Seg1", "EN: Seg2" and "EN: Seg3" described in English, respectively. To do. "EN: Seg1" is data obtained by converting the data indicating the meaning of "EN: hs1" into a document format. "EN: Seg2" is data obtained by converting data indicating the meaning of "EN: hs2" into a document format. "EN: Seg3" is data obtained by converting the data indicating the meaning of "EN: hs3" into a document format.

The decoder 146 is a function of converting data indicating the meaning of the divided first sentence into a document format. The decoder 146, for example, translates "JP: hs1", "JP: hs2" and "JP: hs3" into "JP: Seg1", "JP: Seg2" and "JP: Seg3" described in Japanese, respectively. Convert. “JP: Seg1” is data obtained by converting data indicating the meaning of “JP: hs1” into a document format. “JP: Seg2” is data obtained by converting data indicating the meaning of “JP: hs2” into a document format. “JP: Seg3” is data obtained by converting data indicating the meaning of “JP: hs3” into a document format.

The divided model generation unit 14 includes means for minimizing the difference between the semantic expression data of the divided first sentence and the semantic expression data of the divided second sentence. The means for minimizing the difference is, for example, finding the square of the difference, but is not limited to this.

The encoders 141 and 142, the learning data dividing unit 144, and the decoders 145 and 146 may be, for example, a neural network such as an RSTM (Long Short-Term Memory) or a GRU (Gated Recurrent Unit).

FIG. 8 is an explanatory diagram of the learning data division unit 144. The learning data division unit 144 calculates, for example, the semantic expression data to be divided next based on the learning data and the divided semantic expression data.

The learning data division unit 144 extracts one predetermined semantic expression data based on, for example, either the data of the first sentence or the data of the second sentence and a predetermined value. As the predetermined value, the already divided semantic expression data, the vector obtained by adding the "Attention score" of the already divided sentence, or the like may be used. If there is no divided semantic expression data, a predetermined initial value may be input as the predetermined value.

The already divided semantic expression data is the previously divided semantic expression data. The vector to which the "Attention score" of the already divided sentence is added is included in the document format data which is the output of the decoder 146 based on the semantic expression data divided by the learning data dividing unit 144 immediately before. It is a vector obtained by adding "Attention score" used when outputting the word. The "Attention score" is a value indicating where in the input sentence the generated word focuses.

The learning data division unit 144 may include means for suppressing duplication of semantic expressions between the plurality of divided sentences. The learning data dividing unit 144 includes, for example, a means for extracting predetermined semantic expression data using a predetermined value that does not include a vector obtained by adding the "Attention score" of the sentence previously divided. May be good. The learning data division unit 144 may include, for example, a means for deleting the already divided semantic expression data from the semantic expression data before division and extracting one predetermined semantic expression data from the remaining semantic expression data. Good.

Returning to FIG. 4, the division model generation unit 14 stores the generated division model 144 in the division model storage unit 23 (S15). The division model generation unit 14 may store the division model 144 having the highest division accuracy among the generated plurality of division models 144 in the division model storage unit 23. The split model generation system 10 ends after the process (S15).

FIG. 9 is a processing flow chart of the editorial unit 20. The editing unit 20 processes the input unit 21 (S21), the division target selection unit 22 (S22), the division unit 24 (S23), the division evaluation unit 25 (S24), and the evaluation presentation. It is executed by the process of the unit 26 (S25) and the process of the division result extraction unit 27 (S26 to S28). The editorial unit 20 is started by a user operation.

The process of the editorial unit 20 moves to the process (S21) of the input unit 21. Predetermined document data 331 (see FIG. 11) is input to the input unit 21 by the user (S21). The input unit transmits predetermined document data 331 to the division target selection unit 22.

The division target selection unit 22 extracts a sentence to be divided processing from the predetermined document data 331 (S22). The division target selection unit 22 extracts, for example, a sentence to be divided from the predetermined document data 331 based on the number of words in one sentence. The division target selection unit 22 extracts, for example, one sentence having a predetermined number of words or more as a sentence to be divided. The division target selection unit 22 may extract sentences to be divided based on other criteria, not limited to the number of words. The division target selection unit 22 transmits the data of the extracted sentence to be divided to the division unit 24.

The division unit 24 divides the sentence to be divided into a plurality of sentences for each semantic expression based on the division model 144 stored in the division model storage unit 23 (S23). The division unit 24 generates a divided sentence by using the word data stored in the language server 50. The division unit transmits the division result to the division evaluation unit 25.

The division evaluation unit 25 evaluates the accuracy of the division result (S24). The division evaluation unit 25 evaluates the accuracy from the viewpoints of, for example, the degree of retention of the semantic expression in the sentence before and after the division and the fluency of the sentence after the division.

The division evaluation unit 25 uses, for example, the cosine distance between the vector representing the semantic expression of the original sentence and the vector obtained by combining the semantic expression of the sentence after division, thereby expressing the semantic expression. Evaluate retention. The division evaluation unit 25 evaluates fluency, for example, by using a language model suitable for the divided language. The divided evaluation unit 25 transmits the evaluation result to the evaluation presentation unit 26.

FIG. 10 is an explanatory diagram of the evaluation presentation unit 26. The evaluation presentation unit 26 causes the input / output device 61 to display, for example, the evaluation result 261, the threshold value input field 262, the execute button 263, and the cancel button 264. The evaluation result 261 is an evaluation result of the division evaluation unit 25. The evaluation presentation unit 26 outputs, for example, the retention of the semantic expression as "0.9" and the fluency as "0.8".

The threshold value input field 262 is a function of accepting a predetermined threshold value. The predetermined threshold value is a value used when the division result extraction unit 27 selects the data of the sentence to be converted from the division result. The user sets a predetermined threshold value based on the evaluation result 261 of the division result (process (S24) of FIG. 9).

The execution button 263 is a button for starting the division result extraction unit 27. The cancel button 264 is a button for ending the process of the editorial unit 20.

Returning to FIG. 9, the division result extraction unit 27 compares the predetermined threshold value with the evaluation value of each divided sentence (S26). When the evaluation value of the divided sentence is higher than the predetermined threshold value (S26: Yes), the data of the divided sentence is stored in the conversion data storage unit 28 (S27). When the evaluation value of the divided sentence is lower than the predetermined threshold value (S26: No), the data of the sentence before the division is stored in the conversion data storage unit 28 (S28). After the processing of the division result extraction unit 27 (S27, 28), the processing of the editorial unit 20 ends.

FIG. 11 is an explanatory diagram of the output units 33 and 42. The output unit 33 causes the input / output device 61 to display the predetermined document data 331 and the conversion result 332. The conversion result 332 is a document converted by the conversion unit 31.

The conversion unit 31 acquires sentence data from the conversion data storage unit 28. The conversion unit 31 converts the acquired sentence by using the conversion software stored in the conversion software storage unit 32. The conversion unit 31 generates the converted sentence by acquiring the word data from the language server 50. The conversion unit 31 transmits the data of the sentence before conversion and the data of the sentence after conversion to the output unit 33. Sentence conversion includes machine translation for technical documents, dialogue via voice data, paraphrase generation focusing on the comprehensibility of sentences, and the like.

The conversion unit 31 may generate the converted sentence by preferentially using the words included in the predetermined document data 331 over the word data stored in the language server 50. As a result, the conversion unit 31 can suppress fluctuations in the notation or errors in the symbols.

The output unit 42 causes the input / output device 61 to display the evaluation 421 of the conversion accuracy. The output unit 42 displays, for example, the retention of the semantic expression as "0.9" as the evaluation 421 of the conversion accuracy. The output unit 42 displays fluency as "0.8" as, for example, evaluation 421 of conversion accuracy.

The conversion accuracy evaluation 421 is calculated by the conversion evaluation unit 41. The conversion evaluation unit 41 acquires the data of the sentence after the division from the division unit 24, and acquires the conversion result 332 from the conversion unit 31. The conversion evaluation unit 41 calculates the degree of retention of the semantic expression by, for example, calculating the closeness of the semantic expression between the divided sentence data and the conversion result 332.

The conversion evaluation unit 41 may evaluate the conversion result by using the division model 144 stored in the division model storage unit 23. The conversion evaluation unit 41 uses, for example, the division model 144 to convert the data of the sentence after division and the sentence after conversion into semantic expression data. The conversion evaluation unit 41 compares each semantic expression data and calculates the closeness of the semantic expression between the divided sentence data and the conversion result.

The divided model generation system 10 shown in this embodiment has an input unit 11, a learning data generation unit 12, a learning data storage unit 13, and a division model generation unit 14, so that the sentence is divided for each semantic expression. It is possible to generate a division model 144 to be used. As a result, the division model generation system 10 can divide the sentence into units suitable for conversion. As a result, the quality of sentences such as machine translation processing or natural language processing can be improved.

In the case of technical documents such as patent specifications, there are cases where sentences have a nested structure. If a long sentence is simply divided into a plurality of sentences that are independent of each other, the words that are originally related to each other may be separated into front and back. By generating the division model 144 in which the division model generation system 10 divides the sentence for each semantic expression, the editorial unit 20 can divide the sentence while maintaining the dependency relationship.

When a long sentence is simply divided, it is possible that the later sentence is completely separated from the previous context. The learning data division unit 144 of the division model generation unit 14 refers to the data related to the already divided sentence and extracts the semantic expression data to be divided next, so that the anaphoric relationship with the previous sentence is clear. Can generate statements.

Since there are differences between languages in expressions suitable for translation processing, it is difficult to infer expressions suitable for the target language from the source sentence. The division model generation system 10 learns the division model 144 using the corpus created by the source language of the translation source and the target language of the translation destination, and takes into consideration the characteristics of the target language, and then the translation source. Can be divided into processing units suitable for translation.

The learning data generation unit 12 generates the combination of the first sentence and the second sentence having a predetermined correspondence relationship as learning data, so that the division model generation unit 14 compares the first sentence and the second sentence. The split model 144 can be generated. As a result, the division model generation unit 14 can generate the division model 144 that divides each semantic expression.

Since the first document data 71 and the second document data 72 have a plurality of items arranged by predetermined items, the learning data generation unit 12 extracts learning data based on each item. be able to. As a result, the learning data generation unit 12 can efficiently extract the learning data as compared with the state where there is no reference for each item.

The division model generation unit 14 generates a division model by dividing the first sentence and the second sentence for each semantic expression. As a result, the division model generation unit 14 can generate a division model that divides each semantic expression.

The text segmentation device 1 can divide a predetermined document data 331 based on the division model 144 by including the division model generation system 10 and the editorial unit 20. As a result, the text segmentation device can divide the predetermined document data 331 for each semantic expression.

Further, the text segmentation device 1 can convert the predetermined document data 331 into another sentence by including the conversion processing unit 30. As a result, the text segmentation device 1 can translate the predetermined document data 331 and proofread the predetermined document data 331.

By providing the division evaluation unit 25 and the division result extraction unit 27, the editorial unit 20 can select a sentence suitable for conversion from the sentences divided by the division unit 24. As a result, the quality of the sentence converted by the conversion processing unit 30 can be improved.

Further, the text segmentation device 1 is provided with the conversion evaluation unit 40, so that the conversion result can be evaluated using the division model 144. As a result, the conversion evaluation unit 40 can evaluate the conversion result using the semantic expression data that does not depend on the structural difference between the languages.

Since this embodiment corresponds to a modified example of the first embodiment, the differences from the first embodiment will be mainly described. FIG. 12 is a schematic view of the text segmentation device 1a. The text segmentation device 1a divides one sentence into a plurality of sentences by using the dependency model described later. The text segmentation device 1a includes, for example, a division model generation system 10a, an editing unit 20a, a conversion processing unit 30, a conversion evaluation processing unit 40, and a language server 50.

The division model generation system 10a is a function of generating a division model 144 that divides a sentence and a dependency model that divides a sentence based on the dependency relationship. The divided model generation system 10 includes, for example, an input unit 11, a learning data generation unit 12, a learning data storage unit 13a, a division model generation unit 14a, and a dependency model generation unit 15.

The learning data storage unit 13a is a database for recording learning data. The learning data storage unit 13a supplies learning data to the division model generation unit 14a and the dependency model generation unit 15.

The division model generation unit 14a is a function of generating a division model 144 that divides a document based on learning data. The division model generation unit 14a divides the first sentence and the second sentence for each semantic expression, and generates the division model 144 based on the divided first sentence and the divided second sentence. The division model generation unit 14a stores the division model 144 in the division model storage unit 23. The division model generation unit 14a transmits the division model 144 to the dependency model generation unit 15. The division model generation unit 14a may acquire the dependency model from the dependency model generation unit 15 and generate the division model 144 based on the dependency model.

The dependency model generation unit 14a is a function of generating a dependency model that divides a sentence based on the dependency relationship of the sentence. The dependency model generation unit 14a generates a dependency model based on the learning data and the division model 144 generated by the division model generation unit 14a. The dependency model generation unit 15 stores the dependency model in the division model storage unit 23a.

The editorial unit 20a is a function of dividing one predetermined sentence into a plurality of sentences based on at least one of the division model 144 and the dependency model. The editing unit 20a includes, for example, an input unit 21, a division target selection unit 22, a division model storage unit 23a, a division unit 24a, a division evaluation unit 25, an evaluation presentation unit 26, a division result extraction unit 27, and the like. It has a conversion data storage unit 28.

The division model storage unit 23a is a database that stores the division model 144 and the dependency model. The division model storage unit 23a transmits at least one of the division model 144 and the dependency model to the division unit 24a. The division unit 24a is a function of dividing one sentence to be divided into a plurality of sentences based on at least one of the division model 144 and the dependency model. The division unit 24a transmits the data of the sentence after the division to the division evaluation unit 25.

FIG. 13 is a processing flow chart of the dependency model generation unit 15. The dependency model generation unit 15 acquires learning data from the learning data storage unit 13a (S31). The dependency model generation unit 15 acquires, for example, the first sentence “JP: Sen” from the learning data storage unit 13a (see FIG. 6).

The dependency model generation unit 15 acquires the division result of the learning data from the division model generation unit 14a (S32). The dependency model generation unit 15 acquires, for example, "JP: Seg1", "JP: Seg2", and "JP: Seg3" (see FIG. 7).

The dependency model generation unit 15 generates a dependency model (S33). The dependency model generation unit 15 is one of "JP: Sen" by, for example, comparing "JP: Sen" with "JP: Seg1", "JP: Seg2" and "JP: Seg3". Learn sentence dependency relationships. The dependency model generation unit 15 generates the learning result as a dependency model. The dependency model generation unit 15 stores the dependency model in the split model storage unit 23a (S34).

FIG. 14 is a processing flow chart of the editorial unit 20a. The editing unit 20a includes processing of the input unit 21 (S21), processing of the division target selection unit 22 (S22), processing of the division unit 24a (S41, S42), processing of the division evaluation unit 25 (S24), and so on. It is executed by the processing of the evaluation presentation unit 26 (S25) and the processing of the division result extraction unit 27 (S26 to S28). The editorial unit 20a is started by a user operation.

The processing of the editing unit 20a moves to the processing (S21) of the input unit 21, and the input unit 21 acquires the predetermined document data 331. The division target selection unit 22 extracts a sentence to be divided from the predetermined document data 331 (S22).

The division unit 24a acquires at least one of the division model 144 and the dependency model from the division model storage unit 23a (S41). The division unit 24a divides the division target data based on at least one of the division model 144 and the dependency model (S42). The division unit 24a transmits the division result to the division evaluation unit 25.

The text segmentation device 1a shown in this embodiment includes the dependency model generation unit 15, so that one sentence can be divided into a plurality of sentences based on the dependency relationship of the sentences. As a result, the editorial unit 20a can divide the sentence into a length suitable for converting the document even when proofreading the document in which the language type does not change. As a result, the text segmentation device 1 can improve the quality of sentence conversion.

A computer program for making a computer function as a split model generation system is structured on a computer by including a first document data structured including a plurality of first sentences and a plurality of second sentences. The combination of the input unit into which the second document data is input and the first sentence and the second sentence having a predetermined correspondence relationship among the plurality of first sentences and the plurality of second sentences is used as learning data. A division model that generates a division model 144 that learns based on the training data generation unit to be generated, the learning data storage unit that stores the generated learning data, and the learning data stored in the training data storage unit, and divides the sentence. Realize each of the generators

The present invention is not limited to the above-described embodiment, and includes various modifications. The above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to those having all the described configurations. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, other configurations can be added / deleted / replaced with respect to a part of the configurations of each embodiment.

Each of the above configurations, functions, processing units, processing means, and the like may be realized by hardware, for example, by designing a part or all of them with an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, an SD card, or a DVD.

Further, the technical features included in the above-described embodiment are not limited to the combinations specified in the claims, and can be appropriately combined.

1 ... text division device, 10 ... division model generation system, 11 ... input unit, 12 ... training data generation unit, 13 ... training data storage, 14 ... division model generation unit, 141 ... first target data conversion unit, 142 ... first 1 Source data conversion unit, 143 ... Aggregation unit, 144 ... Learning data division unit, 145 ... Second target data conversion unit, 146 ... Second source data conversion unit, 20 ... Editing unit, 30 ... Conversion processing unit, 40 ... Conversion Evaluation processing unit, 50 ... Language server

Claims (10)

An input unit for inputting a first document data structured including a plurality of first sentences and a second document data structured including a plurality of second sentences.
A learning data generation unit that generates learning data by combining a combination of the first sentence and the second sentence having a predetermined correspondence relationship among the plurality of first sentences and the plurality of second sentences.
A learning data storage unit that stores the generated learning data,
A division model generation system including a division model generation unit that learns based on the learning data stored in the training data storage unit and generates a division model that divides a sentence. The division according to claim 1, wherein the predetermined correspondence relationship is such that the combination of the first sentence and the second sentence includes at least one of a one-to-many relationship, a many-to-one relationship, and a many-to-many relationship. Model generation system. The first document data and the second document data have a plurality of items arranged in a predetermined order, and the learning data generation unit has a plurality of items.
The division model generation according to claim 1, wherein the first sentence and the second sentence are extracted between the corresponding items of each item of the first document data and each item of the second document data. system. The first sentence and the second sentence having a predetermined correspondence relationship include a plurality of a set of semantic expressions.
The split model generator
The first sentence and the second sentence are divided according to the semantic expression of the group.
The divided model generation system according to claim 1, wherein the divided model is generated based on the divided first sentence and the divided second sentence. The division model generation system according to claim 1, further comprising a dependency learning unit that learns the dependency relationship of sentences based on the learning result of the division model generation unit. The split model generation system according to claim 1 and
It is provided with an editorial unit that divides one predetermined sentence into a plurality of sentences based on the division model generated by the division model generation system.
The editorial department
An input unit in which predetermined document data structured including a plurality of sentences is input, and
A division target sentence selection unit that selects a division target sentence from the predetermined document data,
A text segmentation device including a division unit that divides one sentence to be divided into a plurality of sentences based on the division model. The text segmentation device according to claim 6, wherein the text segmentation device includes a conversion processing unit that converts the plurality of divided sentences into other sentences. Furthermore, the editorial department
A division evaluation unit that evaluates the division result of the division unit based on a predetermined evaluation index,
The seventh aspect of claim 7 includes a division result extraction unit that extracts a sentence to be converted by the conversion processing unit from a plurality of sentences divided by the division evaluation unit based on the evaluation of the division evaluation unit. Text segmentation device. The text segmentation device according to claim 7, further comprising a conversion evaluation processing unit that evaluates the conversion accuracy of the conversion processing unit. It is a division model generation method that generates a division model that divides a document by a computer.
The calculator
Of the plurality of first sentences included in the input first document data and the plurality of second sentences included in the input second document data, the first sentence and the second sentence having a predetermined correspondence relationship. Generate a combination with a sentence as learning data,
A division model generation method for generating a division model that divides a document based on the generated learning data.

Images