JP2018077677A - Character string converting device, model learning device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a precise conversion of a character string involving a corrupt word.SOLUTION: A partial character string specifying unit 230 determines, for each character, whether or not an input character string involves a character that is a corrupt word on the basis of a determination model to determine whether or not a character is a corrupt word spelled in several different ways with respect to a normal word that is a normalized expression, and specifies a partial character string constituted by the word determined as the corrupt word. A character string converting unit 232 converts the specified partial character string on the basis of a conversion model to convert the corrupt word into the normal word, thereby creating a character string having the corrupt word involved in the character string converted into the normal word.SELECTED DRAWING: Figure 8

Description

  The present invention relates to a character string conversion device, a model learning device, a method, and a program, and in particular, a character string conversion device and model learning for robustly analyzing a broken notation which is a notation that does not appear in a normal writing method such as colloquial tone. The present invention relates to an apparatus, a method, and a program.

  2. Description of the Related Art Conventionally, a technique for performing normalized morphological analysis based on a character string normalization pattern is known as a technique for normalizing a broken notation. For example, as shown in FIG. 14, a normalized character string is set in advance, and analysis is performed while expanding the dictionary and input sentence using the character string (see Non-Patent Document 1 and Non-Patent Document 2). Examples of conversion from a broken character string to a regular character string are, for example, "-" → "null", "-" → "U", "tsu" → "null", "Shi" → "Shi" is there.

  Also, a technique is known in which an input sequence x is projected onto a vector space by an encoder-decoder type neural network, and an output sequence y is predicted while referring to the vector space (Non-patent Document 3). In this technique, the probability of each character output for each position t is calculated from an encoder-decoder type neural network model as shown in FIG. 15 according to the following expression, and the character sequence having the highest probability is output.

Here, c _t is a weighted average of h _t with a _t as a weight, and is calculated as follows for each position on the input source side (x).

Kenta Katsuki, Shinpei Kanno, Daisuke Kawahara, Ikuo Kurohashi, “Robust Morphological Analysis for Various Language Variations on the Web”, (2011), Proc. Of the 17th Annual Conference Saito, Sadamitsu, Asano, Matsuo, “Japanese Morphological Analysis Based on Normalized-Disrupted String Alignment and Character Type Conversion”, 20th Annual Conference of the Association for Natural Language Processing, 2014/3/10 Minh-Thang Luong, Hieu Pham, Christopher D. Manning, "Effective Approaches to Attention-based Neural Machine Translation" Computer Science Department, Stanford University, Stanford, CA 94305, 2015

  In the conventional techniques of Non-Patent Document 1 and Non-Patent Document 2, when normalization is performed by setting a character string level pattern, there is a problem that only a character string set in advance can be normalized. In addition, there is a problem that only context information within a limited range of the set character string can be considered.

  The technique of Non-Patent Document 3 has a problem that the learning time is long because the conversion is performed in consideration of the entire character string. In addition, particularly when learning data is small or when an input character string is long, there is a problem that degradation such as conversion of characters that do not need to be converted increases.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a character string conversion device, method, and program capable of accurately converting a character string including a broken word. To do.

  It is another object of the present invention to provide a model learning apparatus, method, and program capable of learning a model that can convert a character string including a broken word with high accuracy.

  In order to achieve the above object, the character string conversion device according to the first aspect of the present invention is a notation that fluctuates with respect to a regular word that is a normalized expression for each character with respect to the input character string. Based on the determination model for determining whether or not it is a broken word character, it is determined whether or not it is a broken word character, and a partial character string consisting of characters determined to be a broken word character The partial character string is converted based on a conversion model for converting the broken word into the regular word for the partial character string specifying unit to be specified and the partial character string specified by the partial character string specifying unit. Thus, a character string conversion unit that generates a character string obtained by converting the corrupted word included in the character string into the regular word is configured.

  In the character string conversion device according to the first invention, the conversion model converts the corrupted word into the regular word in consideration of a context before the corrupted word and a context after the corrupted word. The conversion model, the context before the partial character string in the character string, and the part of the partial character string specified by the partial character string specifying unit. The partial character string may be converted based on the context after the character string.

  Further, in the character string conversion device according to the first invention, the determination model receives as input a character to be determined, a character string before the character to be determined, and a character string after the character to be determined, The neural network that outputs a probability that the character to be determined is a broken word character, and the conversion model includes a partial character string to be converted, a character string before the partial character string to be converted, and the conversion target. It is also possible to use a neural network that inputs a character string after the partial character string and outputs the probability that each character of the converted partial character string is a character for each character of the converted partial character string. Good.

  Further, in the model learning device according to the second aspect of the present invention, a collapsed sentence including a normalized sentence composed of a normal word that is an input normalized expression and a broken word that is a distorted expression with respect to the regular word. Based on a plurality of pairs, for each of the plurality of pairs, a character string alignment unit for obtaining a correspondence relationship between each character included in the normalized sentence and each character included in the collapsed sentence, and the character A determination model learning unit that learns a determination model for determining whether or not the character is a broken word based on the correspondence relationship for each of the plurality of pairs obtained by a column alignment unit; and the character A conversion model learning unit that learns a conversion model for converting the broken word into the regular word based on the correspondence relationship for each of the plurality of pairs obtained by the column alignment unit; It is configured to include a.

  In the character string conversion method according to the third invention, the partial character string specifying unit is a notation that is swaying with respect to a regular word that is a normalized expression for each character with respect to the input character string. Based on the judgment model for judging whether or not it is a word character, it is judged whether or not it is a broken word character, and a partial character string made up of characters that are judged to be a broken word character is specified And a step of converting the partial character string based on a conversion model for converting the broken word into the regular word with respect to the partial character string specified by the partial character string specifying unit. And generating a character string obtained by converting the corrupted word contained in the character string into the regular word by performing the conversion.

  According to a fourth aspect of the present invention, there is provided the character string conversion method, wherein the character string alignment unit is a normalized sentence composed of a normal word that is a normalized expression that is input, and a distorted notation with respect to the normal word. Obtaining a correspondence relationship between each character included in the normalized sentence and each character included in the corrupted sentence for each of the plurality of pairs based on a plurality of pairs with a corrupted sentence including a word; A determination model learning unit learns a determination model for determining whether or not the character is a broken word based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit. And a conversion model for converting the corrupted word into the regular word based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit. And executes comprising the steps of learning the Le, the.

  A program according to a fifth aspect is a program for causing a computer to function as each part of the character string conversion device according to any one of claims 1 to 3.

  A program according to a sixth invention is a program for causing a computer to function as each part of the model learning device according to the second invention.

  According to the character string conversion apparatus, method, and program of the present invention, characters of a corrupted word that is a notation that fluctuates with respect to a regular word that is a normalized expression for each character of the input character string. Based on the determination model for determining whether or not it is, it is determined whether or not it is a character of a broken word, and a partial character string consisting of characters determined to be a broken word character is specified and specified By converting a partial character string based on a conversion model for converting a broken word into a regular word, a character string obtained by converting a broken word contained in a character string into a regular word By generating, it is possible to obtain an effect that a character string including a broken word can be converted with high accuracy.

  Further, according to the model learning device, method, and program of the present invention, an input normalized sentence composed of a normal word that is a normalized expression and a corrupted word that is a notation that fluctuates with respect to the normal word. Based on the multiple pairs with the collapsed sentence included, for each of the multiple pairs, the correspondence between each character included in the normalized sentence and each character included in the collapsed sentence is obtained and obtained by the character string alignment unit. Further, based on the correspondence relationship for each of the plurality of pairs, a determination model for determining whether the character is a collapsible word, based on the correspondence relationship for each of the plurality of pairs, By learning a conversion model for converting a broken word into a regular word, an effect is obtained that a model that can accurately convert a character string including a broken word can be obtained.

It is a block diagram which shows the structure of the model learning apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the correspondence of each change label (for determination model learning) of the character of a collapsed sentence, and a partial change character string set (for conversion model learning). It is a figure which shows an example of the correct data produced | generated from the pair of a normalized sentence and a collapse sentence. It is a figure which shows the example of learning of the determination model by the change label of a correspondence. It is a figure which shows an example of the LSTM type neural network used in a determination model. It is a figure which shows the example of learning of the conversion model based on the partial change character string set of a correspondence. It is a figure which shows an example of the encoder-decoder type neural network used in a conversion model. It is a block diagram which shows the structure of the character string converter which concerns on embodiment of this invention. It is a figure which shows an example of determination of whether it is a character of a broken word by a determination model. It is a figure which shows an example of the conversion to each regular word of the partial character string of the broken word by a conversion model. FIG. 5 is a schematic diagram showing the overall processing flow of the character string converter 200. It is a flowchart which shows the model learning process routine in the model learning apparatus which concerns on embodiment of this invention. It is a flowchart which shows the character string conversion process routine in the character string conversion apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the normalization morpheme analysis based on a character string normalization pattern. It is a figure which shows an example of an Encoder-decoder type neural network model.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline according to Embodiment of the Present Invention>

  First, an outline of the embodiment of the present invention will be described.

  The problems in Non-Patent Document 1 and Non-Patent Document 2 are dealt with by estimating a character conversion candidate while considering information of the entire sentence by using a determination model based on a neural network. Thereby, it is not necessary to fix the length of the character string to be considered in advance, and a wider context can be considered. In addition, since an abstract context is considered rather than a character string itself, it is possible to consider similarities such as “the surface is different but semantically similar”.

  For the problem in Non-Patent Document 3, by specifying a partial character string to be converted and introducing a mechanism for converting only the specified partial character string, only the converted part is normalized efficiently and accurately. To do. The characteristics of the partial character string to be converted and the conversion model of the partial character string can be learned separately, and the model learning can be performed efficiently. Since only the characters identified to be converted are converted, it is possible to suppress degradation, and it is difficult to be affected by the character length of the input sentence.

<Configuration of Model Learning Device According to Embodiment of the Present Invention>

  Next, the configuration of the model learning device according to the embodiment of the present invention will be described. As shown in FIG. 1, a model learning device 100 according to an embodiment of the present invention includes a CPU, a RAM, and a ROM that stores a program for executing a model learning processing routine described later and various data. Can be configured with a computer. Functionally, the model learning apparatus 100 includes an input unit 10 and a calculation unit 20 as shown in FIG.

  The input unit 10 includes, as correct answer data, a normalized sentence including a normal word that is a normalized expression, and a broken sentence that includes a broken word that is a fluctuation expression with respect to the regular word that is a normalized expression. Accept multiple pairs.

  The calculation unit 20 includes a character string alignment unit 30, a determination model learning unit 32, a conversion model learning unit 34, a determination model 40, and a conversion model 42.

  The character string alignment unit 30 is included in the normalized sentence for each of a plurality of pairs based on a plurality of pairs of a normalized sentence composed of regular words and a corrupted sentence including a corrupted word received by the input unit 10. The correspondence relationship between each character to be read and each character included in the collapsed sentence is obtained. As the correspondence relationship, as shown in FIG. 2, the change labels (for determination model learning) and the partial change character string set (for conversion model learning) of each character of the collapsed sentence are obtained.

  Specifically, as shown in FIG. 3, with respect to a pair of a normalized sentence and a broken sentence, a part surrounded by a dotted line is a broken character string composed of broken characters to be converted, and the rest is a character string without conversion. Create correct data. At the time of model learning by the determination model learning unit 32, it is necessary to associate character levels of correct data in order to determine whether or not a character included in a broken sentence is a broken word (whether or not to be converted). Therefore, a change label of “1” is assigned to the changed character, and a change label of “0” is assigned to the character that has not changed.

  When determining the correspondence between each character included in the normalized sentence and each character included in the corrupted sentence, the character string level of the corrupted string of the corrupted sentence and the normalized character string of the corresponding normalized sentence are used. Perform DP alignment. Also, from the alignment result, a partial character string delimited by the unit of minimum change in the normalized sentence is created as correct answer data of the partial change character string to be converted.

  The determination model learning unit 32 learns a determination model for determining whether or not the character is a collapsing word based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit 30. Save as model 40.

  Specifically, as shown in FIG. 4, the weight parameter of the determination model 40 that estimates the change label for each character of the broken sentence is learned based on the change label of the correspondence relationship. FIG. 5 shows an LSTM type neural network used in the determination model 40 of the present embodiment. This neural network outputs the label output probability p (j) of each character position j in the following equation (1). Here, the change label is output as a binary value of 1 or 0 based on the label output probability p (j).

... (1)

Here, the hidden layer _{h fj} at character position j of the forward LSTM, at character position j of the reverse LSTM hidden layer _{h bj,} characters embedding character position j and _{e j.} W _fj , W _bj , and W _j are respective weight parameters.

  The conversion model learning unit 34 learns a conversion model for converting a broken word into a regular word based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit 30, and stores it as a conversion model 42. To do.

  Specifically, as shown in FIG. 6, the weight parameter of the conversion model 42 that estimates the regular word corresponding to the broken word of the broken sentence is learned based on the partial change character string set of the correspondence relationship. FIG. 7 shows an encoder-decoder type neural network used in the conversion model 42 of the present embodiment. For the partial change character string that is a broken word to be converted, the output for each character after conversion is formulated as follows.

... (2)

It is based on the existing attention based encoder-decoder model, but considers the conversion in units of the input partial change string, the left context (previous context), and the right context (later context). The point is different. Here, the hidden layer of the right context LSTM is h _cr , and the hidden layer of the left context LSTM is h _cl . h _c represents the hidden layer in both directions LSTM, corresponding to c _t in Encoder-decoder neural network model described Non-Patent Document 3. h _t represents the hidden layer at the current decoder position t. W _ht , W _c , W _cl , and W _cr are respective weight parameters.

  Finally, the probability vector p (t) of the t-th character is estimated by the following equation (3). The t-th character after conversion is determined by this probability vector p (t).

... (3)

  As described above, the conversion model 42 is a conversion model for converting a collapsed word into a regular word in consideration of a context before the collapsed word and a context after the collapsed word.

<Configuration of Character String Conversion Device According to Embodiment of the Present Invention>

  Next, the configuration of the character string conversion device according to the embodiment of the present invention will be described. As shown in FIG. 8, a character string conversion device 200 according to an embodiment of the present invention includes a CPU, a RAM, a ROM for storing a program and various data for executing a character string conversion processing routine described later, Can be configured with a computer including Functionally, the character string conversion device 200 includes an input unit 210, a calculation unit 220, and an output unit 250 as shown in FIG.

  The input unit 210 receives a character string including a broken word to be converted.

  The calculation unit 220 includes a partial character string specifying unit 230, a character string conversion unit 232, a determination model 240, and a conversion model 242.

  The determination model 240 is the same model as the determination model 40 that is learned by the model learning device 100 and determines whether or not the character is a broken word. Further, the determination model 240 receives the determination target character, the character string before the determination target character, and the character string after the determination target character, and outputs the probability that the determination target character is a broken word character. It is a neural network.

  The conversion model 242 is the same model as the conversion model 42 that is learned by the model learning device 100 and converts a corrupted word into a regular word. The conversion model 242 is a partial character string to be converted and a partial character string to be converted. Input the previous character string and the character string after the conversion target partial character string, and output the probability of each character for each character of the converted partial character string converted from the conversion target partial character string It is a neural network.

  The partial character string specifying unit 230 determines whether the character string received by the input unit 210 is a broken word character or not based on the determination model 240 for each character. The partial character string which consists of the character determined to be is specified.

  For example, as shown in FIG. 9, it is determined whether each character of the character string is a broken word character using the determination model 240. A character is input to the LSTM type neural network of the determination model 240, and the output of the label output probability p (j) at the character position j is obtained according to the above equation (1) to determine whether the character is a broken word. A change label “1” is output for characters determined to be a broken word character, and a change label “0” is output to a character determined not to be a broken word character. Is identified.

  The character string conversion unit 232, for each partial character string specified by the partial character string specifying unit 230, the conversion model 242, the context before the partial character string in the character string, and the partial character string after the partial character string By converting the partial character string based on the context, a character string in which a broken word included in the character string is converted into a regular word is generated.

For example, as shown in FIG. 10, each broken character partial character string is converted into a regular word using a conversion model 242. Each character of the partial character string, each character in the context before the partial character string, and each character in the context after the partial character string are input to an encoder-decoder type neural network, and the above equation (2) according, taking into account the context of the previous context and after with respect to the partial string, for t th character substring, computes the o _t by calculating the ~ h _t. Then, according to the above equation (3), the probability vector p (t) of the character t is estimated and converted to the character with the highest probability. If it is a substring “Kyo”, it is converted to “today”, and if it is a character “—”, it is converted to “null”. Then, the partial character strings converted into the original character strings are integrated and output to the output unit 250 as normalized character strings. FIG. 11 is a schematic diagram showing the overall processing flow of the character string conversion apparatus 200.

<Operation of Model Learning Device According to Embodiment of Present Invention>

  Next, the operation of the model learning device 100 according to the embodiment of the present invention will be described. When the input unit 10 receives a plurality of pairs of a regular sentence composed of regular words and a broken sentence including a broken word, the model learning device 100 executes a model learning processing routine shown in FIG.

  First, in step S100, each of a plurality of pairs is included in a normalized sentence based on a plurality of pairs of a normalized sentence composed of regular words and a corrupted sentence including a corrupted word received by the input unit 10. The correspondence between each character and each character included in the collapsed sentence is obtained.

  Next, in step S102, based on the correspondence relationship for each of the plurality of pairs obtained in step S100, a determination for determining whether the character is a broken word using an LSTM type neural network. The model is learned and stored as the determination model 40.

  In step S104, a conversion model for converting a broken word into a regular word using an encoder-decoder type neural network based on the correspondence relationship for each of a plurality of pairs obtained by the character string alignment unit 30. Learn and save as conversion model 42.

  As described above, according to the model learning device according to the present embodiment, a normalized sentence composed of a normal word that is a normalized expression that has been input, and a distorted expression that is distorted with respect to the normal word. Based on a plurality of pairs with a collapsed sentence including words, for each of the plurality of pairs, a correspondence relationship between each character included in the normalized sentence and each character included in the corrupted sentence is obtained, and the character string alignment unit Based on the correspondence relationship for each of the plurality of obtained pairs, a determination model for determining whether or not the character is a broken word is learned, and based on the correspondence relationship for each of the plurality of pairs Thus, by learning a conversion model for converting a broken word into a regular word, it is possible to learn a model that can accurately convert a character string including the broken word.

<Operation of Character String Conversion Device According to Embodiment of the Present Invention>

  Next, the operation of the character string conversion device 200 according to the embodiment of the present invention will be described. When the input unit 210 receives a character string including a broken word to be converted, the character string conversion device 200 executes a character string conversion processing routine shown in FIG.

  First, in step S200, it is determined for each character whether the character string received by the input unit 210 is a broken word character based on the determination model 240, and is determined to be a broken word character. A substring consisting of the specified characters is specified.

  Next, in step S202, for each partial character string specified by the partial character string specifying unit 230, the conversion model 242, the context before the partial character string in the character string, and the partial character string after the partial character string are determined. By converting the partial character string based on the context, a character string obtained by converting the broken word included in the character string into a regular word is generated and output to the output unit 250.

  As described above, according to the character string conversion device according to the embodiment of the present invention, for the input character string, for each character, a notation that fluctuates with respect to a regular word that is a normalized expression Based on the determination model for determining whether or not it is a broken word character, it is determined whether or not it is a broken word character, and a partial character consisting of characters determined to be a broken word character By identifying the column and converting the partial character string based on the conversion model for converting the broken word into a regular word for the identified partial character string, the broken word contained in the character string is converted into the regular word. By generating a character string converted into, a character string including a broken word can be converted with high accuracy.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, in the above-described embodiment, the case where a neural network is used for the determination model 40 (240) and the conversion model 42 (242) has been described as an example. A determination model and a conversion model to which other methods are applied may be used as long as they can be output.

DESCRIPTION OF SYMBOLS 10 Input part 20 Operation part 30 Character string alignment part 32 Determination model learning part 34 Conversion model learning part 40,240 Determination model 42,242 Conversion model 100 Model learning apparatus 200 Character string conversion apparatus 210 Input part 220 Operation part 230 Partial character string Specific part 232 Character string conversion part 250 Output part

Claims (8)

Based on a determination model for determining whether or not each character is a character of a broken word that is a swaying expression with respect to a regular word that is a normalized expression for each character, A partial character string identifying unit that determines whether or not the character is a broken word, and identifies a partial character string composed of characters determined to be a broken word character;
Included in the character string by converting the partial character string based on a conversion model for converting the broken word into the regular word for the partial character string specified by the partial character string specifying unit A character string conversion unit that generates a character string obtained by converting the corrupted word into the regular word;
String converter including The conversion model is a conversion model for converting the broken word into the regular word in consideration of the context before the broken word and the context after the broken word,
The character string conversion unit, for the partial character string specified by the partial character string specifying unit, the conversion model, the context before the partial character string and the context after the partial character string in the character string The character string conversion device according to claim 1, wherein the partial character string is converted based on: The determination model receives a determination target character, a character string before the determination target character, and a character string after the determination target character, and determines the probability that the determination target character is a broken word character. An output neural network,
The conversion model takes as input a partial character string to be converted, a character string before the partial character string to be converted, and a character string after the partial character string to be converted, and converts the partial character string to be converted. 3. The character string conversion device according to claim 1, wherein the character string conversion device is a neural network that outputs a probability of being a character for each character of the converted partial character string. The plurality of pairs based on a plurality of pairs of a normalized sentence composed of regular words that are input normalized expressions and a corrupted sentence that includes a corrupted word that is a distorted expression with respect to the regular word For each of the above, a character string alignment unit for obtaining a correspondence relationship between each character included in the normalized sentence and each character included in the collapsed sentence,
A determination model learning unit that learns a determination model for determining whether or not the character is a broken word, based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit;
A conversion model learning unit that learns a conversion model for converting the broken word into the regular word based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit;
Model learning device including To determine whether the partial character string specifying unit is a broken word character that is a swaying expression with respect to a regular word that is a normalized expression for each character with respect to the input character string Determining whether the character is a broken word based on the determination model, and specifying a partial character string consisting of characters determined to be a broken word character;
By converting the partial character string based on a conversion model for converting the broken word into the regular word for the partial character string specified by the partial character string specifying unit. Generating a character string obtained by converting the collapsible word included in the character string into the regular word;
String conversion method including The character string alignment unit is based on a plurality of pairs of a normalization sentence including a normal word that is a normalized expression that has been input and a collapse sentence that includes a collapse word that is a distorted expression with respect to the regular word. Obtaining a correspondence relationship between each character included in the normalized sentence and each character included in the collapsed sentence for each of the plurality of pairs;
A determination model learning unit learns a determination model for determining whether or not the character is a broken word based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit. Steps,
A conversion model learning unit learning a conversion model for converting the corrupted word into the regular word based on the correspondence relationship for each of the plurality of pairs obtained by the character string alignment unit;
Model learning method including   The program for functioning a computer as each part of the character string converter of any one of Claims 1-3.   The program for functioning a computer as each part of the model learning apparatus of Claim 4.