JP6211995B2 - Language model creation device, language model creation method and program - Google Patents

Description

  The present invention relates to a technique for creating a language model used in speech recognition, and more particularly, to a technique for constructing a language model that matches a target task by mixing a plurality of back-off n-gram language models.

  Speech recognition requires a language model for linguistic prediction. In speech recognition, an n-gram language model is generally used as a language model. The learning method of the n-gram language model is a known technique (for example, see Non-Patent Document 1). The n-gram language model is very compatible with speech recognition decoding. The n-gram language model can be easily learned if there is a learning text, and various learning methods have been proposed so far. Various language models have been proposed in addition to the n-gram language model, but language models other than the n-gram language model are very rarely used for speech recognition in compatibility with decoding of speech recognition. .

  As a method for realizing highly accurate speech recognition, it is known to use a language model specialized for a speech task to be recognized (hereinafter referred to as a target task). Specialization means giving a high probability of occurrence to a language phenomenon often used in the target task. That is, it is effective to use a language model specialized for news speech when recognizing news speech and to use a language model specialized for call center speech when recognizing call center speech. The language model specialized for this specific target task is generally learned by using the text (hereinafter also referred to as the transcription text) obtained by manually transcribe the speech of the target task. Is the method. However, collecting a large amount of human transcripts is very difficult in terms of costs such as time and labor costs. Therefore, it is common to construct a language model by making good use of another language resource related to the target task without using the transcription of the target task. This technique is called language model task adaptation. For example, in the case of a contact center, a plurality of language resources such as documents (for example, Web text) related to the contact center are used in combination. Language model task adaptation is a well-known research area, so please refer to the description in Non-Patent Document 2. As a typical technique of language model task adaptation, the technique of Non-Patent Document 3 can be cited. In this technology, a back-off n-gram language model is constructed for each of a plurality of external language resources, and the language models are constructed by adding the respective probabilities with weights (by linear interpolation). At that time, a framework for optimizing the mixture weight of each model using text data of the target task called development data has been established. Despite the very simple technique, this technology is still widely used. The reason is that, as an external language resource related to the target task, it operates very powerfully when text data of the same quality as the transcription text is obtained.

With reference to FIG. 1, a conventional technique for constructing a language model by mixing a plurality of back-off n-gram language models will be described. A conventional language model creation device includes a development data storage unit 10 that stores development data, and K back-off n-gram language model storage units 20 ₁ ,..., 20 _K that store back-off n-gram language models. A language model mixing unit 30 that optimally mixes development data and K backoff n-gram language models to generate a single mixed backoff n-gram language model, and a mixed backoff n-gram language model The mixed back-off n-gram language model storage unit 60 is stored. Here, a back-off 3-gram language model will be described as an example.

_Pk (w _i | w _i-2 , w _i-1 ) is the 3-gram probability of the k-th back-off n-gram language model for the word strings w _i-2 , w _i-1 , w _i and mixed If the weight is λ _k , the probability value P _mix (w _i | w _i−2 , w _i−1 ) of the 3-gram mixture is expressed by the following equation (1).

Note that P _k (w _i | w _i−2 , w _i−1 ) is expressed as in equation (2) in consideration of backoff.

Here, α _k (w _i−2 , w _i−1 ) is a coefficient applied at the time of back-off, and can be obtained by a known method.

Each mixing weight λ _{k at} this time can be optimized using development data. When the word string of the development data is w ₁ ... W _L , the mixture weight λ _k can be recursively obtained as shown in Equation (3) based on the EM algorithm.

Here, λ _k on the right side is the blending weight obtained one time before, and λ ′ _{k on} the left side is the updated blending weight. As a result, it is possible to determine the optimum mixing weight for the development data.

  As shown in FIG. 2, the back-off n-gram language model has a structure including a probability model for each n-gram order n. For example, the back-off 3-gram language model has a 1-gram language model, a 2-gram language model, and a 3-gram language model internally. Has a structure that gives a probability in a low-order order probability model.

Kenji Kita, “Language and Computation (4) Stochastic Language Model”, The University of Tokyo Press, November 1999, pp. 57-62 Kiyohiro Shikano, Katsunobu Ito, Tatsuya Kawahara, Kazuya Takeda, Mikio Yamamoto, “IT Text Speech Recognition System”, Ohmsha, May 2001, p. 68 R. Iyer, M. Ostendorf, “Modeling long distance dependence in language: topic mixture versus dynamic cache models”, IEEE Transactions on Speech and Audio Processing, vol. 7, no. 1, pp. 30-39, Jan 1999.

  In the prior art, a mixing weight is given in units of n-gram language models, that is, in units of original language resources. That is, if there are three n-gram language models, three blending weights are given. However, the n-gram language model has a structure called back-off, and has a structure including a probability model for each n-gram order. Nevertheless, in the prior art, the mixture weight is given to the entire language model without considering the included structure. In the conventional method, the mixture weight of the entire language model is optimized, but the mixture weight of the low-order order is not optimized. Therefore, in the language model mixed in the prior art, when a word entry is found in a higher order, high performance can be realized. Is not optimized. In particular, for unusual word chains in the target task, it is often necessary to use 1-gram probabilities, and high performance cannot be realized even if unoptimized probability values are used.

  The present invention aims to implement task adaptation that can realize high performance even for an unusual word chain in a target task. It can be said that the problem of the prior art is that the n-gram language model cannot be optimized for the lower order when mixing. Accordingly, an object of the present invention is to realize optimal model mixing even for low-order orders.

  In order to solve the above-described problems, a language model creation device according to the present invention includes a development data storage unit that stores development data, and a plurality of 1-gram language models, 2-gram language models, and 3-gram language models. A 1-gram language model that generates a mixed 1-gram language model by using a backoff n-gram language model storage unit that stores a back-off n-gram language model and development data and multiple 1-gram language models A mixing unit, a 2-gram language model mixing unit that generates a mixed 2-gram language model using development data, a plurality of 2-gram language models, and a mixed 1-gram language model, and development data and a plurality of 3-gram language models A 3-gram language model mixing unit that generates a mixed 3-gram language model using a gram language model, a mixed 1-gram language model, and a mixed 2-gram language model, and a mixed 1-gram language model and a mixed 2-gram Integrated back-off n-gram language model by integrating language model and mixed 3-gram language model Including a language model integration unit for generating a.

  According to the language model creation technique of the present invention, by providing a blending weight for each order of an n-gram language model, optimal model blending is realized even for low-order orders. As a result, a language model optimized for the target task can be constructed. In particular, when many unknown word chains are included, higher-accuracy speech recognition than before can be expected.

FIG. 1 is a diagram illustrating a functional configuration of a conventional language model creation device. FIG. 2 is a diagram conceptually illustrating the structure of the back-off n-gram language model. FIG. 3 is a diagram illustrating a functional configuration of the language model creation device according to the embodiment. FIG. 4 is a diagram illustrating a processing flow of the language model creation method according to the embodiment.

  Hereinafter, embodiments of the present invention will be described in detail. In addition, the same number is attached | subjected to the component which has the same function in drawing, and duplication description is abbreviate | omitted.

  In the present invention, when a plurality of n-gram language models are mixed, a different mixing weight is given to each order of the order n. The back-off n-gram language model has a structure in which higher-order orders depend on lower-order orders, so optimization in consideration of back-off is realized by optimizing in order from lower-order orders. can do.

[Embodiment]
As shown in FIG. 3, the language model creation apparatus of the embodiment includes, for example, a development data storage unit 10 and a mixed back-off n-gram language model storage unit 60, as in the related art, and further includes K back-off n. -gram language model storage unit 20 ₁ ,..., 20 _K , 1-gram language model mixing unit 31, 2-gram language model mixing unit 32, 3-gram language model mixing unit 33, mixed 1-gram language model storage unit 41 The mixed 2-gram language model storage unit 42, the mixed 3-gram language model storage unit 43, and the language model integration unit 50 are included, for example.

  The language model creation device is, for example, a special program configured by reading a special program into a known or dedicated computer having a central processing unit (CPU), a main memory (RAM), and the like. Device. For example, the language model creation device executes each process under the control of the central processing unit. Data input to the language model creation device and data obtained in each process are stored in the main storage device, for example, and the data stored in the main storage device is read out as necessary and used for other processing. Is done. In addition, at least a part of each processing unit of the language model creation device may be configured by hardware such as an integrated circuit.

  Each storage unit included in the language model creation device includes, for example, a main storage device such as a RAM (Random Access Memory), an auxiliary storage device configured by a semiconductor memory element such as a hard disk, an optical disk, or a flash memory (Flash Memory), or It can be configured with middleware such as a relational database or key-value store. Each storage unit included in the language model creation device is only required to be logically divided, and may be stored in one physical storage device.

Development data is stored in the development data storage unit 10. The development data is text data related to the target task of speech recognition. Hereinafter, the word string included in the development data is assumed to be w ₁ ... W _L.

The back-off n-gram language model storage unit 20 _k (k = 1,..., K) stores the k-th back-off n-gram language model. The back-off n-gram language model has a structure including a probability model for each n-gram order. In this embodiment, as shown in FIG. 2, the back-off n-gram language model 20 includes a 1-gram language model 21, a 2-gram language model 22, and a 3-gram language model 23. When the probability is not given by the probability model of the next order, it has a back-off structure that gives the probability by the probability model of the lower order. K back-off n-gram language models are constructed from different external language resources. As the external language resource, any text that matches the target task of speech recognition may be selected.

In this embodiment, the back-off n-gram language model is configured to be stored in each of the _K back-off n-gram language model storage units 20 ₁ ,..., 20 _K. The off-n-gram language model storage unit 20 may be configured to store K back-off n-gram language models.

  With reference to FIG. 4, the language model creation method of the embodiment will be described.

In step S1, 1-gram language model mixing unit 31, a development data stored in the developed data storage unit 10, K number of backoff n-gram language model storage unit 20 _1, ..., are respectively stored in the 20 _K One mixed 1-gram language model is generated using as input the 1-gram language model included in the backoff n-gram language model. The generated mixed 1-gram language model is stored in the mixed 1-gram language model storage unit 41.

Hereinafter, the process of the 1-gram language model mixing unit 31 will be described more specifically. When the 1-gram probability of the k-th backoff n-gram language model for the word w _i is P _k (w _i ) and the mixture weight is λ _k , the probability value P _mix (w _i ) of the 1-gram mixture is It is represented by Formula (4).

Each mixing weight λ _{k at} this time can be optimized using development data. If the word string of the development data is w ₁ ... W _L , λ _k can be recursively obtained as shown in Equation (5) based on the EM algorithm.

Here, λ _k on the right side is the blending weight obtained one time before, and λ ′ _{k on} the left side is the updated blending weight.

In the EM algorithm, the initial blend weight is generally equivalent to all blend weights. For example, if K = 2, λ _k = 1/2 may be given. Note that the number of repetitions of the EM algorithm can be set in advance, for example, a fixed number. For example, repeat 100 times. As a result, the value of the mixing weight λ _k can be optimized, so that the probability value P _mix (w _i ) of 1-gram mixing can be determined. For example, if P ₁ (today) = 0.01, P ₂ (today) = 0.05, and λ ₁ = 0.4 and λ ₂ = 0.6, then P _mix (today) = 0.01 * 0.4 + 0.05 * 0.6 can be determined.

In step S2, 2-gram language model mixing unit 32, a development data stored in the developed data storage unit 10, K number of backoff n-gram language model storage unit 20 _1, ..., are respectively stored in the 20 _K The back-off 2-gram language model included in the back-off n-gram language model and the mixed 1-gram language model stored in the mixed 1-gram language model storage unit 41 are used as inputs. Generate a gram language model. The generated mixed 2-gram language model is stored in the mixed 2-gram language model storage unit 42.

Hereinafter, the process of the 2-gram language model mixing unit 32 will be described more specifically. If the 2-gram probability of the k-th backoff n-gram language model for the word sequence w _i-1 , w _i is P _k (w _i | w _i-1 ) and the mixture weight is λ _k , 2-gram The mixing probability value P _mix (w _i | w _i-1 ) is expressed by equation (6).

Here, α (w _i-1 ) is a coefficient term applied at the time of back-off, and can be obtained by a known method.

Each mixing weight λ _{k at} this time can be optimized using development data. If the word string of the development data is w ₁ ... W _L , the mixture weight λ _k can be recursively obtained as shown in Equation (7) based on the EM algorithm.

Here, λ _k on the right side is the blending weight obtained one time before, and λ ′ _{k on} the left side is the updated blending weight. By optimizing the same as in the case of 1-gram mixing, a mixed 2-gram language model can be obtained.

In step S3, 3-gram language model mixing unit 33, a development data stored in the developed data storage unit 10, K number of backoff n-gram language model storage unit 20 _1, ..., are respectively stored in the 20 _K The back-off 3-gram language model included in the back-off n-gram language model, the mixed 1-gram language model stored in the mixed 1-gram language model storage unit 41, and the mixed 2-gram language model storage unit 42 The mixed 2-gram language model stored in is input, and one mixed 3-gram language model is generated. The generated mixed 3-gram language model is stored in the mixed 3-gram language model storage unit 43.

Hereinafter, the process of the 3-gram language model mixing unit 33 will be described more specifically. P _k (w _i | w _i-2 , w _i-1 ), the mixture weight of the 3-gram probability of the k-th back-off n-gram language model for the word sequence w _i-2 , w _i-1 , w _i Is a λ _k , the probability value P _mix (w _i | w _i−2 , w _i−1 ) of the 3-gram mixture is expressed by equation (8).

Here, α (w _i−2 , w _i−1 ) is a coefficient term applied at the time of back-off, and can be obtained by a known method.

Each mixing weight λ _{k at} this time can be optimized using development data. When the word string of the development data is w ₁ ... W _L , the mixture weight λ _k can be recursively obtained as shown in Equation (9) based on the EM algorithm.

Here, λ _k on the right side is the blending weight obtained one time before, and λ ′ _{k on} the left side is the updated blending weight. A mixed 2-gram language model can be obtained by optimizing the same as in the case of 1-gram mixing or 2-gram mixing.

  In step S4, the language model integration unit 50 includes the mixed 1-gram language model stored in the mixed 1-gram language model storage unit 41 and the mixed 2-gram language model stored in the mixed 2-gram language model storage unit 42. By inputting the model and the mixed 3-gram language model stored in the mixed 3-gram language model storage unit 43, each language model is combined into one to obtain one mixed back-off n-gram language Generate a model. The generated mixed back-off n-gram language model is stored in the mixed back-off n-gram language model storage unit 60.

[Application example]
In the above embodiment, the back-off 3-gram language model is mixed as an example, but the back-off 4-gram language model and the back-off 5-gram language model can also be mixed in the same framework. For example, when mixing back-off 4-gram language models, add a 4-gram language model mixing unit that mixes back-off 4-gram language models included in K back-off n-gram language models. The language model integration unit 60 may be configured to integrate the mixed back-off n-gram language models from 1-gram to 4-gram to generate a mixed back-off n-gram language model. Similarly, when mixing back-off 5-gram language models, a language model mixing unit may be added and the mixed back-off 5-gram language model may be input to the language model integration unit 60. In this way, optimal mixing is possible for each order by mixing in order from the lower order.

  The present invention is not limited to the above-described embodiment, and it goes without saying that modifications can be made as appropriate without departing from the spirit of the present invention. The various processes described in the above embodiment may be executed not only in time series according to the order of description, but also in parallel or individually as required by the processing capability of the apparatus that executes the processes or as necessary.

[Program, recording medium]
When various processing functions in each device described in the above embodiment are realized by a computer, the processing contents of the functions that each device should have are described by a program. Then, by executing this program on a computer, various processing functions in each of the above devices are realized on the computer.

  The program describing the processing contents can be recorded on a computer-readable recording medium. As the computer-readable recording medium, for example, any recording medium such as a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory may be used.

  The program is distributed by selling, transferring, or lending a portable recording medium such as a DVD or CD-ROM in which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of the server computer and transferring the program from the server computer to another computer via a network.

  A computer that executes such a program first stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. When executing the process, the computer reads a program stored in its own recording medium and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from a portable recording medium and execute processing according to the program, and the program is transferred from the server computer to the computer. Each time, the processing according to the received program may be executed sequentially. Also, the program is not transferred from the server computer to the computer, and the above-described processing is executed by a so-called ASP (Application Service Provider) type service that realizes the processing function only by the execution instruction and result acquisition. It is good. Note that the program in this embodiment includes information that is used for processing by an electronic computer and that conforms to the program (data that is not a direct command to the computer but has a property that defines the processing of the computer).

  In this embodiment, the present apparatus is configured by executing a predetermined program on a computer. However, at least a part of these processing contents may be realized by hardware.

10 Development Data Storage Unit 20 Back-off n-gram Language Model Storage Unit 30 Language Model Mixing Unit 31 1-gram Language Model Mixing Unit 32 2-gram Language Model Mixing Unit 33 3-gram Language Model Mixing Unit 41 Mixed 1-gram Language Model storage unit 42 Mixed 2-gram language model storage unit 43 Mixed 3-gram language model storage unit 50 Language model integration unit 60 Mixed back-off n-gram language model storage unit

Claims (7)

A development data storage unit for storing development data;
A back-off n-gram language model storage unit for storing a plurality of back-off n-gram language models including a 1-gram language model, a 2-gram language model, and a 3-gram language model;
A 1-gram language model mixing unit that generates a mixed 1-gram language model using the development data and the plurality of 1-gram language models;
A 2-gram language model mixing unit that generates a mixed 2-gram language model using the development data, the plurality of 2-gram language models, and the mixed 1-gram language model;
A 3-gram language model mixing unit that generates a mixed 3-gram language model using the development data, the plurality of 3-gram language models, the mixed 1-gram language model, and the mixed 2-gram language model;
A language model integration unit that generates a mixed back-off n-gram language model by integrating the mixed 1-gram language model, the mixed 2-gram language model, and the mixed 3-gram language model;
Language model creation device including The language model creation device according to claim 1,
w _i is a word included in the development data, P _k (w _i ) is a 1-gram probability of the 1-gram language model, λ _k is a mixture weight, 0 ≦ λ _k ≦ 1, and Σ _{k = 1} ^K λ _k = 1,
The 1-gram language model mixing unit generates a mixed 1-gram language model by calculating a probability value P _mix (w _i ) of 1-gram mixing according to the following equation:

The 2-gram language model mixing unit, α (w _i-1) is the back-off factor, 2-gram mixture of the probability values P _mix the following equation | said mixture by calculating the (w _i w _i-1) A 2-gram language model,

The 3-gram language model mixing unit uses α (w _i-2 , w _i-1 ) as a backoff coefficient, and the probability value P _mix (w _i | w _i-2 , w _i-1 ) is calculated to generate the above mixed 3-gram language model

Language model creation device. A development data storage unit for storing development data;
A back-off n-gram language model storage unit for storing a plurality of back-off n-gram language models including n-gram language models from 1-gram to N-gram, where N is an integer greater than or equal to 2,
An n-gram language that generates a mixed n-gram language model using the development data, multiple n-gram language models, and a mixed (n-1) -gram language model, where n is an integer of 2 to N A model mixing section;
A language model integration unit that generates a mixed back-off n-gram language model by integrating the mixed n-gram language model from 1-gram to N-gram;
Language model creation device including 1-gram language model mixing step for generating a mixed 1-gram language model using development data and a plurality of 1-gram language models included in a plurality of backoff n-gram language models;
A 2-gram language model that generates a mixed 2-gram language model using the development data, a plurality of 2-gram language models included in the plurality of back-off n-gram language models, and the mixed 1-gram language model A mixing step;
A mixed 3-gram language model using the development data, a plurality of 3-gram language models included in the plurality of back-off n-gram language models, the mixed 1-gram language model, and the mixed 2-gram language model 3-gram language model mixing step to generate
A language model integration step of integrating the mixed 1-gram language model, the mixed 2-gram language model, and the mixed 3-gram language model to generate a mixed back-off n-gram language model;
Language model creation method including The language model creation method according to claim 4,
w _i is a word included in the development data, P _k (w _i ) is a 1-gram probability of the 1-gram language model, λ _k is a mixture weight, 0 ≦ λ _k ≦ 1, and Σ _{k = 1} ^K λ _k = 1,
In the 1-gram language model mixing step, the mixed 1-gram language model is generated by calculating the probability value P _mix (w _i ) of 1-gram mixing by the following equation:

In the 2-gram language model mixing step, α (w _i-1 ) is a back-off coefficient, and the probability value P _mix (w _i | w _i-1 ) of 2-gram mixing is calculated by A 2-gram language model,

In the 3-gram language model mixing step, α (w _i-2 , w _i-1 ) is used as a back-off coefficient, and the probability value P _mix (w _i | w _i-2 , w _i-1 ) is calculated to generate the above mixed 3-gram language model

Language model creation method. The back-off n-gram language model storage unit stores a plurality of back-off n-gram language models including n-gram language models from 1-gram to N-gram, where N is an integer equal to or greater than 2.
An n-gram language model that generates a mixed n-gram language model using development data, the above-mentioned multiple n-gram language models, and the mixed (n-1) -gram language model, where n is an integer of 2 to N A mixing step;
A language model integration step of generating a mixed back-off n-gram language model by integrating the above-mentioned mixed n-gram language model from 1-gram to N-gram;
Language model creation method including   A program for causing a computer to function as the language model creation device according to any one of claims 1 to 3.

Images