JP2022019422A - Learning device, inference device, learning method, inference method and program - Google Patents

Abstract

To provide a learning device, an inference device, a learning method, an inference method and a program, with which it is possible to realize a sequence alignment which can derive, and enable use of, a more complicated feature representation and also which can derive, and enable use of, a monotonous and contiguous corresponding function, without relying on the use of manually designed feature representations.SOLUTION: A learning device comprises: an attention mechanism for generating, using a first feature sequence based on a first sequence and a second feature sequence based on a second sequence, a weight sequence that represents the probability that the elements of the first and second feature sequences have a correspondence relation; an objective function value derivation unit for deriving, on the basis of the weight sequence, a label that represents whether or not the first and second sequences belong to the same class and an objective function value that corresponds to the first and second feature sequences; and an update unit for executing a prescribed learning process on the basis of the objective function value and thereby generating a learning result.SELECTED DRAWING: Figure 2

Description

The present invention relates to a learning device, an inference device, a learning method, an inference method and a program.

An array is a series of data arranged in order. Examples of arrays are audio signals, acoustic signals, and biological signals. Each data in the array is a numerical value, a numerical vector, etc., and is called an element of the array. Each piece of data in the array is identified using a subscript such as a natural number.

Sequence alignment is the alignment of the elements of each sequence so that regions similar to each other can be identified in a plurality of sequences. Sequence alignment is important in many application problems such as motion recognition, speech analysis, biosignal classification and signature authentication, as clues to know the relationship of sequences are given by sequence alignment. In particular, if there is a non-linear time variation between the two sequences with respect to local and velocity changes, sequence alignment is required. As a typical method of sequence alignment, there is a dynamic time expansion / contraction method (see Non-Patent Document 1).

In the dynamic time expansion and contraction method, the distance between each element in the two arrays is derived. The correspondence between the elements in the two arrays is detected so that the sum of the distances between the elements in the correspondence is minimized. The correspondence relationship is a combination of two elements corresponding to each other or a combination of subscripts of two elements corresponding to each other.

In the dynamic time expansion / contraction method, it is difficult to parallelize the processes. For this reason, it is difficult to combine the dynamic time expansion and contraction method with deep learning. Also, the dynamic time expansion and contraction method relies on the use of manually designed feature representations and is inadequate in performance when more complex feature representations are required. Therefore, the dynamic time expansion and contraction method is often not optimal for a given application problem.

In the fields of machine translation, speech synthesis, speech conversion, etc., there is a method of using an attention mechanism as a method of sequence alignment that can be easily combined with deep learning (see Non-Patent Documents 2 and 3). The attention mechanism derives the weight of each element of the first array for each element of the second array with respect to the two arrays, the first array and the second array. Each derived weight represents the probability that each element of the two arrays, the first array and the second array, has a correspondence relationship. In the method of sequence alignment using the attention mechanism, the sequence alignment is realized by rearranging each element of the first array based on the weight of each element of the first array with respect to each element of the second array.

Hiroaki Sakoe and Seibi Chiba, "Dynamic programming algorithm optimization for spoken word recognition," IEEE Transactions on Acoustics, Speech and Signal Processing, Vol. 26, No. 1, pp. 43-49, 1978. Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio, "Neural machine translation by jointly learning to align and translate," In ICLR, 2015. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning, "Effective approaches to attention-based neural machine translation," In EMNLP, pp. 1412-1421, 2015.

Using a function (hereinafter referred to as "correspondence function") in which the subscript of the second array is the independent variable and the subscript of the first array, which is in correspondence with the subscript of the second array, is the dependent variable, the two arrays The correspondence between each element is represented. In applied problems such as collation or classification, the corresponding functions are often monotonous and continuous in two arrays belonging to the same class. On the other hand, in two arrays belonging to different classes, the corresponding functions are often non-monotonic or discontinuous.

By utilizing such a property, it is possible to determine whether or not two arrays belong to the same class. For example, a monotonous and continuous correspondence function can be derived from two arrays, and the total distance between the elements in the correspondence can be derived. When this sum is large, it can be determined that two arrays belong to different classes.

As a typical sequence alignment method utilizing such a property, there is a dynamic time expansion / contraction method. However, the dynamic time expansion and contraction method relies on the use of manually designed feature representations and is inadequate in performance when more complex feature representations are required. Therefore, the dynamic time expansion and contraction method is often not optimal for a given application problem.

Attention mechanisms, on the other hand, do not rely on manually designed feature representations. However, in the past, attention mechanisms have not been used to solve applied problems such as collation or classification. This is because even if the conventional attention mechanism derives the probability that each element of the two arrays has a correspondence relationship, the corresponding function cannot be derived from the probability. Also, even if the conventional attention mechanism derives the corresponding function, there is no way to guarantee that the corresponding function is monotonous and continuous.

Therefore, when the distance between sequences aligned using conventional attention mechanisms is applied to application problems such as collation or classification, the distance between sequences is often derived very small. For this reason, it is often not possible to correctly distinguish between two arrays belonging to different classes.

FIG. 10 is a diagram showing an example of a weight matrix. The weight matrix is a matrix that represents the probability that each element of the two arrays has a correspondence relationship. In FIG. 10, the first sequence is "LISTEN" as an example, and the second sequence is "SILENT" as an example. The elements of the weight matrix having a value of "1" indicate that the corresponding elements have a correspondence relationship.

The weight matrix shown on the left side in FIG. 10 is a weight matrix derived by a conventional attention mechanism. Thus, the conventional attention mechanism derives a non-monotonic and discontinuous correspondence function. Even if two arrays belong to different classes, the weight matrix shown on the left side in FIG. 10 correctly distinguishes the two arrays because the total distance between the corresponding elements is 0. I haven't been able to.

Therefore, in application problems such as collation or classification, there is a need for an array alignment method that can derive and use a monotonous and continuous correspondence function like the arrangement of "1" in the weight matrix shown on the right side in FIG. Has been done. By such an array alignment method, the distance or similarity between sequences can be correctly derived, and it is possible to correctly infer whether or not the sequences belong to different classes.

In application problems such as speech synthesis or speech conversion, the purpose is to convert the first array to the second array. If there is a non-linear time variation between the first and second sequences with respect to local and velocity changes, sequence alignment is required. For example, when converting Japanese English voice to American English voice, it is necessary to align the arrangement of voice signals because the tempo of the English voice fluctuates. That is, the correspondence between each element of the two arrays needs to be estimated, the first sequence aligned using the estimated correspondence, and the aligned first sequence converted to the second array. .. Even in such cases, the corresponding function between the two arrays is often monotonous and continuous.

However, the conventional method using the attention mechanism does not have a function of guiding the learning of the mathematical model so that the attention mechanism can derive a monotonous and continuous correspondence function. For this reason, it often takes a long learning time before the attention mechanism can provide sufficient performance.

Therefore, the above-mentioned arrangement method is also required for application problems such as speech synthesis or speech conversion. By such an arrangement method, it is possible to achieve both improvement of inference accuracy such as speech synthesis or speech conversion and shortening of learning time.

In view of the above circumstances, the present invention can derive and use more complex feature representations without relying on the use of manually designed feature representations, while at the same time deriving and using monotonous and continuous correspondence functions. It is an object of the present invention to provide a learning device, an inference device, a learning method, an inference method, and a program capable of realizing a possible sequence arrangement.

In one aspect of the present invention, the first feature sequence based on the first sequence and the second feature sequence based on the second sequence are used, and the elements of the first feature sequence and the second feature sequence correspond to each other. A caution mechanism for generating a weighting matrix, which is a matrix representing a certain probability, a label indicating whether or not the first array and the second array belong to the same class, the first feature array, and the second feature array. An objective function value derivation unit that derives an objective function value that is a value according to the above weight matrix based on the weight matrix, and an update unit that generates a learning result by executing a predetermined learning process based on the objective function value. It is a learning device provided with.

In one aspect of the present invention, the first feature sequence based on the first sequence and the second feature sequence based on the second sequence are used, and the elements of the first feature sequence and the second feature sequence correspond to each other. Based on the attention mechanism that creates a weighting matrix, which is a matrix representing a certain probability, and the weight of each element of the first feature array with respect to the elements of the second array at the current time, and the current first feature array. Based on the decoding unit that derives the elements of the second array at time, the objective function value derivation unit that derives the objective function value that is the value corresponding to the correct array and the second array, and the objective function value. It is a learning device including an update unit that generates a learning result by executing a predetermined learning process.

In one aspect of the present invention, the first feature sequence based on the first sequence and the second feature sequence based on the second sequence are used, and the elements of the first feature sequence and the second feature sequence correspond to each other. The distance between the first array and the second array based on the attention mechanism that generates the weight matrix, which is a matrix representing a certain probability, and the first feature array, the second feature array, and the weight matrix. This is an inference device including a collation unit for deriving the above and an inference unit for generating an inference result by executing a predetermined inference process based on the distance.

In one aspect of the present invention, the first feature sequence based on the first sequence and the second feature sequence based on the second sequence are used, and the elements of the first feature sequence and the second feature sequence correspond to each other. A caution mechanism that generates a weight matrix that represents a certain probability, a decoding unit that derives a second array based on the first feature array and the weight matrix, and a predetermined inference based on the second array. It is an inference device including an inference unit that generates an inference result by executing a process.

One aspect of the present invention is a learning method executed by a learning device, wherein the first feature sequence and the first feature sequence are used by using a first feature sequence based on the first sequence and a second feature sequence based on the second sequence. A caution step to generate a weight matrix, which is a matrix representing the probability that each element of the two feature arrays has a correspondence relationship, a label indicating whether or not the first array and the second array belong to the same class, and the above. An objective function value derivation step for deriving an objective function value which is a value corresponding to the first feature array and the second feature array based on the weight matrix, and a predetermined learning process based on the objective function value are executed. It is a learning method including an update step for generating a learning result by doing so.

One aspect of the present invention is an inference method executed by an inference device, wherein the first feature sequence and the first feature sequence based on the second sequence are used. The first feature array is based on the attention step of generating a weight matrix, which is a matrix representing the probability that each element of the two feature array is in a correspondence relationship, and the first feature array, the second feature array, and the weight matrix. It is an inference method including a collation step for deriving a distance between an array and the second array, and an inference step for generating an inference result by executing a predetermined inference process based on the distance.

One aspect of the present invention is a learning method executed by a learning device, wherein the first feature sequence and the first feature sequence are used by using a first feature sequence based on the first sequence and a second feature sequence based on the second sequence. A caution step to generate a weighting matrix, which is a matrix representing the probability that each element of the two feature arrays has a correspondence relationship, and the weight of each element of the first feature array with respect to the element of the second array at the current time, and the above. A decoding step that derives the elements of the second array at the current time based on the first feature array, and an objective function value that derives an objective function value that is a value corresponding to the correct array and the second array. It is a learning method including a derivation step and an update step of generating a learning result by executing a predetermined learning process based on the objective function value.

One aspect of the present invention is an inference method executed by an inference device, wherein the first feature sequence and the first feature sequence based on the second sequence are used. A caution step for generating a weight matrix, which is a matrix representing the probability that each element of the two feature arrays has a correspondence relationship, and a decoding step for deriving a second array based on the first feature array and the weight matrix. , A reasoning method including a reasoning step of generating a reasoning result by executing a predetermined reasoning process based on the second array.

One aspect of the present invention is a program for operating a computer as the learning device described above.

One aspect of the present invention is a program for operating a computer as the inference device described above.

INDUSTRIAL APPLICABILITY According to the present invention, more complicated feature representations can be derived and used without depending on the use of manually designed feature representations, and at the same time, monotonous and continuous correspondence functions can be derived and usable sequence alignment can be performed. It is possible to achieve it.

It is a figure which shows the structural example of the inference apparatus in 1st Embodiment. It is a figure which shows the structural example of the learning apparatus in 1st Embodiment. It is a figure which shows the example of the corresponding arrangement in 1st Embodiment. It is a figure which shows the derivation example of the monotonicity constraint function value in 1st Embodiment. It is a figure which shows the derivation example of the continuity constraint function value in 1st Embodiment. It is a figure which shows the structural example of the inference device in 2nd Embodiment. It is a figure which shows the structural example of the learning apparatus in 2nd Embodiment. It is a figure which shows the hardware configuration example of the inference apparatus in each embodiment. It is a figure which shows the hardware configuration example of the learning apparatus in each embodiment. It is a figure which shows the example of the weight matrix.

Embodiments of the present invention will be described in detail with reference to the drawings.
In the following, attention mechanisms are used in application problems such as sequence matching or classification. This allows for derivation and usable sequence alignment of more complex feature representations without relying on the use of manually designed feature representations.

In the following, a new constraint function value representing at least one of a monotonic constraint and a continuity constraint is proposed. By minimizing the constraint function value so that the attention mechanism can derive a monotonous and continuous correspondence function, it is possible to guide the learning of the mathematical model including the coding part and the attention mechanism. be.

Hereinafter, the monotonic constraint has a correspondence relationship between the elements of the first array and the elements of the second array, and as the subscript (number) of the elements of the second array increases, the correspondence relationship with the elements of the second array becomes. It is a constraint that the subscripts (numbers) of the elements of a certain first array do not decrease. Hereinafter, the continuity constraint has a correspondence relationship between the elements of the first array and the elements of the second array, and when the subscripts (numbers) of adjacent elements in the second array are continuous, in the second array. It is a constraint that the difference between the subscripts of the elements of the first array that correspond to the subscripts of the adjacent elements is not more than a predetermined positive value.

(First Embodiment)
In the first embodiment, a learning method and an inference method are applied to an applied problem such as collation or classification. Applied problems such as collation or classification include, for example, motion recognition, voice recognition, biological signal classification, signature authentication, and the like.

At the learning stage, the learning device uses the attention mechanism to learn the mathematical model. That is, in the learning stage, the learning device learns a mathematical model having a large number of parameters using the learning data. The learning device generates a trained mathematical model by determining the numerical values of the parameters of the mathematical model. At the execution stage, the inference device executes the inference process using the trained mathematical model. For example, the inference device performs a task of interest such as collation or classification.

First, an inference method applied to an applied problem such as collation or classification in the execution stage will be described.

FIG. 1 is a diagram showing a configuration example of the inference device 1 in the first embodiment. In the execution stage of the first embodiment, the inference method is applied to an applied problem such as collation or classification. The inference device 1 acquires the first array and the second array as inputs. For example, in motion recognition, the inference device 1 acquires, as an input, an array in which the coordinates of a plurality of feature points (for example, joint positions) in the human body are arranged in chronological order. In signature authentication, the inference device 1 acquires, as an input, an array in which signature coordinates, pen pressure, and the like on the display of the signature collection device are arranged in chronological order. The inference device 1 derives the distance between the first array and the second array. The inference device 1 executes the inference process based on the distance. The inference device 1 outputs the inference result to a predetermined external device (not shown).

Distances can be used to solve application problems such as matching or classification. For example, in a classification problem, the inference device 1 derives the distance between a learning array whose class is known and a target array whose class is unknown. The inference device 1 estimates the class of the target sequence using the K-nearest neighbor method, a support vector machine, or the like. In the search problem, the inference device 1 derives the distance between the query array and the array in the database. The inference device 1 derives the array having the shortest distance as a search result.

The inference device 1 includes a coding unit 10-1, a coding unit 10-2, an attention mechanism 11, a collation unit 12, and an inference unit 13.

The details of the functional part of the inference device 1 will be described.
<Encoding unit 10>
The coding unit 10-1 acquires the first array as an input. The coding unit 10-2 acquires the second array as an input. The coding unit 10-1 outputs the first feature array (first feature expression) to the attention mechanism 11 and the collation unit 12. The coding unit 10-2 outputs the second feature array (second feature expression) to the attention mechanism 11 and the collation unit 12.

The operation of the coding unit 10-1 is the same as the operation of the coding unit 10-2. Therefore, the operation of the coding unit 10-1 will be described below. In the following, items common to the coding unit 10-1 and the coding unit 10-2 will be referred to as "encoding unit 10" by omitting a part of the code. The coding unit 10 derives an array having a numerical value or a numerical vector as an element as a first feature array based on the first array.

<First example of coding unit 10>
In the first example of the coding unit 10, the coding unit 10 derives the first feature array from the first array by using an artificial neural network. In the learning stage, the parameters of the artificial neural network are determined based on the training data.

The details of the processing of the first example of the coding unit 10 are as follows.
In the first example of the coding unit 10, the coding unit 10 changes the length of the first array to a predetermined length (for example, 1024). This is necessary to enable batch learning or mini-batch learning when learning of the artificial neural network is performed. Each element of the first array is a one-dimensional numerical value or a multidimensional numerical vector.

For each dimension of the elements of the first array whose length has been changed, the coding unit 10 sets the average of all the numerical values in the dimension to 0 and the dispersion of all the numerical values in the dimension to 1. Normalize all numbers in that dimension. The normalized first sequence is, for example, a "1x1024x5" tensor. This "1024" is an example of the length of the sequence. This "5" is an example of the number of dimensions of the elements of the array.

The coding unit 10 inputs the normalized first array to the convolutional neural network. The convolutional neural network includes, for example, one "1x7x64" convolutional layer, one maximum pooling layer, and two "1x3x64" convolutional layers. Immediately after each convolution layer is a batch normalization layer. Following the batch normalization layer, the ReLU layer is provided as an activation function. The final ReLU layer outputs an array whose elements are multidimensional numerical vectors.

The coding unit 10 normalizes all the numerical values of the element so that the L2 norm of all the numerical values of the element becomes 1 for each element of the array having the multidimensional numerical vector as an element. The coding unit 10 outputs the normalized sequence as the first feature array to the attention mechanism 11 and the collating unit 12. In the first example of the coding unit 10, a recurrent neural network or the like may be used instead of the convolutional neural network.

<Second example of coding unit 10>
In the second example of the coding unit 10, the coding unit 10 outputs the input first array as the first feature array to the attention mechanism 11 and the collating unit 12. In the second example of the coding unit 10, the coding unit 10 has no parameters.

<Caution mechanism 11>
The attention mechanism 11 acquires the first feature array from the coding unit 10-1. The attention mechanism 11 acquires the second feature array from the coding unit 10-2. Attention mechanism 11 derives the weight of each element of the first feature array for each element of the second feature array based on each element of the first feature array and each element of the second feature array. The weight of each element of the first feature array for each element of the second feature array represents the probability that the two elements are in a correspondence relationship. The larger the weight, the higher the probability that the two elements are in a correspondence relationship. The attention mechanism 11 outputs the weight matrix to the collating unit 12.

<First example of attention mechanism 11>
In the first example of the attention mechanism 11, the attention mechanism 11 uses an artificial neural network for each element of the second feature array based on each element of the first feature array and each element of the second feature array. The weight of each element of the first feature array is derived. In the learning stage, the parameters of the artificial neural network are determined based on the training data.

The details of the processing of the first example of the attention mechanism 11 are as follows.
In the first example of the attention mechanism 11, the attention mechanism 11 connects the numerical vector which is each element of the first feature array and the numerical vector which is each element of the second feature array along the dimensional direction of the numerical vector. do. Attention mechanism 11 inputs the concatenated numerical vectors into the artificial neural network.

The artificial neural network includes, for example, three fully connected layers. Of the three fully bonded layers, the first fully bonded layer has 64 hidden units, the second fully bonded layer has 16 hidden units, and the third fully bonded layer has 16 hidden units. It has one hidden unit. Immediately after the first fully connected layer, a ReLU layer is provided as an activation function. Immediately after the second fully connected layer, a ReLU layer is provided as an activation function. The third fully connected layer outputs one real number.

For each element of the second feature array, the attention mechanism 11 normalizes an array containing all the real numbers derived using the element and each element of the first feature array using the Softmax function. The array including all the derived real numbers is a collection of the real numbers output for each element of the first feature array as an array. An array containing all the derived real numbers contains the same number of real numbers as the number of elements in the first feature array. Attention mechanism 11 derives a normalized real number as the weight of each element of the first feature array for each element of the second feature array. Attention mechanism 11 outputs a matrix including all the weights of each element of the first feature array to each element of the second feature array to the collating unit 12 as a weight matrix.

<Second example of attention mechanism 11>
The details of the processing of the second example of the attention mechanism 11 are as follows.
In the second example of the attention mechanism 11, the attention mechanism 11 derives the inner product of each element of the first feature array and each element of the second feature array. Attention mechanism 11 normalizes an array including all the inner products of each element of the second feature array and each element of the first feature array for each element of the second feature array by the Softmax function. Attention mechanism 11 derives a normalized inner product as the weight of each element of the first feature array for each element of the second feature array. Attention mechanism 11 outputs a matrix including all the weights of each element of the first feature array to each element of the second feature array to the collating unit 12 as a weight matrix.

In the second example of the attention mechanism 11, the attention mechanism 11 has no parameters. In order to learn a mathematical model including the coding unit 10 and the attention mechanism 11, the mathematical model cannot be learned unless the mathematical model has parameters. Therefore, when the second example of the coding unit 10 is used, the second example of the attention mechanism 11 cannot be used. That is, when the coding unit 10 having no parameters is used, the attention mechanism 11 having no parameters cannot be used.

<Collation unit 12>
The collation unit 12 acquires the first feature array from the coding unit 10-1. The collation unit 12 acquires the second feature array from the coding unit 10-2. The collation unit 12 acquires the weight matrix from the attention mechanism 11. The collation unit 12 derives the distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix. The collation unit 12 outputs the distance (distance information) between the first array and the second array to the inference unit 13. The collation unit 12 may output the distance (distance information) to a predetermined external device (not shown).

<First example of collation unit 12>
In the first example of the collating unit 12, the collating unit 12 uses a weighting matrix to perform weighting on each element of the first feature array. The collation unit 12 derives a new feature array obtained by weighting as a conversion feature array. The collation unit 12 derives the distance between the converted feature array and the second feature array as the distance between the first array and the second array.

The details of the processing of the first example of the collating unit 12 are as follows.
In the first example of the collating unit 12, the collating unit 12 uses the weights of each element of the first feature array for each element of the second feature array for each element of the second feature array, and all of the first feature array. Derivation of the weighted sum of the elements of. As a result, the elements of the first feature array that correspond to each element of the second feature array are specified (extracted or generated) as the weighted sum. That is, the elements of the first feature array that correspond to each element of the second feature array are aligned. Therefore, the non-linear time variation with respect to the local transition and the change in velocity existing between the first sequence and the second sequence is compensated.

The collation unit 12 has a distance (for example, a numerical value or a numerical vector) between each element (numerical value or numerical vector) of the second feature array and the weighted sum (numerical value or numerical vector) of all the elements of the first feature array derived for the element. , Euclidean distance) is derived as a local distance. Since the time variation between the first sequence and the second sequence has already been compensated, there is a high probability that each element of the second feature sequence and the weighted sum derived for the element are in a correspondence relationship. Therefore, the collating unit 12 derives the distance between each element of the second feature array and the weighted sum derived for the element, so that the local difference between the first feature array and the second feature array is obtained. It becomes possible for the collating unit 12 to derive a distance that more accurately represents.

The collation unit 12 derives the sum or average of all local distances for all the elements of the second feature array. The collation unit 12 outputs the sum or average of the local distances to the inference unit 13 as the distance between the first array and the second array. Here, the first feature array is expressed as "X ∈ R ^{W × K} ", and the second feature array is expressed as "Y ∈ R ^{W × K} ". "W" represents the length of the feature array. "K" represents the number of dimensions of a numerical value or a numerical vector which is an element of the feature array. The j-th row vector "x _j ∈ R ^{1 × K} " of "X" represents the j-th element of "X". Similarly, the i-th row vector "y _i ∈ R ^{1 × K} " of "Y" represents the i-th element of "Y".

The weight matrix is written as "P ∈ R ^{W × W} ". The i-th row vector "pi ∈ R ^{1 × W} " of "P" includes the weight " _{pi 1, ..., p i W "} of "x ₁ , ..., X W" with respect to "y _i ". The jth element " _pij " of " _pi " represents the weight of "x _j " with respect to "y _i ".

Since " _pi " is normalized by the Softmax function, the sum of " _pi1 , ..., _piW " is 1. Therefore, the distance between the first sequence and the second sequence is expressed by the equation (1).

Here, " _pi X" represents the weighted sum of "x ₁ , ..., X _W " with respect to "y _i ". "|| p _i X-y _i ||" represents the Euclidean distance between " _pi X" and "y _i ", that is, the local distance.

<Second example of collation unit 12>
In the second example of the collating unit 12, the collating unit 12 derives the distance between each element of the first feature array and each element of the second feature array. The collating unit 12 uses the weighting matrix to perform weighting on the distance. The collating unit 12 derives the distance between the first array and the second array based on the weight.

The details of the processing of the second example of the collating unit 12 are as follows.
In the second example of the collating unit 12, the collating unit 12 derives the distance (for example, the Euclidean distance) between each element of the first feature array and each element of the second feature array as a local distance. The collation unit 12 uses the weight matrix to derive the weighted sum or weighted average of the local distances. The collation unit 12 outputs the weighted sum or weighted average of the local distances to the inference unit 13 as the distance between the first array and the second array.

The weight of each element of the first feature array for each element of the second feature array represents the probability that the two elements are in a correspondence relationship. The larger the weight, the higher the probability that the two elements are in a correspondence relationship. The collation unit 12 gives a larger weight to the two elements having a high probability of being in a correspondence relationship with respect to the local distance between the two elements. The collating unit 12 imparts a smaller weight to the local distance between the two elements to the two elements having a low probability of being in a correspondence relationship.

This compensates for the non-linear time variation with respect to the local and velocity changes that exist between the first and second sequences. Also, the distance between the first array and the second array is more accurately derived.

Similar to the first example of the collating unit 12, in the second example of the collating unit 12, the first feature array is written as “X ∈ R ^{W × K} ” and the second feature array is “Y ∈ R ^{W × K} ”. It is written as. The length of the feature array is written as "W". The j-th element of "X" is written as "x _j ∈ R ^{1 × K} ", and the i-th element of “Y” is written as “y _i ∈ R ^{1 × K} ”. The weight matrix is written as "P ∈ R ^{W × W} ". The weight of "x _j " for "y _i " is expressed as " _pij ∈ P". Therefore, the distance between the first sequence and the second sequence is expressed by the equation (2).

Here, "|| x _j -y _i ||" represents the Euclidean distance between "x _j " and "y _i ", that is, the local distance.

<Inference unit 13>
The inference unit 13 obtains the weighted sum or the weighted average of the local distances from the collation unit 12 as the distance between the first array and the second array. The inference unit 13 executes the inference process based on the distance between the first array and the second array. The inference unit 13 outputs the inference result to a predetermined external device (not shown). The inference process is not limited to a specific inference process. For example, when the authors of a plurality of handwritten signatures are inferred, the signatures unknown to the author (first array) and the signatures known to the author (second array) are input to the trained mathematical model. The inference unit 13 uses the author ID (identification number) of the second array, which has the shortest distance between the first array and the second array acquired from the collation unit 12, as the author ID (inference result) of the first array. Output. When a plurality of second sequences exist for each writer, the reasoning unit 13 may output the writer ID having the shortest average value of distances as an inference result.

Next, a learning method applied to an applied problem such as collation or classification in the learning stage will be described.

FIG. 2 is a diagram showing a configuration example of the learning device 2 in the first embodiment. In the learning stage of the first embodiment, the learning method is applied to applied problems such as collation or classification. The learning device 2 acquires the first array, the second array, and the label as inputs. The learning device 2 derives the objective function value and the constraint function value. The learning device 2 outputs a trained mathematical model (learning result) to a predetermined external device (not shown) based on the objective function value and the constraint function value. Further, the learning device 2 outputs the trained mathematical model to the inference device 1 before the execution stage.

The first array, the second array, and the label are learning data used by the learning device 2 to learn a mathematical model for performing a task of a predetermined purpose (for example, collation or classification). The label indicates whether or not the first array and the second array belong to the same class. The objective function value and the constraint function value are used for the learning device 2 to learn the mathematical model. For example, the learning device 2 is mathematical so that the weighted sum or weighted average of the objective function value and the constraint function value derived using a large number of training data is as small as possible (for example, to be the minimum). Update model parameters. The larger the number of training data, the better the performance of the mathematical model. The number of training data is, for example, about 20,000 to 30,000.

The learning device 2 includes a coding unit 20-1, a coding unit 20-2, an attention mechanism 21, an objective function value derivation unit 22, a constraint function value derivation unit 23, and an update unit 24.

The details of the functional unit of the learning device 2 will be described.
<Encoding unit 20>
The coding unit 20-1 acquires the first array as an input. The coding unit 20-2 acquires the second array as an input. The operation of the coding unit 20-1 is the same as the operation of the coding unit 20-2. The processing of the coding unit 20-1 in the learning stage is the same as the processing of the coding unit 10-1 in the execution stage. The processing of the coding unit 20-2 in the learning stage is the same as the processing of the coding unit 10-2 in the execution stage.

The coding unit 20-1 outputs the first feature array to the attention mechanism 21 and the objective function value deriving unit 22. The coding unit 20-2 outputs the second feature array to the attention mechanism 21 and the objective function value deriving unit 22. In the following, the matters common to the coding unit 20-1 and the coding unit 20-2 will be referred to as "encoding unit 20" by omitting a part of the code.

<Caution mechanism 21>
The attention mechanism 21 acquires the first feature array from the coding unit 20-1. The attention mechanism 21 acquires the second feature array from the coding unit 20-2. The processing of the attention mechanism 21 in the learning stage is the same as the processing of the attention mechanism 11 in the execution stage. The attention mechanism 21 outputs the weight matrix to the objective function value derivation unit 22 and the constraint function value derivation unit 23.

<Objective function value derivation unit 22>
The objective function value derivation unit 22 acquires the label as an input. The objective function value derivation unit 22 acquires the first feature array and the second feature array from the coding unit 20. The objective function value derivation unit 22 acquires the weight matrix from the attention mechanism 21. The objective function value derivation unit 22 derives the difference between the first feature array and the second feature array based on the first feature array, the second feature array, and the weight matrix. The objective function value derivation unit 22 derives the objective function value so that the derived difference is associated with the label.

When the first array and the second array belong to the same class, the larger the difference, the larger the objective function value. When the first array and the second array belong to different classes, the smaller the difference, the larger the objective function value. The objective function value derivation unit 22 outputs such an objective function value to the update unit 24.

<First example of objective function value derivation unit 22>
When the first example of the collation unit 12 is used in the execution stage, the first example of the objective function value derivation unit 22 is used in the learning stage, and the second example of the objective function value derivation unit 22 is used. More desirable than. In the first example of the objective function value derivation unit 22, the objective function value derivation unit 22 uses a weight matrix to perform weighting on each element of the first feature array. The objective function value derivation unit 22 derives a new feature array obtained by weighting as a conversion feature array. The objective function value derivation unit 22 derives the difference between the conversion feature array and the second feature array. The objective function value derivation unit 22 derives the objective function value so that the derived difference is associated with the label.

The details of the processing of the first example of the objective function value derivation unit 22 are as follows.
In the first example of the objective function value derivation unit 22, the objective function value derivation unit 22 uses the weight of each element of the first feature array with respect to each element of the second feature array for each element of the second feature array. The weighted sum of all the elements of the first feature array is derived.

As a result, the elements of the first feature array that correspond to each element of the second feature array are specified (extracted or generated) as the weighted sum. That is, the elements of the first feature array that correspond to each element of the second feature array are aligned. Therefore, the non-linear time variation with respect to the local transition and the change in velocity existing between the first sequence and the second sequence is compensated.

The objective function value derivation unit 22 determines the distance (for example, Euclidean distance) between the weighted sum (numerical value or numerical vector) of all the elements of the first feature array and each element (numerical value or numerical vector) of the second feature array. , Derived as a local distance. The objective function value derivation unit 22 derives the local objective function value using the local distance. When the first array and the second array belong to the same class, the longer the local distance, the larger the local objective function value. When the first array and the second array belong to different classes, the shorter the local distance, the larger the local objective function value.

The objective function value derivation unit 22 derives the sum or average of all local objective function values for all the elements of the second feature array. The objective function value derivation unit 22 outputs the sum or average of the local objective function values to the update unit 24 as the objective function value. Here, the first feature array is expressed as "X ∈ R ^{W × K} ". The second feature array is written as "Y ∈ R ^{W × K} ". The length of the feature array is written as "W". The jth element of "X" is written as "x _j ∈ R ^{1 × K} ". The i-th element of "Y" is written as "y ∈ R ^{1 × K} ".

The weight matrix is expressed as "P ∈ R ^{W × W} ". The i-th row vector "pi ∈ R ^{1 × W} " of "P" includes the weight " _{pi 1, ..., p i W "} of "x ₁ , ..., X W" with respect to "y _i ". The label is written as "z ∈ {0,1}". When the first array and the second array belong to the same class, the label is "z = 1". When the first array and the second array belong to different classes, the label is "z = 0". Therefore, the objective function value is expressed as in Eq. (3).

Here, " _pi X" represents the weighted sum of "x ₁ , ..., X _W " with respect to "y _i ". "|| p _i X-y _i ||" represents the Euclidean distance between " _pi X" and "y _i ", that is, the local distance. "Τ" is a hyperparameter and is a positive real number.

At the learning stage, the updater 24 makes the weighted sum or weighted average of the objective function value and the constraint function value derived using a large number of training data as small as possible (for example, to be minimized). ), The parameters of the mathematical model including the coding unit 20 and the attention mechanism 21 are updated. By minimizing the objective function value, the parameters are updated so that the mathematical model derives a smaller local distance when the first and second arrays belong to the same class.

The first of the objective function value derivation unit 22 so that the mathematical model can more correctly identify the elements of the first feature array that are similar to each element of the second feature array when the first array and the second array belong to the same class. The parameters are updated based on the objective function value of one example. That is, the objective function value so that the mathematical model can more correctly identify the elements of the first feature array that correspond to each element of the second feature array when the first array and the second array belong to the same class. The parameters are updated based on the objective function value of the first example of the derivation unit 22.

By using the mathematical model learned in this way, the correspondence between each element of the first feature array and each element of the second feature array is more accurately specified. The distance between the first array and the second array is derived more correctly. Also, without relying on the use of manually designed feature representations, more complex feature representations can be derived and available sequence alignments can be achieved as compared to the dynamic time expansion and contraction method.

<Second example of objective function value derivation unit 22>
When the second example of the collating unit 12 is used in the execution stage, the first example of the objective function value deriving unit 22 is used when the second example of the objective function value deriving unit 22 is used in the learning stage. More desirable than. In the second example of the objective function value derivation unit 22, the objective function value derivation unit 22 derives the distance between each element of the first feature array and each element of the second feature array. The objective function value derivation unit 22 uses the weight matrix to perform weighting on the distance. The objective function value derivation unit 22 derives the degree of similarity between the first feature array and the second feature array. The objective function value derivation unit 22 derives the objective function value so that the derived similarity is associated with the label.

The details of the processing of the second example of the objective function value derivation unit 22 are as follows.
In the second example of the objective function value derivation unit 22, the objective function value derivation unit 22 uses the distance between each element of the first feature array and each element of the second feature array (for example, the Euclidean distance) as a local distance. Derived. The objective function value derivation unit 22 uses a weight matrix to derive a weighted sum or a weighted average of local distances. The objective function value derivation unit 22 derives the objective function value so that the derived weighted sum or weighted average is associated with the label.

Here, the first feature array is expressed as "X ∈ R ^{W × K} ". The second feature array is written as "Y ∈ R ^{W × K} ". The length of the feature array is written as "W". The jth element of "X" is written as "x _j ∈ R ^{1 × K} ". The i-th element of "Y" is written as "y _i ∈ R ^{1 × K} ". The weight matrix is written as "P ∈ R ^{W × W} ". The weight of "x _j " for "y _i " is expressed as " _pij ∈ P". The label is written as "z ∈ {0,1}". When the first array and the second array belong to the same class, the label is "z = 1". When the first array and the second array belong to different classes, the label is "z = 0". Therefore, the degree of similarity between the first feature sequence and the second feature sequence is expressed by the equation (4).

Here, "|| x _j -y _i ||" represents the Euclidean distance between "x _j " and "y _i ", that is, the local distance. The objective function value is expressed by the equation (5).

In the learning stage, the update unit 24 includes the coding unit 20 and the attention mechanism 21 so that the objective function value derived using a large amount of training data becomes as small as possible (for example, to be minimized). Update the parameters of the mathematical model including. By minimizing the objective function value, the parameters are updated so that the mathematical model derives a smaller local distance when the first and second arrays belong to the same class.

When the first array and the second array belong to the same class, the update unit 24 updates the parameters of the mathematical model so that a larger weight is derived for the two elements having a high probability of being in a correspondence relationship. .. When the first array and the second array belong to the same class, the updater 24 updates the parameters of the mathematical model so that smaller weights are derived for the two elements that are unlikely to be in a correspondence relationship. .. That is, the parameters of the mathematical model are updated so that the correspondence between each element of the first feature array and each element of the second feature array can be more accurately specified.

By using the mathematical model learned in this way, the correspondence between each element of the first feature array and each element of the second feature array is more correctly identified, and the first array and the second array are used. The distance between and can be derived more correctly. In addition, it is possible to derive more complicated feature representations and realize usable sequence alignment as compared with the dynamic time expansion / contraction method, without depending on the use of manually designed feature representations.

<Constraint function value derivation unit 23>
The constraint function value derivation unit 23 acquires the weight matrix from the attention mechanism 21. The constraint function value derivation unit 23 derives the constraint function value using the weight matrix. The constraint function value derivation unit 23 derives the constraint function value so that the greater the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied, the smaller the constraint function value. The constraint function value derivation unit 23 outputs the constraint function value to the update unit 24.

In the mathematical model including the coding unit 20 and the attention mechanism 21, the correspondence between each element of the first feature array and each element of the second feature array is monotonic by minimizing the constraint function value. It is learned to derive a weighting matrix that satisfies at least one of a constraint and a continuity constraint.

The details of the processing of the constraint function value derivation unit 23 are as follows.
The weight matrix is a matrix representing the probability that each element of the first feature array and each element of the second feature array have a correspondence relationship, and is not the correspondence relationship itself. Therefore, the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied cannot be evaluated directly from the weight matrix.

In order to evaluate the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied, it is necessary to transform the weight matrix into a form like a corresponding function. This correspondence function is, for example, a function in which the subscript of each element of the second feature array is an independent variable and the subscript of the element of the first feature array that corresponds to the subscript of each element of the second feature array is a dependent variable. Is.

Therefore, the constraint function value derivation unit 23 derives the product of the weight matrix and the predetermined arithmetic progression as a corresponding array. Arithmetic progression is a sequence that has a common difference for adjacent elements.

FIG. 3 is a diagram showing an example of a corresponding sequence in the first embodiment. On the upper side in FIG. 3, an example of a weight matrix, an example of an arithmetic progression, and an example of a corresponding array are shown for the case where the monotonicity constraint and the continuity constraint are satisfied. On the lower side in FIG. 3, an example of a weight matrix, an example of an arithmetic progression, and an example of a corresponding array are shown for the case where the monotonicity constraint and the continuity constraint are not satisfied. That is, on the left side of the equal sign, the product of the weight matrix and the arithmetic progression "[1,2,3,4] ^T " is represented. Each row of the weight matrix has been normalized, and each row of the weight matrix has a sum of elements of 1. The corresponding array is shown on the right side of the equal sign.

The subscript of the corresponding array derived using the arithmetic progression represents the subscript (number) of each element of the second feature array. The numerical value which is an element of the corresponding array represents the subscript (number) of the element of the first feature array which has a correspondence relationship with each element of the second feature array. The numerical value that is an element of the corresponding array may represent a numerical value that is proportional to the subscript of the element of the first feature array that has a corresponding relationship with each element of the second feature array.

In FIG. 3, a corresponding array is derived using a weight matrix and an arithmetic progression. For example, in the example shown on the upper side in FIG. 3, the corresponding array represents that the first element of the second feature array has a correspondence relationship with the first element of the first feature array. The corresponding array represents that the second element of the second feature array has a correspondence with the second element of the first feature array. The correspondence array represents that the third element of the second feature array has a correspondence with the second element of the first feature array.

The subscripts of the elements of the first feature array, which correspond to the fourth element of the second feature array, are not represented using integers, but are represented as "3.6" using real numbers. There is. By using such a corresponding array, it becomes possible to evaluate the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied.

The constraint function value derived using the corresponding array needs to be smaller as the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied becomes larger. In order for the learning device 2 to learn the mathematical model using the gradient method, it is desirable that the constraint function value is differentiable with respect to the weight matrix or the corresponding array. In addition, it is desirable that parallelization of the derivation of constraint function values is easy in order to enable faster learning.

The constraint function value derivation unit 23 derives at least one of the monotonic constraint function value and the continuity constraint function value as the constraint function value.

<Monotonic constraint function value>
The constraint function value derivation unit 23 compares the size of the element immediately before the element of the corresponding array with the element of the corresponding array for each element of the corresponding array, so that the function value of the local monotonic constraint (the function value of the local monotonic constraint) ( Hereinafter, “local monotonic constraint function value”) is derived. The local monotonic constraint function value is the absolute value of the difference between these two elements when the element immediately preceding the element of the corresponding array is larger than the element of the corresponding array. The local monotonicity constraint function value is 0 when the element immediately preceding the element of the corresponding array is equal to or less than the element of the corresponding array.

The constraint function value derivation unit 23 derives the sum or average of all the local monotonic constraint function values for all the elements in the corresponding array. The constraint function value derivation unit 23 outputs the sum or average of the local monotonic constraint function values to the update unit 24 as the monotonic constraint function value.

Here, the weight matrix is expressed as "P ∈ R ^{W × W} ". The length of the feature array is written as "W". The corresponding array is written as "F ∈ R ^{W × 1} ". The _i -th element of "F" is written as "fi". Therefore, the monotonic constraint function value is expressed by Eq. (6).

Here, "f ₀ " is 0. By implementing equation (6) using a library of convolutional neural networks, the monotonic constraint function value is derived at a higher speed.

FIG. 4 is a diagram showing an example of deriving the monotonic constraint function value in the first embodiment. On the upper side in FIG. 4, an example of deriving the monotonicity constraint function value is shown for the case where the monotonicity constraint and the continuity constraint are satisfied. On the lower side in FIG. 4, an example of deriving the monotonicity constraint function value is shown for the case where the monotonicity constraint and the continuity constraint are not satisfied.

In FIG. 4, in order from the left side, an example of a corresponding array, an example of a filter, an example of the difference between two adjacent elements in the corresponding array, an example of a local monotonic constraint function value, and an example of a monotonic constraint function value. Is expressed. In FIG. 4, the symbol “x” in the circle indicates convolution. "Loss" represents a monotonic constraint function value. The greater the degree to which the corresponding array satisfies the monotonic constraint, the smaller the monotonic constraint function value is derived. The smaller the degree to which the corresponding array satisfies the monotonic constraint, the larger the monotonic constraint function value is derived.

In FIG. 4, as a result of the convolution of the corresponding array and the filter "[1, -1] ^T ", the difference between two adjacent elements in the corresponding array is derived. The constraint function value derivation unit 23 applies "ReLU" as an activation function to an array of differences between two adjacent elements. In this way, the local monotonic constraint function value is derived. By deriving the average of all the elements in the array of local monotonic constraint function values, the monotonic constraint function value as in Eq. (6) can be easily derived.

The filter may be any filter capable of deriving the difference between two elements whose positions are close to each other in the corresponding array. For example, a filter such as "[1,0, -1] ^T " or "[2,1, -1, -2] ^T " may be used in place of "[1, -1] ^T ". ..

<Continuity constraint function value>
The constraint function value derivation unit 23 derives the absolute value of the difference between the element immediately before the element of the corresponding array and the element of the corresponding array for each element of the corresponding array. The constraint function value derivation unit 23 subtracts a predetermined positive number from the derived absolute value. This predetermined positive number is a hyperparameter and is a positive integer such as 1, 2 or 3. Real numbers such as "1.5" may be used as hyperparameters.

The constraint function value derivation unit 23 derives the maximum value of the numerical value of the subtraction result and 0 as the function value of the local continuity constraint (hereinafter referred to as “local continuity constraint function value”). The constraint function value derivation unit 23 derives the sum or average of all the local continuity constraint function values for all the elements in the corresponding array. The constraint function value derivation unit 23 outputs the sum or average of the local continuity constraint function values to the update unit 24 as the continuity constraint function value.

The weight matrix is written as "P ∈ R ^{W × W} ". The length of the feature array is written as "W". The corresponding array is written as "F ∈ R ^{W × 1} ". The _i -th element of "F" is written as "fi". Therefore, the continuity constraint function value is expressed as in Eq. (7).

Here, "f ₀ " is 0. By implementing the equation (7) using the library of the convolutional neural network, the continuity constraint function value is derived at a higher speed.

FIG. 5 is a diagram showing an example of deriving the continuity constraint function value in the first embodiment. On the upper side in FIG. 5, an example of deriving the continuity constraint function value is shown for the case where the monotonicity constraint and the continuity constraint are satisfied. On the lower side in FIG. 5, an example of deriving the continuity constraint function value is shown for the case where the monotonicity constraint and the continuity constraint are not satisfied.

In FIG. 4, in order from the left side, an example of a corresponding array, an example of a filter, an example of a predetermined positive number, and an example of a predetermined positive number are subtracted from the absolute value of the difference between two adjacent elements in the corresponding array. The results, an example of the local continuity constraint function value, and an example of the continuity constraint function value are shown. In FIG. 5, the symbol “x” in the circle indicates convolution. "Loss" represents a continuity constraint function value.

In FIG. 5, the convolution of the corresponding array and the filter "[-1,1] ^T " derives the difference between two adjacent elements in the corresponding array. The constraint function value derivation unit 23 derives the absolute value of each element in the array of the differences between two adjacent elements. The constraint function value derivation unit 23 subtracts a predetermined positive number (1 in FIG. 5) from the derived absolute value. The constraint function value derivation unit 23 applies "ReLU" as an activation function to the array of subtraction results. In this way, the local continuity constraint function value is derived. By deriving the average of all the elements in the array of local continuity constraint function values, the continuity constraint function value as in Eq. (7) can be easily derived.

The filter may be any filter capable of deriving the difference between two elements whose positions are close to each other in the corresponding array. For example, a filter such as "[1,0, -1] ^T " or "[2,1, -1, -2] ^T " may be used in place of "[1, -1] ^T ". ..

As shown in FIG. 5, the greater the degree to which the corresponding array satisfies the continuity constraint, the smaller the continuity constraint function value is derived. The smaller the degree to which the corresponding array satisfies the continuity constraint, the larger the continuity constraint function value is derived.

<Update part 24>
The update unit 24 acquires the objective function value from the objective function value derivation unit 22. The update unit 24 acquires the constraint function value from the constraint function value derivation unit 23. The update unit 24 executes the learning process based on the objective function value and the constraint function value. The learning process is not limited to a specific learning process. The update unit 24 is a mathematical model including the coding unit 20 and the attention mechanism 21 so that the weighted sum or the weighted average of the constraint function value and the objective function value is as small as possible (for example, to be minimized). Update the parameters of. The update unit 24 outputs a trained mathematical model (learning result) to a predetermined external device (not shown).

As described above, in the learning stage, the attention mechanism 21 uses the first feature sequence based on the first sequence and the second feature sequence based on the second sequence, and each of the first feature sequence and the second feature sequence is used. Generate a weighting matrix, which is a matrix that represents the probability that the elements are in a correspondence. The objective function value derivation unit 22 weights the label indicating whether or not the first array and the second array belong to the same class, and the objective function value which is a value corresponding to the first feature array and the second feature array. Derived based on the matrix. The constraint function value derivation unit 23 derives a constraint function value representing at least one of a monotonic constraint and a continuity constraint based on a weight matrix. The update unit 24 generates a learning result by executing a predetermined learning process based on the objective function value and the constraint function value. The objective function value is, for example, a value according to the difference or similarity between the first feature array and the second feature array and the label. The update unit 24 updates the mathematical model.

The mathematical model updated in the learning stage is used to execute the inference process in the execution stage. At the execution stage, the attention mechanism 11 uses the first feature sequence based on the first sequence and the second feature sequence based on the second sequence, and each element of the first feature sequence and the second feature sequence has a correspondence relationship. Generate a weighting matrix, which is a matrix representing a certain probability. The collation unit 12 derives the distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix. The inference unit 13 generates an inference result by executing a predetermined inference process based on the distance.

In this way, a weight matrix that works effectively by the coding unit deriving a feature array using a mathematical model trained using a constraint function value that represents at least one of a monotonic constraint and a continuity constraint. Is generated by the attention mechanism based on the feature array.

This makes it possible to derive and use more complex feature representations without relying on the use of manually designed feature representations, while at the same time deriving and using monotonous and continuous correspondence functions for array alignment. It is possible to do. It is possible to realize more complicated feature representations without relying on the use of manually designed feature representations. In addition, it is possible to improve the inference accuracy and shorten the learning time at the same time.

According to the learning device 2, the learning method and the program, the update unit 24 guides (guides) the learning of the mathematical model when learning the mathematical model so that the attention mechanism 11 can derive a monotonous and continuous correspondence function. Will be possible. By using the attention mechanism 11 in the trained mathematical model, it is possible to correctly derive the distance or similarity between sequences in application problems such as matching or classification. It is possible to correctly infer whether or not the array belongs to a different class. In addition, it is possible to shorten the learning time (time required for learning a mathematical model) until the attention mechanism 11 can provide sufficient performance.

(Second Embodiment)
The second embodiment is an embodiment for applying a learning method and an inference method to an applied problem such as synthesis or conversion of continuous data such as voice. Speech synthesis is the artificial creation of human speech, for example, the synthesis of speech from text. Speech conversion is the conversion of an individual's speech into the speech of another individual or character.

If the discontinuous data (for example, a handwritten signature) is corrected in advance so as to be continuous data, the learning method and the inference method in the second embodiment can be used for the discontinuous data.

The second embodiment is divided into a learning stage and an execution stage. In the learning stage, the learning device uses the training data to train a mathematical model with many parameters. The learning device determines the numerical values of the parameters of the mathematical model. At the execution stage, the inference device uses the trained mathematical model to perform a task of a predetermined purpose (for example, speech synthesis, speech conversion).

First, an inference method applied to an applied problem such as speech synthesis or speech conversion in the execution stage will be described.

FIG. 6 is a diagram showing a configuration example of the inference device 3 in the second embodiment. In speech synthesis, the elements of the first array are, for example, numerical vectors representing the characteristics of each word in the sentence. The characteristic of each word in a sentence is, for example, the One-Hot vector of the word. The element of the second array is, for example, a numerical vector representing the characteristics of each time or each frame of the voice.

In speech conversion, the elements of the first array are, for example, numerical vectors representing the characteristics of each time or frame of speech. The characteristics of each time or frame of speech are, for example, a predetermined extraction method (Reference 1: Masanori Morise, Fumiya Yokomori, Kenji Ozawa, "WORLD: A vocoder-based high-quality speech synthesis system for real-time applications," It is a multidimensional vector containing a Melkeptrum coefficient and a logarithmic F0 pattern extracted using "IEICE Trans. Inf. Syst. 99-D (7): 1877-1884 (2016)). The elements of the second array are, for example, numerical vectors representing the characteristics of each time or each frame in the voice of an individual or character different from the individual voice of the first array.

The inference device 3 includes a first coding unit 30, a second coding unit 31, a caution mechanism 32, a decoding unit 33, and an inference unit 34.

The first coding unit 30 acquires the first array as an input. The first coding unit 30 executes the coding process for the first array only once, for example. The first feature array is derived. The first coding unit 30 outputs the first feature array to the attention mechanism 32 and the decoding unit 33.

The second coding unit 31 acquires the elements of the second array at the previous time from the decoding unit 33. The second coding unit 31 derives the elements of the second feature array at the previous time by executing the coding process for the elements of the second array at the previous time. The second coding unit 31 outputs the element of the second feature array at the previous time to the attention mechanism 32.

The attention mechanism 32 acquires the first feature array from the first coding unit 30. The attention mechanism 32 acquires the element of the second feature array at the previous time from the second coding unit 31. Attention mechanism 32 uses the elements of the second feature array and each element of the first feature array at the previous time, and the weight of each element of the first feature array with respect to the elements of the second array at the current time. Is derived. The attention mechanism 32 outputs the weight of each element of the first feature array to the element of the second array at the current time as a weight matrix to the decoding unit 33.

The decoding unit 33 acquires the first feature array from the first coding unit 30. The decoding unit 33 acquires the weight of each element of the first feature array with respect to the element of the second array at the current time from the attention mechanism 32 as a weight matrix. The decoding unit 33 derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array. The decoding unit 33 outputs the elements of the second array at the current time to the second coding unit 31 and the inference unit 34. The decoding unit 33 may output the elements of the second array at the current time to a predetermined external device (not shown).

The second coding unit 31 acquires the elements of the second array at the current time from the decoding unit 33. The second coding unit 31 uses the elements of the second array at the current time to derive the elements of the second feature array at the current time. The second coding unit 31 outputs the elements of the second feature array at the current time to the attention mechanism 32.

In this way, the inference device 3 has a cycle in which the signal starts from the second coding unit 31, the signal passes through the attention mechanism 32 and the decoding unit 33, and the signal returns to the second coding unit 31 again. exist. After the elements of the second array are initialized at the first time, the initialized elements of the second array are input to the second coding unit 31, and at the last time, the elements of the second array are the decoding unit 33. The inference processing in this cycle is repeated every unit time from to the output.

The attention mechanism 32 outputs a matrix including all the weights of each element of the first feature array to each element of the second array to the decoding unit 33 as a weight matrix. Further, the decoding unit 33 outputs each element of the second array at all times to the inference unit 34 as the second array.

The inference unit 34 acquires the second array from the decoding unit 33. The inference unit 34 generates an inference result based on the second array. In applied problems such as speech synthesis or speech conversion, the inference result is a speech signal. The inference unit 34 outputs the inference result to a predetermined external device (not shown).

The details of the functional part of the inference device 3 will be described.
<First coding unit 30>
The first coding unit 30 acquires the first array as an input. The first coding unit 30 uses the first array to derive an array having a numerical value or a numerical vector as an element as a first feature array. For example, reference numeral 2 (Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ-Skerrv Ryan, Rif A) . Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu, "Natural TTS synthesis by conditioning wavenet on MEL spectrogram predictions," In ICASSP, pp.4779-4783, 2018.) Derived from one array. The first coding unit 30 determines the parameters of the artificial neural network using the training data in the learning stage. The first coding unit 30 outputs the first feature array to the attention mechanism 32 and the decoding unit 33.

The details of the processing of the first coding unit 30 are as follows.
The first sequence is, for example, a "1 x N x 512" tensor. "N" represents the length of the array. "512" is an example of the number of dimensions of the elements of the array. The first coding unit 30 inputs the first array to the artificial neural network.

The artificial neural network has, for example, three "1 x 5 x 512" convolution layers and one bidirectional long short-term memory (BiLSTM) (hereinafter referred to as "bidirectional LSTM"). And. Immediately after each convolution layer is a batch normalization layer. Immediately after the batch normalization layer, a ReLU layer is provided as an activation function. The bidirectional LSTM has a total of 512 hidden units. The bidirectional LSTM of the first coding unit 30 outputs an array having a numerical value or a numerical vector as an element to the attention mechanism 32 and the decoding unit 33 as a first feature array.

<Second coding unit 31>
The second coding unit 31 acquires the second sequence from the decoding unit 33. The element of the second array at the time immediately before is acquired from the decoding unit 33. The second coding unit 31 uses the elements of the second array at the previous time to derive a numerical value or a numerical vector as an element of the second feature array at the previous time. For the derivation of the numerical value or the numerical value vector, for example, the artificial neural network of Reference 2 described above can be used. The parameters of the artificial neural network are determined using the training data at the training stage. The second coding unit 31 outputs the second feature array to the attention mechanism 32.

The details of the processing of the second coding unit 31 are as follows.
Each element of the second array at the previous time is, for example, a 512-dimensional numerical vector. The second coding unit 31 inputs each element of the second array at the previous time to the artificial neural network. This artificial neural network includes, for example, two fully connected layers. Each fully connected layer has 256 hidden units. Immediately after each fully connected layer, a ReLU layer is provided as an activation function. The last fully connected layer outputs a numerical value or a numerical vector to the attention mechanism 32 as an element of the second feature array at the previous time.

<Caution mechanism 32>
The attention mechanism 32 acquires the first feature array from the first coding unit 30. The attention mechanism 32 acquires the second feature array from the second coding unit 31. Attention mechanism 32 uses the elements of the second feature array at the previous time and each element of the first feature array to represent each element of the first feature array with respect to the elements of the second array with respect to the current time. Derive the weight. As the attention mechanism 32, for example, an artificial neural network may be used, or a mathematical model other than the artificial neural network (for example, a linear regression model, a polynomial regression model, a logistic regression model) may be used. The parameters of the artificial neural network are determined using the training data at the training stage. The attention mechanism 32 outputs the weight matrix to the decoding unit 33.

The details of the processing of the attention mechanism 32 are as follows.
Attention mechanism 32 connects the numerical vector which is an element of the second feature array at the previous time and the numerical vector which is each element of the first feature array along the dimensional direction of the numerical vector. Attention mechanism 32 inputs the concatenated numerical vectors into the artificial neural network. The artificial neural network includes, for example, three fully connected layers. Of the three fully bonded layers, the first fully bonded layer has 64 hidden units, the second fully bonded layer has 16 hidden units, and the third fully bonded layer has 16 hidden units. It has one hidden unit. Immediately after the first fully connected layer, a ReLU layer is provided as an activation function. Immediately after the second fully connected layer, a ReLU layer is provided as an activation function. The third fully connected layer outputs one real number.

Attention mechanism 32 normalizes an array containing all the real numbers derived by using the elements of the second feature array and each element of the first feature array at the previous time by the Softmax function. The array including all the derived real numbers is a collection of the real numbers output for each element of the first feature array as an array. An array containing all the derived real numbers contains the same number of real numbers as the number of elements in the first feature array. Attention mechanism 32 derives a normalized real number as the weight of each element of the first feature array with respect to the elements of the second array at the current time. The attention mechanism 32 outputs a matrix including all the weights of each element of the first feature array to the elements of the second array at the current time as a weight matrix to the decoding unit 33.

<Decoding unit 33>
The decoding unit 33 acquires the first feature array from the first coding unit 30. The decoding unit 33 acquires the weight matrix from the attention mechanism 32. The decoding unit 33 performs weighting for each element of the first feature array by using the weight of each element of the first feature array for the elements of the second array at the current time. The decoding unit 33 uses the numerical value or the numerical value vector obtained by weighting to derive the elements of the second array at the current time. For example, the decoding unit 33 uses the artificial neural network of Reference 2 described above to derive the elements of the second array at the current time. The decoding unit 33 determines the parameters of the artificial neural network using the learning data in the learning stage. The decoding unit 33 outputs the second array to the inference unit 34.

The details of the processing of the decoding unit 33 are as follows.
The decoding unit 33 derives the weighted sum of all the elements of the first feature array by using the weight of each element of the first feature array with respect to the elements of the second array at the current time. As a result, the elements of the first feature array that correspond to the elements of the second array at the current time are specified (extracted or generated) as the weighted sum. That is, the elements of the first feature array that correspond to the elements of the second array at the current time are aligned. Therefore, the non-linear time variation with respect to the local transition and the change in velocity existing between the first sequence and the second sequence is compensated.

Here, the first feature array is expressed as "X ∈ R ^{W × K} ". The weight matrix is written as "P ∈ R ^{W × W} ". The row vector containing all the weights of each element of the first feature array with respect to the elements of the second array at the current time is expressed as " _pi ∈ R ^{1 × W} ". The current time is written as "i". The weighted sum of all the elements of the first feature array with respect to the elements of the second array at the current time is expressed as " _pi X".

The weighted sum is, for example, a 128-dimensional numerical vector. The decoding unit 33 inputs this numerical vector into the artificial neural network. The artificial neural network includes, for example, two bidirectional LSTMs and one fully connected layer. Each bidirectional LSTM has 1024 hidden units. The fully connected layer outputs a numerical value or a numerical vector to the inference unit 34 as an element of the second array at the current time.

The decoding unit 33 may derive the second array by using the second feature array, the first feature array, and the weight matrix output from the second coding unit 31. In this case, the decoding unit 33 concatenates the numerical vector which is the weighted sum and the numerical vector which is the element of the second feature array at the previous time along the dimension direction of the numerical vector. The decoding unit 33 inputs the concatenated numerical vectors to the artificial neural network.

<Inference unit 34>
The inference unit 34 acquires the second array from the decoding unit 33. The inference unit 34 generates an inference result based on the second array. In applied problems such as speech synthesis or speech conversion, the inference result is a speech signal. The inference unit 34 is, for example, a predetermined generation method (Reference 3: Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, Koray Kavukcuoglu, "WaveNet: A generative model for raw audio, "SSW 2016: 125.) is used to generate an audio signal based on the second array. The inference unit 34 outputs the inference result to a predetermined external device (not shown).

Next, a learning method applied to an applied problem such as speech synthesis or speech conversion in the learning stage will be described.

FIG. 7 is a diagram showing a configuration example of the learning device 4 in the second embodiment. In the learning stage of the second embodiment, the learning method is applied to an applied problem such as speech synthesis or speech conversion. The learning device 4 acquires the first array and the correct answer array as inputs. The learning device 4 derives the objective function value and the constraint function value. The learning device 4 learns a mathematical model based on the objective function value and the constraint function value, and outputs the trained mathematical model (learning result) to a predetermined external device (not shown). Further, the learning device 4 outputs the trained mathematical model to the inference device 3 before the execution stage.

The first array and the correct array are training data used to train a mathematical model for performing a task of a predetermined purpose (for example, speech synthesis or speech conversion). The objective function value and the constraint function value are used for the learning device 4 to learn the mathematical model. For example, the learning device 4 is designed so that the weighted sum or weighted average of the objective function value and the constraint function value derived using a large number of training data is as small as possible (for example, to be the minimum). Update the parameters of the mathematical model. The larger the number of training data, the better the performance of the mathematical model. The number of training data is, for example, about 20,000 to 30,000.

The learning device 4 includes a first coding unit 40, a second coding unit 41, an attention mechanism 42, a decoding unit 43, an objective function value derivation unit 44, a constraint function value derivation unit 45, and an update unit. It is equipped with 46.

The first coding unit 40 acquires the first array as an input. The first coding unit 40 executes the coding process for the first array only once, for example. The first feature array is derived. The first coding unit 40 outputs the first feature array to the attention mechanism 42 and the decoding unit 43.

The second coding unit 41 acquires the elements of the second array at the previous time from the decoding unit 43. The second coding unit 41 derives the elements of the second feature array at the previous time by executing the coding process for the elements of the second array at the previous time.

The attention mechanism 42 acquires the first feature array from the first coding unit 40. The attention mechanism 42 acquires the element of the second feature array at the previous time from the second coding unit 41. Attention mechanism 42 uses the elements of the second feature array and each element of the first feature array at the previous time, and the weight of each element of the first feature array with respect to the elements of the second array at the current time. Is derived. The attention mechanism 32 outputs the weight of each element of the first feature array to the element of the second array at the current time as a weight matrix to the decoding unit 43.

The decoding unit 43 acquires the first feature array from the first coding unit 40. The decoding unit 43 acquires the weight of each element of the first feature array with respect to the element of the second array at the current time from the attention mechanism 42 as a weight matrix. The decoding unit 43 derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array. The decoding unit 43 outputs the elements of the second array at the current time to the second coding unit 41 and the objective function value derivation unit 44.

The second coding unit 41 acquires the elements of the second array at the current time from the decoding unit 43. The second coding unit 41 uses the elements of the second array at the current time to derive the elements of the second feature array at the current time. The second coding unit 41 outputs the elements of the second feature array at the current time to the attention mechanism 42.

In this way, the learning device 4 has a cycle in which the signal starts from the second coding unit 41, passes through the attention mechanism 42 and the decoding unit 43, and returns to the second coding unit 41 again. exist. In this cycle, the elements of the second array are initialized at the first time, then the initialized elements of the second array are input to the second coding unit 41, and the elements of the second array are input at the last time. The learning process is repeated every unit time until the output is output from the decoding unit 43.

The attention mechanism 42 outputs a matrix including all the weights of each element of the first feature array to each element of the second array to the decoding unit 43 as a weight matrix. Further, the decoding unit 43 outputs each element of the second array at all times to the second coding unit 41 and the objective function value derivation unit 44 as the second array.

The objective function value derivation unit 44 acquires the correct answer array as an input. The objective function value derivation unit 44 acquires the second array from the decoding unit 43. The objective function value derivation unit 44 derives the objective function value based on the correct answer array and the second array. The process of deriving the objective function value by the objective function value deriving unit 44 is executed only once, for example. The objective function value derivation unit 44 outputs the objective function value to the update unit 46.

The constraint function value derivation unit 45 acquires the weight matrix from the attention mechanism 42. The constraint function value derivation unit 45 derives the constraint function value using the weight matrix. The process of deriving the constraint function value by the constraint function value deriving unit 45 is executed only once, for example. The constraint function value derivation unit 45 outputs the constraint function value to the update unit 46.

The update unit 46 acquires the objective function value from the objective function value derivation unit 44. The update unit 46 acquires the constraint function value from the constraint function value derivation unit 45. The update unit 46 executes the learning process based on the objective function value and the constraint function value. The update unit 46 has a first coding unit 40 and a second coding unit 41 so that the weighted sum or the weighted average of the constraint function value and the objective function value is as small as possible (for example, to be the minimum). The mathematical model including the attention mechanism 42 and the decoding unit 43 is updated. The update unit 46 outputs a trained mathematical model (learning result) to a predetermined external device (not shown).

The details of the functional unit of the learning device 4 will be described.
<First coding unit 40>
The first coding unit 40 acquires the first array as an input. The process executed by the first coding unit 40 in the learning stage is the same as the process executed by the first coding unit 30 in the execution stage. The first coding unit 40 outputs the first feature array to the attention mechanism 42 and the decoding unit 43.

<Second coding unit 41>
The second coding unit 41 acquires the second sequence from the decoding unit 43 and outputs the second feature sequence to the attention mechanism 42. The processing of the second coding unit 41 in the learning stage is the same as the processing of the second coding unit 31 in the execution stage. The second coding unit 41 in the learning stage may use the correct answer sequence as an input instead of using the second array as an input. In this case, all the processing performed on the second array is performed on the correct array used in place of the second array.

<Caution mechanism 42>
The attention mechanism 42 acquires the first feature array from the first coding unit 40. The attention mechanism 42 acquires the second feature array from the second coding unit 41. The processing of the attention mechanism 42 in the learning stage is the same as the processing of the attention mechanism 32 in the execution stage. The attention mechanism 42 outputs the weight matrix to the decoding unit 43 and the constraint function value derivation unit 45.

<Decoding unit 43>
The decoding unit 43 acquires the first feature array from the first coding unit 40. The decoding unit 43 acquires the weight matrix from the attention mechanism 42. The processing of the decoding unit 43 in the learning stage is the same as the processing of the decoding unit 33 in the execution stage. The decoding unit 43 outputs the second array to the objective function value derivation unit 44.

<Objective function value derivation unit 44>
The objective function value derivation unit 44 acquires the correct answer array as an input. The objective function value derivation unit 44 acquires the second array from the decoding unit 43. The objective function value derivation unit 44 derives the difference between the correct answer array and the second array. The objective function value derivation unit 44 derives the objective function value so that the larger the derived difference is, the larger the value is. The objective function value derivation unit 44 outputs the objective function value to the update unit 46.

The details of the processing of the objective function value derivation unit 44 are as follows.
The objective function value derivation unit 44 derives, for example, the residual sum of squares (similarity) between the correct answer array and the second array as the objective function value. Here, the correct answer sequence is written as "Z ^* ". The second sequence is written as "Z". Therefore, the objective function value is expressed as in Eq. (8).

Here, "|| · ||" represents the L2 norm.

<Constraint function value derivation unit 45>
The constraint function value derivation unit 45 acquires the weight matrix from the attention mechanism 42. The constraint function value derivation unit 45 derives the constraint function value using the weight matrix. Here, the constraint function value is derived so that the greater the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied, the smaller the constraint function value. The constraint function value derivation unit 45 outputs the constraint function value to the update unit 46.

A weight matrix in which the correspondence between each element of the first feature array and each element of the second array satisfies at least one of the monotonic constraint and the continuity constraint by minimizing the constraint function value. The mathematical model is trained to derive. This mathematical model includes a first coding unit 40, a second coding unit 41, an attention mechanism 42, and a decoding unit 43.

The details of the processing of the constraint function value derivation unit 45 are as follows.
The weight matrix is a matrix that represents the probability that each element of the first feature array and each element of the second array have a correspondence relationship. The weight matrix is not the correspondence itself. Therefore, the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied cannot be directly evaluated from the weight matrix.

The weight matrix needs to be transformed in order to be able to evaluate the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied. For example, a function (correspondence function) in which the time of each element of the second array is an independent variable and the subscript of the element of the first feature array having a correspondence with the time of each element of the second array is a dependent variable. The weight matrix needs to be transformed into a form. For this purpose, the constraint function value derivation unit 45 derives the product of the weight matrix and the predetermined arithmetic progression as a corresponding array. Arithmetic progression is a sequence obtained by adding a certain number (tolerance) to the term (element) immediately before each term (each element).

For example, in FIG. 3, "[1,2,3,4] ^T " is an arithmetic progression. In the corresponding array derived using the arithmetic progression, the subscript of the corresponding array represents the time of each element of the second array. The numerical value that is an element of the corresponding array represents a subscript or a numerical value proportional to the subscript of the element of the first feature array that has a corresponding relationship with each element of the second array. In the example shown on the upper side in FIG. 3, the first element of the second array has a correspondence relationship with the first element of the first feature array. The second element of the second array has a correspondence with the second element of the first feature array. The correspondence array indicates that the third element of the second array has a correspondence relationship with the second element of the first feature array. The subscripts of the elements of the first feature array that correspond to the fourth element of the second array are not represented using integers, but are represented as "3.6" using real numbers. .. By using such a corresponding array, it becomes possible to evaluate the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied.

The constraint function value derived using the corresponding array needs to become smaller as the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied is larger. In order for the learning device 4 to learn the mathematical model using the gradient method, it is desirable that the constraint function value is differentiable with respect to the weight matrix or the corresponding array. In addition, it is desirable that parallelization of the derivation of constraint function values is easy in order to enable faster learning.

The constraint function value derivation unit 45 derives at least one of the monotonic constraint function value and the continuity constraint function value as the constraint function value.

<Monotonic constraint function value>
The description of the monotonic constraint function value in the second embodiment is the same as the description of the monotonic constraint function value in the first embodiment.

<Continuity constraint function value>
The description of the continuity constraint function value in the second embodiment is the same as the description of the continuity constraint function value in the first embodiment.

<Update part 46>
The update unit 46 acquires the objective function value from the objective function value derivation unit 44. The update unit 46 acquires the constraint function value from the constraint function value derivation unit 45. The update unit 46 executes the learning process based on the objective function value and the constraint function value. The update unit 46 outputs a trained mathematical model (learning result) to a predetermined external device (not shown). The learning process is not limited to a specific learning process.

As described above, the attention mechanism 42 uses the first feature sequence based on the first sequence and the second feature sequence based on the second sequence, and each element of the first feature sequence and the second feature sequence has a correspondence relationship. Generates a weighting matrix, which is a matrix representing the probabilities in. The decoding unit 43 derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array. The objective function value derivation unit 44 derives the objective function value which is a value corresponding to the correct answer array and the second array. The constraint function value derivation unit 45 derives the constraint function value based on the weight matrix. The update unit 46 executes a predetermined learning process based on the objective function value and the constraint function value, whereby the first coding unit 40, the second coding unit 41, the attention mechanism 42, and the decoding unit 43 are combined. Update the parameters of the included mathematical model and generate the training result. The objective function value is, for example, the difference or residual sum of squares between the correct array and the second array. The update unit 46 updates the mathematical model.

The mathematical model updated in the learning stage is used to execute the inference process in the execution stage. At the execution stage, the attention mechanism 32 uses the first feature sequence based on the first sequence and the second feature sequence based on the second sequence, and each element of the first feature sequence and the second feature sequence has a correspondence relationship. Generate a weighting matrix, which is a matrix representing a certain probability. The decoding unit 33 derives the second array based on the first feature array and the weight matrix. The inference unit 34 generates an inference result by executing a predetermined inference process based on the second array.

In this way, a weight matrix that works effectively by the coding unit deriving a feature array using a mathematical model trained using constraint function values that represent at least one of a monotonic constraint and a continuity constraint. The attention mechanism produces.

This makes it possible to derive and use more complex feature representations for applied problems such as speech synthesis or speech conversion without relying on the use of manually designed feature representations, while at the same time being monotonous and continuous. It is possible to derive various corresponding functions and realize usable array alignment. It is possible to realize more complicated feature expressions for application problems such as speech synthesis or speech conversion without depending on the use of manually designed feature expressions. In addition, it is possible to achieve both improvement of inference accuracy such as speech synthesis or speech conversion and shortening of learning time.

FIG. 8 is a diagram showing a hardware configuration example of the inference device 1 in each embodiment. A part or all of each functional unit of the inference device 1 is stored in a storage unit 200 in which a processor 100 such as a CPU (Central Processing Unit) has a non-volatile recording medium (non-temporary recording medium). It is realized as software by executing the program. The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, flexible disks, optomagnetic disks, portable media such as ROM (Read Only Memory) and CD-ROM (Compact Disc Read Only Memory), and storage of hard disks built into computer systems. It is a non-temporary recording medium such as a device. The communication unit 300 transmits the processing result of the inference device 1 to an external device (not shown). The communication unit 300 may receive the program via the communication line. The display unit 400 displays the processing result of the inference device 1. The display unit 400 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

A part or all of each functional part of the inference device 1 includes, for example, an LSI (Large Scale Integration circuit), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), an FPGA (Field Programmable Gate Array), or the like. It may be realized using hardware including the used electronic circuit (electronic circuit or circuitry). The hardware configuration example of the inference device 3 is the same as the hardware configuration example of the inference device 1.

FIG. 9 is a diagram showing a hardware configuration example of the learning device 2 in each embodiment. In a part or all of each functional unit of the learning device 2, a processor 101 such as a CPU executes a program stored in a storage unit 201 having a non-volatile recording medium (non-temporary recording medium). Is realized as software. The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a flexible disk, a magneto-optical disk, a portable medium such as a ROM or a CD-ROM, or a non-temporary recording medium such as a storage device such as a hard disk built in a computer system. The communication unit 301 transmits the processing result of the learning device 2 to an external device (not shown). The communication unit 301 may receive the program via the communication line. The display unit 401 displays the processing result by the learning device 2. The display unit 401 is, for example, a liquid crystal display or an organic EL display.

A part or all of each functional part of the learning device 2 may be realized by using hardware including an electronic circuit (electronic circuit or circuitry) using, for example, LSI, ASIC, PLD, FPGA or the like. The hardware configuration example of the learning device 4 is the same as the hardware configuration example of the learning device 2.

Although the embodiment of the present invention has been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and the design and the like within a range not deviating from the gist of the present invention are also included.

The present invention is applicable to learning devices and inference devices.

1 ... Inference device, 2 ... Learning device, 3 ... Inference device, 4 ... Learning device, 10 ... Coding unit, 11 ... Attention mechanism, 12 ... Collation unit, 13 ... Inference unit, 20 ... Coding unit, 21 ... Caution Mechanism, 22 ... Objective function value derivation unit, 23 ... Constraint function value derivation unit, 24 ... Update unit, 30 ... First coding unit, 31 ... Second coding unit, 32 ... Attention mechanism, 33 ... Decoding unit, 34 ... Inference unit, 40 ... First coding unit, 41 ... Second coding unit, 42 ... Attention mechanism, 43 ... Decoding unit, 44 ... Objective function value derivation unit, 45 ... Constraint function value derivation unit, 46 ... Update unit, 100 ... Processor, 101 ... Processor, 200 ... Storage unit, 201 ... Storage unit, 300 ... Communication unit, 301 ... Communication unit, 400 ... Display unit, 401 ... Display unit

Claims (15)

It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention mechanism to generate weight matrix and
A label indicating whether or not the first array and the second array belong to the same class, and an objective function value which is a value corresponding to the first feature array and the second feature array are based on the weight matrix. Objective function value derivation part to be derived from
A learning device including an update unit that generates a learning result by executing a predetermined learning process based on the objective function value. In the objective function value derivation unit, the difference or similarity between the first feature array and the second feature array or between the second feature array and the feature array derived from the weight matrix is displayed on the label. Determine the objective function value so that it can be associated,
The learning device according to claim 1. It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention mechanism to generate weight matrix and
A decoding unit that derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array.
An objective function value derivation unit that derives an objective function value that is a value corresponding to the correct array and the second array, and
A learning device including an update unit that generates a learning result by executing a predetermined learning process based on the objective function value. When there is a correspondence between the elements of the first array and the elements of the second array, a constraint function that derives a constraint function value representing at least one of a monotonic constraint and a continuity constraint based on the weight matrix. Equipped with a value derivator
The monotonic constraint is a constraint that the subscripts of the elements of the first array that correspond to the elements of the second array do not decrease as the subscripts of the elements of the second array increase.
The continuity constraint is that when the subscripts of adjacent elements in the second array are continuous, the subscripts of the elements of the first array that correspond to the subscripts of the adjacent elements in the second array It is a constraint that the difference is less than or equal to a predetermined positive value.
The update unit generates a learning result by executing a predetermined learning process based on the objective function value and the constraint function value.
The learning device according to any one of claims 1 to 3. The constraint function value derivation unit reduces the constraint function value as the degree to which at least one of the monotonic constraint and the continuity constraint is satisfied is greater.
The learning device according to claim 4. The constraint function value derivation unit derives the product of the weight matrix and a predetermined equality sequence as a corresponding array, and sums or sums all the local monotonic constraint function values for all the elements in the corresponding array. Derivation of the mean as the constraint function value of monotonicity,
The learning device according to claim 4 or 5. The constraint function value derivation unit derives the product of the weight matrix and a predetermined equality sequence as a corresponding array, and for each element of the corresponding array, the element immediately preceding the element of the corresponding array and the corresponding array. The absolute value of the difference from the element is derived, a predetermined positive number is subtracted from the derived absolute value, and the maximum value of the numerical value of the subtraction result and 0 is derived as the function value of the local continuity constraint. Then, the sum or average of the function values of all the local continuity constraints for all the elements in the corresponding array is derived as the constraint function values of continuity.
The learning device according to claim 4 or 5. It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention mechanism to generate weight matrix and
A collation unit for deriving the distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix.
An inference device including an inference unit that generates an inference result by executing a predetermined inference process based on the distance. It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention mechanism to generate weight matrix and
A decoding unit that derives a second array based on the first feature array and the weight matrix,
An inference device including an inference unit that generates an inference result by executing a predetermined inference process based on the second array. It is a learning method executed by the learning device.
It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention step to generate weight matrix and
A label indicating whether or not the first array and the second array belong to the same class, and an objective function value which is a value corresponding to the first feature array and the second feature array are based on the weight matrix. The objective function value derivation step to be derived from
A learning method including an update step that generates a learning result by executing a predetermined learning process based on the objective function value. It is an inference method executed by an inference device.
It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention step to generate weight matrix and
A collation step for deriving the distance between the first sequence and the second sequence based on the first feature sequence, the second feature sequence, and the weight matrix.
An inference method that includes an inference step that produces an inference result by performing a predetermined inference process based on the distance. It is a learning method executed by the learning device.
It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention step to generate weight matrix and
A decoding step for deriving the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array.
An objective function value derivation step for deriving an objective function value that is a value corresponding to the correct array and the second array, and
A learning method including an update step that generates a learning result by executing a predetermined learning process based on the objective function value. It is an inference method executed by an inference device.
It is a matrix representing the probability that each element of the first feature sequence and the second feature sequence has a correspondence relationship by using the first feature array based on the first sequence and the second feature sequence based on the second sequence. Attention step to generate weight matrix and
A decoding step for deriving a second sequence based on the first feature array and the weight matrix,
An inference method including an inference step that generates an inference result by executing a predetermined inference process based on the second array. A program for operating a computer as the learning device according to any one of claims 1 to 7. A program for operating a computer as the inference device according to claim 8 or 9.

Images