JP7340199B2 - Learning device, reasoning device, learning method, reasoning method and program - Google Patents

Description

The present invention relates to a learning device, an inference device, a learning method, an inference method, and a program.

An array is a series of data arranged in order. Examples of arrays include audio signals, acoustic signals, and biological signals. Each piece of data in the array is a numeric value, a numeric vector, etc., and is called an element of the array. Each piece of data in the array is identified using a subscript such as a natural number.

Sequence alignment refers to arranging the elements of each array so that mutually similar regions in multiple arrays can be identified. Sequence alignment is important in many applications, such as motion recognition, speech analysis, biosignal classification, and signature authentication, because it provides clues to sequence relationships. In particular, array alignment is required when there are non-linear temporal variations between the two arrays with respect to local displacements and velocity changes. A typical method for array alignment is the dynamic time warping method (see Non-Patent Document 1).

In the dynamic time warping method, the distance between each element in two arrays is derived. The correspondence between each element in the two arrays is detected so that the sum of the distances between the corresponding elements is minimized. A correspondence relationship is a combination of two elements that correspond to each other, or a combination of subscripts of two elements that correspond to each other.

In the dynamic time warping method, parallelization of processing is difficult. For this reason, it is difficult to combine dynamic time warping and deep learning. Additionally, dynamic time warping methods rely on the use of manually designed feature representations and have insufficient performance when more complex feature representations are required. Therefore, dynamic time warping methods are often not optimal for a given purpose application problem.

In fields such as machine translation, speech synthesis, and speech conversion, there is a method of using an attention mechanism as a sequence alignment method that can be easily combined with deep learning (see Non-Patent Documents 2 and 3). The attention mechanism derives the weight of each element of the first array relative to each element of the second array for two arrays, the first array and the second array. Each derived weight represents the probability that each element of the two arrays, the first array and the second array, has a corresponding relationship. In a method of array alignment using an attention mechanism, array alignment is achieved by rearranging each element of the first array based on the weight of each element of the first array relative to each element of the second array.

Hiroaki Sakoe and Seibi Chiba, "Dynamic programming algorithm optimization for spoken word recognition," IEEE Transactions on Acoustics, Speech and Signal Processing, Vol. 26, No. 1, pp. 43-49, 1978. Dzmitry Bahdanau, KyungHyun Cho, and Yoshua Bengio, "Neural machine translation by jointly learning to align and translate," In ICLR, 2015. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning, "Effective approaches to attention-based neural machine translation," In EMNLP, pp. 1412-1421, 2015.

Using a function (hereinafter referred to as "correspondence function") that takes the subscript of the second array as an independent variable and the subscript of the first array that has a correspondence relationship with the subscript of the second array as a dependent variable, The correspondence between each element is expressed. In applied problems such as matching or classification, the correspondence functions between two arrays belonging to the same class are often monotone and continuous. On the other hand, in two arrays belonging to different classes, the corresponding functions are often non-monotonic or discontinuous.

By utilizing such properties, it is possible to determine whether two arrays belong to the same class. For example, a monotonic, continuous correspondence function can be derived from two arrays, and the sum of distances between corresponding elements can be derived. If this sum is large, it can be determined that the two arrays belong to different classes.

A dynamic time warping method is a typical array alignment method that utilizes this property. However, dynamic time warping methods rely on the use of manually designed feature representations and perform poorly when more complex feature representations are required. Therefore, dynamic time warping methods are often not optimal for a given purpose application problem.

In contrast, attention mechanisms do not rely on manually designed feature representations. However, conventionally, attention mechanisms cannot be used to solve application problems such as matching or classification. This is because even if the conventional attention mechanism derives the probability that each element of two arrays has a corresponding relationship, it is not possible to derive the correspondence function from the probability. Further, even if a conventional attention mechanism derives a correspondence function, there is no way to guarantee that the correspondence function is monotone and continuous.

Therefore, when distances between arrays aligned using conventional attention mechanisms are applied to applied problems such as matching or classification, the distances between arrays are often derived to be very small. For this reason, it is often impossible to correctly distinguish between two arrays belonging to different classes.

FIG. 10 is a diagram showing an example of a weight matrix. The weight matrix is a matrix that represents the probability that each element of two arrays has a corresponding relationship. In FIG. 10, the first array is, for example, "LISTEN", and the second array is, for example, "SILENT". An element of the weight matrix having a value of "1" indicates that the corresponding element has a corresponding relationship.

The weight matrix shown on the left side of FIG. 10 is a weight matrix derived by a conventional attention mechanism. Conventional attention mechanisms thus derive non-monotonic and discontinuous correspondence functions. Even if two arrays belong to different classes, in the weight matrix shown on the left side of Figure 10, the sum of the distances between corresponding elements is 0, so the two arrays cannot be correctly distinguished. I haven't been able to do that.

For this reason, in applied problems such as matching or classification, there is a need for an array alignment method that can derive and use monotonous and continuous correspondence functions, such as the arrangement of "1"s in the weight matrix shown on the right side of Figure 10. has been done. With such a sequence alignment method, the distance or similarity between sequences can be correctly derived, and it is possible to correctly infer whether the sequences belong to different classes.

In applied problems such as speech synthesis or speech conversion, the objective is to convert a first array into a second array. Sequence alignment is required when there is a non-linear temporal variation in local displacement and velocity change between the first and second arrays. For example, when converting Japanese English speech into American English speech, it is necessary to align the array of audio signals because there are variations in the tempo of the English speech. That is, it is necessary to estimate the correspondence between each element of the two arrays, align the first array using the estimated correspondence, and convert the aligned first array into the second array. . Even in such cases, the correspondence function between the two arrays is often monotone and continuous.

However, the conventional method using an attention mechanism does not have a function to guide learning of a mathematical model so that the attention mechanism can derive a monotonous and continuous correspondence function. For this reason, long learning times are often required before the attention mechanism can provide sufficient performance.

Therefore, the above-mentioned array alignment method is also required in applied problems such as speech synthesis or speech conversion. By using such an array alignment method, it is possible to both improve the inference accuracy of speech synthesis or speech conversion and shorten the learning time.

In view of the above circumstances, the present invention makes it possible to derive and use more complex feature representations without relying on the use of manually designed feature representations, and at the same time derive and use monotonous and continuous correspondence functions. It is an object of the present invention to provide a learning device, an inference device, a learning method, an inference method, and a program that can realize possible array alignments.

One aspect of the present invention is to use a first feature array based on a first array and a second feature array based on a second array, so that each element of the first feature array and the second feature array is in a corresponding relationship. an attention mechanism that generates a weight matrix that is a matrix representing a certain probability; a label that represents whether the first array and the second array belong to the same class; and the first feature array and the second feature array. an objective function value deriving unit that derives an objective function value that is a value according to the weight matrix based on the weight matrix; and an updating unit that generates a learning result by executing a predetermined learning process based on the objective function value. It is a learning device equipped with.

One aspect of the present invention is to use a first feature array based on a first array and a second feature array based on a second array, so that each element of the first feature array and the second feature array is in a corresponding relationship. an attention mechanism that generates a weight matrix, which is a matrix representing a certain probability, and a weight of each element of the first feature array with respect to the elements of the second array at the current time, and the first feature array. a decoding unit that derives the elements of the second array at a time; an objective function value deriving unit that derives an objective function value that is a value corresponding to the correct answer array and the second array; The learning device includes an updating unit that generates a learning result by executing a predetermined learning process.

One aspect of the present invention is to use a first feature array based on a first array and a second feature array based on a second array, so that each element of the first feature array and the second feature array is in a corresponding relationship. an attention mechanism that generates a weight matrix that is a matrix representing a certain probability; and a distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix. This inference device includes a matching unit that derives the distance, and an inference unit that generates an inference result by executing a predetermined inference process based on the distance.

One aspect of the present invention is to use a first feature array based on a first array and a second feature array based on a second array, so that each element of the first feature array and the second feature array is in a corresponding relationship. an attention mechanism that generates a weight matrix that is a matrix representing a certain probability; a decoding unit that derives a second array based on the first feature array and the weight matrix; and a predetermined inference based on the second array. The inference device includes an inference unit that generates inference results by executing processing.

One aspect of the present invention is a learning method executed by a learning device, which uses a first feature array based on a first array and a second feature array based on a second array. a caution step of generating a weight matrix, which is a matrix representing the probability that each element has a correspondence relationship with the two feature arrays, and a label representing whether the first array and the second array belong to the same class; an objective function value derivation step of deriving an objective function value that is a value corresponding to the first feature array and the second feature array based on the weight matrix; and executing a predetermined learning process based on the objective function value. This learning method includes an updating step of generating a learning result by performing the following steps.

One aspect of the present invention is an inference method executed by an inference device, which uses a first feature array based on a first array and a second feature array based on a second array. a step of generating a weight matrix that is a matrix representing the probability that each element has a correspondence relationship with the first feature array, the second feature array, and the weight matrix; The inference method includes a matching step of deriving a distance between an array and the second array, and an inference step of generating an inference result by performing a predetermined inference process based on the distance.

One aspect of the present invention is a learning method executed by a learning device, which uses a first feature array based on a first array and a second feature array based on a second array. a step of generating a weight matrix, which is a matrix representing the probability that each element has a correspondence relationship with the two feature arrays; a decoding step for deriving the elements of the second array at the current time based on the first feature array; and an objective function value for deriving an objective function value that is a value according to the correct answer array and the second array. The learning method includes a deriving step and an updating step of generating a learning result by executing a predetermined learning process based on the objective function value.

One aspect of the present invention is an inference method executed by an inference device, which uses a first feature array based on a first array and a second feature array based on a second array. a caution step of generating a weight matrix, which is a matrix representing the probability that each element has a correspondence relationship with the two feature arrays; and a decoding step of deriving a second array based on the first feature array and the weight matrix. and an inference step of generating an inference result by performing a predetermined inference process based on the second array.

One aspect of the present invention is a program for causing a computer to function as the learning device described above.

One aspect of the present invention is a program for causing a computer to function as the inference device described above.

The present invention allows for the derivation and use of more complex feature representations without relying on the use of manually designed feature representations, while at the same time providing array alignment that allows the derivation and use of monotonic and continuous correspondence functions. It is possible to achieve this.

FIG. 3 is a diagram illustrating a configuration example of an inference device in the first embodiment. It is a diagram showing an example of the configuration of a learning device in the first embodiment. It is a figure showing an example of correspondence arrangement in a 1st embodiment. FIG. 6 is a diagram showing an example of deriving a monotonicity constraint function value in the first embodiment. FIG. 3 is a diagram showing an example of deriving a continuity constraint function value in the first embodiment. FIG. 7 is a diagram illustrating a configuration example of an inference device in a second embodiment. It is a figure showing an example of composition of a learning device in a 2nd embodiment. FIG. 2 is a diagram illustrating an example of the hardware configuration of an inference device in each embodiment. It is a diagram showing an example of the hardware configuration of a learning device in each embodiment. FIG. 3 is a diagram showing an example of a weight matrix.

Embodiments of the present invention will be described in detail with reference to the drawings.
In the following, attention mechanisms are used in applied problems such as sequence matching or classification. This provides an array alignment that allows more complex feature representations to be derived and used without relying on the use of manually designed feature representations.

In the following, a new constraint function value representing at least one of the monotonicity constraint and the continuity constraint is proposed. By minimizing the constraint function value so that the attention mechanism can derive a monotone and continuous correspondence function, it is possible to guide the learning of the mathematical model that includes the encoding part and the attention mechanism. be.

Hereinafter, the monotonicity constraint means that there is a correspondence between the elements of the first array and the elements of the second array, and as the subscript (number) of the element of the second array increases, the correspondence with the element of the second array increases. This is a restriction that the subscript (number) of an element in a certain first array does not decrease. Hereinafter, a continuity constraint means that there is a correspondence between elements in the first array and elements in the second array, and if the subscripts (numbers) of adjacent elements in the second array are consecutive, then in the second array This is a constraint that the difference between the subscripts of elements in the first array that correspond to the subscripts of adjacent elements is less than or equal to a predetermined positive value.

(First embodiment)
In the first embodiment, the learning method and the inference method are applied to applied problems such as matching or classification. Examples of applied problems such as matching or classification include motion recognition, voice recognition, biological signal classification, and signature authentication.

In the learning phase, the learning device uses an attention mechanism to perform learning of the mathematical model. That is, in the learning stage, the learning device learns a mathematical model having a large number of parameters using learning data. The learning device generates a learned mathematical model by determining numerical values of parameters of the mathematical model. In the execution stage, the inference device uses the learned mathematical model to execute inference processing. For example, the reasoning device performs a desired task such as matching or classification.

First, the inference method applied to applied problems such as matching or classification in the execution stage will be explained.

FIG. 1 is a diagram showing a configuration example of an inference device 1 in the first embodiment. In the execution stage of the first embodiment, the inference method is applied to applied problems such as matching or classification. The inference device 1 obtains the first array and the second array as input. For example, in motion recognition, the inference device 1 obtains as input an array in which the coordinates of a plurality of feature points (for example, joint positions) on a human body are arranged in chronological order. In signature authentication, the inference device 1 obtains as input an array in which signature coordinates or pen pressure on the display of the signature collection device are arranged in chronological order. The inference device 1 derives the distance between the first array and the second array. The inference device 1 executes inference processing based on distance. The inference device 1 outputs the inference result to a predetermined external device (not shown).

Distance can be used to solve application problems such as matching or classification. For example, in a classification problem, the inference device 1 derives the distance between a learning array whose class is known and a target array whose class is unknown. The inference device 1 estimates the class of the target array using the K-nearest neighbor method, support vector machine, or the like. In the search problem, the inference device 1 derives the distance between the query sequence and the sequences in the database. The inference device 1 derives the array with the shortest distance as a search result.

The inference device 1 includes an encoding section 10-1, an encoding section 10-2, a caution mechanism 11, a collation section 12, and an inference section 13.

The details of the functional units of the inference device 1 will be explained.
<Encoding unit 10>
The encoding unit 10-1 obtains the first array as input. The encoding unit 10-2 obtains the second array as input. The encoding unit 10-1 outputs the first feature array (first feature expression) to the attention mechanism 11 and the matching unit 12. The encoding unit 10-2 outputs the second feature array (second feature expression) to the attention mechanism 11 and the matching unit 12.

The operation of encoding section 10-1 is similar to that of encoding section 10-2. Therefore, the operation of the encoding section 10-1 will be explained below. Further, in the following, items common to the encoding unit 10-1 and the encoding unit 10-2 will be referred to as "encoding unit 10" with a part of the code omitted. Based on the first array, the encoding unit 10 derives an array whose elements are numerical values or numerical vectors as a first feature array.

<First example of encoding unit 10>
In a first example of the encoding unit 10, the encoding unit 10 uses an artificial neural network to derive a first feature array from the first array. In the learning phase, parameters of the artificial neural network are determined based on training data.

Details of the first example of processing by the encoding unit 10 are as follows.
In the first example of the encoding unit 10, the encoding unit 10 changes the length of the first array to a predetermined length (for example, 1024). This is necessary to be able to use batch or mini-batch learning when training the artificial neural network is performed. Each element of the first array is a one-dimensional numerical value or a multidimensional numerical vector.

For each dimension of the elements of the first array whose length has been changed, the encoding unit 10 performs the following: Normalize all numbers in that dimension. The normalized first array is, for example, a "1x1024x5" tensor. This "1024" is an example of the length of the array. This "5" is an example of the number of dimensions of the elements of the array.

The encoding unit 10 inputs the normalized first array to the convolutional neural network. The convolutional neural network includes, for example, one "1x7x64" convolutional layer, one maximum pooling layer, and two "1x3x64" convolutional layers. A batch normalization layer is provided immediately after each convolutional layer. Following the batch normalization layer, a ReLU layer is provided as an activation function. The final ReLU layer outputs an array whose elements are multidimensional numerical vectors.

For each element of an array whose elements are multidimensional numerical vectors, the encoding unit 10 normalizes all the numerical values of the element so that the L2 norm of all the numerical values of the element becomes 1. The encoding unit 10 outputs the normalized array to the attention mechanism 11 and the matching unit 12 as a first feature array. In the first example of the encoding unit 10, a recurrent neural network or the like may be used instead of a convolutional neural network.

<Second example of encoding unit 10>
In the second example of the encoding unit 10, the encoding unit 10 outputs the input first array to the attention mechanism 11 and the matching unit 12 as the first feature array. In the second example of the encoding unit 10, the encoding unit 10 does not have parameters.

<Caution mechanism 11>
The attention mechanism 11 acquires the first feature array from the encoding unit 10-1. The attention mechanism 11 acquires the second feature array from the encoding unit 10-2. The attention mechanism 11 derives the weight of each element of the first feature array with respect to each element of the second feature array based on each element of the first feature array and each element of the second feature array. The weight of each element of the first feature array with respect to each element of the second feature array represents the probability that the two elements have a corresponding relationship. The larger the weight, the higher the probability that two elements have a corresponding relationship. The attention mechanism 11 outputs the weight matrix to the matching unit 12.

<First example of attention mechanism 11>
In the first example of the attention mechanism 11, the attention mechanism 11 uses an artificial neural network to perform a calculation for each element of the second feature array based on each element of the first feature array and each element of the second feature array. Derive the weight of each element of the first feature array. In the learning phase, parameters of the artificial neural network are determined based on training data.

Details of the first example of processing by the attention mechanism 11 are as follows.
In the first example of the attention mechanism 11, the attention mechanism 11 connects the numerical vector that is each element of the first feature array and the numerical vector that is each element of the second feature array along the dimension direction of the numerical vector. do. The attention mechanism 11 inputs the concatenated numerical vectors into the artificial neural network.

The artificial neural network comprises, for example, three fully connected layers. Among the three fully connected layers, the first fully connected layer has 64 hidden units, the second fully connected layer has 16 hidden units, and the third fully connected layer has 64 hidden units. Contains 1 hidden unit. Immediately after the first fully connected layer, a ReLU layer is provided as an activation function. Immediately after the second fully connected layer, a ReLU layer is provided as an activation function. The third fully connected layer outputs one real number.

For each element of the second feature array, the attention mechanism 11 normalizes an array including all real numbers derived using the element and each element of the first feature array using the Softmax function. The array including all the derived real numbers is an array of real numbers output for each element of the first feature array. The array containing all the derived real numbers contains the same number of real numbers as the number of elements of the first feature array. The attention mechanism 11 derives a normalized real number as the weight of each element of the first feature array with respect to each element of the second feature array. The attention mechanism 11 outputs a matrix including all the weights of each element of the first feature array for each element of the second feature array to the matching unit 12 as a weight matrix.

<Second example of attention mechanism 11>
Details of the second example of processing by the attention mechanism 11 are as follows.
In a second example of the attention mechanism 11, the attention mechanism 11 derives the inner product of each element of the first feature array and each element of the second feature array. For each element of the second feature array, the attention mechanism 11 normalizes an array including all inner products of each element of the second feature array and each element of the first feature array using a Softmax function. The attention mechanism 11 derives a normalized inner product as a weight of each element of the first feature array with respect to each element of the second feature array. The attention mechanism 11 outputs a matrix including all the weights of each element of the first feature array for each element of the second feature array to the matching unit 12 as a weight matrix.

In the second example of the attention mechanism 11, the attention mechanism 11 has no parameters. In order to learn a mathematical model including the encoding unit 10 and the attention mechanism 11, the mathematical model cannot be learned unless the mathematical model has parameters. Therefore, when the second example of the encoder 10 is used, the second example of the caution mechanism 11 cannot be used. That is, when the encoding unit 10 without parameters is used, the attention mechanism 11 without parameters cannot be used.

<Verification section 12>
The matching unit 12 obtains the first feature array from the encoding unit 10-1. The matching unit 12 obtains the second feature array from the encoding unit 10-2. The matching unit 12 acquires the weight matrix from the attention mechanism 11. The matching unit 12 derives the distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix. The matching unit 12 outputs the distance (distance information) between the first array and the second array to the inference unit 13. Note that the matching unit 12 may output the distance (distance information) to a predetermined external device (not shown).

<First example of matching unit 12>
In the first example of the matching unit 12, the matching unit 12 weights each element of the first feature array using a weight matrix. The matching unit 12 derives the new feature array obtained by weighting as a transformed feature array. The matching unit 12 derives the distance between the converted feature array and the second feature array as the distance between the first array and the second array.

Details of the first example of processing by the matching unit 12 are as follows.
In the first example of the matching unit 12, for each element of the second feature array, the matching unit 12 uses the weight of each element of the first feature array for each element of the second feature array to calculate all of the first feature array. Derive the weighted sum of the elements of . As a result, elements of the first feature array that have a corresponding relationship with each element of the second feature array are specified (extracted or generated) as a weighted sum. That is, the elements of the first feature array that have a corresponding relationship with each element of the second feature array are arranged. Therefore, non-linear temporal variations in local displacements and velocity changes that exist between the first and second arrays are compensated for.

The matching unit 12 calculates the distance (for example, , Euclidean distance) is derived as the local distance. Since the time variation between the first array and the second array has already been compensated for, there is a high probability that each element of the second feature array and the weighted sum derived for the element have a corresponding relationship. Therefore, by deriving the distance between each element of the second feature array and the weighted sum derived for the element, the matching unit 12 calculates the local difference between the first feature array and the second feature array. It becomes possible for the matching unit 12 to derive a distance that more accurately represents the distance.

The matching unit 12 derives the sum or average of all local distances regarding all elements of the second feature array. The matching unit 12 outputs the sum or average of the local distances to the inference unit 13 as the distance between the first array and the second array. Here, the first feature array is expressed as "XεR ^W×K ", and the second feature array is expressed as "YεR ^W×K ". "W" represents the length of the feature array. “K” represents the number of dimensions of a numerical value or a numerical vector that is an element of the feature array. The j-th row vector “x _j ∈R ^1×K ” of “X” represents the j-th element of “X”. Similarly, the i-th row vector “y _i ∈R ^1×K ” of “Y” represents the i-th element of “Y”.

The weight matrix is expressed as "P∈R ^W×W ". The i-th row vector “p _i εR ^1×W ” of “P” includes weights “p _i1 ,…, p _iW ” of “x ₁ , . . . , x _W ” with respect to “y _i ”. The j-th element “p _ij ” of “p _i ” represents the weight of “x _j ” with respect to “y _i ”.

Since “p _i ” has been normalized by the Softmax function, the sum of “p _i1 , . . . , p _iW ” is 1. Therefore, the distance between the first array and the second array is expressed as in equation (1).

Here, "p _i X" represents the weighted sum of "x ₁ , . . . , x _W " with respect to "y _{i "} . “||p _i X−y _i ||” represents the Euclidean distance between “p _i X” and “y _i ”, that is, the local distance.

<Second example of matching unit 12>
In the second example of the matching unit 12, the matching unit 12 derives the distance between each element of the first feature array and each element of the second feature array. The matching unit 12 weights the distance using a weight matrix. The matching unit 12 derives the distance between the first array and the second array based on the weight.

Details of the second example of processing by the matching unit 12 are as follows.
In the second example of the matching unit 12, the matching unit 12 derives the distance (for example, Euclidean distance) between each element of the first feature array and each element of the second feature array as a local distance. The matching unit 12 uses the weight matrix to derive a weighted sum or weighted average of local distances. The matching unit 12 outputs the weighted sum or weighted average of local distances to the inference unit 13 as the distance between the first array and the second array.

The weight of each element of the first feature array relative to each element of the second feature array represents the probability that the two elements have a corresponding relationship. The larger the weight, the higher the probability that two elements have a corresponding relationship. The matching unit 12 assigns a larger weight to the local distance between two elements that have a high probability of being in a corresponding relationship. The collation unit 12 assigns a smaller weight to the local distance between two elements that have a low probability of being in a correspondence relationship.

This compensates for non-linear temporal variations in local displacements and velocity changes that exist between the first and second arrays. Furthermore, the distance between the first array and the second array can be derived more accurately.

Similar to the first example of the matching unit 12, in the second example of the matching unit 12, the first feature array is expressed as “X∈R ^W×K ”, and the second feature array is expressed as “Y∈R ^W×K ”. It is written as The length of the feature array is denoted as "W". The j-th element of "X" is written as "x _j ∈R ^1xK ", and the i-th element of "Y" is written as "y _i ∈R ^1xK ". The weight matrix is expressed as "P∈R ^W×W ". The weight of “x _j ” with respect to “y _i ” is expressed as “p _ij ∈P”. Therefore, the distance between the first array and the second array is expressed as in equation (2).

Here, “||x _j −y _i ||” represents the Euclidean distance between “x _j ” and “y _i ”, that is, the local distance.

<Inference part 13>
The inference unit 13 obtains the weighted sum or weighted average of local distances from the matching unit 12 as the distance between the first array and the second array. The inference unit 13 performs inference processing based on the distance between the first array and the second array. The inference unit 13 outputs the inference result to a predetermined external device (not shown). Inference processing is not limited to specific inference processing. For example, when the authors of handwritten signatures of multiple people are inferred, signatures whose authors are unknown (first array) and signatures whose authors are known (second array) are input to a trained mathematical model. The inference unit 13 uses the author ID (identification number) of the second array with the shortest distance between the first array and the second array obtained from the collation unit 12 as the author ID (inference result) of the first array. Output. If a plurality of second arrays exist for each author, the inference unit 13 may output the author ID with the shortest average distance value as the inference result.

Next, a learning method applied to applied problems such as matching or classification in the learning stage will be explained.

FIG. 2 is a diagram showing a configuration example of the learning device 2 in the first embodiment. In the learning stage of the first embodiment, the learning method is applied to applied problems such as matching or classification. The learning device 2 obtains the first array, the second array, and the label as input. The learning device 2 derives an objective function value and a constraint function value. The learning device 2 outputs the learned mathematical model (learning result) to a predetermined external device (not shown) based on the objective function value and the constraint function value. Further, the learning device 2 outputs the learned mathematical model to the inference device 1 before the execution stage.

The first array, the second array, and the label are learning data used by the learning device 2 to learn a mathematical model for executing a task for a predetermined purpose (for example, matching or classification). The label indicates whether the first array and the second array belong to the same class. The objective function value and the constraint function value are used by the learning device 2 to learn the mathematical model. For example, the learning device 2 uses a mathematical method to Update model parameters. The larger the number of training data, the better the performance of the mathematical model. The number of learning data is, for example, about 20,000 to 30,000.

The learning device 2 includes an encoding section 20-1, an encoding section 20-2, an attention mechanism 21, an objective function value deriving section 22, a constraint function value deriving section 23, and an updating section 24.

The details of the functional units of the learning device 2 will be explained.
<Encoding unit 20>
The encoding unit 20-1 obtains the first array as input. The encoding unit 20-2 obtains the second array as input. The operation of encoding section 20-1 is similar to that of encoding section 20-2. The processing of the encoding unit 20-1 in the learning stage is the same as the processing of the encoding unit 10-1 in the execution stage. The processing of the encoding unit 20-2 in the learning stage is the same as the processing of the encoding unit 10-2 in the execution stage.

The encoding unit 20-1 outputs the first feature array to the attention mechanism 21 and the objective function value deriving unit 22. The encoding unit 20-2 outputs the second feature array to the attention mechanism 21 and the objective function value deriving unit 22. Hereinafter, items common to the encoding unit 20-1 and the encoding unit 20-2 will be referred to as "encoding unit 20" with a part of the code omitted.

<Caution mechanism 21>
The attention mechanism 21 acquires the first feature array from the encoding unit 20-1. The attention mechanism 21 acquires the second feature array from the encoding unit 20-2. The processing of the attention mechanism 21 in the learning stage is the same as the processing of the attention mechanism 11 in the execution stage. The attention mechanism 21 outputs the weight matrix to the objective function value deriving unit 22 and the constraint function value deriving unit 23.

<Objective function value deriving unit 22>
The objective function value deriving unit 22 obtains the label as input. The objective function value deriving unit 22 obtains the first feature array and the second feature array from the encoding unit 20. The objective function value deriving unit 22 acquires the weight matrix from the attention mechanism 21. The objective function value deriving unit 22 derives the difference between the first feature array and the second feature array based on the first feature array, the second feature array, and the weight matrix. The objective function value deriving unit 22 derives an objective function value so that the derived difference is associated with a label.

When the first array and the second array belong to the same class, the larger the difference, the larger the objective function value. When the first array and the second array belong to different classes, the smaller the difference, the larger the objective function value. The objective function value deriving unit 22 outputs such an objective function value to the updating unit 24.

<First example of objective function value deriving unit 22>
When the first example of the matching unit 12 is used in the execution stage, it is better to use the first example of the objective function value deriving unit 22 in the learning stage, and it is better to use the second example of the objective function value deriving unit 22 in the learning stage. more desirable than In the first example of the objective function value deriving unit 22, the objective function value deriving unit 22 weights each element of the first feature array using a weight matrix. The objective function value deriving unit 22 derives the new feature array obtained by weighting as a transformed feature array. The objective function value deriving unit 22 derives the difference between the transformed feature array and the second feature array. The objective function value deriving unit 22 derives an objective function value so that the derived difference is associated with a label.

Details of the first example of processing by the objective function value deriving unit 22 are as follows.
In the first example of the objective function value deriving unit 22, the objective function value deriving unit 22 uses, for each element of the second feature array, the weight of each element of the first feature array with respect to each element of the second feature array, Derive a weighted sum of all elements of the first feature array.

As a result, elements of the first feature array that have a corresponding relationship with each element of the second feature array are specified (extracted or generated) as a weighted sum. That is, the elements of the first feature array that have a corresponding relationship with each element of the second feature array are arranged. Therefore, non-linear temporal variations in local displacements and velocity changes that exist between the first and second arrays are compensated for.

The objective function value deriving unit 22 calculates the distance (e.g., Euclidean distance) between the weighted sum of all elements (numeric value or numerical vector) of the first feature array and each element (numeric value or numerical vector) of the second feature array. , is derived as a local distance. The objective function value deriving unit 22 derives a local objective function value using the local distance. When the first array and the second array belong to the same class, the longer the local distance, the larger the local objective function value. When the first array and the second array belong to different classes, the shorter the local distance, the larger the local objective function value.

The objective function value deriving unit 22 derives the sum or average of all local objective function values regarding all elements of the second feature array. The objective function value deriving unit 22 outputs the sum or average of the local objective function values to the updating unit 24 as an objective function value. Here, the first feature array is expressed as "X∈R ^W×K ". The second feature array is written as “YεR ^W×K ”. The length of the feature array is denoted as "W". The j-th element of "X" is written as "x _j ∈R ^1×K ". The i-th element of "Y" is written as "yi∈R ^1×K ".

The weight matrix is expressed as "P∈R ^W×W ". The i-th row vector “p _i εR ^1×W ” of “P” includes weights “p _i1 ,…, p _iW ” of “x ₁ , . . . , x _W ” with respect to “y _i ”. The label is written as "z∈{0,1}". When the first array and the second array belong to the same class, the label becomes "z=1". When the first array and the second array belong to different classes, the label becomes "z=0". Therefore, the objective function value is expressed as in equation (3).

Here, "p _i X" represents the weighted sum of "x ₁ , . . . , x _W " with respect to "y _{i "} . “||p _i X−y _i ||” represents the Euclidean distance between “p _i X” and “y _i ”, that is, the local distance. “τ” is a hyperparameter and is a positive real number.

In the learning stage, the updating unit 24 updates the update unit 24 so that the weighted sum or weighted average of the objective function value and the constraint function value derived using a large amount of learning data is as small as possible (for example, the weighted average is minimized). ), the parameters of the mathematical model including the encoding unit 20 and the attention mechanism 21 are updated. By minimizing the objective function value, the parameters are updated so that the mathematical model derives a smaller local distance when the first array and the second array belong to the same class.

When the first array and the second array belong to the same class, the objective function value deriving unit 22 Parameters are updated based on one example objective function value. In other words, when the first array and the second array belong to the same class, the objective function value is The parameters are updated based on the first example objective function value of the derivation unit 22.

By using the mathematical model learned in this way, the correspondence between each element of the first feature array and each element of the second feature array can be specified more accurately. The distance between the first array and the second array is more correctly derived. Furthermore, without relying on the use of manually designed feature representations, array alignment is realized that allows more complex feature representations to be derived and used than with dynamic time warping methods.

<Second example of objective function value deriving unit 22>
When the second example of the matching unit 12 is used in the execution stage, it is better to use the second example of the objective function value deriving unit 22 in the learning stage, and it is better to use the first example of the objective function value deriving unit 22 in the learning stage. more desirable than In the second example of the objective function value deriving unit 22, the objective function value deriving unit 22 derives the distance between each element of the first feature array and each element of the second feature array. The objective function value deriving unit 22 weights the distance using a weight matrix. The objective function value deriving unit 22 derives the degree of similarity between the first feature array and the second feature array. The objective function value deriving unit 22 derives an objective function value so that the derived degree of similarity is associated with a label.

Details of the second example of processing by the objective function value deriving unit 22 are as follows.
In the second example of the objective function value deriving unit 22, the objective function value deriving unit 22 calculates the distance (for example, Euclidean distance) between each element of the first feature array and each element of the second feature array as a local distance. Derive. The objective function value deriving unit 22 derives a weighted sum or weighted average of local distances using a weight matrix. The objective function value deriving unit 22 derives an objective function value such that the derived weighted sum or weighted average is associated with a label.

Here, the first feature array is expressed as "XεR ^W×K ". The second feature array is expressed as “YεR ^W×K ”. The length of the feature array is expressed as "W". The j-th element of "X" is expressed as "x _j ∈R ^1×K ". The i-th element of "Y" is written as "y _i ∈R ^1×K ". The weight matrix is expressed as "P∈R ^W×W ". The weight of “x _j ” with respect to “y _i ” is expressed as “p _ij ∈P”. The label is written as "z∈{0,1}". When the first array and the second array belong to the same class, the label becomes "z=1". When the first array and the second array belong to different classes, the label becomes "z=0". Therefore, the degree of similarity between the first feature array and the second feature array is expressed as in equation (4).

Here, “||x _j −y _i ||” represents the Euclidean distance between “x _j ” and “y _i ”, that is, the local distance. The objective function value is expressed as in equation (5).

In the learning stage, the updating unit 24 updates the encoding unit 20 and the attention mechanism 21 so that the objective function value derived using a large amount of learning data is as small as possible (for example, minimized). Update the parameters of the mathematical model, including By minimizing the objective function value, the parameters are updated so that the mathematical model derives a smaller local distance when the first array and the second array belong to the same class.

When the first array and the second array belong to the same class, the updating unit 24 updates the parameters of the mathematical model so that a larger weight is derived for two elements with a high probability of corresponding relationship. . When the first array and the second array belong to the same class, the updating unit 24 updates the parameters of the mathematical model so that a smaller weight is derived for two elements with a low probability of corresponding relationship. . That is, the parameters of the mathematical model are updated so that the correspondence between each element of the first feature array and each element of the second feature array can be specified more accurately.

By using the mathematical model learned in this way, the correspondence between each element of the first feature array and each element of the second feature array can be more accurately specified, and It is possible to more accurately derive the distance between Moreover, without relying on the use of manually designed feature representations, it is possible to realize array alignment that allows more complex feature representations to be derived and used than with the dynamic time warping method.

<Constraint function value derivation unit 23>
The constraint function value deriving unit 23 acquires the weight matrix from the attention mechanism 21. The constraint function value deriving unit 23 derives a constraint function value using the weight matrix. The constraint function value deriving unit 23 derives the constraint function value such that the greater the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied, the smaller the constraint function value becomes. The constraint function value deriving unit 23 outputs the constraint function value to the updating unit 24.

The mathematical model including the encoding unit 20 and the attention mechanism 21 minimizes the constraint function value so that the correspondence between each element of the first feature array and each element of the second feature array is monotonic. The weight matrix is trained to derive a weight matrix that satisfies at least one of the constraints and the continuity constraint.

Details of the processing of the constraint function value deriving unit 23 are as follows.
The weight matrix is a matrix representing the probability that each element of the first feature array and each element of the second feature array are in a correspondence relationship, and is not the correspondence relationship itself. Therefore, the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied cannot be directly evaluated from the weight matrix.

In order to evaluate the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied, it is necessary to transform the weight matrix into a form like a correspondence function. This correspondence function is, for example, a function in which the independent variable is the subscript of each element of the second feature array, and the dependent variable is the subscript of the element of the first feature array that has a corresponding relationship with the subscript of each element of the second feature array. It is.

Therefore, the constraint function value deriving unit 23 derives the product of the weight matrix and a predetermined arithmetic progression as a corresponding array. An arithmetic sequence is a sequence in which adjacent elements have a common difference.

FIG. 3 is a diagram showing an example of a corresponding arrangement in the first embodiment. In the upper part of FIG. 3, examples of weight matrices, examples of arithmetic progressions, and examples of corresponding arrays are shown for the case where the monotonicity constraint and the continuity constraint are satisfied. The lower part of FIG. 3 shows an example of a weight matrix, an example of an arithmetic progression, and an example of a corresponding array for the case where the monotonicity constraint and the continuity constraint are not satisfied. That is, the left side of the equal sign represents the product of the weight matrix and the arithmetic progression "[1,2,3,4] ^T ". Each row of the weight matrix has been normalized, and the sum of the elements in each row of the weight matrix is 1. The corresponding array is shown on the right side of the equal sign.

The subscript of the corresponding array derived using the arithmetic progression represents the subscript (number) of each element of the second feature array. Numerical values that are elements of the correspondence array represent subscripts (numbers) of elements of the first feature array that have a correspondence relationship with each element of the second feature array. Note that the numerical values that are the elements of the correspondence array may represent numerical values that are proportional to the subscripts of the elements of the first feature array that have a correspondence relationship with each element of the second feature array.

In FIG. 3, a corresponding array is derived using a weight matrix and an arithmetic progression. For example, in the example shown in the upper part of FIG. 3, the correspondence array indicates that the first element of the second feature array is in a correspondence relationship with the first element of the first feature array. The correspondence array indicates that the second element of the second feature array is in a correspondence relationship with the second element of the first feature array. The correspondence array indicates that the third element of the second feature array is in a correspondence relationship with the second element of the first feature array.

The subscript of the element of the first feature array that corresponds to the fourth element of the second feature array is not expressed using an integer, but is expressed as "3.6" using a real number. There is. By using such a correspondence array, it becomes possible to evaluate the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied.

The constraint function value derived using the correspondence array needs to become smaller as the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied is greater. Note that in order for the learning device 2 to learn a mathematical model using the gradient method, it is desirable that the constraint function value is differentiable with respect to the weight matrix or the corresponding array. Furthermore, in order to enable faster learning, it is desirable that the derivation of constraint function values can be easily parallelized.

The constraint function value deriving unit 23 derives at least one of the monotonicity constraint function value and the continuity constraint function value as a constraint function value.

<Monotonicity constraint function value>
The constraint function value deriving unit 23 calculates the local monotonicity constraint function value ( (hereinafter referred to as "local monotonicity constraint function value"). When the element immediately before the element of the corresponding array is larger than the element of the corresponding array, the local monotonicity constraint function value becomes the absolute value of the difference between these two elements. The local monotonicity constraint function value becomes 0 when the element immediately before the element of the corresponding array is less than or equal to the element of the corresponding array.

The constraint function value deriving unit 23 derives the sum or average of all local monotonicity constraint function values regarding all elements in the corresponding array. The constraint function value deriving unit 23 outputs the sum or average of the local monotonicity constraint function values to the updating unit 24 as a monotonicity constraint function value.

Here, the weight matrix is expressed as "P∈R ^W×W ". The length of the feature array is denoted as "W". The corresponding array is expressed as "F∈R ^W×1 ". The i-th element of "F" is written as "f _i ". Therefore, the monotonicity constraint function value is expressed as in equation (6).

Here, "f ₀ " is 0. Equation (6) is implemented using a library of convolutional neural networks to derive the monotonicity constraint function value faster.

FIG. 4 is a diagram showing an example of deriving the monotonicity constraint function value in the first embodiment. The upper part of FIG. 4 shows an example of deriving the monotonicity constraint function value in the case where the monotonicity constraint and the continuity constraint are satisfied. The lower part of FIG. 4 shows an example of deriving the monotonicity constraint function value in a case where the monotonicity constraint and the continuity constraint are not satisfied.

FIG. 4 shows, from the left, an example of a corresponding array, an example of a filter, a difference between two adjacent elements in a corresponding array, an example of a local monotonicity constraint function value, and an example of a monotonicity constraint function value. is expressed. In FIG. 4, a circle with an "x" symbol represents convolution. "Loss" represents the monotonicity constraint function value. The greater the degree to which the corresponding arrays satisfy the monotonicity constraint, the smaller the monotonicity constraint function value is derived. The smaller the degree to which the corresponding arrays satisfy the monotonicity constraint, the larger the monotonicity constraint function value is derived.

In FIG. 4, the difference between two adjacent elements in the corresponding array is derived as a result of the convolution of the corresponding array and the filter "[1,-1] ^T ". The constraint function value deriving unit 23 applies "ReLU" as an activation function to the array of differences between two adjacent elements. In this way, the local monotonicity constraint function value is derived. By deriving the average of all elements in the array of local monotonicity constraint function values, the monotonicity constraint function value as shown in Equation (6) can be easily derived.

Note that the filter may be any filter capable of deriving the difference between two elements whose positions are close to each other in the corresponding array. For example, a filter such as "[1,0,-1] ^T " or "[2,1,-1,-2] ^T " may be used instead of "[1,-1] ^T " .

<Continuity constraint function value>
The constraint function value deriving unit 23 derives, for each element of the corresponding array, the absolute value of the difference between the element immediately before the element of the corresponding array and the element of the corresponding array. The constraint function value deriving unit 23 subtracts a predetermined positive number from the derived absolute value. This predetermined positive number is a hyperparameter and is, for example, a positive integer such as 1, 2, or 3. A real number such as "1.5" may be used as a hyperparameter.

The constraint function value deriving unit 23 derives the maximum value between the numerical value of the subtraction result and 0 as a function value of a local continuity constraint (hereinafter referred to as "local continuity constraint function value"). The constraint function value deriving unit 23 derives the sum or average of all local continuity constraint function values regarding all elements in the corresponding array. The constraint function value deriving unit 23 outputs the sum or average of the local continuity constraint function values to the updating unit 24 as a continuity constraint function value.

The weight matrix is expressed as "P∈R ^W×W ". The length of the feature array is denoted as "W". The corresponding array is expressed as "F∈R ^W×1 ". The i-th element of "F" is written as "f _i ". Therefore, the continuity constraint function value is expressed as in equation (7).

Here, "f ₀ " is 0. Equation (7) is implemented using a library of convolutional neural networks to derive continuity constraint function values faster.

FIG. 5 is a diagram showing an example of deriving continuity constraint function values in the first embodiment. The upper part of FIG. 5 shows an example of deriving the continuity constraint function value in the case where the monotonicity constraint and the continuity constraint are satisfied. The lower part of FIG. 5 shows an example of deriving the continuity constraint function value in a case where the monotonicity constraint and the continuity constraint are not satisfied.

FIG. 4 shows, from the left, an example of a corresponding array, an example of a filter, an example of a predetermined positive number, and an example of a predetermined positive number subtracted from the absolute value of the difference between two adjacent elements in the corresponding array. The results, examples of local continuity constraint function values, and examples of continuity constraint function values are shown. In FIG. 5, a circle with an "x" symbol represents convolution. "Loss" represents the continuity constraint function value.

In FIG. 5, the difference between two adjacent elements in the corresponding array is derived by convolving the corresponding array with the filter “[−1,1] ^T ”. The constraint function value deriving unit 23 derives the absolute value of each element in the array of differences between two adjacent elements. The constraint function value deriving unit 23 subtracts a predetermined positive number (1 in FIG. 5) from the derived absolute value. The constraint function value deriving unit 23 applies "ReLU" as an activation function to the subtraction result array. In this way, the local continuity constraint function value is derived. By deriving the average of all elements in the array of local continuity constraint function values, the continuity constraint function value as shown in equation (7) can be easily derived.

Note that the filter may be any filter capable of deriving the difference between two elements whose positions are close to each other in the corresponding array. For example, a filter such as "[1,0,-1] ^T " or "[2,1,-1,-2] ^T " may be used instead of "[1,-1] ^T " .

As shown in FIG. 5, the greater the degree to which the corresponding array satisfies the continuity constraint, the smaller the continuity constraint function value is derived. The smaller the degree to which the corresponding array satisfies the continuity constraint, the larger the continuity constraint function value is derived.

<Update section 24>
The updating unit 24 obtains the objective function value from the objective function value deriving unit 22. The updating unit 24 obtains the constraint function value from the constraint function value deriving unit 23. The updating unit 24 executes learning processing based on the objective function value and the constraint function value. The learning process is not limited to a specific learning process. The updating unit 24 updates the mathematical model including the encoding unit 20 and the attention mechanism 21 so that the weighted sum or weighted average of the constraint function value and the objective function value is as small as possible (for example, minimized). Update the parameters of The updating unit 24 outputs the trained mathematical model (learning result) to a predetermined external device (not shown).

As described above, in the learning stage, the attention mechanism 21 uses the first feature array based on the first array and the second feature array based on the second array to A weight matrix is generated, which is a matrix representing the probability that elements have a corresponding relationship. The objective function value deriving unit 22 calculates the objective function value, which is a value corresponding to the label indicating whether the first array and the second array belong to the same class, and the first feature array and the second feature array, using weights. Derive based on the matrix. The constraint function value deriving unit 23 derives a constraint function value representing at least one of the monotonicity constraint and the continuity constraint based on the weight matrix. The updating unit 24 generates a learning result by executing a predetermined learning process based on the objective function value and the constraint function value. The objective function value is, for example, a value according to the difference or similarity between the first feature array and the second feature array and the label. The updating unit 24 updates the mathematical model.

The mathematical model updated in the learning stage is used to execute inference processing in the execution stage. In the execution stage, the attention mechanism 11 uses the first feature array based on the first array and the second feature array based on the second array to establish a correspondence relationship between each element of the first feature array and the second feature array. Generate a weight matrix, which is a matrix that represents a certain probability. The matching unit 12 derives the distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix. The inference unit 13 generates an inference result by executing a predetermined inference process based on the distance.

In this way, the encoding unit derives the feature array using the mathematical model learned using the constraint function value representing at least one of the monotonicity constraint and the continuity constraint, thereby creating an effective weight matrix. is generated by the attention mechanism based on the feature array.

This makes it possible to derive and use more complex feature representations without relying on the use of manually designed feature representations, and at the same time realizes array alignment that allows for the derivation and use of monotonic and continuous correspondence functions. It is possible to do so. It is possible to realize more complex feature representations without relying on the use of manually designed feature representations. Furthermore, it is possible to both improve inference accuracy and shorten learning time.

According to the learning device 2, the learning method, and the program, the updating unit 24 guides learning of the mathematical model when learning the mathematical model so that the attention mechanism 11 can derive a monotonous and continuous correspondence function. becomes possible. By using the attention mechanism 11 in the learned mathematical model, it is possible to correctly derive the distance or similarity between sequences in applied problems such as matching or classification. It is possible to correctly infer whether the arrays belong to different classes. Furthermore, it is possible to shorten the learning time (the time required for learning the mathematical model) until the attention mechanism 11 can provide sufficient performance.

(Second embodiment)
The second embodiment is an embodiment for applying a learning method and an inference method to applied problems such as synthesis or conversion of continuous data such as speech. Speech synthesis refers to artificially creating human speech, for example, synthesizing speech from text. Voice conversion is the conversion of an individual's voice into the voice of another individual or character.

Note that if the discontinuous data (for example, a handwritten signature) is corrected in advance so as to become continuous data, the learning method and inference method in the second embodiment can be used for the discontinuous data.

The second embodiment is divided into a learning stage and an execution stage. In the learning phase, the learning device uses training data to learn a mathematical model with a large number of parameters. The learning device determines numerical values of parameters of the mathematical model. In the execution stage, the inference device uses the learned mathematical model to execute a task for a predetermined purpose (eg, speech synthesis, speech conversion).

First, an inference method applied to applied problems such as speech synthesis or speech conversion in the execution stage will be explained.

FIG. 6 is a diagram showing a configuration example of the inference device 3 in the second embodiment. In speech synthesis, the elements of the first array are, for example, numerical vectors representing the characteristics of each word in a sentence. The feature of each word in a sentence is, for example, the word's One-Hot vector. The elements of the second array are, for example, numerical vectors representing the characteristics of each time or frame of audio.

In audio conversion, the elements of the first array are, for example, numerical vectors representing the characteristics of each time or frame of audio. The characteristics of each time or frame of audio can be extracted using, for example, a predetermined extraction method (Reference 1: Masanori Morise, Fumiya Yokomori, Kenji Ozawa, "WORLD: A vocoder-based high-quality speech synthesis system for real-time applications, "IEICE Trans. Inf. Syst. 99-D (7): 1877-1884 (2016)) is a multidimensional vector containing mel cepstral coefficients and a logarithm F0 pattern. The elements of the second array are, for example, numerical vectors representing the characteristics of each time or each frame in the voice of an individual or character different from the individual in the voice of the first array.

The inference device 3 includes a first encoding section 30, a second encoding section 31, a caution mechanism 32, a decoding section 33, and an inference section 34.

The first encoding unit 30 obtains the first array as input. The first encoding unit 30 performs encoding processing on the first array, for example, only once. Derive a first feature array. The first encoding unit 30 outputs the first feature array to the attention mechanism 32 and the decoding unit 33.

The second encoding unit 31 acquires the elements of the second array at the previous time from the decoding unit 33. The second encoding unit 31 derives the elements of the second feature array at the previous time by performing encoding processing on the elements of the second array at the previous time. The second encoding unit 31 outputs the elements of the second feature array at the previous time to the attention mechanism 32.

The attention mechanism 32 acquires the first feature array from the first encoding unit 30. The attention mechanism 32 acquires the elements of the second feature array at the previous time from the second encoding unit 31. The attention mechanism 32 uses the elements of the second feature array at the previous time and each element of the first feature array to determine the weight of each element of the first feature array with respect to the element of the second array at the current time. Derive. The attention mechanism 32 outputs the weight of each element of the first feature array with respect to the element of the second array at the current time to the decoding unit 33 as a weight matrix.

The decoding unit 33 acquires the first feature array from the first encoding unit 30. The decoding unit 33 obtains the weight of each element of the first feature array with respect to the element of the second array at the current time from the attention mechanism 32 as a weight matrix. The decoding unit 33 derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array. The decoding unit 33 outputs the elements of the second array at the current time to the second encoding unit 31 and the inference unit 34. Note that the decoding unit 33 may output the elements of the second array at the current time to a predetermined external device (not shown).

The second encoding unit 31 obtains the elements of the second array at the current time from the decoding unit 33. The second encoding unit 31 uses the elements of the second array at the current time to derive the elements of the second feature array at the current time. The second encoding unit 31 outputs the elements of the second feature array at the current time to the attention mechanism 32.

In this way, a cycle in which the signal departs from the second encoding unit 31, passes through the attention mechanism 32 and the decoding unit 33, and returns to the second encoding unit 31 again occurs in the inference device 3. exist. After the elements of the second array are initialized at the first time, the initialized elements of the second array are input to the second encoding unit 31, and at the last time, the elements of the second array are input to the decoding unit 33. The inference process in this cycle is repeated every unit time from to output.

The attention mechanism 32 outputs a matrix including all the weights of each element of the first feature array for each element of the second array to the decoding unit 33 as a weight matrix. Further, the decoding unit 33 outputs each element of the second array at all times to the inference unit 34 as a second array.

The inference unit 34 obtains the second array from the decoding unit 33. The inference unit 34 generates an inference result based on the second array. In applied problems such as speech synthesis or speech conversion, the inference result is an audio signal. The inference unit 34 outputs the inference result to a predetermined external device (not shown).

The details of the functional units of the inference device 3 will be explained.
<First encoding unit 30>
The first encoding unit 30 obtains the first array as input. The first encoding unit 30 uses the first array to derive an array whose elements are numerical values or numerical vectors as a first feature array. For example, the first encoding unit 30 may be configured as described in Reference 2 (Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ-Skerrv Ryan, Rif A Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu, "Natural TTS synthesis by conditioning wavenet on MEL spectrogram predictions," In ICASSP, pp.4779-4783, 2018. Derived from one array. The first encoding unit 30 determines parameters of the artificial neural network using learning data in a learning stage. The first encoding unit 30 outputs the first feature array to the attention mechanism 32 and the decoding unit 33.

Details of the processing of the first encoding unit 30 are as follows.
The first array is, for example, a “1×N×512” tensor. "N" represents the length of the array. “512” is an example of the number of dimensions of the elements of the array. The first encoding unit 30 inputs the first array to the artificial neural network.

The artificial neural network, for example, has three 1×5×512 convolutional layers and one Bidirectional Long Short-Term Memory (BiLSTM) (hereinafter referred to as ``bidirectional LSTM''). Equipped with. A batch normalization layer is provided immediately after each convolutional layer. Immediately after the batch normalization layer, a ReLU layer is provided as an activation function. The bidirectional LSTM has a total of 512 hidden units. The bidirectional LSTM of the first encoding unit 30 outputs an array having numerical values or numerical vectors as elements to the attention mechanism 32 and the decoding unit 33 as a first feature array.

<Second encoding unit 31>
The second encoding unit 31 obtains the second array from the decoding unit 33. The elements of the second array at the previous time are acquired from the decoding unit 33. The second encoding unit 31 uses the elements of the second array at the previous time to derive a numerical value or a numerical value vector as an element of the second feature array at the previous time. For deriving a numerical value or a numerical vector, for example, the artificial neural network of Reference 2 mentioned above can be used. The parameters of the artificial neural network are determined using training data during the training phase. The second encoding unit 31 outputs the second feature array to the attention mechanism 32.

Details of the processing by the second encoding unit 31 are as follows.
Each element of the second array at the previous time is, for example, a 512-dimensional numerical vector. The second encoding unit 31 inputs each element of the second array at the previous time to the artificial neural network. This artificial neural network includes, for example, two fully connected layers. Each fully connected layer has 256 hidden units. Immediately after each fully connected layer, a ReLU layer is provided as an activation function. The final fully connected layer outputs a numerical value or a numerical vector to the attention mechanism 32 as an element of the second feature array at the previous time.

<Caution mechanism 32>
The attention mechanism 32 obtains the first feature array from the first encoding unit 30. The attention mechanism 32 acquires the second feature array from the second encoding unit 31. The attention mechanism 32 uses the elements of the second feature array at the previous time and each element of the first feature array to determine the relationship between each element of the first feature array and the elements of the second array at the current time. Derive the weights. As the attention mechanism 32, for example, an artificial neural network may be used, or a mathematical model other than the artificial neural network (for example, a linear regression model, a polynomial regression model, a logistic regression model) may be used. The parameters of the artificial neural network are determined using training data during the training phase. The attention mechanism 32 outputs the weight matrix to the decoding unit 33.

Details of the processing by the attention mechanism 32 are as follows.
The attention mechanism 32 connects the numerical vector that is an element of the second feature array at the previous time and the numerical vector that is each element of the first feature array along the dimension direction of the numerical vector. The attention mechanism 32 inputs the concatenated numerical vectors into the artificial neural network. The artificial neural network comprises, for example, three fully connected layers. Among the three fully connected layers, the first fully connected layer has 64 hidden units, the second fully connected layer has 16 hidden units, and the third fully connected layer has 64 hidden units. Contains 1 hidden unit. Immediately after the first fully connected layer, a ReLU layer is provided as an activation function. Immediately after the second fully connected layer, a ReLU layer is provided as an activation function. The third fully connected layer outputs one real number.

The attention mechanism 32 normalizes an array including all real numbers derived using the elements of the second feature array and each element of the first feature array at the previous time using the Softmax function. The array including all the derived real numbers is an array of real numbers output for each element of the first feature array. The array containing all the derived real numbers contains the same number of real numbers as the number of elements of the first feature array. The attention mechanism 32 derives a normalized real number as the weight of each element of the first feature array relative to the element of the second array at the current time. The attention mechanism 32 outputs a matrix including all the weights of each element of the first feature array to the elements of the second array at the current time to the decoding unit 33 as a weight matrix.

<Decoding unit 33>
The decoding unit 33 acquires the first feature array from the first encoding unit 30. The decoding unit 33 acquires the weight matrix from the attention mechanism 32. The decoding unit 33 weights each element of the first feature array using the weight of each element of the first feature array with respect to the element of the second array at the current time. The decoding unit 33 uses the numerical value or numerical vector obtained by weighting to derive the elements of the second array at the current time. For example, the decoding unit 33 uses the artificial neural network of Reference 2 mentioned above to derive the elements of the second array at the current time. The decoding unit 33 determines parameters of the artificial neural network using learning data in the learning stage. The decoding unit 33 outputs the second array to the inference unit 34.

Details of the processing of the decoding unit 33 are as follows.
The decoding unit 33 derives a weighted sum of all elements of the first feature array using the weight of each element of the first feature array with respect to the element of the second array at the current time. As a result, the elements of the first feature array that have a corresponding relationship with the elements of the second array at the current time are specified (extracted or generated) as a weighted sum. That is, the elements of the first feature array that have a corresponding relationship with the elements of the second array at the current time are arranged. Therefore, non-linear temporal variations in local displacements and velocity changes that exist between the first and second arrays are compensated for.

Here, the first feature array is expressed as "X∈R ^W×K ". The weight matrix is expressed as "P∈R ^W×W ". A row vector including all the weights of each element of the first feature array with respect to the elements of the second array at the current time is expressed as "p _i ∈R ^1×W ". The current time is written as "i". The weighted sum of all elements of the first feature array relative to the elements of the second array at the current time is denoted as "p _i X."

The weighted sum is, for example, a 128-dimensional numerical vector. The decoding unit 33 inputs this numerical vector to the artificial neural network. The artificial neural network comprises, for example, two bidirectional LSTMs and one fully connected layer. Each bidirectional LSTM has 1024 hidden units. The fully connected layer outputs the numerical value or numerical vector to the inference unit 34 as an element of the second array at the current time.

Note that the decoding unit 33 may derive the second array using the second feature array, the first feature array, and the weight matrix output from the second encoding unit 31. In this case, the decoding unit 33 connects the numerical vector that is the weighted sum and the numerical vector that is the element of the second feature array at the previous time along the dimension direction of the numerical vector. The decoding unit 33 inputs the concatenated numerical vectors to the artificial neural network.

<Inference section 34>
The inference unit 34 obtains the second array from the decoding unit 33. The inference unit 34 generates an inference result based on the second array. In applied problems such as speech synthesis or speech conversion, the inference result is an audio signal. The inference unit 34 uses, for example, a predetermined generation method (Reference 3: Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, Koray Kavukcuoglu, "WaveNet: A generative model for raw audio, "SSW 2016: 125.) is used to generate an audio signal based on the second array. The inference unit 34 outputs the inference result to a predetermined external device (not shown).

Next, a learning method applied to applied problems such as speech synthesis or speech conversion in the learning stage will be explained.

FIG. 7 is a diagram showing a configuration example of the learning device 4 in the second embodiment. In the learning stage of the second embodiment, the learning method is applied to applied problems such as speech synthesis or speech conversion. The learning device 4 receives the first array and the correct array as input. The learning device 4 derives an objective function value and a constraint function value. The learning device 4 learns a mathematical model based on the objective function value and the constraint function value, and outputs the learned mathematical model (learning result) to a predetermined external device (not shown). Further, the learning device 4 outputs the learned mathematical model to the inference device 3 before the execution stage.

The first array and the correct array are training data used to learn a mathematical model for performing a task for a predetermined purpose (eg, speech synthesis or speech conversion). The objective function value and the constraint function value are used by the learning device 4 to learn the mathematical model. For example, the learning device 4 performs the following steps so that the weighted sum or weighted average of the objective function value and the constraint function value derived using a large amount of learning data is as small as possible (for example, minimized). Update the parameters of the mathematical model. The larger the number of training data, the better the performance of the mathematical model. The number of learning data is, for example, about 20,000 to 30,000.

The learning device 4 includes a first encoding section 40, a second encoding section 41, a caution mechanism 42, a decoding section 43, an objective function value deriving section 44, a constraint function value deriving section 45, and an updating section. 46.

The first encoding unit 40 obtains the first array as input. The first encoding unit 40 performs encoding processing on the first array, for example, only once. Derive a first feature array. The first encoding unit 40 outputs the first feature array to the attention mechanism 42 and the decoding unit 43.

The second encoding unit 41 acquires the elements of the second array at the previous time from the decoding unit 43. The second encoding unit 41 derives the elements of the second feature array at the previous time by performing encoding processing on the elements of the second array at the previous time.

The attention mechanism 42 obtains the first feature array from the first encoding unit 40 . The attention mechanism 42 acquires the elements of the second feature array at the previous time from the second encoding unit 41. The attention mechanism 42 uses the elements of the second feature array at the previous time and each element of the first feature array to determine the weight of each element of the first feature array with respect to the element of the second array at the current time. Derive. The attention mechanism 32 outputs the weight of each element of the first feature array with respect to the element of the second array at the current time to the decoding unit 43 as a weight matrix.

The decoding unit 43 acquires the first feature array from the first encoding unit 40. The decoding unit 43 acquires the weight of each element of the first feature array with respect to the element of the second array at the current time from the attention mechanism 42 as a weight matrix. The decoding unit 43 derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array. The decoding unit 43 outputs the elements of the second array at the current time to the second encoding unit 41 and the objective function value deriving unit 44.

The second encoding unit 41 obtains the elements of the second array at the current time from the decoding unit 43. The second encoding unit 41 uses the elements of the second array at the current time to derive the elements of the second feature array at the current time. The second encoding unit 41 outputs the elements of the second feature array at the current time to the attention mechanism 42.

In this way, a cycle in which the signal departs from the second encoding section 41, passes through the attention mechanism 42 and the decoding section 43, and returns to the second encoding section 41 again occurs in the learning device 4. exist. In this cycle, the elements of the second array are initialized at the first time, the initialized elements of the second array are input to the second encoding unit 41, and the elements of the second array are input at the final time. The learning process is repeated every unit time until output from the decoding unit 43.

The attention mechanism 42 outputs a matrix including all the weights of each element of the first feature array for each element of the second array to the decoding unit 43 as a weight matrix. Further, the decoding unit 43 outputs each element of the second array at every time as a second array to the second encoding unit 41 and the objective function value deriving unit 44.

The objective function value deriving unit 44 obtains the correct answer array as input. The objective function value deriving unit 44 obtains the second array from the decoding unit 43. The objective function value deriving unit 44 derives an objective function value based on the correct array and the second array. The process in which the objective function value derivation unit 44 derives the objective function value is executed, for example, only once. The objective function value deriving unit 44 outputs the objective function value to the updating unit 46.

The constraint function value deriving unit 45 obtains the weight matrix from the attention mechanism 42 . The constraint function value deriving unit 45 derives a constraint function value using the weight matrix. The process of deriving a constraint function value by the constraint function value deriving unit 45 is executed, for example, only once. The constraint function value deriving unit 45 outputs the constraint function value to the updating unit 46.

The updating unit 46 obtains the objective function value from the objective function value deriving unit 44. The updating unit 46 obtains the constraint function value from the constraint function value deriving unit 45. The updating unit 46 executes learning processing based on the objective function value and the constraint function value. The updating unit 46 updates the first encoding unit 40 and the second encoding unit 41 so that the weighted sum or weighted average of the constraint function value and the objective function value is as small as possible (for example, minimized). The mathematical model including the attention mechanism 42 and the decoding unit 43 is updated. The updating unit 46 outputs the learned mathematical model (learning result) to a predetermined external device (not shown).

The details of the functional units of the learning device 4 will be explained.
<First encoding unit 40>
The first encoding unit 40 obtains the first array as input. The processing executed by the first encoding unit 40 in the learning stage is the same as the processing executed by the first encoding unit 30 in the execution stage. The first encoding unit 40 outputs the first feature array to the attention mechanism 42 and the decoding unit 43.

<Second encoding unit 41>
The second encoding unit 41 acquires the second array from the decoding unit 43 and outputs the second feature array to the attention mechanism 42. The processing of the second encoding unit 41 in the learning stage is the same as the processing of the second encoding unit 31 in the execution stage. Note that the second encoding unit 41 in the learning stage may use the correct array as an input instead of using the second array as an input. In this case, all processing performed on the second array is performed on the ground truth array used in place of the second array.

<Caution mechanism 42>
The attention mechanism 42 obtains the first feature array from the first encoding unit 40 . The attention mechanism 42 acquires the second feature array from the second encoding unit 41. The processing of the attention mechanism 42 in the learning stage is the same as the processing of the attention mechanism 32 in the execution stage. The attention mechanism 42 outputs the weight matrix to the decoding section 43 and the constraint function value deriving section 45.

<Decoding unit 43>
The decoding unit 43 acquires the first feature array from the first encoding unit 40. The decoding unit 43 acquires the weight matrix from the attention mechanism 42. The processing of the decoding unit 43 in the learning stage is the same as the processing of the decoding unit 33 in the execution stage. The decoding unit 43 outputs the second array to the objective function value deriving unit 44.

<Objective function value deriving unit 44>
The objective function value deriving unit 44 obtains the correct answer array as input. The objective function value deriving unit 44 obtains the second array from the decoding unit 43. The objective function value deriving unit 44 derives the difference between the correct array and the second array. The objective function value deriving unit 44 derives an objective function value such that the larger the derived difference, the larger the value. The objective function value deriving unit 44 outputs the objective function value to the updating unit 46.

Details of the processing of the objective function value deriving unit 44 are as follows.
The objective function value deriving unit 44 derives, for example, the sum of squared residuals (similarity) between the correct array and the second array as the objective function value. Here, the correct array is written as "Z ^* ". The second array is written as "Z". Therefore, the objective function value is expressed as in equation (8).

Here, “||·||” represents the L2 norm.

<Constraint function value deriving unit 45>
The constraint function value deriving unit 45 obtains the weight matrix from the attention mechanism 42 . The constraint function value deriving unit 45 derives a constraint function value using the weight matrix. Here, the constraint function value is derived such that the greater the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied, the smaller the constraint function value is. The constraint function value deriving unit 45 outputs the constraint function value to the updating unit 46.

A weight matrix in which the correspondence between each element of the first feature array and each element of the second array satisfies at least one of the monotonicity constraint and the continuity constraint by minimizing the constraint function value. The mathematical model is trained to derive . This mathematical model includes a first encoding section 40, a second encoding section 41, a caution mechanism 42, and a decoding section 43.

Details of the processing by the constraint function value deriving unit 45 are as follows.
The weight matrix is a matrix representing the probability that each element of the first feature array and each element of the second array have a corresponding relationship. The weight matrix is not a correspondence relationship itself. Therefore, the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied cannot be directly evaluated from the weight matrix.

In order to be able to evaluate the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied, the weight matrix needs to be transformed. For example, a function (correspondence function) where the time of each element of the second array is the independent variable and the subscript of the element of the first feature array that has a correspondence relationship with the time of each element of the second array is the dependent variable. The weight matrix needs to be transformed into For this purpose, the constraint function value deriving unit 45 derives the product of the weight matrix and a predetermined arithmetic progression as a corresponding array. An arithmetic progression is a progression in which each term (each element) is obtained by adding a constant number (tolerance) to the term (element) immediately before it.

For example, in FIG. 3, "[1,2,3,4] ^T " is an arithmetic progression. In the correspondence array derived using the arithmetic progression, the subscript of the correspondence array represents the time of each element of the second array. The numerical values that are the elements of the correspondence array represent the subscripts of the elements of the first feature array that are in correspondence with the respective elements of the second array, or the numerical values that are proportional to the subscripts. In the example shown in the upper part of FIG. 3, the first element of the second array has a corresponding relationship with the first element of the first feature array. The second element of the second array corresponds to the second element of the first feature array. The correspondence array indicates that the third element of the second array is in a correspondence relationship with the second element of the first feature array. The subscript of the element of the first feature array that corresponds to the fourth element of the second array is not expressed using an integer, but is expressed as "3.6" using a real number. . By using such a correspondence array, it becomes possible to evaluate the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied.

The constraint function value derived using the correspondence array needs to be smaller as the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied is greater. Note that in order for the learning device 4 to learn a mathematical model using the gradient method, it is desirable that the constraint function value is differentiable with respect to the weight matrix or the corresponding array. Furthermore, in order to enable faster learning, it is desirable that the derivation of constraint function values can be easily parallelized.

The constraint function value deriving unit 45 derives at least one of the monotonicity constraint function value and the continuity constraint function value as a constraint function value.

<Monotonicity constraint function value>
The explanation regarding the monotonicity constraint function value in the second embodiment is the same as the explanation regarding the monotonicity constraint function value in the first embodiment.

<Continuity constraint function value>
The explanation regarding the continuity constraint function value in the second embodiment is the same as the explanation regarding the continuity constraint function value in the first embodiment.

<Update section 46>
The updating unit 46 obtains the objective function value from the objective function value deriving unit 44. The updating unit 46 obtains the constraint function value from the constraint function value deriving unit 45. The updating unit 46 executes learning processing based on the objective function value and the constraint function value. The updating unit 46 outputs the learned mathematical model (learning result) to a predetermined external device (not shown). The learning process is not limited to a specific learning process.

As described above, the attention mechanism 42 uses the first feature array based on the first array and the second feature array based on the second array, so that each element of the first feature array and the second feature array has a corresponding relationship. Generate a weight matrix, which is a matrix representing the probability that . The decoding unit 43 derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array. The objective function value deriving unit 44 derives an objective function value that is a value according to the correct array and the second array. The constraint function value deriving unit 45 derives a constraint function value based on the weight matrix. The updating unit 46 updates the first encoding unit 40, the second encoding unit 41, the attention mechanism 42, and the decoding unit 43 by executing a predetermined learning process based on the objective function value and the constraint function value. Update the parameters of the included mathematical model and generate learning results. The objective function value is, for example, the difference or residual sum of squares between the correct array and the second array. The updating unit 46 updates the mathematical model.

The mathematical model updated in the learning stage is used to execute inference processing in the execution stage. In the execution stage, the attention mechanism 32 uses the first feature array based on the first array and the second feature array based on the second array to establish a correspondence between each element of the first feature array and the second feature array. Generate a weight matrix, which is a matrix that represents a certain probability. The decoding unit 33 derives a second array based on the first feature array and the weight matrix. The inference unit 34 generates an inference result by performing a predetermined inference process based on the second array.

In this way, the encoding unit derives the feature array using the mathematical model learned using the constraint function value representing at least one of the monotonicity constraint and the continuity constraint, thereby creating an effective weight matrix. is generated by the attention mechanism.

This makes it possible to derive and use more complex feature representations for applied problems such as speech synthesis or speech conversion, without relying on the use of manually designed feature representations, while at the same time providing a monotonous and continuous representation. It is possible to derive a corresponding function and realize a usable array alignment. It is possible to realize more complex feature representations for application problems such as speech synthesis or speech conversion without relying on the use of manually designed feature representations. Furthermore, it is possible to simultaneously improve the accuracy of inferences such as speech synthesis or speech conversion and shorten the learning time.

FIG. 8 is a diagram showing an example of the hardware configuration of the inference device 1 in each embodiment. Some or all of the functional units of the inference device 1 are stored by a processor 100 such as a CPU (Central Processing Unit) in a storage unit 200 having a non-volatile recording medium (non-temporary recording medium). It is realized as software by executing the program. The program may be recorded on a computer-readable recording medium. Computer-readable recording media include, for example, portable media such as flexible disks, magneto-optical disks, ROM (Read Only Memory), and CD-ROM (Compact Disc Read Only Memory), and storage such as hard disks built into computer systems. It is a non-temporary recording medium such as a device. The communication unit 300 transmits the processing results by the inference device 1 to an external device (not shown). The communication unit 300 may receive the program via a communication line. The display unit 400 displays the processing results by the inference device 1. The display unit 400 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display.

Some or all of the functional units of the inference device 1 may include, for example, an LSI (Large Scale Integration circuit), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). It may also be realized using hardware including the electronic circuit or circuitry used. Note that the hardware configuration example of the inference device 3 is similar to the hardware configuration example of the inference device 1.

FIG. 9 is a diagram showing an example of the hardware configuration of the learning device 2 in each embodiment. Some or all of the functional units of the learning device 2 are configured such that a processor 101 such as a CPU executes a program stored in a storage unit 201 having a non-volatile recording medium (non-temporary recording medium). This is realized as software. The program may be recorded on a computer-readable recording medium. The computer-readable recording medium is a non-temporary recording medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or other portable medium, or a hard disk or other storage device built into a computer system. The communication unit 301 transmits the processing results by the learning device 2 to an external device (not shown). The communication unit 301 may receive the program via a communication line. The display unit 401 displays the processing results by the learning device 2. The display unit 401 is, for example, a liquid crystal display or an organic EL display.

Some or all of the functional units of the learning device 2 may be realized using hardware including an electronic circuit using an LSI, an ASIC, a PLD, an FPGA, or the like, for example. Note that the example hardware configuration of the learning device 4 is similar to the example hardware configuration of the learning device 2.

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

The present invention is applicable to learning devices and reasoning devices.

DESCRIPTION OF SYMBOLS 1...Inference device, 2...Learning device, 3...Inference device, 4...Learning device, 10...Encoding unit, 11...Attention mechanism, 12...Verification unit, 13...Inference unit, 20...Encoding unit, 21...Attention mechanism, 22... objective function value deriving unit, 23... constraint function value deriving unit, 24... updating unit, 30... first encoding unit, 31... second encoding unit, 32... attention mechanism, 33... decoding unit, 34... Reasoning section, 40... First encoding section, 41... Second encoding section, 42... Attention mechanism, 43... Decoding section, 44... Objective function value deriving section, 45... Constraint function value deriving section, 46... Update unit, 100... Processor, 101... Processor, 200... Storage unit, 201... Storage unit, 300... Communication unit, 301... Communication unit, 400... Display unit, 401... Display unit

Claims (14)

A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention mechanism that generates a weight matrix;
An objective function value, which is a value corresponding to a label indicating whether the first array and the second array belong to the same class, and the first feature array and the second feature array, is determined based on the weight matrix. an objective function value derivation unit that derives the
an updating unit that generates a learning result by executing a predetermined learning process based on the objective function value ;
If there is a correspondence between the elements of the first array and the elements of the second array, a constraint function that derives a constraint function value representing at least one of a monotonicity constraint and a continuity constraint based on the weight matrix. Value derivation part and
Equipped with
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The updating unit generates a learning result by executing a predetermined learning process based on the objective function value and the constraint function value.
learning device. The objective function value deriving unit calculates the difference or similarity between the first feature array and the second feature array or between the second feature array and the feature array derived from the weight matrix into the label. determining the objective function value so as to be associated with it;
The learning device according to claim 1. A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention mechanism that generates a weight matrix;
a decoding unit that derives the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array;
an objective function value derivation unit that derives an objective function value that is a value according to the correct answer array and the second array;
an updating unit that generates a learning result by executing a predetermined learning process based on the objective function value ;
If there is a correspondence between the elements of the first array and the elements of the second array, a constraint function that derives a constraint function value representing at least one of a monotonicity constraint and a continuity constraint based on the weight matrix. Value derivation part and
Equipped with
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The updating unit generates a learning result by executing a predetermined learning process based on the objective function value and the constraint function value.
learning device. The constraint function value deriving unit decreases the constraint function value as the degree to which at least one of the monotonicity constraint and the continuity constraint is satisfied is greater;
The learning device according to any one of claims 1 to 3 . The constraint function value deriving unit derives the product of the weight matrix and a predetermined arithmetic progression as a corresponding array, and calculates the sum or sum of all local monotonicity constraint function values for all elements in the corresponding array. deriving the mean as said constraint function value of monotonicity;
The learning device according to any one of claims 1 to 4. The constraint function value deriving unit derives the product of the weight matrix and a predetermined arithmetic progression as a corresponding array, and for each element of the corresponding array, the element immediately before the element of the corresponding array and the element of the corresponding array. Derive the absolute value of the difference with the element, subtract a predetermined positive number from the derived absolute value, and derive the maximum value between the subtraction result and 0 as the local function value of the continuity constraint. and deriving the sum or average of all the local continuity constraint function values for all elements in the corresponding array as the continuity constraint function value;
The learning device according to any one of claims 1 to 4. A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention mechanism that generates a weight matrix;
a matching unit that derives a distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix;
an inference unit that generates an inference result by executing a predetermined inference process based on the distance ;
The objective function value is a value according to a label indicating whether the first array and the second array belong to the same class, and the first feature array and the second feature array,
When there is a correspondence between the elements of the first array and the elements of the second array, the constraint function value represents at least one of a monotonicity constraint and a continuity constraint,
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The inference unit generates the inference result using a learning result generated by executing a predetermined learning process based on the objective function value and the constraint function value.
Reasoning device. A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention mechanism that generates a weight matrix;
a decoding unit that derives a second array based on the first feature array and the weight matrix;
an inference unit that generates an inference result by performing a predetermined inference process based on the second array ,
The objective function value is a value according to the correct answer array and the second array,
When there is a correspondence between the elements of the first array and the elements of the second array, the constraint function value represents at least one of a monotonicity constraint and a continuity constraint,
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The inference unit generates the inference result using a learning result generated by executing a predetermined learning process based on the objective function value and the constraint function value.
Reasoning device. A learning method executed by a learning device, comprising:
A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention step of generating a weight matrix;
An objective function value, which is a value corresponding to a label indicating whether the first array and the second array belong to the same class, and the first feature array and the second feature array, is determined based on the weight matrix. an objective function value derivation step,
an updating step of generating a learning result by executing a predetermined learning process based on the objective function value ;
If there is a correspondence between the elements of the first array and the elements of the second array, a constraint function that derives a constraint function value representing at least one of a monotonicity constraint and a continuity constraint based on the weight matrix. value derivation step and
including;
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The updating step includes generating a learning result by performing a predetermined learning process based on the objective function value and the constraint function value.
How to learn. An inference method executed by an inference device,
A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention step of generating a weight matrix;
a matching step of deriving a distance between the first array and the second array based on the first feature array, the second feature array, and the weight matrix;
an inference step of generating an inference result by performing a predetermined inference process based on the distance ;
The objective function value is a value according to a label indicating whether the first array and the second array belong to the same class, and the first feature array and the second feature array,
When there is a correspondence between the elements of the first array and the elements of the second array, the constraint function value represents at least one of a monotonicity constraint and a continuity constraint,
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The inference step includes generating the inference result using a learning result generated by executing a predetermined learning process based on the objective function value and the constraint function value.
Reasoning method. A learning method executed by a learning device, comprising:
A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention step of generating a weight matrix;
a decoding step of deriving the elements of the second array at the current time based on the weight of each element of the first feature array with respect to the elements of the second array at the current time and the first feature array;
an objective function value deriving step of deriving an objective function value that is a value according to the correct answer array and the second array;
an updating step of generating a learning result by executing a predetermined learning process based on the objective function value ;
If there is a correspondence between the elements of the first array and the elements of the second array, a constraint function that derives a constraint function value representing at least one of a monotonicity constraint and a continuity constraint based on the weight matrix. value derivation step and
including;
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The updating step includes generating a learning result by performing a predetermined learning process based on the objective function value and the constraint function value.
How to learn. An inference method executed by an inference device,
A matrix representing a probability that each element of the first feature array and the second feature array has a corresponding relationship using a first feature array based on the first array and a second feature array based on the second array. an attention step of generating a weight matrix;
a decoding step of deriving a second array based on the first feature array and the weight matrix;
an inference step of generating an inference result by performing a predetermined inference process based on the second array ;
The objective function value is a value according to the correct answer array and the second array,
When there is a correspondence between the elements of the first array and the elements of the second array, the constraint function value represents at least one of a monotonicity constraint and a continuity constraint,
The monotonicity constraint is a constraint that as the subscript of the element of the second array increases, the subscript of the element of the first array that has a corresponding relationship with the element of the second array does not decrease,
The continuity constraint means that when the subscripts of adjacent elements in the second array are consecutive, the subscripts of the elements of the first array that have a corresponding relationship with the subscripts of adjacent elements in the second array are The constraint is that the difference is less than or equal to a predetermined positive value,
The inference step includes generating the inference result using a learning result generated by executing a predetermined learning process based on the objective function value and the constraint function value.
Reasoning method. A program for causing a computer to function as the learning device according to any one of claims 1 to 6 . A program for causing a computer to function as the inference device according to claim 7 or 8 .

Images