JP2023139731A - Learning method and learning apparatus for gesture generation model - Google Patents

Abstract

To provide a learning method and a learning apparatus for a gesture generation model, configured to naturally choice a gesture suitable for the content of speech.SOLUTION: A learning method for a gesture generation model includes the steps of: preparing training data subdivided into multiple segments; preparing a generator 118 configured to receive, as input, a voice matrix included in each segment, a noise matrix, and a seed pose vector 184 from a previous segment, and generate a gesture matrix 126 with respect to the input voice matrix 128; preparing a discriminator 120 configured to output an evaluation value 130 of an input gesture matrix 200 with respect to the input voice matrix 128; and training the generator 118 and the discriminator 120 by adversarial training that evaluates a gesture matrix with respect to a voice matrix by switching between the gesture matrix 126 obtained from the generator 118 and a gesture matrix 124 of the training data.SELECTED DRAWING: Figure 3

Description

Application for application of Article 30, Paragraph 2 of the Patent Act has been filed At the GENEA WORKSHOP 2021 of the 23rd ACM International Conference on Multimodal Interaction, which was published on the web on July 6, 2021, “Probabilis tic Human-like Gesture Synthesis from Speech using Published in a paper titled “GRU-based WGAN”

The present invention relates to a learning method, a learning device, and a gesture generation model for a gesture generation model that generates speech gestures from voice.

When a person interacts with a humanoid robot, a computer graphics (CG) avatar, or the like (hereinafter simply referred to as a "robot"), it is desirable to prevent the person from feeling uncomfortable as much as possible. In such situations, not only the content of the utterance and the form of the utterance, but also the robot's gestures during the utterance have great significance.

Conventionally, research has been conducted on generating robot gestures according to the content of speech. A typical conventional method is a method of one-to-one mapping of speech content to gestures. However, in such a method, if the content of the utterance is the same, the gestures will also be the same. In reality, it is rare for people to make the same gesture even with the same utterance content. Therefore, if such mapping is adopted, the other party will feel uncomfortable with the robot's gestures.

On the other hand, Patent Document 1 discloses that grammatical rules prepared in advance are applied to utterance content to select several gesture candidates. In Patent Document 1, a distribution of weights assigned in advance to these gesture candidates is used, and one of the gesture candidates is determined based on a value comparison obtained from the distribution of weights for a specified parameter. is selected probabilistically. Furthermore, when the robot moves according to the selected gesture candidate, a value based on a random number is added to the trajectory of the control point to change the robot's movement.

According to Patent Document 1, basic gesture candidates are selected stochastically. Therefore, different gestures may be selected even if the content of the utterance is the same. Furthermore, since the robot's movements are ultimately changed based on random numbers, it is said to have the effect of making the robot's gestures look natural to humans.

Special Publication No. 2014-504959

According to the technology disclosed in Patent Document 1, the robot's gestures are different even if the content of the utterance is the same. However, the technology disclosed in Patent Document 1 only says that the robot's gestures are different even if the content of the utterance is the same, so it feels natural, and there is no disclosure as to whether the gestures themselves are more natural than human gestures. Even if the gesture is made random in relation to the content of the utterance, it is meaningless if the gesture itself is unnatural.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a learning method and a learning device for a gesture generation model that allows a user to naturally select and use gestures appropriate to the content of an utterance.

A learning method for a gesture generation model according to a first aspect of the present invention is a learning method for a gesture generation model for generating speech gestures from speech sounds, the method comprising: the step of preparing training data divided into a plurality of segments having overlapping periods, each of the plurality of segments including a plurality of frames of a constant length, and each of the plurality of segments including A computer generates a segment that includes an audio matrix whose components are predetermined acoustic features obtained from speech uttered by a person in a frame, and a gesture matrix consisting of a pose vector representing the pose of the person in each frame included in the segment. For each segment, a seed pose consisting of a voice matrix included in the segment, a noise matrix whose components are noise sampled from a predetermined distribution, and a predetermined number of pose vectors forming part of the gesture matrix of the immediately preceding segment. preparing a first neural network for generating and outputting a gesture matrix corresponding to the input speech matrix based on the speech matrix, the noise matrix, and the seed pose vector; , has a first input and a second input, receives a speech matrix input to the first neural network as the first input, and corresponds to the input speech matrix of the gesture matrix received as the second input. a step of preparing a second neural network for outputting an evaluation value as a gesture matrix for the voice matrix; The method includes training the first neural network and the second neural network by adversarial learning evaluated as a gesture matrix.

Preferably, the loss function in adversarial learning is a first neural network that represents the difference between the conditional distribution of output from the first neural network and the conditional distribution of correct data from the second neural network with respect to the input speech matrix. a loss function, a gradient penalty of the loss function in the second neural network, a predetermined number of pose vectors at the end of the gesture matrix of the immediately preceding learning data, and a predetermined number at the beginning of the gesture matrix output from the first neural network. and a third loss function representing the difference from the number of pose vectors.

More preferably, the predetermined time length is selected from a range of 1 second or more and 2 seconds or less.

More preferably, the predetermined time length is selected from a range of 1.3 seconds or more and 1.7 seconds or less.

Preferably, the predetermined overlap period is selected from a range of 10 ms or more and 30 ms or less.

More preferably, the predetermined overlap period is selected from a range of 15 milliseconds or more and 25 milliseconds or less.

More preferably, the predetermined acoustic feature amount includes F0 or power or both.

Preferably, the adversarial learning is performed by an unrolled-GAN (Generative Adversarial Network).

More preferably, the second neural network receives as input a gesture matrix received at the second input and a speech matrix inputted to the first neural network, and the second neural network receives as input the gesture matrix received at the second input. It includes a convolutional neural network that outputs a scalar representing an evaluation value as a gesture corresponding to a speech matrix.

More preferably, the first neural network receives as input a matrix obtained by concatenating a speech matrix, a noise matrix, and a seed pose vector in the learning data, and generates gestures corresponding to the speech matrix input to the first neural network. It includes a bi-GRU (Gated Recurrent Unit) network for outputting matrices.

A learning device for a gesture generation model according to a second aspect of the present invention is a learning device for a gesture generation model for generating human gestures at the time of speech from speech sounds, each of which has a predetermined overlap for a predetermined length of time. a learning data storage device that stores learning data divided into a plurality of segments having a period, each of the plurality of segments including a plurality of frames of a certain length, and each of the plurality of segments For each of the plurality of segments, the segment includes an audio matrix whose components are predetermined acoustic features obtained from the speech uttered by the person in each included frame, and a gesture matrix representing the gesture of the person in the frame. , a noise matrix sampled from a predetermined distribution, and a predetermined seed pose vector. a first neural network for generating and outputting a gesture matrix; receiving a gesture matrix output from the first neural network; and a speech matrix input to the first neural network; A second neural network for outputting an evaluation value of a gesture matrix to be inputted with respect to an input speech matrix, a first neural network, and a second neural network are connected to a gesture matrix from learning data, and adversarial learning means that performs training by adversarial learning that switches between the gesture matrix output from the neural network and the gesture matrix output from the neural network.

The above and other objects, features, aspects and advantages of the present invention will become apparent from the following detailed description of the invention, understood in conjunction with the accompanying drawings.

FIG. 1 is a diagram schematically showing how to use a gesture generation model according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing an outline of a learning method for a gesture generation model according to an embodiment of the present invention. FIG. 3 is a block diagram showing the functional configuration of a gesture generation model learning device according to an embodiment of the present invention. FIG. 4 is a block diagram of a gesture generation model generator according to an embodiment of the present invention. FIG. 5 is a block diagram of a gesture generation model discriminator according to an embodiment of the present invention. FIG. 6 is a schematic diagram for explaining the operation of the gesture generation model during learning according to the embodiment of the present invention. FIG. 7 is a schematic diagram for explaining a method of calculating part of the cost during learning of the gesture generation model according to the embodiment of the present invention. FIG. 8 is a flowchart showing the control structure of a program that implements the gesture generation model learning method according to the embodiment of the present invention. FIG. 9 is a functional block diagram of a gesture generation device using a gesture generation model according to an embodiment of the present invention. FIG. 10 is a schematic diagram for explaining interpolation of gesture segments generated in the gesture generation model according to the embodiment of the present invention. FIG. 11 is a diagram showing, in a table format, an objective evaluation of experiments conducted on the gesture generation model according to the embodiment of the present invention. FIG. 12 is a diagram showing, in a table format, items for subjective evaluation of experiments conducted on the gesture generation model according to the embodiment of the present invention. FIG. 13 is a graph showing the results of a subjective evaluation of an experiment conducted on the gesture generation model according to the embodiment of the present invention. FIG. 14 is a diagram showing, in a table format, differences in the effect of weights on continuity loss in the gesture generation model according to the embodiment of the present invention. FIG. 15 is a graph showing the smoothness of gesture generation by the gesture generation model according to the embodiment of the present invention. FIG. 16 is a diagram showing the external appearance of a gesture generation method and apparatus using a gesture generation model, and a computer system for realizing the gesture generation method, according to an embodiment of the present invention. FIG. 17 is a block diagram showing the hardware configuration of the computer system shown in FIG. 16.

In the following description and drawings, identical parts are provided with the same reference numerals. Therefore, detailed description thereof will not be repeated.

1 First embodiment A. Configuration In controlling a robot, there may be a method in which the robot's actions are described step by step in a program in response to the robot's utterances. However, it is clear that there are limits to how the robot can operate with such a method. Furthermore, in order to make the gestures obtained by a written program natural, it is necessary to repeat experiments and programs. There is also an infinite number of utterance contents. Therefore, in practice, it is impossible to describe the robot's actions in response to utterances.

A promising method for solving such problems is to obtain a model 54 that generates a gesture 52 corresponding to the robot's utterance from an audio signal 50 corresponding to the robot's utterance, as shown in FIG. If a model 54 that can produce natural gestures in response to any audio signal can be obtained, there is no need to go through the trouble of describing the robot's movements. The gesture generation model learning method according to the present embodiment, which will be described below, is a method for this purpose.

a. Overall Configuration FIG. 2 schematically shows the entire learning method of the gesture generation model according to this embodiment. Referring to FIG. 2, a learning system 100 that implements this gesture generation model learning method includes a learning data storage device 110 for storing learning data. The learning data storage device 110 stores a plurality of learning data sets. Each training data set includes audio signals obtained from human utterances. The audio signal is divided into segments each having a fixed length, and further divided into a plurality of frames. Then, the audio feature extraction unit 116 extracts F0 and power, which are prosody information of the audio, from each frame every 10 milliseconds and vectorizes the extracted information. This vector is referred to as a speech vector in this specification. The audio vector is a two-dimensional vector.

In this embodiment, one segment is 1.5 seconds with an overlap of 0.2 seconds between adjacent segments. That is, one segment is substantially 1.7 seconds long. In this embodiment, one segment is divided into 34 frames. One frame corresponds to 0.05 seconds. That is, one segment's worth of audio signals is made up of 34 two-dimensional audio vectors, forming a matrix of 34 rows and 2 columns. This matrix is referred to as a speech matrix in this specification.

Each segment also includes a sequence of pose vectors corresponding to each frame. One pose vector is a vector whose components are rotation angles of 13 joints in the upper body obtained from the person's posture in the corresponding frame. Since the rotation angle of each joint is three-dimensional, the pose vector per frame includes 3×13=39 components. Since there are 34 pose vectors included in one segment, pose information for one segment is a matrix of 34 rows and 39 columns. Gestures are generated by consecutive poses. Therefore, this matrix is referred to as a gesture matrix in this specification. Furthermore, one segment of gestures generated based on this gesture matrix is called a gesture chunk.

The reason why we limited the joints for obtaining the gesture matrix to the upper body is because most of the gestures that people make when speaking are concentrated in the upper body. In addition, the reason why we shortened the segment length to 1.5 seconds excluding overlaps was because long gestures can be considered to be a series of short gestures, and if the length of the generated gesture is too long, the gesture This is because it was thought that it would be difficult. As will be described later, in this embodiment, a gesture matrix is generated for each short segment of 1.5 seconds, and the gesture generation model is further trained so that the connections between adjacent segments are smooth.

For the reasons stated above, the segment length is preferably selected from a range of 1 second or more and 2 seconds or less, preferably 1.3 seconds or more and 1.7 seconds or less. Also, if the overlapping period is too long, it is not desirable, and if it is short, the gesture transition will not be smooth. Therefore, for example, select from a range of 10 ms or more and 30 ms or less, preferably 15 ms or more and 25 ms or less. is preferred.

The learning system 100 further receives as input a speech matrix 128 having speech features as components from the speech feature extraction unit 116 and a noise matrix 132 containing random noise sampled from a predetermined distribution as input, and performs predictions from the speech. The gesture matrix 126 includes a generator 118 that is trained to output a gesture matrix 126 that is information representing the gestures made. In this embodiment, the noise is sampled from a standard normal distribution. Further, the noise vectors constituting the noise matrix 132 are 20-dimensional vectors generated corresponding to each frame. Therefore, the noise matrix 132 is a matrix of 34 rows and 20 columns.

The learning system 100 further identifies whether the input gesture matrix is a gesture matrix composed of pose vectors 112 of actual data or a fake gesture matrix generated by the generator 118 from random noise; A discriminator 120 that is trained to output an evaluation value 130 representing the distance between the distribution of pose vectors 118 learned and the actual distribution of pose vectors, and a gesture matrix output from the generator 118 for this training. 126 and a real gesture matrix 124 generated from the actual pose vectors 112 stored in the learning data storage device 110 .

b. Gesture Model Learning Device FIG. 3 shows the configuration of a learning system 100 according to this embodiment. Referring to FIG. 3, the learning system 100 includes an audio capture unit 166 for capturing uttered audio and converting it into an audio signal, frames the audio signal output from the audio capture unit 166, and converts the audio signal F0 from each frame into frames. and an F0/power extraction unit 168 for calculating and outputting the power as a voice vector, and a motion capture unit 162 for capturing the movement of a predetermined part from the movement of the speaker's upper body and outputting motion capture data. A pose vector generation unit 164 calculates the rotation angles of 12 joints of the upper body and outputs a pose vector at a rate of 34 frames per 1.5 seconds, and an audio vector output from the F0/power extraction unit 168. It includes a learning data generation device 170 for generating learning data in association with the pose vector of the same frame outputted by the pose vector generation unit 164.

For learning, a plurality of pairs of audio signals of utterances and gesture data for the utterances are prepared. The above-described process is executed for each set, and multiple sets of learning data are obtained. The learning system 100 further includes a learning data storage device 110 shown in FIG. 2 for storing the plurality of sets of learning data obtained in this manner.

The learning system 100 further includes a generator 118, a discriminator 120, and a selector 122 shown in FIG. It includes a gesture model learning device 150 for training. The generator 118 after the learning by the gesture model learning device 150 is completed is used for gesture generation.

The generator 118 has three inputs, first to third, and is for generating and outputting a gesture matrix 126 corresponding to the audio signal 114 from the data received at the first to third inputs. A first input is provided with an audio signal 114 (speech matrix) from the learning data storage device 110. A noise matrix 132 is provided to the second input. To this end, the gesture model learning device 150 includes a noise vector generation unit 180 that generates a noise matrix 132 of 34 rows and 20 columns from noise sampled from a normal distribution and provides the generated noise matrix 132 to the second input of the generator 118. A third input of the generator 118 is provided with a seed pose vector 184, which will be described later.

The gesture model learning device 150 further includes an LPF (Long Pass Filter) for removing and smoothing high frequency components of each component of the gesture matrix 126 output from the generator 118 and providing the result to the first input of the selector 122. Contains 190. A second input of selector 122 is provided with a real gesture matrix 124 from training data storage 110 . That is, the selector 122 selects either the gesture matrix 126 output from the generator 118 or the real gesture matrix 124 provided from the learning data storage device 110 and outputs it as the gesture matrix 200.

Discriminator 120 has two inputs. The first input is provided with the audio matrix 128 from the training data storage 110 . A gesture matrix 200, which is the output of the selector 122, is given to the second input. The discriminator 120 generates an evaluation value 130 representing whether the gesture matrix 200 received at the second input is a genuine gesture matrix relative to the speech matrix 128 received at the first input or a fake gesture matrix generated by the generator 118. Output.

The gesture model learning device 150 further controls the discriminator 120, the generator 118, and the selector 122 using a control signal 196, and performs adversarial learning to train the discriminator 120 and the selector 122 by adversarial learning as described below. section 192, and a parameter storage section 194 where the adversarial learning section 192 temporarily saves each parameter of the neural network constituting the classifier 120 during adversarial learning.

The gesture model learning device 150 further extracts and stores pose vectors for the last four frames of the real gesture matrix 124, and uses the pose vectors of the generator 118 as a seed pose vector when the generator 118 generates a gesture matrix next time. It includes a seed pose vector extraction unit 182 for providing to the third input.

Referring to FIG. 4, the generator 118 includes a fully connected layer 250 that has the first to third inputs described above and receives the speech matrix 128, the noise matrix 132, and the seed pose vector 184 input thereto; 2-layer bi-GRU 252 connected to receive the output of the connection layer 250; and a fully connected layer 254 connected to receive the output of the 2-layer bi-GRU 252 for outputting the gesture matrix 126. .

The inputs to the fully connected layer 250 of the generator 118 are a gesture matrix in which four pose vectors are formatted into 34 rows and 39 columns, an audio signal matrix of 34 rows and two columns consisting of one segment's worth of audio vectors, and noise. This is a 34 row by 20 column noise matrix generated from . The output of the fully connected layer 250 is a matrix of 34 rows and 256 columns.

The input of the 2-layer bi-GRU 252 is a matrix with 34 rows and 256 columns, and the output is also a matrix with 34 rows and 256 columns.

The input to the fully connected layer 254 is a matrix of 34 rows and 256 columns, and the output is a matrix of 34 rows and 39 columns, similar to the matrix consisting of pose vectors.

Referring to FIG. 5, the discriminator 120 has two inputs, a fully connected layer 300 receiving the speech matrix 128 and the gesture matrix 200 from the selector 122, and a fully connected layer 300 connected to receive the output of the fully connected layer 300. 1-D convolution layer 302 and a fully connected layer 304 for receiving the output of the 1-D convolution layer 302 and outputting an evaluation value 130.

The input to the fully connected layer 300 is a 34 row x 2 column audio matrix consisting of audio vectors for one segment, and a 34 row x 2 column audio matrix consisting of real or false pose vectors received from the selector 122 shown in FIG. It is a matrix with 39 columns.

c. Details of Adversarial Learning FIG. 6 shows an outline of training of the generator 118 and the discriminator 120 by the gesture model learning device 150. In this embodiment, training is performed according to the order of training the discriminator 120 and training the generator 118.

Although not shown, first, the discriminator 120 is given a speech matrix obtained from one segment of speech and a gesture matrix consisting of a pose vector obtained from that segment. This gesture matrix is a real gesture matrix. The classifier 120 performs calculations using the parameters of each internal layer, and outputs an evaluation value indicating whether the gesture matrix is a gesture for the speech matrix. At the beginning of training, this evaluation value cannot be trusted. During training, the parameters of the classifier 120 are updated so that this evaluation value approaches 0.

Next, the generator 118 is provided with the pose vector 354 of the last four frames of the segment immediately before the segment to be processed in the learning data, the audio matrix 350 consisting of feature quantities extracted from the audio, and the audio matrix 350 that is sampled from a normal distribution. A noise vector 352 generated by noise is input. The generator 118 performs calculations using parameters in each layer on these inputs, and outputs a gesture matrix 356 consisting of pose vectors. Gesture matrix 356 is a fake gesture matrix.

This gesture matrix 356 and voice matrix 350 are provided to the discriminator 120. The classifier 120 performs calculations on the gesture matrix 356 and the audio matrix 350 using parameters of each layer, and outputs an evaluation value 358 indicating whether the gesture matrix 356 is a pose vector corresponding to the audio matrix 350. Here, first, the parameters of the classifier 120 are updated so that the evaluation value 358 becomes larger. Furthermore, the parameters of the generator 118 are updated so that the evaluation value 358 approaches 0.

Thereafter, similar processing is repeated for each segment. At this time, during learning, not the gesture matrix 356 output by the generator 118, but the actual pose vector that is learning data is used as a seed pose vector for the next process. The above is an outline of adversarial learning for the generator 118 and the classifier 120.

However, in reality, a method called unrolled-GAN is used for this adversarial learning. Furthermore, as for the loss function used as the evaluation value during learning, a loss function different from the normal one is used as described below.

The loss function used in this embodiment is shown below.

上の式において、ｙはポーズベクトルを、ｓは音声ベクトルを、ｚはノイズベクトルを、ｍはセグメント内のフレーム数を、それぞれ表す。ｋは前後のセグメント間で強制的に重複させるポーズベクトルの数を示すハイパーパラメータである。本実施形態ではｋ＝４である。上式において、「Ｈｕｂｅｒ」はＨｕｂｅｒ損失を指し、その定義は以下の文献に記載されている。
Ross Girshick. 2015. Fast r-cnn. In IEEE International Conference on Computer Vision. 1440-1448.
この文献に記載されている定義は以下のとおりである。

In the above equation, y represents a pose vector, s represents a speech vector, z represents a noise vector, and m represents the number of frames within a segment. k is a hyperparameter indicating the number of pose vectors that are forcibly overlapped between the preceding and succeeding segments. In this embodiment, k=4. In the above formula, "Huber" refers to Huber loss, and its definition is described in the following document.
Ross Girshick. 2015. Fast r-cnn. In IEEE International Conference on Computer Vision. 1440-1448.
The definitions described in this document are as follows.

Lcritic is a loss function used in normal WGAN (Wasserstein GAN). In this embodiment, the discriminator 120 is trained in advance by WGAN to be able to calculate Lcritic, and the value is used as Lcritic in the learning of the generator 118 and the discriminator 120 described above. Lgp is a gradient penalty and is a regularization term for stabilizing learning. The gradient penalty is for imposing a penalty so that the norm of the gradient of the classifier is equal to 1. In this embodiment, the discriminator has two inputs. Experiments showed that imposing this gradient penalty on both inputs had little difference in effect compared to imposing it on only one input. Therefore, in this embodiment, this penalty is imposed only on the one receiving the input of the generator among the two inputs.

The last Lcontinuity is a loss function that did not exist in the past, and was introduced in this embodiment in order to maintain continuity of the operation sequence. FIG. 7 shows the purpose of this function regarding continuity loss.

In this embodiment, one segment of audio signal 114 input to generator 118 is as short as 1.5 seconds. Therefore, the gesture matrix 126 generated from the audio signal 114 and the noise matrix 132 at one time is only for 1.5 seconds. For longer utterances, it is necessary to generate gesture chunks for multiple segments and concatenate them. Therefore, in this embodiment, the pose vectors 450 of the first few (k) frames of the gesture matrix 126 of the segment output from the generator 118 are the pose vectors 450 of the same number (k) of frames at the end of the segment just before it. This loss function is imposed to closely resemble the gesture (pose) of the frame, ie, the seed pose vector 184 input to the generator 118. By doing so, a smooth gesture can be obtained even if gesture chunks generated from gesture matrices generated for consecutive segments are directly connected.

FIG. 8 shows, in flowchart form, the control structure of the main part of the program for realizing the unrolled-GAN, which is used to train the generator 118 and the discriminator 120 in this embodiment.

Referring to FIG. 8, this program includes step 500 in which the following process 502 is repeated for all learning data.

The process 502 includes a process 510 of updating only the parameters of the classifier 120 using real gesture data, and a step 512 of saving the parameters of the classifier 120 updated in the process 510 to the parameter storage unit 194 shown in FIG. , step 512 of repeating process 514 for updating only the parameters of classifier 120 using real gesture data a predetermined number of times, similar to process 510.

The process 510 includes sampling 540 a batch of real data, sampling 542 a batch of noise data, and using the data generated from the sampled real data and noise in steps 540 and 542 to run the classifier 120 . The method includes step 544 of calculating a gradient, and step 546 of updating parameters of the discriminator 120 based on the gradient calculated in step 544.

The contents of process 516 are also the same as process 510.

Process 502 further continues process 514 with a step 518 of sampling a batch of real data, a step 520 of sampling a batch of noise data, and using the data sampled in steps 518 and 520 to generate a generator 118 and a discriminator. step 522 of calculating the gradient of the generator 118 based on the evaluation value obtained from the discriminator 120 by operating the discriminator 120; and step 524 of updating the parameters of the generator 118 based on the gradient calculated in step 522. , step 525 of further updating the parameters of the generator 118 by continuity loss, and step 526 of restoring the parameters of the discriminator 120 saved in the parameter storage unit 194 of FIG. 3 in step 512 and ending the process 502. include.

In this way, in unrolled-GAN, the parameters of the discriminator are updated multiple times before the generator, and after the parameters of the generator are updated, the parameters of the discriminator are returned to the time of the first update. Such processing is said to have the effect of preventing a phenomenon called mode collapse that often occurs in GAN learning.

d. Gesture Generation Device Using the generator 118 that has been trained by the learning system 100, gesture information for the voice vector can be obtained from the voice vector. FIG. 9 shows the configuration of a gesture generation device 560 using the generator 118.

Referring to FIG. 9, gesture generation device 560 includes an audio capture unit 166 having the same configuration as that shown in FIG. , an F0/power extraction unit 168 having the same configuration as that shown in FIG. An audio segment unit 580 for generating an audio matrix for one segment by combining 34 audio vector sequences output by the power extraction unit 168; and an audio segment unit 580 for storing the audio matrix output by the audio segment unit 580. data storage device 582.

Gesture generation device 560 further includes a gesture generation unit 584 for reading each segment of audio data stored in audio data storage device 582 and generating gesture data 586 corresponding to the audio data.

The gesture generation unit 584 includes a noise vector generation unit 180 similar to that shown in FIG. 3, a generator 118, and the last four frames of the gesture chunk output from the generator 118 as a seed pose vector for the next segment. and a seed pose vector extraction unit 590 for extracting and temporarily storing.

A first input of generator 118 is provided with an audio matrix from audio data storage 582 . A noise matrix sampled by the noise vector generator 180 is given to the second input. A seed pose matrix obtained by shaping the pose vectors for the last four frames of the immediately preceding segment into the same form as the other inputs is supplied to the third input from the seed pose vector extraction unit 590.

The gesture generation section 584 further includes an LPF 190 similar to that shown in FIG. 3, connected so that the generator 118 receives the output, and a gesture chunk storage section 592 for temporarily storing gesture chunks output from the LPF 190. Among the plurality of gesture chunks stored in the gesture chunk storage unit 592, between the gesture chunks of adjacent segments, each of the plurality of (four in this embodiment) pose vectors at the end of the preceding gesture chunk is , a gesture interpolation unit for interpolating gestures between segments by adding the same number of pose vectors at the beginning of the subsequent gesture chunk and corresponding to the preceding pose vectors with equal weights. 594. The gesture interpolated between the segments by the gesture interpolation unit 594 is output as gesture data 586.

FIG. 10 shows the configuration of gesture interpolation section 594. Referring to FIG. 10, gesture interpolation unit 594 receives gesture chunk 600 generated from the preceding audio segment and gesture chunk 606 generated from the segment immediately following that segment.

The gesture chunk 600 includes a plurality of pose vectors 604 at the end and pose vectors 602 in the other half. The gesture chunk 606 includes a plurality of pose vectors 610 at the beginning and pose vectors 608 in the other half. In this embodiment, pose vectors 604 and 610 each include four pose vectors.

The gesture interpolation unit 594 includes a multiplication unit 612 for extracting the last four pose vectors 604 from the gesture chunk 600 and multiplying each component by 0.5, and a multiplication unit 612 for extracting the last four pose vectors 604 from the gesture chunk 600 and multiplying each component by 0.5. 610 and multiplies each component by 0.5, each of the four pose vectors output by the multiplier 612, and the four pose vectors output by the multiplier 614. Among them, an adding unit 616 that adds corresponding ones to each other to generate four new interpolated pose vectors 618 is included.

By concatenating the pose vector 602 of the first half of the gesture chunk 600, the interpolated pose vector 618, and the pose vector 608 of the second half of the gesture chunk 606 in this order, one gesture is obtained from the two segments.

B. Action a. Learning Prior to learning, learning data is generated and stored in the learning data storage device 110. Before learning the generator 118 and the classifier 120 using the unrolled-GAN, the classifier 120 is trained using the WGAN. This learning enables the classifier 120 to calculate Lcritic in unrolled-GAN.

Referring to FIG. 3, of the learning data stored in learning data storage device 110, the speech matrix from the first segment is input to generator 118 and discriminator 120. Also, among the learning data, the gesture matrix of the first segment is given to the selector 122. The adversarial learning unit 192 causes the selector 122 to select a gesture matrix from the learning data. As a result, the real gesture matrix from the training data is input to the classifier 120. The classifier 120 uses the input speech matrix and gesture matrix to calculate an evaluation value based on the loss function described above.

On the other hand, the seed pose vector extraction unit 182 usually extracts a seed pose vector 184 from the end of the immediately preceding segment, formats it into a matrix, and supplies it to the generator 118. The noise vector generator 180 generates a noise matrix 132 by sampling from a normal distribution and provides it to the generator 118. The generator 118 generates the gesture matrix 126 using the speech matrix of the input learning data, the noise matrix 132, and the seed pose vector 184. The gesture matrix 126 from which high frequency components have been removed is provided to the input of the selector 122 . The selector 122 selects the output of the LPF 190 and supplies it to the discriminator 120. This input is a fake pose vector. The classifier 120 then calculates an evaluation value 130 between the audio matrix 128 and the fake pose vector.

The adversarial learning unit 192 calculates the parameter gradient of the classifier 120 from the first two terms of the loss function based on the evaluation value 130 obtained from the real pose vector and the evaluation value 130 obtained from the fake pose vector. Then, the parameters of the classifier 120 are updated based on the learning rate and this gradient. As a result of this update, the parameters of the classifier 120 are updated so that the evaluation value is close to 0 for real data and becomes large for fake data. The adversarial learning unit 192 saves the updated parameters to the parameter storage unit 194.

The adversarial learning unit 192 further repeats the process of updating the parameters of the classifier 120 a predetermined number of times. However, at this time, the process of saving the updated parameters to the parameter storage unit 194 is not performed. Through this process, the learning of the parameters of the classifier 120 has temporarily progressed to a generation earlier than the parameters of the generator 118.

After that, the parameters of the generator 118 are updated based on the evaluation value obtained from the discriminator 120 based on the learning data and the evaluation value obtained from the discriminator 120 based on the noise data in the same manner. be exposed. In this update, the parameters of the generator 118 are updated so that the evaluation value output by the discriminator 120 becomes smaller for false data.

Furthermore, Lcontinuity is calculated using the last four pose vectors of the preceding segment and the first four pose vectors of the following segment, and the generator 118 parameters are updated. Thereafter, the compared parameters of the classifier 120 are restored to the classifier 120.

In this way, learning using all the learning data is repeated, and when a predetermined termination condition is met, the learning of the generator 118 ends. For example, the termination condition may be that training has been completed a predetermined number of times.

Through this learning, the generator 118 has learned the distribution of gesture data represented by the learning data.

b. Inference During inference, gesture generator 560 (FIG. 9) operates as follows. Referring to FIG. 9, audio capture unit 166 captures audio that is a target of gesture generation and converts it into an audio signal. The F0/power extraction unit 168 calculates F0 and power for each frame of this audio signal and vectorizes it. The audio segment unit 580 divides the vector string output from the F0/power extraction unit 168 into a segment string of 1.5 seconds each with an overlapping portion of 0.2 seconds between segments, and stores the segment strings in the audio data storage device 582. Store.

Generator 118 reads the audio matrix of the segment of interest from audio data storage 582 . Further, the noise vector generator 180 samples noise from the standard normal distribution and provides it to the generator 118 as a noise matrix. The seed pose vector extraction unit 590 extracts the last four pose vectors of the gesture chunk obtained from the output of the generator 118 for the immediately preceding segment as seed pose vectors, and provides them to the generator 118.

The generator 118 performs calculations on these inputs using parameters obtained through learning, and outputs a gesture matrix consisting of a sequence of pose vectors corresponding to the input speech matrix. This gesture matrix is stored in the gesture chunk storage unit 592 via the LPF 190. The last four pose vectors of the stored gesture matrix are read out by the seed pose vector extraction unit 590 as seed pose vectors for processing the next segment.

The gesture interpolation unit 594 adds 0.5 to the corresponding vector among the four pose vectors existing in the overlapping part of two adjacent gesture matrices among the plurality of gesture chunks stored in the gesture chunk storage unit 592. Interpolate gestures between two segments by multiplying and adding.

C. Effects According to the above embodiment, it becomes possible to generate gestures that give a smoother and more natural impression than conventional methods. Furthermore, since gestures are generated stochastically, different gestures can be generated for the same utterance. According to the experiments described below, it was found that the above-mentioned effects can be obtained by both objective evaluation and subjective evaluation.

Second experiment A. Experimental Settings In the experiment, 1094 sets of training data consisting of 1094 pairs of motions (gestures) and Japanese utterances were obtained by performing motion capture on a person speaking. The total time for these data is 6 hours. To train the model, the learning rate was set to 10 ⁻⁴ for both generator 118 and discriminator 120. The batch size was 128. λc and λgp in the loss function were set to 1 and 10, respectively. The normal distribution used for noise sampling had a mean of 0 and a variance of 1. It took 6 hours to train the model using a high-performance computer equipped with a GPU (Graphics Processing Unit).

Kernel Density Estimation (KDE) is said to be suitable for evaluating distributions because it outputs the log likelihood of a certain distribution in a predetermined distribution. Therefore, KED was used for objective evaluation of gestures generated in the following experiments. Furthermore, as a subjective evaluation, one based on the user's perception was used. That is, in order to summarize the evaluation of the performance of the above embodiment, a survey of users was conducted. The rotation angle of a joint is not intuitively easy for humans to understand. However, by moving the avatar image based on the obtained gesture model, the user's evaluation of the gesture can be obtained.

In the experiment, the trained generator according to the embodiment described above was compared with various control groups using video clips. Its contents are as follows.

・Correct answer (Ground Truth: GT)
Real data from real human gestures obtained through motion capture. Before applying it to the avatar, preprocessing (smoothing) was performed using a low-pass filter to remove jerky movements. This jerky movement is probably caused by the device used for motion capture.

・Baseline (reference data)
As the baseline model, we used a state-of-the-art model for generating coordinate values of joints, which was trained using the same data set as the data set used for learning in the above embodiment. This model is a probabilistic gesture generation model based on GAN. For comparison, we used a neural network to convert joint coordinate data into corresponding joint rotation angles.

- Embodiment The model of the embodiment generates gestures stochastically. Therefore, the model of this embodiment allows different gesture sequences to be generated from the same speech segment. In order to evaluate the results of such different gestures, for each utterance set, two videos were generated using the model of the embodiment as a first plan and a second plan, which will be described later. When generating this video, the model used different noise vectors for the same audio input.

B. Experimental results a. Objective Evaluation Gestures generated by the embodiment and baseline models were both fitted to the KDE model. The optimal frequency band was determined using a grid search with triple cross-validation. Therefore, the larger the output value, the closer the gestures were to the actual data distribution. The results obtained using the various models described above are shown in FIG. 11.

Referring to FIG. 11, the correct answer result was calculated by calculating the likelihood of the correct answer data using a KDE model fitted by the correct answer data itself. This result can be seen as the best possible result. The closer the log-likelihood of a model is to the log-likelihood of the correct answer, the better the model can be considered to fit the correct data distribution.

As shown in FIG. 11, the model according to the embodiment of the present application was able to achieve a log likelihood closer to the correct data.

b. Subjective Evaluation Three scales were used to evaluate the model according to the embodiment. That is, the naturalness of the gesture, the consistency between the gesture and the utterance, and the semantic consistency. In order to obtain these evaluation values, questions as shown in FIG. 12 were set.

For comparison, a video was created using the baseline model, first and second plans in which gestures were generated from different noise vectors in the embodiment of the present application, and correct data, and used for comparison. In the experiment, we selected five separate utterances, and created video clips of avatar movements that reproduced four types of gestures (correct data, baseline, first plan, and second plan) for each utterance. The order of the videos was randomized. After watching each video, subjects were asked to indicate on a scale of 1 to 7 (1: strongly disagree, 7: strongly agree) the avatar's movements for each item shown in Figure 12. required to be allocated.

Thirty-three people gathered through a crowdsourcing service were used as subjects. Of these 33, 17 were men and 16 were women. All were native speakers of Japanese. The mean age was 38 years and the variance was 9.5 years.

The results are shown in graphical form in FIG.

An analysis of variance was performed to statistically examine whether there were significant differences in scores using the settings in this experiment. All scale scores passed analysis of variance at p<0.01.

Furthermore, Tukey HSD (Tukey's Honestly Significant Difference test) was conducted to examine whether there was a significant difference between the groups for each group combination. As shown in FIG. 13, in terms of naturalness, there was a significant difference between the baseline and the correct answer at p<0.05. Significant differences were found between the baseline and the first plan and between the baseline and the second plan at p<0.01. There was no significant difference between the correct data and the first plan, p<0.77, and no significant difference was found between the correct data and the second plan, p=0.77. Furthermore, no significant difference was found between the first and second plans at p=0.9.

Furthermore, regarding temporal consistency, there were significant differences at p<0.01 between the baseline and the correct answer, between the baseline and the first plan, and between the baseline and the second plan. was there. There was no significant difference between the correct answer and the first plan, the correct answer and the second plan, and the first plan and the second plan, with p=0.9.

Regarding semantic consistency, there was a significant difference between the baseline and the correct answer at p<0.01. There was a significant difference between the baseline and the first plan at p<0.05, and a significant difference between the baseline and the second plan at p<0.01. No significant difference was found between the correct answer and the first plan, and between the correct answer and the second plan, with p=0.9. No significant difference was found between the first plan and the second plan at p=0.86.

As described above, according to this embodiment, a natural gesture that is subjectively the same as a correct answer was obtained.

Note that in the above embodiment, the continuity loss Lc is used as the loss, and is multiplied by λc in the loss function. Using this loss has two effects. The first effect is that continuity of gestures can be obtained by forcibly making the pose vectors of the overlapping portions of two gesture matrices generated adjacently the same. Second, by using the last pose vector of the gesture chunk of the preceding segment as a seed pose vector when generating the next segment, subsequent segments are given context and semantic consistency can be maintained between segments. The effect is that it can be done.

In order to investigate the effect on gestures when changing the value of the coefficient λc of this continuity loss Lc, we performed learning by changing only the value of the coefficient λc and keeping other hyperparameters constant. built the model. As a result, it was found that when the coefficient λc=0, gestures appear to be separated for each segment. That is, a gesture has at least two stages: preparation and stroke, and the stroke must be completely completed before preparation, but if no continuity loss Lc is used, the final stage of the stroke I got the impression that the stroke was stopped before it reached that point, and preparations for the next gesture began. Furthermore, when only interpolation between gestures is used without using the continuity loss Lc, the gestures become smoother, but give an unnatural impression.

Furthermore, from observation of the generated gestures, it can be said that although continuity is an issue with gestures generated by a model learned without using continuity loss, these gestures can all be said to be movements within the correct range. Therefore, we hypothesized that the log likelihoods of joint angles and positions in generated gestures should not differ significantly between models that include continuity loss and models that do not, and in order to verify this. An experiment was conducted. We also experimentally verified the hypothesis that the log-likelihood of joint velocities should be significantly different between models trained using continuity loss and models that were not trained, based on the observed results of generated gestures. did. The experiment to test this hypothesis is as follows.

First, the output of the joint angle was converted to the coordinate position of the joint. Furthermore, the velocity of each joint was calculated from the converted data and evaluated using KDE. FIG. 14 shows the evaluation results of joint positions and velocities using KDE. Referring to FIG. 14, it was found that by introducing the continuity loss, a distribution of positions and velocities generated by the model that is similar to the correct solution can be obtained.

Furthermore, as shown in FIG. 15, it can be seen that the abnormal velocity of the joints mainly occurs in the transition section of the generated gesture. Referring to FIG. 15, an abnormally large speed does not occur in the speed graph 720 obtained from the correct data. Further, no abnormal increase in speed is observed in the graph 724 of the embodiment in which λc=1. On the other hand, in the case of a model obtained by learning with λc=0 and no continuity loss, it can be seen that the speed periodically increases abnormally, as seen in graph 722. These sections 700 to 714 are all boundary portions between segments, that is, transition sections from the preceding gesture chunk to the next gesture chunk. In particular, it can be seen that when no continuity loss is used, abnormally large velocities occur at the beginning and end of gesture chunks. Although the cause of this phenomenon is unknown, this result shows that such problems can be avoided by at least introducing continuity loss.

Third Implementation by Computer FIG. 16 is an external view of an example of a computer system that implements the above embodiment. FIG. 17 is a block diagram showing an example of the hardware configuration of the computer system shown in FIG. 16.

Referring to FIG. 16, this computer system 950 includes a computer 970 having a DVD (Digital Versatile Disc) drive 1002, a keyboard 974, a mouse 976, and a monitor for interacting with the user, all connected to the computer 970. 972. Of course, these are examples of configurations for when user interaction is required, and any general hardware and software that can be used for user interaction (e.g. touch panel, voice input, pointing device in general) can be used. Available.

Referring to FIG. 17, a computer 970 includes, in addition to a DVD drive 1002, a CPU (Central Processing Unit) 990, a GPU 992, a bus 1010 connected to the CPU 990, the GPU 992, and the DVD drive 1002; , a ROM (Read-Only Memory) 996 that stores a boot-up program for the computer 970, and a RAM (Random Access Memory) 998 that is connected to the bus 1010 and stores instructions, system programs, work data, etc. that make up the program. and an SSD (Solid State Drive) 1000 that is a nonvolatile memory connected to a bus 1010. The SSD 1000 is for storing programs executed by the CPU 990 and GPU 992, data used by the programs executed by the CPU 990 and GPU 992, and the like. The computer 970 further includes a network I/F (Interface) 1008 that provides a connection to a network 986 that enables communication with other terminals, and a USB (Universal Serial Bus) memory 984 that is removable. 970.

The computer 970 is further connected to a microphone 982, a speaker 980, a motion capture device (not shown), and a bus 1010, and transmits audio signals, video signals, motion data, text data, etc. generated by the CPU 990 and stored in the RAM 998 or the SSD 1000 to the CPU 990. Read the signal according to the instructions, perform analog conversion and amplification processing to drive the speaker 980, or digitize the analog audio signal from the microphone 982 and save it in an arbitrary address specified by the CPU 990 in the RAM 998 or SSD 1000. , an input/output I/F 1004 for receiving a motion capture signal from a motion capture device and storing it at an address designated by the CPU 990.

In the above embodiment, the learning system 100 and the gesture generation device 560 or a program for realizing the generator 118, the discriminator 120, the adversarial learning unit 192, etc. that are part of the learning system 100, the neural network parameters, the neural network program, etc. are stored in, for example, the SSD 1000, RAM 998, DVD 978, or USB memory 984 shown in FIG. 17, or a storage medium of an external device (not shown) connected via the network I/F 1008 and network 986. Typically, these data and parameters are written into the SSD 1000 from the outside, for example, and loaded into the RAM 998 when executed by the computer 970.

A computer program for operating this computer system so as to realize the functions of the learning system 100 and the gesture generation device 560 shown in FIGS. , are transferred from the DVD drive 1002 to the SSD 1000. Alternatively, these programs are stored in the USB memory 984, the USB memory 984 is attached to the USB port 1006, and the programs are transferred to the SSD 1000. Alternatively, this program may be transmitted to computer 970 via network 986 and stored on SSD 1000.

The program is loaded into RAM 998 during execution. Of course, a source program may be input using the keyboard 974, monitor 972, and mouse 976, and the compiled object program may be stored in the SSD 1000. In the case of a script language, a script input using the keyboard 974 or the like may be stored in the SSD 1000. In the case of a program that runs on a virtual machine, it is necessary to install the program that functions as a virtual machine on the computer 970 in advance. Since the training and testing of a neural network involves a large amount of calculation, the program portion that is the entity that performs numerical calculations is implemented as an object program consisting of computer native code rather than a script language. is preferable.

The CPU 990 reads the program from the RAM 998 according to the address indicated by an internal register called a program counter (not shown), interprets the instruction, and stores the data necessary for executing the instruction in the RAM 998 according to the address specified by the instruction. , the SSD 1000 or other devices and executes the processing specified by the command. The CPU 990 stores the data of the execution result at an address specified by the program, such as the RAM 998, the SSD 1000, or a register within the CPU 990. At this time, the value of the program counter is also updated by the program. Computer programs may be loaded directly into RAM 998 from DVD 978, from USB memory 984, or via a network. Note that in the program executed by the CPU 990, some tasks (mainly numerical calculations) are dispatched to the GPU 992 according to instructions included in the program or according to an analysis result when the CPU 990 executes the instructions.

A program that realizes the functions of each part according to the embodiment described above in cooperation with the computer 970 includes a plurality of instructions written and arranged to cause the computer 970 to operate to realize those functions. Some of the basic functionality necessary to execute this instruction is provided by an operating system (OS) or third party programs running on computer 970 or by modules of various toolkits installed on computer 970. Therefore, this program does not necessarily include all the functions necessary to implement the system and method of this embodiment. This program may be activated by statically linking appropriate functions or Programming Tool Kit functions within the instructions in a controlled manner to achieve the desired results, or when the program is run. By dynamically linking these functions, it is sufficient to include only the instructions for executing the operations of each of the above-mentioned devices and their constituent elements. The manner in which computer 970 operates for this purpose is well known and will not be repeated here.

Note that the GPU 992 can perform parallel processing, and can execute a large amount of calculations associated with machine learning simultaneously in parallel or in a pipeline manner. For example, parallel computing elements found in a program when the program is compiled or parallel computing elements discovered when the program is executed are dispatched from the CPU 990 to the GPU 992 and executed, and the results are sent directly or to the RAM 998. It is returned to the CPU 990 via a predetermined address, and is substituted into a predetermined variable in the program.

Fourth Modification In the above embodiment, prosody information called F0 and power is used as the acoustic feature amount. However, the invention is not limited to such embodiments. Other acoustic features related to linguistic information such as MFCC (Mel-frequency cepstral coefficient) may also be used. Further, the number of dimensions of each vector and matrix used in the above embodiment is merely an example. It goes without saying that the number of dimensions may be changed as appropriate depending on the application. Further, in the above embodiment, four vectors are used as seed pose vectors. This corresponds to a value obtained by dividing 0.2 seconds, which is the overlapping period of audio segments, by 0.05 seconds, which is the time of one frame. Therefore, if the overlapping period of audio segments is changed or the time of one frame is changed, the number of seed pose vectors may be changed accordingly.

In the above embodiment, a standard normal distribution is used as a distribution for sampling noise. However, this invention is not limited to such. A distribution different from the standard normal distribution may be used. Furthermore, the configurations of the generator 118 and the discriminator 120 are not limited to those described above. The configuration may be changed as appropriate depending on the application. Furthermore, in the above embodiment, unrolled-GAN is used for adversarial learning. This is because unrolled-GAN is less likely to cause mode collapse. Therefore, other algorithms may be used as long as they are algorithms for adversarial learning that do not cause mode collapse.

The embodiment disclosed this time is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim, with reference to the description of the detailed description of the invention, and all changes within the scope and meaning equivalent to the words described therein are defined. include.

50, 114 Audio signal 52 Gesture 54 Model 100 Learning system 110 Learning data storage device 112, 354, 450, 602, 604, 608, 610 Pose vector 116 Audio feature extractor 118 Generator 120 Discriminator 122 Selector 124, 126 , 200, 356 Gesture matrix 128, 350 Audio matrix 130, 358 Evaluation value 132 Noise matrix 150 Gesture model learning device 170 Learning data generation device 180 Noise vector generation unit 184 Seed pose vector 192 Adversarial learning unit 250, 254, 300, 304 Fully connected layer 252 2-layer bi-GRU
302 1-D convolution layer 352 Noise vector 560 Gesture generation device 580 Voice segment section 582 Voice data storage device 584 Gesture generation section 586 Gesture data 592 Gesture chunk storage section 594 Gesture interpolation section 600, 606 Gesture chunk 612, 614 Multiplication section 616 Addition Section 618 Interpolated pose vector

Claims (11)

A learning method for a gesture generation model for generating speech gestures from speech sounds, the method comprising:
the computer preparing training data divided into a plurality of segments each having a predetermined length of time and a predetermined overlapping period;
each of the plurality of segments includes a plurality of frames of a certain length;
Each of the plurality of segments includes an audio matrix whose components are predetermined acoustic features obtained from speech uttered by a person in each frame included in the segment, and a pose representing the pose of the person in each frame included in the segment. a gesture matrix consisting of vectors;
A computer generates, for each of the segments, the audio matrix included in the segment, a noise matrix whose components are noise sampled from a predetermined distribution, and a predetermined number of gesture matrices forming part of the gesture matrix of the immediately preceding segment. a seed pose vector consisting of pose vectors, and generates and outputs the gesture matrix corresponding to the input voice matrix based on the voice matrix, the noise matrix, and the seed pose vector. a step of preparing a neural network in step 1;
a computer having a first input and a second input, receiving the speech matrix input to the first neural network at the first input; preparing a second neural network for outputting an evaluation value as the gesture matrix corresponding to the input voice matrix;
The computer switches the gesture matrix output from the first neural network and the gesture matrix of the learning data, and performs adversarial learning to evaluate the gesture matrix as a gesture matrix for the voice matrix. 2. A learning method for a gesture generation model, the method comprising the step of training a neural network according to No. 2. The loss function in the adversarial learning is
a first loss function representing a difference between a conditional distribution of output from the first neural network and a conditional distribution of correct data from the second neural network with respect to the input speech matrix;
a gradient penalty of the loss function in the second neural network;
A first neural network representing a difference between the predetermined number of pose vectors at the end of the gesture matrix of the immediately preceding learning data and the predetermined number of pose vectors at the head of the gesture matrix output from the first neural network. 3. The gesture generation model learning method according to claim 1, comprising a loss function of 3. 3. The gesture generation model learning method according to claim 1, wherein the predetermined time length is selected from a range of 1 second or more and 2 seconds or less. 4. The gesture generation model learning method according to claim 3, wherein the predetermined time length is selected from a range of 1.3 seconds or more and 1.7 seconds or less. 5. The gesture generation model learning method according to claim 3, wherein the predetermined overlapping period is selected from a range of 10 milliseconds or more and 30 milliseconds or less. 6. The gesture generation model learning method according to claim 5, wherein the predetermined overlapping period is selected from a range of 15 milliseconds or more and 25 milliseconds or less. 7. The gesture generation model learning method according to claim 1, wherein the predetermined acoustic feature includes F0, power, or both. The gesture generation model learning method according to any one of claims 1 to 7, wherein the adversarial learning is performed by an unrolled-GAN. The second neural network receives as input the gesture matrix received at the second input and the voice matrix inputted to the first neural network, and receives the gesture matrix received at the second input. The learning method for a gesture generation model according to any one of claims 1 to 8, comprising a convolutional neural network that outputs a scalar representing an evaluation value of a matrix as a gesture corresponding to the audio matrix. The first neural network receives as input a matrix in which the audio matrix, the noise matrix, and the seed pose vector in the learning data are connected, and applies a matrix to the audio matrix input to the first neural network. The learning method of a gesture generation model according to any one of claims 1 to 9, comprising a bi-GRU network for outputting the corresponding gesture matrix. A learning device for a gesture generation model for generating human gestures during speech from speech sounds, comprising:
a learning data storage device storing learning data divided into a plurality of segments each having a predetermined length of time and a predetermined overlapping period;
each of the plurality of segments includes a plurality of frames of a certain length;
Each of the plurality of segments includes an audio matrix whose components are predetermined acoustic features obtained from speech uttered by a person in each frame included in the segment, and a gesture matrix representing the gesture of the person in the frame. including,
For each of the plurality of segments, the audio matrix included in the segment, a noise matrix sampled from a predetermined distribution, and a predetermined seed pose vector are input, and the audio matrix, the noise matrix, and the seed a first neural network for generating and outputting the gesture matrix corresponding to the input voice matrix based on a pose vector;
The gesture matrix output from the first neural network and the voice matrix input to the first neural network are received, and the input voice of the gesture matrix output from the first neural network is received. a second neural network for outputting evaluation values for the matrix;
Adversarial training in which the first neural network and the second neural network are trained by adversarial learning that switches between the gesture matrix from the training data and the gesture matrix output from the first neural network. A learning device for a gesture generation model, comprising a learning means.

Images