JP2020082246A - Posture data generation device, learning tool, computer program, learning data, posture data generation method and learning model generation method - Google Patents

Abstract

To provide a posture data generation device that can automatically generate a gesture for presentation, and to provide a learning tool, a computer program, learning data, a posture data generation method and a learning model generation method.SOLUTION: A posture data generation device includes: a learning tool that is produced using uttered voice data and human body posture data as learning data; an acquisition unit for acquiring the uttered voice data; and a generating unit for generating posture data on the basis of the uttered voice data acquired by the acquisition unit and the learning tool.SELECTED DRAWING: Figure 1

Description

The present invention relates to a posture data generation device, a learning device, a computer program, learning data, a posture data generation method, and a learning model generation method.

In recent years, agents such as robots and avatars have advanced into society, and the opportunities for agents to come into contact with each other are increasing in daily life. An example of such an agent is to give a presentation.

Patent Document 1 discloses a service robot that includes an application for facing a person like a presentation and that can rotate a joint of an arm or a head to change a posture to provide highly reliable communication. It is disclosed.

JP, 2006-297531, A

However, in order to realize natural communication with people, agents need to create gestures that look like natural communication. However, the production of natural gestures requires a creator with considerable skill. Moreover, in order to realize various operation patterns, it is necessary to manually create programs and motion data in advance. For this reason, there is a problem in that it takes a long time to produce a natural gesture and the cost becomes high.

The present invention has been made in view of the above circumstances, and includes a posture data generation device, a learning device, a computer program, learning data, a posture data generation method, and a learning model that can automatically generate a gesture of a presentation. It is intended to provide a generation method.

A posture data generation device according to an embodiment of the present invention is a learning device that is generated by using utterance voice data and posture data of a human body as learning data, an acquisition unit that acquires utterance voice data, and the acquisition unit. And a generation unit that generates posture data based on the utterance voice data acquired in step 1 and the learning device.

The learning device according to the embodiment of the present invention is generated using utterance voice data and human posture data as learning data.

The figure computer program according to an embodiment of the present invention is acquired by a process of acquiring utterance voice data in a computer and a learner generated using utterance voice data and human body posture data as learning data. A process of inputting utterance voice data and generating posture data is executed.

The learning data according to the embodiment of the present invention is learning data including time-series data of utterance voice data and time-series data of posture data of a human body extracted from a presentation moving image, and the posture data is a plurality of human body data. Processing for giving time-series data of the 3D data of the joint positions over a plurality of frames of the presentation moving image to the output node of the recursive neural network, and 1 frame of the presentation moving image. A process of giving time series data of the time series data of the utterance voice data sampled a required number of times to the input node of the recursive neural network, and the node given to the output node and the input, respectively. And a process of learning the recurrent neural network based on time series data.

A posture data generation method according to an embodiment of the present invention acquires utterance voice data, and acquires the utterance voice data in a learning device that is generated by using the utterance voice data and the posture data of the human body as learning data. Input and generate posture data.

A learning model generation method according to an embodiment of the present invention acquires utterance voice data and human body posture data, and uses the acquired utterance voice data and human body posture data as learning data.

According to the present invention, a presentation gesture can be automatically generated.

It is a block diagram which shows an example of a structure of the gesture generation apparatus of this Embodiment. It is a schematic diagram which shows an example of utterance voice data. It is a schematic diagram which shows an example of the time series data of the pitch and energy of speech voice. It is a schematic diagram which shows an example of three-dimensional posture data. It is a schematic diagram which shows an example of a structure of a learning model. It is a schematic diagram which shows an example of the mode of the output of the posture data by a learning model. It is a schematic diagram which shows the 1st example of the connection method of the gesture produced|generated for every utterance sentence. It is a schematic diagram which shows the 2nd example of the connection method of the gesture produced|generated for every utterance sentence. It is a schematic diagram which shows an example of the gesture which the gesture generation apparatus of this Embodiment produced|generated. It is a schematic diagram which shows the example of calculation of time T of the gesture to replace among the generated gestures. It is explanatory drawing which shows an example of the relationship between a keyword and gesture data. It is a schematic diagram which shows an example of the resampling method of the gesture to replace. It is a schematic diagram which shows an example of the result of resampling. It is a schematic diagram which shows an example of the gesture after replacing with the gesture corresponding to a keyword. It is a schematic diagram which shows an example of interpolation of the gesture after replacement. It is a block diagram which shows an example of a structure of the learning model production|generation part of this Embodiment. It is a schematic diagram which shows an example of the standard which divides an utterance sentence into learning units. It is a schematic diagram which shows an example of the generation method of a learning model. It is a flow chart which shows an example of the processing procedure of gesture generation by the gesture generation device of this embodiment. It is a flow chart which shows an example of a processing procedure of learning model generation by a learning model generation part.

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing an example of the configuration of the gesture generation device 50 according to the present embodiment. The gesture generation device 50 includes a control unit 51 that controls the entire device, an acquisition unit 52, a storage unit 53, a processing unit 54, and a generation unit 57. Further, the processing unit 54 includes a learning model 55 as a learning device, and a correction unit 56. The control unit 51 can be composed of a CPU, a ROM, a RAM, and the like.

The acquisition unit 52 can acquire utterance voice data.

FIG. 2 is a schematic diagram showing an example of uttered voice data. In FIG. 2, the vertical axis represents the amplitude of the waveform, which can be represented by a voltage level, for example. The horizontal axis represents time.

The control unit 51 has a voice analysis function and can extract the pitch and energy of the utterance voice from the utterance voice data acquired by the acquisition unit 52.

FIG. 3 is a schematic diagram showing an example of time-series data of pitch and energy of speech voice. The pitch of the uttered voice is the frequency of the voice waveform and can represent the pitch of the voice. The energy of the uttered voice is the energy of the voice and can represent the strength of the voice. The time series data of the pitch and energy of the uttered voice are collectively referred to as voice prosody time series data.

The acquisition unit 52 can also acquire time-series data of the pitch and energy of the spoken voice. In this case, the control unit 51 does not need to extract the pitch and energy of the spoken voice. The acquisition unit 52 has, for example, a function of reading utterance voice data stored in an external storage device, or time-series data of pitch and energy of utterance voice, or a function of receiving it via a communication network such as the Internet. be able to.

The storage unit 53 can store the uttered voice data acquired by the acquisition unit 52 or the time-series data of the pitch and energy of the uttered voice. The storage unit 53 also stores a plurality of keywords and time-series data of transmission three-dimensional data that transmits the meaning of each of the plurality of keywords in association with each other. Details of the keywords and the transmitted three-dimensional data will be described later.

The processing unit 54 is, for example, a hardware such as a CPU (for example, a multi-processor including a plurality of processor cores), a GPU (Graphics Processing Units), a DSP (Digital Signal Processors), and an FPGA (Field-Programmable Gate Arrays). Can be configured by combining. It is also possible to combine quantum processors.

The learning model 55 is generated by using the speech data and the posture data of the human body as learning data. For example, the learning model 55 can be generated by using the speech data of the person who gives the presentation and the posture data indicating the movement of the person as the learning data.

More specifically, the learning model 55 is generated by using the time-series data of the pitch and energy of the uttered voice illustrated in FIG. 3 as the learning data. The learning model 55 is generated by using time-series data of three-dimensional data of a plurality of joint positions of the human body as learning data.

FIG. 4 is a schematic diagram showing an example of three-dimensional posture data. In the figure, symbols P1 to P9 indicate the positions of the joints of the upper half of the human body. Although FIG. 4 shows nine joints, the number of joints is not limited to nine. It suffices for the plurality of joint positions to include a portion in which a person's movement is remarkably displayed at the time of presentation, and as shown in FIG. 4, the plurality of joint positions of the upper body including the waist, arms, shoulders, neck, head and the like are set. be able to. The three-dimensional data can be xyz coordinates in a reference coordinate system.

Thereby, the learning model 55 can learn the relationship between the utterance of a person and the movement of the body accompanying the utterance. Further, the speaker's intention and enthusiasm at the time of utterance are expressed as voice prosody, that is, changes in pitch and energy of the uttered voice. Therefore, by using the phonetic prosody time series data as the learning data, the learning model 55 can output the posture data expressing the intention or enthusiasm.

The learning model 55 only needs to use time-series data as learning data, and can be, for example, a recurrent neural network (Recurrent Neural Network), but is not limited thereto. The learning model 55 may use another machine learning. Details of the learning model 55 will be described later.

The generation unit 57 can generate posture data based on the acquired utterance voice data (more specifically, time-series data of each pitch and energy of the utterance voice) and the learning model 55.

Since the learning model 55 is generated in advance using the utterance voice data and the posture data of the human body as learning data, when the acquired utterance voice data is input to the learning model 55, the learning model 55 receives the input utterance voice. Outputs posture data that is related to the data. Thereby, the utterance of the person and the gesture (movement of the body) accompanying the utterance can be generated, and the gesture of the presentation can be automatically generated. Also, the cost of gesture production can be reduced.

FIG. 5 is a schematic diagram showing an example of the configuration of the learning model 55. The learning model 55 includes an encoder 551 and a decoder 552. The encoder 551 encodes the uttered voice data (specifically, the time-series data of the pitch and energy of the uttered voice) input to the input node. The decoder 552 decodes the encoded data and outputs the posture data of the human body (specifically, the time series data of the three-dimensional data of a plurality of joint positions) from the output node.

The encoder 551 and the decoder 552 have a plurality of intermediate layers called LSTMs (long short term memories). The LSTM has a memory cell (not shown), can hold necessary time series data in the past, and can output the hidden state ht to the LSTM at the next time. The encoder 551 and the decoder 552 can convert the time series data into another time series data. In the figure, <EOS> is a "delimiter", which is used as a signal notifying the decoder 552 of the start of generation of time-series data, and is also used as an end signal. Note that, in FIG. 5, for convenience, a plurality of LSTM layers are collectively shown as one LSTM. In addition, an embedding layer, a fully connected layer, etc. are omitted.

FIG. 6 is a schematic diagram showing an example of how posture data is output by the learning model 55. In the example of FIG. 6, for convenience, three frames (time points t, t+1, and t+2) of three-dimensional posture time series data are output from the output node. Since there are xyz coordinate values per joint, where the number of joint positions is nine, there are 27 (=9×3) data per frame. In this case, assuming that the sampling number of the speech prosody time series data in one frame is n, the speech prosody time series data at the time point t is (X ₁ ,..., X _n ) and the speech at the time point t+1. The prosody time series data is (X _n+1 ,..., X _2n ), and the speech prosody time series data at the time point t+2 is (X _2n+1 ,..., X _3n ). Here, X is a physical quantity including the pitch and energy of the spoken voice.

The generation unit 57 can generate time-series data of three-dimensional data of a plurality of joint positions of the human body. Thereby, the generation unit 57 can generate posture data indicating a plurality of joint positions of the upper body, which change with the passage of time, and can automatically generate a presentation gesture.

Further, the generation unit 57 can generate time-series data of three-dimensional data of a plurality of joint positions of the human body in units of uttered sentences. The utterance sentence is a unit in which a voice and a gesture are few at the beginning and end of the utterance (basic posture). By learning and generating a gesture for each utterance sentence, it is possible to avoid a sudden change in the movement of the gesture before and after the connection point when the generated gesture is connected.

Even if the body movement is smooth in the utterance sentence unit, the body movement may not be smooth between the utterance sentences, and the generated gesture may be unnatural between the utterance sentences. Hereinafter, a method for smoothing gestures between spoken sentences will be described.

FIG. 7 is a schematic diagram showing a first example of a method of connecting gestures generated for each uttered sentence. As shown in Fig. 7, the utterance sentence is "one orange on the surface of the moon", and the utterance sentence "is as small as observed" connected to the utterance sentence unit S1. The unit is S2. The speech prosody time series data of the uttered sentence unit S1 is input to the learning model 55, and the generation unit 57 generates a plurality of gestures (time series data of three-dimensional data of a plurality of joint positions). As shown in FIG. 7, the penultimate gesture and the final gesture of the uttered sentence unit S1 are represented as G12 and G11.

Similarly, the phonetic prosody time series data of the uttered sentence unit S2 is input to the learning model 55, and the generation unit 57 generates a plurality of gestures (time series data of three-dimensional data of a plurality of joint positions). As shown in FIG. 7, the first gesture and the next gesture of the utterance sentence unit S2 are represented as G21 and G22.

The correction unit 56 has a function as a first correction unit, and is three-dimensional data in one utterance sentence unit that is connected to three-dimensional data in another utterance sentence unit that is connected to one utterance sentence unit. Correct the data.

In the example of FIG. 7, the gesture corrected by the correction unit 56 is illustrated as the gesture after connection. As shown in FIG. 7, the corrected gesture of the gesture G11 can be generated by performing linear interpolation of the gestures G11 and G21. Specifically, the data of the gesture G11 is weighted by 34%, the data of the gesture G21 is weighted by 66%, and the sum of the weighted data of the gestures G11 and G21 is corrected. Gesture.

The corrected gesture of the gesture G12 can be generated by linearly interpolating the gestures G12 and G21. Specifically, the data of the gesture G12 is weighted by 67%, the data of the gesture G21 is weighted by 33%, and the sum of the weighted data of the gestures G12 and G21 is corrected. Gesture.

The correction of the gesture G21 and the correction of the gesture G22 can be performed in the same manner. In the example of FIG. 7, the gestures G12 and G11 are corrected and the gestures G21 and G22 are corrected. However, the present invention is not limited to this, and either the gestures G12 and G11 or the gestures G21 and G22 is performed. Only one may be corrected. Further, the weighting ratio (%) is an example, and is not limited to the example of FIG. 7.

Within the utterance sentence unit, it is possible to generate posture data indicating natural movement and smooth movement. However, between the utterance sentence and the next utterance sentence, the temporal change of the posture data becomes large, and an unnatural gesture may be generated. Therefore, as shown in FIG. 7, by correcting the posture data of the portion where the utterance sentences are connected, the change in the posture data between the utterance sentences can be smoothed and a natural gesture can be generated.

FIG. 8 is a schematic diagram showing a second example of a method of connecting gestures generated for each uttered sentence. In the example of FIG. 7, the last two gestures in the utterance sentence unit S1 and the first two gestures in the utterance sentence unit S2 are corrected, but in FIG. 8, the last gesture in the utterance sentence unit S1 is corrected. And the first gesture in the uttered sentence unit S2 are corrected.

As shown in FIG. 8, the data of the gesture G11 is weighted by 50%, the data of the gesture G21 is weighted by 50%, and the sum of the weighted data of the gestures G11 and G21 is calculated. It is the corrected gesture. The same applies to the correction of the gesture G21. Further, only one of the gestures G11 and G21 may be corrected. Further, the weighting ratio (%) is an example, and is not limited to the example of FIG. 8.

Next, an example of correction using a gesture that expresses the meaning of a word will be described.

FIG. 9 is a schematic diagram showing an example of a gesture generated by the gesture generation device 50 according to the present embodiment. The gesture shown in FIG. 9 is a gesture generated from the phonetic prosody time series data extracted from the utterance sentence “There is a large elephant in the center of the galaxy”, and for convenience, the gesture for 10 frames is illustrated.

The correction unit 56 determines whether or not there is a specific word (also referred to as “keyword”) in the utterance sentence, and if there is a keyword, specifies the utterance timing of the keyword. The utterance timing can be determined by the start point and the end point of the keyword utterance. Note that the utterance start time and utterance length may be used.

FIG. 10 is a schematic diagram showing an example of calculating the time T of a gesture to be replaced among the generated gestures. As shown in FIG. 10, the correction unit 56 determines the first time point and the last time point of the gesture to be replaced in the generated gesture. The first time point can be a time point t1 seconds before the start time of the speech time. t1 can be, for example, 645 milliseconds, but is not limited thereto. t1 can be the time when body movement begins prior to the utterance of the word. Further, the last time point can be a time point t2 seconds after the end time of the utterance time. t2 can be, for example, 555 milliseconds, but is not limited to this. t2 can be the time when the body movement ends after the word utterance ends.

The correction unit 56 divides the time T of the gesture to be replaced by the frame rate (the time at which each frame is displayed, that is, the reciprocal of fps) to calculate the frame number F of the gesture to be replaced.

FIG. 11 is an explanatory diagram showing an example of the relationship between keywords and gesture data. Keywords are characterized by the movement that expresses the meaning of the keyword, such as "big", "small", "eat", "laugh", and "long". It is a related word. The gesture data is time-series data of gestures that express the movement of the body when speaking the corresponding keyword. For example, when a person speaks "large", the body movement can be represented by five gestures g1, g2, g3, g4, and g5. The relationship between the keyword and the gesture data can be stored in the storage unit 53.

FIG. 12 is a schematic diagram showing an example of a resampling method of a replacement gesture. As shown in FIG. 12, the gestures expressing the keyword “large” are five gestures g1, g2, g3, g4, and g5. The correction unit 56 calculates the sampling interval Δt for resampling. The sampling interval Δt can be obtained by dividing the time from the start point of the gesture g1 to the end point of the gesture g5 by (F-1). Here, F is the number of frames of the gesture to be replaced. In the example of FIG. 12, the time from the start point of the gesture g1 to the end point of the gesture g5 is divided into three intervals.

The sampled gestures are G1, G2, G3, and G4. The gesture g1 uses the gesture g1 as it is. The gesture G4 uses the gesture g5 as it is. As the gesture G2, a gesture obtained by linearly interpolating the gestures g2 and g3 is used according to the sampling timing. The gesture G3 uses a linear interpolation of the gestures g3 and g4 according to the sampling timing. As a result, the changes in the sampled gestures G1 to G4 become smooth, and a natural movement can be achieved.

FIG. 13 is a schematic diagram showing an example of the result of resampling. Gestures g1, g2, g3, g4, and g5 expressing “large” are resampled into four gestures G1, G2, G3, and G4.

FIG. 14 is a schematic diagram showing an example of a gesture after replacement with a gesture corresponding to a keyword. The original gesture is replaced with four gestures G1, G2, G3, and G4 starting from a time point t1 seconds before the start time point of the utterance of the keyword (“large” in the example of FIG. 14). The first gesture G1 of the replaced gestures and the gesture before the gesture G1 may not be smoothly changed. Similarly, the change of the last gesture G4 of the replaced gestures and the gesture after the gesture G4 may not be smooth. Therefore, the gesture can be interpolated as follows.

FIG. 15 is a schematic diagram illustrating an example of gesture interpolation after replacement. In the example of FIG. 15, two gestures before the replaced gesture G1 are linearly interpolated. Specifically, the gesture before the gesture G1 can have a greater weight than the gesture before the gesture G1. In addition, the two gestures behind are also linearly interpolated by the replaced gesture G4. Specifically, the gesture one behind the gesture G4 can make the weight of the gesture G4 heavier than the gesture two behind.

As described above, the correction unit 56 has a function as the second correction unit, and corresponds to the keyword (for example, “large”) in the utterance sentence corresponding to the time-series data of the three-dimensional data generated by the generation unit 57. The time series data of the three-dimensional data to be processed can be corrected using the time series data of the transmission three-dimensional data (g1 to g5, or G1 to G4, etc.) associated with the keyword. The series data is, for example, data indicating temporal changes in three-dimensional data representing a body movement expressing a keyword, whereby a gesture can be corrected with a movement expressing the meaning of a word in a spoken sentence. Therefore, it is possible to generate a natural gesture including a motion involving information transmission.

Next, a method of generating the learning model 55 will be described.

FIG. 16 is a block diagram showing an example of the configuration of the learning model generation unit 60 of this embodiment. The learning model generation unit 60 may be incorporated in the gesture generation device 50, or may be incorporated in another learning server. The learning model generation unit 60 includes a presentation moving image acquisition unit 61, a speech voice data extraction unit 62, a frame image extraction unit 63, a pitch and energy extraction unit 64, a two-dimensional posture extraction unit 65, and a three-dimensional posture estimation unit 66.

The presentation moving image acquisition unit 61 acquires a presentation moving image. In order to collect a large amount of data for learning, it is possible to use, for example, a presentation moving image that is disclosed and available on the web for everyone.

The utterance voice data extraction unit 62 extracts utterance voice data from the presentation moving image. The pitch and energy extraction unit 64 has a voice analysis function and can extract the energy and pitch of the uttered voice from the uttered voice data at a required sampling period. The speech prosody time series data extracted by the pitch and energy extraction unit 64 is given to the input node of the learning model 55 as learning data.

The frame image extraction unit 63 extracts an image from the presentation moving image in frame units. The two-dimensional posture extraction unit 65 identifies parts such as a human face, arm, and foot from each frame image, connects the identified parts, and two-dimensional coordinates of a plurality of joints (also referred to as two-dimensional pose information). To extract.

The three-dimensional posture estimation unit 66 includes a database in which two-dimensional posture information and three-dimensional posture information are associated with each other in advance, and based on the two-dimensional posture information extracted by the two-dimensional posture extraction unit 65, the closest three-dimensional posture is obtained. Estimate attitude information. The three-dimensional posture information includes the three-dimensional coordinates of a plurality of joints. The three-dimensional posture estimation unit 66 can extract the three-dimensional posture data illustrated in FIG. 4 in frame units. The three-dimensional posture time-series data extracted by the three-dimensional posture estimation unit 66 is given to the output node of the learning model 55 as learning data.

As described above, the learning model 55 can be generated using the time-series data of the pitch and energy of the uttered voice extracted from the uttered voice data as the learning data. The pitch of the uttered voice is the frequency of the voice waveform and can represent the pitch of the voice. The energy of the uttered voice is the energy of the voice and can represent the strength of the voice. The time series data of the pitch and energy of the uttered voice are collectively referred to as voice prosody time series data.

The intention and enthusiasm of the speaker at the time of utterance are expressed as voice prosody, that is, changes in pitch and energy of the uttered voice. Therefore, by using the phonetic prosody time series data as the learning data, the learning model 55 can output the posture data expressing the intention or enthusiasm.

The learning model 55 can be generated using time-series data of three-dimensional data of a plurality of joint positions of the human body as learning data. It is sufficient that the plurality of joint positions include a portion in which a person's movement is remarkably shown at the time of presentation, and may be, for example, positions of a plurality of joints in the upper body. The three-dimensional data can be xyz coordinates in a reference coordinate system.

The utterance voice data and the posture data of the human body are extracted from the presentation moving image for each utterance sentence, and the learning model 55 can generate the utterance voice data and the posture data of the human body for each utterance sentence as a set of learning data. it can. The utterance sentence is a unit in which a voice and a gesture are few at the beginning and end of the utterance (basic posture). By learning and generating a gesture for each utterance sentence, it is possible to avoid a sudden change in the movement of the gesture before and after the connection point when the generated gesture is connected.

The three-dimensional data of a plurality of joint positions of the human body is extracted for each frame of the presentation moving image, and the learning model 55 can be generated by using the time-series data over a plurality of frames of the extracted three-dimensional data as learning data. .. When the presentation moving image is composed of images of 10 frames per second (10 fps), it is possible to generate a gesture for 1 second by using the time-series data of 10 frames as learning data. Thereby, the gesture of the required time can be generated.

The utterance voice data and the human body posture data are extracted in units of subtitle text insertion from the presentation video in which the subtitles are inserted, and the learning model 55 uses the extracted utterance voice data and the human body posture data as a set of learning data. Can be generated.

FIG. 17 is a schematic diagram showing an example of criteria for dividing an uttered sentence into learning units. FIG. 17 illustrates a main part of subtitle information. A subtitle is divided into a plurality of subtitle texts, and a start time indicating an insertion start time point of each subtitle text and a time length indicating an insertion time are recorded in association with each other. ing. For example, in the presentation moving image, the subtitle text Text1 starts to be displayed at time 0.00 and is displayed for 2.00 seconds. The subtitle text Text2 is started to be displayed at time 2.00 and is displayed for 1.45 seconds. The same applies to other subtitle texts.

By using the data of the subtitle text insertion unit as a set of learning data, it is possible to learn by using the voice utterance data and the posture data within the time when the body movement is smooth. Therefore, the learning model 55 has a smooth body movement. It is possible to generate posture data within a certain period of time, and it is possible to generate a natural gesture linked to the uttered voice.

FIG. 18 is a schematic diagram showing an example of a method of generating the learning model 55. As shown in FIG. 18, the output node of the recurrent neural network extends over a plurality of frames (three frames which are utterance sentence units in the example of FIG. 18) of a presentation moving image of three-dimensional data of a plurality of joint positions of a human body. Given the time series data, the plurality of frames of the time series data of the uttered voice data sampled the required number of times (40 times in the example of FIG. 18) during one frame of the presentation moving image are input to the input node of the recursive neural network. A learning model can be generated by giving time-series data (for 3 frames).

For example, the number of frames of the presentation moving image is 3, and the time points of the frames are t, t+1, and t+2. The output node of the recurrent neural network is given three-dimensional data at time points t, t+1, and t+2. Assuming that the number of samples of the uttered voice data per frame is n (n=40 in the example of FIG. 18), the input node of the recursive neural network corresponds to the time point (X ₁ ,..., X _n ) the time-series data of the utterance voice data is given, and the time-series data of the utterance voice data of (X _n+1 ,..., X _2n ) is given corresponding to the time point t+1, and corresponding to the time point t+2. , (X _2n+1 ,..., X _3n ) are given as time series data of the uttered voice data.

As a result, the relationship between the utterance (prosody of the utterance) and the information on the body movement can be learned, and a gesture that matches the utterance (prosody of the utterance) can be generated.

FIG. 19 is a flow chart showing an example of a procedure for generating a gesture by the gesture generation device 50 according to the present embodiment. Hereinafter, for convenience, the main body of processing will be described as the control unit 51. The control unit 51 acquires the utterance voice data (S11), and extracts the pitch and energy of the utterance voice at a required sampling cycle for each utterance sentence (S12).

The control unit 51 inputs the speech prosody time series data to the learning model 55 (S13) and outputs the three-dimensional posture time series data (S14). The control unit 51 corrects the other utterance sentence connected to the one utterance sentence and the three-dimensional posture data of at least one of the one utterance sentence (S15).

The control unit 51 determines whether or not there is a keyword in the uttered sentence (S16), and when the keyword is present (YES in S16), the gesture data associated with the keyword is used to determine the 3D corresponding to the keyword. The posture data is corrected (S17), and the process of step S18 described later is performed.

When there is no keyword in the uttered sentence (NO in S16), the control unit 51 determines whether or not to end the process (S18), and when the process is not to be ended (NO in S18), the processes of step S12 and thereafter. When the processing is to be ended (YES in S18), the processing is ended.

FIG. 20 is a flowchart showing an example of a processing procedure of learning model generation by the learning model generation unit 60. The learning model generation unit 60 acquires the presentation moving image (S31) and extracts the uttered voice data and the frame image for each subtitle text insertion unit (S32). The learning model generation unit 60 extracts time-series data (speech prosody time-series data) of pitch and energy of utterance voice from the utterance voice data (S33), extracts two-dimensional posture information from the frame image, and three-dimensional posture information. Is estimated (S34).

The learning model generation unit 60 extracts the three-dimensional posture time series data based on the estimated three-dimensional posture information (S35), and uses the phonetic prosody time series data and the three-dimensional posture time series data as the learning data 55. Is generated (S36), and the process ends.

The gesture generation device 50 or the learning model generation unit 60 according to the present embodiment can also be realized by using a general-purpose computer including a CPU (processor), GPU, RAM (memory), and the like. That is, as shown in FIG. 19 or FIG. 20, by loading a computer program that defines the procedure of each process into a RAM (memory) provided in the computer and executing the computer program by a CPU (processor), Thus, the gesture generation device 50 or the learning model generation unit 60 can be realized. The computer program may be recorded in a recording medium and distributed, or may be installed in the gesture generation device 50 via a network.

According to the present embodiment, it is possible to automatically generate a natural gesture that is linked to a spoken voice, and reduce the cost required for gesture production.

Further, according to the present embodiment, the impression received from the generated gesture can be changed according to the presentation moving image used for generating the learning model 55. For example, when the presentation animation used includes a large number of speakers, the individualities of the speakers are averaged and the generated gestures can be average. On the contrary, when the learning model is generated using the presentation moving image of the specific speaker, it is possible to generate the gesture reflecting the individuality.

The posture data generation device of the present embodiment is obtained by the learning device that is generated by using the utterance voice data and the posture data of the human body as learning data, the acquisition unit that acquires the utterance voice data, and the acquisition unit. And a generation unit that generates posture data based on the speech data and the learning device.

The learning device of the present embodiment is generated by using utterance voice data and human posture data as learning data.

The computer program of the present embodiment, a process of acquiring utterance voice data to a computer, a learner generated by using the utterance voice data and the posture data of the human body as learning data, the acquired utterance voice data. A process of inputting and generating attitude data is executed.

The posture data generation method of the present embodiment acquires utterance voice data, inputs the obtained utterance voice data to a learning device that is generated by using the utterance voice data and the posture data of the human body as learning data. Generate attitude data.

The learning model generation method of the present embodiment acquires utterance voice data and human body posture data, and uses the acquired utterance voice data and human body posture data as learning data.

The learning device (learning model) is generated by using the uttered voice data and the posture data of the human body as learning data. For example, it is possible to generate a learning device by using speech data of a person who gives a presentation and posture data indicating the movement of the person as learning data. Accordingly, the learning device can learn the relationship between the utterance of a person and the movement of the body accompanying the utterance. The learning device may be any device that uses time-series data as learning data, and can be, for example, a recurrent neural network (Recurrent Neural Network), but is not limited thereto.

The acquisition unit acquires the speech voice data, and the generation unit generates the posture data based on the acquired speech voice data and the learned device. Since the learning device is generated in advance using the utterance voice data and the posture data of the human body as learning data, when the acquired utterance voice data is input to the learning device, the learning device associates with the input utterance voice data. Output posture data that has a certain property. Thereby, the utterance of the person and the gesture (movement of the body) accompanying the utterance can be generated, and the gesture of the presentation can be automatically generated. Also, the cost of gesture production can be reduced.

In the posture data generation device of the present embodiment, the generation unit generates time-series data of three-dimensional data of a plurality of joint positions of a human body.

The learning device according to the present embodiment is generated using time-series data of three-dimensional data of a plurality of joint positions of the human body as learning data.

The learning device is generated using time-series data of three-dimensional data of a plurality of joint positions of the human body as learning data. It is sufficient that the plurality of joint positions include a portion in which a person's movement is remarkably shown at the time of presentation, and may be, for example, positions of a plurality of joints in the upper body. The three-dimensional data can be xyz coordinates in a reference coordinate system.

The generation unit generates time-series data of three-dimensional data of a plurality of joint positions of a human body. Accordingly, the generation unit can generate posture data indicating a plurality of joint positions of the upper body, which change with the passage of time, and can automatically generate a gesture for presentation.

In the posture data generation device according to the present embodiment, the generation unit generates time-series data of three-dimensional data of a plurality of joint positions of a human body in units of uttered sentences.

The learning device of the present embodiment extracts the utterance voice data and the posture data of the human body from the presentation video for each utterance sentence, and uses the utterance voice data and the posture data of the human body for each utterance sentence as a set of learning data. Has been generated.

The learning device extracts the utterance voice data and the human body posture data for each utterance sentence from the presentation moving image, and generates the utterance voice data and the human body posture data for each utterance sentence as a set of learning data. The utterance sentence is a unit in which a voice and a gesture are few at the beginning and end of the utterance (basic posture). By learning and generating a gesture for each utterance sentence, it is possible to avoid a sudden change in the movement of the gesture before and after the connection point when the generated gesture is connected.

The generation unit generates time-series data of three-dimensional data of a plurality of joint positions of the human body in units of uttered sentences. As a result, it is possible to generate posture data within a time period in which the body movement is smooth, and it is possible to generate a natural gesture that is linked to the uttered voice.

The posture data generation device according to the present embodiment corrects three-dimensional data in one utterance sentence unit, which is connected to the one utterance sentence unit and three-dimensional data in another utterance sentence unit. A first correction unit is provided.

The first correction unit corrects the three-dimensional data in one utterance sentence unit and the three-dimensional data connected to the other utterance sentence unit in another utterance sentence unit. Within the utterance sentence unit, it is possible to generate posture data indicating natural movement and smooth movement. However, between the utterance sentence and the next utterance sentence, there is a possibility that the temporal change of the posture data becomes large and an unnatural gesture is generated. Then, by correcting the posture data at the place where the utterance sentences are connected, the change in the posture data between the utterance sentences can be smoothed and a natural gesture can be generated.

The posture data generation device according to the present embodiment stores a plurality of keywords and a storage unit that associates and stores time series data of transmission three-dimensional data that transmits the meaning of each of the plurality of keywords, and 3 generated by the generation unit. A second correction unit for correcting the time series data of the three-dimensional data corresponding to the keyword in the utterance sentence corresponding to the time series data of the three-dimensional data using the time series data of the transmission three-dimensional data associated with the keyword. Prepare

The storage unit stores a plurality of keywords and time-series data of transmission three-dimensional data that transmits the meaning of each of the plurality of keywords in association with each other. A keyword is a word related to a motion for the purpose of information transmission that is characterized by a motion that expresses the meaning of the keyword. For example, words such as “large”, “small”, “eat”, and “laugh” are used. Including. The time-series data of the transmission three-dimensional data is, for example, data indicating a temporal change of the three-dimensional data representing the movement of the body expressing the keyword.

The second correction unit sets the time-series data of the three-dimensional data corresponding to the keyword in the utterance sentence corresponding to the time-series data of the three-dimensional data generated by the generation unit to the time of the transmission three-dimensional data associated with the keyword. Correct using the series data. With this, the gesture can be corrected by the movement that expresses the meaning of the word in the spoken sentence, and thus a natural gesture including the movement accompanied by information transmission can be generated.

The learning device according to the present embodiment is generated by using the time-series data of the pitch and energy of the speech voice extracted from the speech data as learning data.

The learning device is generated using the time-series data of the pitch and energy of the utterance voice extracted from the utterance voice data as learning data. The pitch of the uttered voice is the frequency of the voice waveform and can represent the pitch of the voice. The energy of the uttered voice is the energy of the voice and can represent the strength of the voice. The time series data of the pitch and energy of the uttered voice are collectively referred to as voice prosody time series data.

The intention and enthusiasm of the speaker at the time of utterance are expressed as voice prosody, that is, changes in pitch and energy of the uttered voice. Therefore, by using the phonetic prosody time series data as the learning data, the learning device can output the posture data expressing the intention or enthusiasm.

The learning device of the present embodiment extracts utterance voice data and human body posture data in a subtitle text insertion unit from a presentation video in which subtitles are inserted, and learns a set of extracted utterance voice data and human body posture data. It is generated by using it as data.

The learner extracts the utterance voice data and the posture data of the human body for each subtitle text insertion unit from the presentation video in which the subtitles are inserted, and generates them by using the extracted utterance voice data and the posture data of the human body as a set of learning data. I am doing it. By using the data for each subtitle text insertion unit as a set of learning data, it is possible to learn by voice utterance data and posture data within the time when the body movement is smooth. Posture data can be generated, and a natural gesture linked to the uttered voice can be generated.

The learning device according to the present embodiment extracts three-dimensional data of a plurality of joint positions of a human body for each frame of a presentation moving image and generates time-series data over a plurality of frames of the extracted three-dimensional data as learning data. I am doing it.

The learning device extracts three-dimensional data of a plurality of joint positions of the human body for each frame of the presentation moving image, and uses the extracted time-series data of a plurality of frames of the three-dimensional data as learning data. When the presentation moving image is composed of images of 10 frames per second (10 fps), it is possible to generate a gesture for 1 second by using the time-series data of 10 frames as learning data. Thereby, the gesture of the required time can be generated.

The learning device of the present embodiment provides time series data for a plurality of frames of a presentation moving image of three-dimensional data of a plurality of joint positions of a human body, which is given to an output node of the recurrent neural network, and an input node of the recurrent neural network. And the time-series data of the time-series data of the uttered voice data sampled a required number of times during one frame of the presentation moving image over the plurality of frames are used as learning data.

The learning data of the present embodiment is learning data having time-series data of utterance voice data and time-series data of posture data of a human body extracted from a presentation moving image, and the posture data is a plurality of joint positions of a human body. Between the process of giving the output node of the recursive neural network the time-series data of the three-dimensional data of the plurality of joint positions over a plurality of frames of the presentation moving image, and one frame of the presentation moving image. A process of applying time-series data of the time-series data of the uttered voice data sampled a required number of times over the plurality of frames to an input node of the recurrent neural network, and the time-series data applied to the output node and the input node, respectively. And a process of learning the recurrent neural network based on the above.

The learner gives time series data over a plurality of frames of a presentation moving image of three-dimensional data of a plurality of joint positions of a human body to an output node of the recursive neural network, and the presentation moving image given to an input node of the recurrent neural network. Is generated by using the time-series data of the utterance voice data sampled the required number of times during one frame of the above as the learning data.

For example, the number of frames of the presentation moving image is 3, and the time points of the frames are t, t+1, and t+2. The output node of the recurrent neural network is given three-dimensional data at time points t, t+1, and t+2. Assuming that the number of samples of the uttered voice data per frame is n, the time-series data of the uttered voice data of (X ₁ ,..., X _n ) is associated with the time t at the input node of the recurrent neural network. Given, time-series data of the utterance voice data of (X _n+1 ,..., X _2n ) is given corresponding to the time point t ₊₁ , and (X _2n+1 ,..., X _3n is given corresponding to the time point t+2. ) The time-series data of the utterance voice data is given.

As a result, the relationship between the utterance (prosody of the utterance) and the information on the body movement can be learned, and a gesture that matches the utterance (prosody of the utterance) can be generated.

50 gesture generation device 51 control unit 52 acquisition unit 53 storage unit 54 processing unit 55 learning model 551 encoder 552 decoder 56 correction unit 57 generation unit 60 learning model generation unit 61 presentation video acquisition unit 62 uttered voice data extraction unit 63 frame image extraction unit 64 pitch and energy extraction unit 65 two-dimensional posture extraction unit 66 three-dimensional posture estimation unit

Claims (16)

A learning device generated using utterance voice data and human posture data as learning data,
An acquisition unit for acquiring utterance voice data,
A posture data generation device, comprising: a generation unit that generates posture data based on the utterance voice data acquired by the acquisition unit and the learning device. The generator is
The posture data generation device according to claim 1, wherein time-series data of three-dimensional data of a plurality of joint positions of a human body is generated. The generator is
The posture data generation device according to claim 1 or 2, wherein time-series data of three-dimensional data of a plurality of joint positions of a human body is generated for each utterance sentence. The 3rd data in one utterance sentence unit, Comprising: The 1st correction|amendment part which correct|amends 3 dimensional data connected with the 3 dimensional data in another utterance sentence unit connected with the said 1 utterance sentence is provided. Attitude data generator. A storage unit that stores a plurality of keywords and time series data of transmission three-dimensional data that transmits the meaning of each of the plurality of keywords in association with each other,
Using the time-series data of the transmission three-dimensional data associated with the keyword, the time-series data of the three-dimensional data corresponding to the keyword in the utterance sentence corresponding to the time-series data of the three-dimensional data generated by the generation unit. The attitude data generation device according to claim 3 or 4, further comprising a second correction unit that corrects. A learning device generated using utterance voice data and human body posture data as learning data. 7. The learning device according to claim 6, wherein the learning device is generated by using time-series data of each of pitch and energy of the utterance voice extracted from the utterance voice data as learning data. The learning device according to claim 6 or 7, which is generated by using time-series data of three-dimensional data of a plurality of joint positions of a human body as learning data. The utterance voice data and the human body posture data are extracted from the presentation video for each utterance sentence, and the utterance voice data and the human body posture data for each utterance sentence are generated as a set of learning data. Item 9. The learning device according to any one of items 8. Claimed speech data and human body posture data are extracted in units of subtitle text insertion from a presentation video in which subtitles are inserted, and generated by using the extracted speech voice data and human body posture data as a set of learning data. The learning device according to any one of claims 6 to 9. 7. The method according to claim 6, wherein three-dimensional data of a plurality of joint positions of the human body is extracted for each frame of the presentation moving image, and time-series data over a plurality of frames of the extracted three-dimensional data is used as learning data. The learning device according to any one of 10. Time-series data for a plurality of frames of a presentation moving image of three-dimensional data of a plurality of joint positions of a human body, which is given to an output node of a recurrent neural network
The time-series data of the time-series data of the uttered voice data sampled a required number of times during one frame of the presentation moving image, which is given to the input node of the recurrent neural network, is used as learning data. The learning device according to any one of claims 6 to 11. On the computer,
A process of acquiring speech data,
A computer program that causes a learner, which is generated by using utterance voice data and human posture data as learning data, to input the obtained utterance voice data and generate posture data. Learning data having time-series data of speech data extracted from a presentation video and time-series data of posture data of a human body,
The posture data has three-dimensional data of a plurality of joint positions of the human body,
A process of giving time series data over a plurality of frames of a presentation moving image of three-dimensional data of a plurality of joint positions to an output node of a recursive neural network;
A process of giving time series data of the time series data of the utterance voice data sampled a required number of times during one frame of the presentation moving image over the plurality of frames to an input node of the recursive neural network;
Learning data used for performing processing for learning the recursive neural network based on the time series data given to the output node and the input node, respectively. Acquire speech data,
A posture data generation method for generating posture data by inputting the acquired voice data to a learning device that is generated by using utterance voice data and human posture data as learning data. Acquire speech data and posture data of the human body,
A method for generating a learning model using the acquired utterance voice data and human posture data as learning data.

Images