TWI766499B - Method and apparatus for driving interactive object, device and storage medium - Google Patents

Abstract

The present disclosure relates to a method and an apparatus for driving interactive object, a device and a storage medium. The interactive object is displayed in a display device, and the method comprises: obtaining a phoneme sequence corresponding to sound driving data of the interactive object; obtaining a value of pose parameter of the interactive object matched with the phoneme sequence; and controlling a posture of the interactive object displayed by the display device according to the value of posture parameter.

Description

Driving method, device, device and storage medium for interactive objects

The present disclosure relates to the field of computer technology, and in particular, to a driving method, apparatus, device and storage medium for interactive objects.

Most of the human-computer interaction methods are based on keystrokes, touch, and voice input, and respond by presenting images, texts or virtual characters on the display screen. At present, virtual characters are mostly improved on the basis of voice assistants.

Embodiments of the present disclosure provide a driving solution for an interactive object.

According to an aspect of the present disclosure, there is provided a method for driving an interactive object, where the interactive object is displayed on a display device, the method comprising: obtaining a phoneme sequence corresponding to sound driving data of the interactive object; obtaining a phoneme sequence corresponding to the sound driving data of the interactive object; The matched gesture parameter value of the interactive object; controlling the gesture of the interactive object displayed by the display device according to the gesture parameter value. With reference to any of the embodiments provided in the present disclosure, the method further includes: controlling the display device to output speech and/or text according to the phoneme sequence.

With reference to any of the embodiments provided in the present disclosure, the obtaining the gesture parameter value of the interactive object matching the phoneme sequence includes: performing feature encoding on the phoneme sequence to obtain feature information of the phoneme sequence; obtaining The gesture parameter value of the interactive object corresponding to the feature information of the phoneme sequence.

With reference to any of the implementation manners provided in the present disclosure, performing feature encoding on the phoneme sequence to obtain feature information of the phoneme sequence includes: for each phoneme in the plurality of phonemes included in the phoneme sequence, generating the phoneme respectively according to the coding values of the coding sequences corresponding to the phonemes and the corresponding durations of various phonemes in the phoneme sequence respectively, obtain the feature information of the coding sequences of the phonemes; The feature information of the coding sequence is obtained, and the feature information of the phoneme sequence is obtained.

With reference to any of the implementation manners provided in the present disclosure, for each phoneme in the plurality of phonemes included in the phoneme sequence, generating a separate coding sequence of the phoneme includes: detecting whether each time point corresponds to the phoneme; The coding value at the time point with the phoneme is set as the first numerical value, and the coding value at the time point without the phoneme is set as the second numerical value, so as to obtain the coding sequence corresponding to the phoneme.

With reference to any of the embodiments provided in the present disclosure, according to the coding values of the coding sequences corresponding to the multiple phonemes and the durations corresponding to the multiple phonemes, the feature information of the coding sequence corresponding to the multiple phonemes is obtained, including: For each phoneme in the plurality of phonemes, for the coding sequence corresponding to the phoneme, use a Gaussian filter to perform a Gaussian convolution operation on the continuous values of the phoneme in time to obtain the feature of the coding sequence corresponding to the phoneme News.

With reference to any of the embodiments provided in the present disclosure, the posture parameters include facial posture parameters, the facial posture parameters include facial muscle control coefficients, and the facial muscle control coefficients are used to control the motion state of at least one facial muscle; according to the posture parameters The value controls the posture of the interactive object displayed by the display device, including: according to the facial muscle control coefficient value matching the phoneme sequence, driving the interactive object to make a face matching each phoneme in the phoneme sequence action.

With reference to any of the embodiments provided in the present disclosure, the method further includes: acquiring driving data of the body posture associated with the facial posture parameter value; controlling the interactive object displayed by the display device according to the posture parameter value The gesture includes: driving the interactive object to perform body movements according to the driving data of the body gesture associated with the facial gesture parameter value.

With reference to any of the embodiments provided in the present disclosure, acquiring the gesture parameter value of the interactive object corresponding to the feature information of the phoneme sequence includes: sampling the feature information of the phoneme sequence at a set time interval to obtain a first sample Sampling feature information corresponding to time; inputting the sampling feature information corresponding to the first sampling time into a pre-trained neural network to obtain the attitude parameter value of the interactive object corresponding to the sampling feature information.

With reference to any of the embodiments provided in the present disclosure, the neural network includes a long short-term memory network and a fully connected network, and the sampling feature information corresponding to the first sampling time is input into a pre-trained neural network and obtaining the attitude parameter value of the interactive object corresponding to the sampling feature information, comprising: inputting the sampling feature information corresponding to the first sampling time into the long-term and short-term memory network, according to the first sampling time The previously sampled feature information, and output associated feature information; input the associated feature information into the fully connected network, and determine the attitude parameter value corresponding to the associated feature information according to the classification result of the fully connected network; Wherein, each category in the classification result corresponds to a set of attitude parameter values.

With reference to any of the embodiments provided in the present disclosure, the neural network is obtained by training phoneme sequence samples; the method further includes: acquiring a video segment of a character uttering speech; acquiring a plurality of video segments including the character according to the video segment a first image frame, and multiple audio frames corresponding to the multiple first image frames; converting the first image frame into a second image frame including the interactive object, and obtaining the The attitude parameter value corresponding to the second image frame; according to the attitude parameter value corresponding to the second image frame, the audio frame corresponding to the first image frame is marked; The audio frame of the pose parameter value is obtained to obtain the phoneme sequence sample.

In combination with any of the embodiments provided in the present disclosure, the method further includes: performing feature encoding on the phoneme sequence samples, obtaining feature information corresponding to the second sampling time, and labeling the feature information with corresponding attitude parameter values , obtain feature information samples; train the initial neural network according to the feature information samples, and train the neural network after the change of the network loss satisfies the convergence condition, wherein the network loss includes the initial neural network The difference between the pose parameter value predicted by the network and the labeled pose parameter value.

With reference to any of the embodiments provided in the present disclosure, the network loss includes a two-norm of the difference between the attitude parameter value predicted by the initial neural network and the marked attitude parameter value; the network loss It further includes a norm of the attitude parameter value predicted by the initial neural network.

According to an aspect of the present disclosure, there is provided an apparatus for driving an interactive object, where the interactive object is displayed on a display device, and the apparatus includes: a phoneme sequence acquisition unit configured to acquire a phoneme sequence corresponding to sound driving data of the interactive object ; a parameter acquisition unit for acquiring the gesture parameter value of the interactive object matched with the phoneme sequence; a driving unit for controlling the gesture of the interactive object displayed by the display device according to the gesture parameter value.

According to an aspect of the present disclosure, an electronic device is provided, the device includes a memory and a processor, the memory is used to store computer instructions that can be executed on the processor, and the processor is used to execute the computer instructions. At the same time, the driving method of the interactive object described in any one of the implementation manners provided by the present disclosure is implemented.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method for driving an interactive object according to any one of the implementation manners provided in the present disclosure.

The method, device, device, and computer-readable storage medium for driving an interactive object according to one or more embodiments of the present disclosure, by acquiring the phoneme sequence corresponding to the sound driving data of the interactive object displayed by the display device, and obtaining the phoneme sequence matching the phoneme sequence. The gesture parameter value of the interactive object, and according to the gesture parameter value of the interactive object matching the phoneme sequence, the gesture of the interactive object displayed by the display device is controlled, so that the interactive object makes the same The target object communicates or responds to the matching gesture of the target object, so that the target object feels that it is communicating with the interactive object, and the interactive experience between the target object and the interactive object is improved.

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

The term "and/or" in this article is only a relationship to describe related objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein refers to any combination of any one of a plurality or at least two of a plurality, for example, including at least one of A, B, and C, and may mean including those composed of A, B, and C. Any one or more elements selected in the collection.

At least one embodiment of the present disclosure provides a method for driving an interactive object. The driving method can be executed by an electronic device such as a terminal device or a server, and the terminal device can be a fixed terminal or a mobile terminal, such as a mobile phone, a tablet computer, a game computer, desktop computer, advertising machine, all-in-one computer, vehicle-mounted terminal, etc., the server includes a local server or a cloud server, etc., the method can also call the computer-readable instructions stored in the memory through the processor. to fulfill.

In the embodiment of the present disclosure, the interactive object may be any virtual image capable of interacting with the target object. In one embodiment, the interactive object may be a virtual character, and may also be a virtual animal, a virtual item, a cartoon image, or other virtual images capable of realizing interactive functions. The presentation form of the interactive object may be either a 2D form or a 3D form, which is not limited in the present disclosure. The target object may be a user, a robot, or other smart devices. The interaction mode between the interactive object and the target object may be an active interaction mode or a passive interaction mode. In one example, the target object can issue a demand by making gestures or body movements, and trigger the interactive object to interact with it through active interaction. In another example, the interactive object may actively say hello, prompt the target object to make an action, etc., so that the target object interacts with the interactive object in a passive manner.

The interactive object can be displayed by a terminal device, and the terminal device can be a TV, an all-in-one machine with a display function, a projector, a virtual reality (Virtual Reality, VR) device, an augmented reality (Augmented Reality, AR) equipment, etc., the present disclosure does not limit the specific form of the terminal equipment.

FIG. 1 illustrates a display device proposed by at least one embodiment of the present disclosure. As shown in FIG. 1 , the display device has a transparent display screen, and a stereoscopic image can be displayed on the transparent display screen, so as to present a virtual scene with a stereoscopic effect and interactive objects. For example, the interactive objects displayed on the transparent display screen in FIG. 1 include virtual cartoon characters. In some embodiments, the terminal device described in the present disclosure may also be the above-mentioned display device with a transparent display screen. The display device is configured with a memory and a processor, and the memory is used to store a computer that can run on the processor. The processor is configured to implement the driving method of the interactive object provided by the present disclosure when executing the computer instruction, so as to drive the interactive object displayed on the transparent display screen to communicate or respond to the target object.

In some embodiments, in response to the sound driving data for driving the interactive object to output the speech, the interactive object may emit the specified speech to the target object. The terminal device can generate voice-driven data according to the actions, expressions, identities, preferences, etc. of the target objects around the terminal device, so as to drive the interactive objects to communicate or respond by issuing specified voices, thereby providing anthropomorphic services for the target objects. It should be noted that the sound-driven data can also be generated in other ways, for example, generated by a server and sent to a terminal device.

During the interaction between the interactive object and the target object, when the interactive object is driven to emit a specified voice according to the sound driving data, the interactive object may not be driven to make facial movements synchronized with the specified voice, so that the interactive object is rigid when it emits a voice. , Unnatural, affecting the interactive experience between the target object and the interactive object. Based on this, at least one embodiment of the present disclosure provides a method for driving an interactive object, so as to improve the experience of interacting between a target object and an interactive object.

FIG. 2 shows a flowchart of a method for driving an interactive object according to at least one embodiment of the present disclosure. As shown in FIG. 2 , the method includes steps 201 to 203 .

Step 201: Acquire a phoneme sequence corresponding to the sound driving data of the interactive object.

The sound-driven data may include audio data (voice data), text, and the like. In response to the sound-driven data being audio data, the audio data can be used to directly drive the interactive object to output voice, that is, the terminal device directly outputs the voice through the audio data; The corresponding phoneme is generated, and the generated phoneme is used to drive the interactive object to output the speech. The sound driving data may also be other forms of driving data, which are not limited in the present disclosure.

In the embodiment of the present disclosure, the sound driving data may be the driving data generated by the server or the terminal device according to the action, expression, identity, preference, etc. of the target object interacting with the interactive object, or it may be the driving data generated by the terminal device from the internal Sound-driven data for memory recall. The present disclosure does not limit the acquisition manner of the sound driving data.

In response to the sound driving data being audio data, the audio data can be split into a plurality of audio frames, and the audio frames can be combined according to the state of the audio frames to form phonemes; each phoneme formed according to the audio data forms a phoneme sequence. Among them, the phoneme is the smallest phonetic unit divided according to the natural attributes of the voice, and one pronunciation action of a real person can form a phoneme.

In response to the sound driving data being text, the phoneme corresponding to the morpheme may be obtained according to the morpheme contained in the text, thereby obtaining a corresponding phoneme sequence.

Those skilled in the art should understand that the phoneme sequence corresponding to the sound driving data may also be obtained in other manners, which is not limited in the present disclosure.

Step 202: Obtain the gesture parameter value of the interactive object matching the phoneme sequence.

In the embodiment of the present disclosure, the attitude parameter value of the interactive object matching the phoneme sequence can be obtained according to the acoustic features of the phoneme sequence; the feature encoding can also be performed on the phoneme sequence to determine the corresponding value of the feature code. attitude parameter value, so as to determine the attitude parameter value corresponding to the phoneme sequence.

The attitude parameter is used to control the attitude of the interactive object, and the interactive object can be driven to make a corresponding attitude by using different attitude parameter values. The gesture parameters include facial gesture parameters, and in some embodiments, the gesture parameters may also include limb gesture parameters. Among them, the facial posture parameter is used to control the facial posture of the interactive object, including expressions, mouth shapes, facial features and head posture, etc.; the limb posture parameter is used to control the limb posture of the interactive object, that is, used to drive all The interactive object makes a physical action. In the embodiment of the present disclosure, a correspondence relationship between a certain feature of the phoneme sequence and the gesture parameter value of the interactive object can be established in advance, so that the corresponding gesture parameter value can be obtained through the phoneme sequence. The specific method for acquiring the gesture parameter value of the interactive object matching the phoneme sequence will be described in detail later. The specific form of the pose parameter can be determined according to the type of the interactive object model.

Step 203: Control the posture of the interactive object displayed by the display device according to the posture parameter value.

Wherein, the gesture parameter value matches the phoneme sequence corresponding to the sound driving data of the interactive object, and the gesture of the interactive object is controlled according to the gesture parameter value, so that the gesture of the interactive object and the interactive object can be aligned with the target. The communication or response made by the object matches. For example, when the interactive object is communicating or responding to the target object with speech, the gestures made are synchronized with the output speech, thereby giving the target object a feeling that the interactive object is talking.

In the embodiment of the present disclosure, by acquiring the phoneme sequence corresponding to the sound driving data of the interactive object displayed by the display device, the gesture parameter value of the interactive object matching the phoneme sequence is obtained, and according to the phoneme sequence matching the phoneme sequence The gesture parameter value of the interactive object controls the gesture of the interactive object displayed by the display device, so that the interactive object makes a matching gesture for communicating with the target object or responding to the target object, Thereby, the target object feels that it is communicating with the interactive object, and the interactive experience of the target object is improved.

In some embodiments, the method is applied to a server, including a local server or a cloud server, etc., the server processes the sound-driven data of the interactive object, generates the gesture parameter value of the interactive object, and generates a gesture parameter value of the interactive object according to The attitude parameter value is rendered by a three-dimensional rendering engine to obtain the animation of the interactive object. The server can send the animation to the terminal for display to communicate or respond to the target object, and can also send the animation to the cloud, so that the terminal can obtain the animation from the cloud to communicate or communicate with the target object. respond. After the server generates the attitude parameter value of the interactive object, the attitude parameter value may also be sent to the terminal, so that the terminal can complete the process of rendering, generating animation and displaying.

In some embodiments, the method is applied to a terminal, and the terminal processes sound-driven data of an interactive object, generates an attitude parameter value of the interactive object, and uses a three-dimensional rendering engine to perform rendering according to the attitude parameter value, The animation of the interactive object is obtained, and the terminal can display the animation to communicate or respond to the target object.

In some embodiments, the display device may be controlled to output speech and/or display text according to the phoneme sequence. And while controlling the display device to output speech and/or display text according to the phoneme sequence, the gesture of the interactive object displayed by the display device can be controlled according to the gesture parameter value.

In the embodiment of the present disclosure, since the gesture parameter value matches the phoneme sequence, the output speech and/or displayed text according to the phoneme sequence is different from the gesture of controlling the interactive object according to the gesture parameter value. In the case of synchronization, the gesture made by the interactive object is synchronized with the output speech and/or the displayed text, giving the target object the feeling that the interactive object is speaking.

Since the output of sound needs to maintain continuity, in one embodiment, the time window is moved on the phoneme sequence, and the phonemes in the time window during each movement are output, wherein the set duration is used as the time of each movement The step size of the window. For example, the length of the time window can be set to 1 second, and the set duration can be set to 0.1 seconds. While outputting the phonemes in the time window, obtain the gesture parameter value corresponding to the phoneme or the feature information of the phoneme at the set position of the time window, and use the gesture parameter value to control the gesture of the interactive object; the set position is the distance time The starting position of the window is the position where the duration is set. For example, when the length of the time window is set to 1s, the distance between the set position and the starting position of the time window can be 0.5s. With each movement of the time window, while outputting the phonemes in the time window, the posture of the interactive object is controlled by the corresponding attitude parameter value at the set position of the time window, so that the posture of the interactive object is synchronized with the output voice, giving the target The object feels like the interactive object is talking.

By changing the set duration, you can change the time interval (frequency) for obtaining the attitude parameter value, thereby changing the frequency of the interactive object making the gesture. The set duration can be set according to the actual interactive scene, so as to make the posture change of the interactive object more natural.

In some embodiments, feature encoding can be performed on the phoneme sequence to obtain feature information of the phoneme sequence; and the gesture parameter value of the interactive object is determined according to the feature information.

The embodiment of the present disclosure performs feature encoding on the phoneme sequence corresponding to the sound driving data of the interactive object, and obtains the corresponding attitude parameter value according to the obtained feature information, so as to output the sound according to the phoneme sequence, according to the feature information. The corresponding posture parameter value controls the posture of the interactive object, especially driving the interactive object to make facial movements according to the facial posture parameter value corresponding to the feature information, so that the expression of the interactive object and the sound emitted are synchronized , which makes the target object feel like the interactive object is talking, which improves the interactive experience of the target object.

In some embodiments, the feature encoding of the phoneme sequence may be performed in the following manner to obtain feature information of the phoneme sequence.

First, for the multiple phonemes included in the phoneme sequence, a coding sequence corresponding to the multiple phonemes is generated.

In an example, it is detected whether there is a first phoneme corresponding to each time point, and the first phoneme is any one of the plurality of phonemes; the encoding value at the time point where the first phoneme exists is set as the first phoneme A numerical value, the coding value at the time point without the first phoneme is set as the second numerical value, and the coding sequence corresponding to the first phoneme can be obtained after assigning the coding value at each time point. For example, the code value at the time point with the first phoneme may be set to 1, and the code value at the time point without the first phoneme may be set to 0. That is, for each phoneme in the plurality of phonemes included in the phoneme sequence, it is detected whether the phoneme corresponds to the phoneme at each time point; The encoding value at the time point of the phoneme is set as the second value, and the encoding sequence corresponding to the phoneme can be obtained after assigning the encoding value at each time point. Those skilled in the art should understand that the above setting of the encoding value is only an example, and the encoding value may also be set to other values, which is not limited in the present disclosure.

Then, according to the coding value of the coding sequence corresponding to each phoneme and the duration of each phoneme in the phoneme sequence, the feature information of the coding sequence corresponding to each phoneme is obtained.

In an example, for the coding sequence corresponding to the first phoneme, a Gaussian convolution operation is performed on continuous values of the first phoneme in time using a Gaussian filter to obtain feature information of the coding sequence corresponding to the first phoneme; The first phoneme is any one of the plurality of phonemes.

Finally, the feature information of the phoneme sequence is obtained according to the set of feature information of each coding sequence.

FIG. 3 shows a schematic diagram of a process of feature encoding for a phoneme sequence. As shown in FIG. 3 , the phoneme sequence 310 includes phonemes j, i1, j, and ie4 (for the sake of brevity, only some phonemes are shown), and for each phoneme j, i1, and ie4, codes corresponding to the above phonemes are obtained respectively. Sequence 321, 322, 323. In each encoding sequence, the corresponding encoding value at the time point with the phoneme is set to the first numerical value (for example, 1), and the corresponding encoding value at the time point without the phoneme is set to the second numerical value (for example, is 0). Taking the coding sequence 321 as an example, the value of the coding sequence 321 is the first numerical value at the time point when the phoneme sequence 310 has phoneme j, and the value of the coding sequence 321 is the second numerical value at the time point when there is no phoneme j in the phoneme sequence 310 . All coding sequences 321 , 322 , 323 constitute the total coding sequence 320 .

According to the coding values of the coding sequences 321, 322 and 323 corresponding to the phonemes j, i1 and ie4 respectively, and the durations of the corresponding phonemes in the three coding sequences, that is, the duration of j in the coding sequence 321, the duration of the coding sequence in the coding sequence The duration of i1 in 322 and the duration of ie4 in the coding sequence 323 can obtain the characteristic information of the coding sequences 321 , 322 and 323 .

For example, Gaussian convolution operation can be performed on successive temporal values of phonemes j, i1, ie4 in the encoded sequences 321, 322, 323 using a Gaussian filter to obtain the feature information of the encoded sequences. That is, the Gaussian convolution operation is performed on the continuous values of the phoneme in time through the Gaussian filter, so that the change stage of the coding value in each coding sequence from the second value to the first value or from the first value to the second value becomes smooth. Gaussian convolution operation is performed on each coding sequence 321, 322, 323, respectively, so as to obtain the feature value of each coding sequence, wherein the feature value is a parameter constituting feature information, and the phoneme sequence is obtained according to the set of feature information of each coding sequence. Feature information 330 corresponding to 310 . Those skilled in the art should understand that other operations may also be performed on each coding sequence to obtain characteristic information of the coding sequence, which is not limited in the present disclosure.

In the embodiment of the present disclosure, the characteristic information of the coding sequence is obtained according to the duration of each phoneme in the phoneme sequence, so that the change phase of the coding sequence is smooth. For example, the values of the coding sequence show an intermediate state except for 0 and 1. value, such as 0.2, 0.3, etc., and the attitude parameter values obtained according to the values of these intermediate states make the changes of the interactive characters' poses more gentle and natural, especially the changes of the interactive characters' expressions are more gentle and natural, and improve the The interactive experience of the target object.

In some embodiments, the facial pose parameters may include facial muscle control coefficients.

The movement of the human face, from an anatomical point of view, is the result of cooperative deformation of the muscles of various parts of the face. Therefore, the facial muscle model is obtained by dividing the facial muscles of the interactive object, and the movement of each muscle (region) obtained by the division is controlled by the corresponding facial muscle control coefficient, that is, the contraction/expansion control is performed on it, then It can make the faces of interactive characters make various expressions. For each muscle of the facial muscle model, motion states corresponding to different muscle control coefficients can be set according to the facial position where the muscle is located and the motion characteristics of the muscle itself. For example, for the upper lip muscle, the value range of its control coefficient is 0~1. Different values in this range correspond to different contraction/expansion states of the upper lip muscle. By changing the value, the longitudinal opening and closing of the mouth can be realized; For the muscle of the left corner of the mouth, the value of the control coefficient ranges from 0 to 1. Different values in this range correspond to the contraction/expansion state of the muscle at the left corner of the mouth. By changing the value, the lateral change of the mouth can be achieved.

While outputting the sound according to the phoneme sequence, the interactive object is driven to make a facial expression according to the facial muscle control coefficient value corresponding to the phoneme sequence, so that when the display device outputs the sound, the interactive object can synchronously make the sound. The expression of the voice, so that the target object feels that the interactive object is talking, and the interactive experience of the target object is improved.

In some embodiments, the facial action of the interactive object may be associated with a body posture, that is, the facial posture parameter value corresponding to the facial action may be associated with the body posture, and the body posture may include body movements, Gestures, walking gestures, etc.

During the driving process of the interactive object, obtain the driving data of the body posture associated with the facial posture parameter value; while outputting the sound according to the phoneme sequence, according to the driving data of the body posture associated with the facial posture parameter value , and drive the interactive object to make body movements. That is, while driving the interactive object to make a facial action according to the sound driving data of the interactive object, the driving data of the associated body posture is also obtained according to the facial posture parameter value corresponding to the facial action, so as to output the sound. When the interactive object is activated, it can drive the interactive object to make corresponding facial and body movements synchronously, so that the speaking state of the interactive object is more vivid and natural, and the interactive experience of the target object is improved.

In some embodiments, the gesture parameter value of the interactive object corresponding to the feature information of the phoneme sequence can be obtained by the following method.

First, the feature information of the phoneme sequence is sampled at set time intervals to obtain sampled feature information corresponding to each first sampling time. For example, if the set time interval is 0.1s, each first sampling time may be 0.1s, 0.2s, 0.3s, and so on.

Referring to FIG. 3 , the feature information 330 is time-based information. Therefore, when the feature information is sampled at a set time interval, the sampled feature information corresponding to each first sampling time can be obtained.

Next, the sampling feature information corresponding to the first sampling time is input into the pre-trained neural network, and then the attitude parameter value of the interactive object corresponding to the sampling feature information can be obtained. Based on the sampling feature information corresponding to each first sampling time, the attitude parameter value of the interactive object corresponding to each first sampling time can be obtained.

As mentioned above, in the case of outputting phonemes by moving the time window on the phoneme sequence, the feature information at the set position of the time window is obtained, that is, the feature information at the first sampling time corresponding to the set position of the time window is obtained, By obtaining the gesture parameter value corresponding to the feature information to control the gesture of the interactive object, the interactive object can be made to make a gesture suitable for the voice issued, so that the process of the interactive object issuing voice is more vivid and natural.

In some embodiments, the neural network includes a Long Short-Term Memory (LSTM) network and a fully connected network. Among them, the long short-term memory network is a time recurrent neural network, which can learn the historical information of the input sampling feature information; and the long short-term memory network and the fully connected network are jointly trained.

In the case where the neural network includes a long-term and short-term memory network and a fully connected network, first input the sampling feature information corresponding to the first sampling time into the long-term and short-term memory network, and the long-term and short-term memory network The sampling feature information before the first sampling time is described, and the associated feature information is output. That is, the information output by the long short-term memory network includes the influence of the historical feature information on the current feature information. Next, input the associated feature information into the fully connected network, and determine a posture parameter value corresponding to the associated feature information according to the classification result of the fully connected network; wherein each classification corresponds to a The group pose parameter value, that is, the distribution state corresponding to a facial muscle control coefficient.

In the embodiment of the present disclosure, the posture parameter value corresponding to the sampled feature information of the phoneme sequence can be predicted through the long short-term memory network and the fully connected network, and the related historical feature information and the current feature information can be fused, thereby The historical attitude parameter value has an influence on the change of the current attitude parameter value, so that the change of the attitude parameter value of the interactive character is more gentle and natural.

In some embodiments, the neural network can be trained in the following manner.

First, a phoneme sequence sample is obtained, and the phoneme sequence sample includes the gesture parameter value of the interactive object marked at the second sampling time of the set time interval. As shown in the phoneme sequence sample shown in FIG. 4 , the dotted line represents the second sampling time, and the gesture parameter values of the interactive objects are marked at each second sampling time.

Next, feature encoding is performed on the phoneme sequence samples to obtain feature information corresponding to each second sampling time, and corresponding attitude parameter values are marked for the feature information to obtain feature information samples. That is, the feature information sample includes the gesture parameter value of the interactive object marked at the second sampling time.

After the characteristic information sample is obtained, the neural network can be trained according to the characteristic information sample, and the training is completed when the network loss is less than the set loss value, wherein the network loss includes the prediction obtained by the neural network. The difference between the pose parameter value of and the annotated pose parameter value.

（1）In one example, the network loss function is expressed as Equation (1):

(1)

是神經網路預測得到的第

個姿態參數值；

是所標註的第

個姿態參數值，也即真實值；

表示向量的二範數。in,

is predicted by the neural network

an attitude parameter value;

is the marked

A pose parameter value, that is, the true value;

Represents the two-norm of a vector.

By adjusting the network parameter values of the neural network to minimize the network loss function, when the change of the network loss satisfies the convergence condition, for example, when the change of the network loss is less than the set threshold, or the number of iterations reaches the set number When the training is completed, the trained neural network is obtained.

（2）In another example, the network loss function is expressed as Equation (2):

(2)

是神經網路預測得到的第

個姿態參數值；

是所標註的第

個姿態參數值，也即真實值；

表示向量的二範數；

表示向量的一範數。in,

is predicted by the neural network

an attitude parameter value;

is the marked

A pose parameter value, that is, the true value;

represents the two-norm of the vector;

Represents the one-norm of a vector.

By adding a norm of the predicted pose parameter value to the network loss function, the constraint on the sparsity of facial parameters is increased.

In some embodiments, phoneme sequence samples may be obtained by the following methods.

First, a video clip of a character uttering a voice is obtained. For example, a video clip of a real person speaking may be obtained.

For the video segment, a plurality of first image frames including the character, and a plurality of audio frames corresponding to the first image frames are acquired. That is, the video segment is divided into image frames and audio frames, and each image frame corresponds to each audio frame, that is, for one image frame, it can be determined that the character is making The audio frame corresponding to the sound produced when the image frame is expressed.

Next, convert the first image frame, that is, the image frame containing the character, into a second image frame containing the interactive object, and obtain the pose parameter value corresponding to the second image frame . Taking the first image frame as an image frame containing a real person as an example, the image frame of the real person can be converted into a second image frame containing an image represented by an interactive object, and the real person's image frame can be converted into a second image frame containing an image represented by an interactive object. The attitude parameter value corresponds to the attitude parameter value of the interactive object, so that the attitude parameter value of the interactive object in each second image frame can be acquired.

Then, according to the attitude parameter value corresponding to the second image frame, the audio frame corresponding to the first image frame is marked, and phoneme sequence samples are obtained according to the audio frame marked with the attitude parameter value.

In an embodiment of the present disclosure, a video segment of a character is split into corresponding image frames and audio frames, and a first image frame containing a real person is converted into a second image frame containing interactive objects To obtain the attitude parameter value corresponding to the phoneme sequence, the correspondence between the phoneme and the attitude parameter value is better, and a higher quality phoneme sequence sample can be obtained.

FIG. 5 shows a schematic structural diagram of an apparatus for driving an interactive object according to at least one embodiment of the present disclosure. The interactive object is displayed on a display device. As shown in FIG. 5 , the apparatus may include: a phoneme sequence acquisition unit 501 for Acquire the phoneme sequence corresponding to the sound driving data of the interactive object; the parameter acquisition unit 502 is used for acquiring the attitude parameter value of the interactive object matching the phoneme sequence; the driving unit 503 is used for according to the attitude parameter value Control the gesture of the interactive object displayed by the display device.

In some embodiments, the apparatus further includes an output unit for controlling the display device to output speech and/or display text according to the phoneme sequence.

In some embodiments, the parameter obtaining unit is specifically configured to: perform feature encoding on the phoneme sequence to obtain feature information of the phoneme sequence; obtain gesture parameters of the interactive object corresponding to the feature information of the phoneme sequence value.

In some embodiments, when the feature encoding is performed on the phoneme sequence to obtain feature information of the phoneme sequence, the parameter obtaining unit is specifically configured to: for each phoneme in the plurality of phonemes included in the phoneme sequence, generating coding sequences corresponding to multiple phonemes respectively; according to the coding values of the coding sequences corresponding to the multiple phonemes and the duration corresponding to the multiple phonemes respectively in the phoneme sequence, obtain the feature information of the coding sequences corresponding to the multiple phonemes respectively; The feature information of the phoneme sequence is obtained according to the feature information of the coding sequences corresponding to the plurality of phonemes respectively.

In some embodiments, when generating encoding sequences corresponding to multiple phonemes respectively for multiple phonemes included in the phoneme sequence, the parameter obtaining unit is specifically configured to: detect whether each time point corresponds to a first phoneme, so The first phoneme is any one of the plurality of phonemes; by setting the encoding value at the time point with the first phoneme as the first value, the encoding value at the time point without the first phoneme is set Set as the second value to obtain the coding sequence corresponding to the first phoneme.

In some embodiments, when the feature information of the coding sequences corresponding to the various phonemes is obtained according to the coding values of the coding sequences corresponding to the various phonemes and the durations corresponding to the various phonemes in the phoneme sequence, the The parameter obtaining unit is specifically used to: for the coding sequence corresponding to the first phoneme, use a Gaussian filter to perform a Gaussian convolution operation on the continuous values of the first phoneme in time, and obtain the coding sequence corresponding to the first phoneme. feature information; the first phoneme is any one of the plurality of phonemes.

In some embodiments, the posture parameters include facial posture parameters, and the facial posture parameters include facial muscle control coefficients, which are used to control the motion state of at least one facial muscle; the driving unit is specifically configured to: according to matching with the phoneme sequence The facial muscle control coefficient of , drives the interactive object to make facial actions matching each phoneme in the phoneme sequence.

In some embodiments, the device further includes an action driving unit for acquiring driving data of the body posture associated with the facial posture parameter; The interactive object makes a physical action.

In some embodiments, when acquiring the gesture parameter value of the interactive object corresponding to the feature information of the phoneme sequence, the parameter acquisition unit is specifically configured to: sample the feature information of the phoneme sequence at a set time interval , obtain the sampling feature information corresponding to the first sampling time; input the sampling feature information corresponding to the first sampling time into the pre-trained neural network, and obtain the attitude parameter value of the interactive object corresponding to the sampling feature information.

In some embodiments, the neural network includes a long short-term memory network and a fully connected network; after inputting the sampling feature information corresponding to the first sampling time into a pre-trained neural network, obtain a When the feature information corresponds to the attitude parameter value of the interactive object, the parameter acquisition unit is specifically configured to: input the sampling feature information corresponding to the first sampling time into the long short-term memory network, and according to the first sampling time Sampling feature information before time, and output associated feature information; input the associated feature information into the fully connected network, and determine the attitude parameter value corresponding to the associated feature information according to the classification result of the fully connected network ; wherein, each category in the classification result corresponds to a set of attitude parameter values.

In some embodiments, the neural network is trained on phoneme sequence samples. The device further includes a sample acquisition unit for: acquiring a video segment of a character uttering a voice; acquiring a plurality of first image frames including the character and a plurality of first image frames corresponding to the first image frames according to the video segment. a plurality of audio frames; converting the first image frame into a second image frame containing the interactive object, and obtaining the posture parameter value corresponding to the second image frame; The audio frame corresponding to the first image frame is marked; according to the audio frame marked with the attitude parameter value, the phoneme sequence sample is obtained.

At least one embodiment of the present specification also provides an electronic device, as shown in FIG. 6 , the device includes a memory and a processor, where the memory is used to store computer instructions that can be executed on the processor, and the processor is used to execute all computer instructions. The method for driving an interactive object described in any embodiment of the present disclosure is implemented when the computer instruction is used.

At least one embodiment of the present specification further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the method for driving an interactive object described in any embodiment of the present disclosure.

As will be appreciated by one skilled in the art, one or more embodiments of this specification may be provided as a method, system or computer program product. Accordingly, one or more embodiments of this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may be implemented on one or more computer-usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) having computer-usable program code embodied therein in the form of a computer program product.

Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to the partial description of the method embodiment.

The foregoing describes specific embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the acts or steps recited in the claims may be performed in an order different from that of the embodiments and still achieve desirable results. Additionally, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuits, in tangible embodiment of computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or A combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, ie, one or more of computer program instructions encoded on a tangible, non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus. Multiple mods. Alternatively or additionally, program instructions may be encoded on artificially generated propagating signals, such as machine-generated electrical, optical or electromagnetic signals, which are generated to encode and transmit information to suitable receiver devices for data retrieval. The processing device executes. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or sequential access memory device, or a combination of one or more of these.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, eg, an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

Computers suitable for the execution of a computer program include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from read-only memory and/or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operably coupled to, such mass storage devices to receive data therefrom, one or more mass storage devices, such as magnetic disks, optical disks, magneto-optical disks, or optical disks, etc., for storing data Or send data to it, or both. However, the computer does not have to have such a device. Furthermore, the computer may be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, global positioning system (GPS) receiver, or a universal serial bus (USB ) flash drives for portable storage devices, to name a few.

Computer readable media suitable for storage of computer program instructions and data include all forms of non-volatile memory, media, and memory devices including, for example, semiconductor memory devices (eg, EPROM, EEPROM, and flash memory devices), magnetic disks (eg, internal hard disks or removable disks), compact disks, magneto-optical disks, and CD-ROMs and DVD-ROMs. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or what may be claimed, but rather are used primarily to describe features of specific embodiments of particular inventions. Certain features that are described in this specification in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function as described above in certain combinations and even be originally claimed as such, one or more features from a claimed combination may in some cases be removed from the combination and the claimed A protected combination may point to a subcombination or a variation of a subcombination.

Similarly, although operations in the figures are depicted in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown or sequentially, or that all illustrated operations be performed, to achieve the desired results . In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated together in a single software product , or packaged into multiple software products.

Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claim may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above descriptions are only preferred embodiments of one or more embodiments of this specification, and are not intended to limit one or more embodiments of this specification. All within the spirit and principles of one or more embodiments of this specification, Any modifications, equivalent replacements, improvements, etc. made should be included within the protection scope of one or more embodiments of this specification.

201: the step of acquiring the phoneme sequence corresponding to the sound-driven data of the interactive object 202: the step of acquiring the gesture parameter value of the interactive object matched with the phoneme sequence 203: the step of controlling the posture of the interactive object displayed by the display device according to the posture parameter value 501: Phoneme sequence acquisition unit 502: Parameter acquisition unit 503: Drive unit

FIG. 1 is a schematic diagram of a display device in a method for driving an interactive object provided by at least one embodiment of the present disclosure. FIG. 2 is a flowchart of a method for driving an interactive object provided by at least one embodiment of the present disclosure. FIG. 3 is a schematic diagram of a process of feature encoding for a phoneme sequence proposed by at least one embodiment of the present disclosure. FIG. 4 is a schematic diagram of a phoneme sequence sample proposed by at least one embodiment of the present disclosure. FIG. 5 is a schematic structural diagram of a driving device for an interactive object according to at least one embodiment of the present disclosure. FIG. 6 is a schematic structural diagram of an electronic device provided by at least one embodiment of the present disclosure.

201: the step of acquiring the phoneme sequence corresponding to the sound-driven data of the interactive object

202: the step of acquiring the gesture parameter value of the interactive object matched with the phoneme sequence

203: the step of controlling the posture of the interactive object displayed by the display device according to the posture parameter value

Claims (14)

A method for driving an interactive object, wherein the interactive object is displayed on a display device, the method includes: acquiring a phoneme sequence corresponding to sound driving data of the interactive object; acquiring a gesture of the interactive object matching the phoneme sequence parameter values, the posture parameter values include facial posture parameters and limb posture parameters, the facial posture parameters include facial muscle control coefficients, and the facial muscle control coefficients are used to control the motion state of at least one facial muscle; according to the posture parameters The value controls the posture of the interactive object displayed by the display device, at least including, according to the facial muscle control coefficient value matched with the phoneme sequence, driving the interactive object to make a match with each phoneme in the phoneme sequence. facial movements. The driving method according to claim 1, further comprising: controlling the display device to output speech and/or display text according to the phoneme sequence. The driving method according to claim 1 or 2, wherein obtaining the gesture parameter value of the interactive object matching the phoneme sequence includes: performing feature encoding on the phoneme sequence to obtain feature information of the phoneme sequence ; Obtain the gesture parameter value of the interactive object corresponding to the feature information of the phoneme sequence. The driving method according to claim 3, wherein the feature encoding is performed on the phoneme sequence to obtain feature information of the phoneme sequence, comprising: For each phoneme in the plurality of phonemes included in the phoneme sequence, a coding sequence corresponding to the phoneme is generated; according to the coding value of the coding sequence corresponding to the phoneme and the duration corresponding to the phoneme, obtain the corresponding phoneme. The feature information of the coding sequence; the feature information of the phoneme sequence is obtained according to the feature information of the coding sequence corresponding to the plurality of phonemes respectively. The driving method according to claim 4, wherein, for each phoneme in the plurality of phonemes included in the phoneme sequence, generating a coding sequence corresponding to the phoneme includes: detecting whether each time point corresponds to the phoneme; The encoding sequence corresponding to the phoneme is obtained by setting the encoding value at the time point with the phoneme as the first value, and setting the encoding value at the time point without the phoneme as the second value. The driving method according to claim 4, wherein the feature information of the coding sequences corresponding to the plurality of phonemes is obtained according to the coding values of the coding sequences corresponding to the plurality of phonemes and the durations corresponding to the plurality of phonemes respectively , including: for each phoneme in the plurality of phonemes, for the coding sequence corresponding to the phoneme, using a Gaussian filter to perform a Gaussian convolution operation on the continuous values of the phoneme in time to obtain the code corresponding to the phoneme. Characteristic information for the sequence. The driving method according to claim 1, further comprising: Acquiring the driving data of the body posture associated with the facial posture parameter value; and driving the interactive object to perform body movements according to the driving data of the body posture associated with the facial posture parameter value. The driving method according to claim 3, wherein acquiring the attitude parameter value of the interactive object corresponding to the feature information of the phoneme sequence comprises: sampling the feature information of the phoneme sequence at a set time interval to obtain the first Sampling feature information corresponding to the sampling time; inputting the sampling feature information corresponding to the first sampling time into a pre-trained neural network to obtain the attitude parameter value of the interactive object corresponding to the sampling feature information. The driving method according to claim 8, wherein the pre-trained neural network includes a long short-term memory network and a fully connected network, and the sampling feature information corresponding to the first sampling time is input to the pre-trained neural network network, and obtaining the attitude parameter value of the interactive object corresponding to the sampling feature information includes: inputting the sampling feature information corresponding to the first sampling time into the long short-term memory network, according to Sampling feature information before the first sampling time, output associated feature information; input the associated feature information into the fully connected network, and determine the associated feature according to the classification result of the fully connected network The attitude parameter values corresponding to the information; wherein, in the classification result, each category corresponds to a group of the attitude parameter values. The driving method according to claim 8, wherein the neural network is obtained by training phoneme sequence samples; the method further comprises: acquiring a video segment of a character uttering speech; acquiring a plurality of segments including the a first image frame of the character, and multiple audio frames corresponding to the multiple first image frames; converting the first image frame into a second image frame including the interactive object, and obtaining The attitude parameter value corresponding to the second image frame; according to the attitude parameter value corresponding to the second image frame, the audio frame corresponding to the first image frame is marked; The audio frame of the gesture parameter value, the phoneme sequence sample is obtained. The driving method according to claim 10, further comprising: performing feature encoding on the phoneme sequence samples to obtain feature information corresponding to the second sampling time, and labeling the feature information with corresponding attitude parameter values to obtain feature information samples The initial neural network is trained according to the feature information samples, and the neural network is obtained by training after the change of the network loss meets the convergence condition, wherein the network loss includes the predicted value obtained by the initial neural network. The difference between the attitude parameter value and the marked attitude parameter value; wherein, the network loss includes the difference between the attitude parameter value predicted by the initial neural network and the marked attitude parameter value The second norm of ; The network loss further includes a norm of the attitude parameter value predicted by the initial neural network. A driving device for an interactive object, the interactive object is displayed on a display device, the device comprises: a phoneme sequence acquisition unit for acquiring a phoneme sequence corresponding to sound driving data of the interactive object; a parameter acquisition unit for acquiring The posture parameter value of the interactive object matched with the phoneme sequence, the posture parameter value includes a facial posture parameter and a limb posture parameter, the facial posture parameter includes a facial muscle control coefficient, and the facial muscle control coefficient is used to control The motion state of at least one facial muscle; a driving unit, configured to control the gesture of the interactive object displayed by the display device according to the gesture parameter value; wherein, the driving unit is specifically configured to The facial muscle controls the coefficient value to drive the interactive object to make facial actions matching each phoneme in the phoneme sequence. An electronic device, comprising a memory and a processor, the memory is used to store computer instructions that can be executed on the processor, and the processor is used to implement any one of claim items 1 to 11 when executing the computer instructions the described driving method. A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the driving method described in any one of claim 1 to 11.

Images