JP2022531057A - Interactive target drive methods, devices, devices, and recording media - Google Patents

Abstract

The present invention discloses a method, apparatus, device and recording medium for driving an interactive object, wherein the interactive object is displayed on a display device, and the method includes a phoneme sequence corresponding to voice driving data of the interactive object obtaining a pose parameter value of the interactive object that matches the phoneme sequence; and controlling a pose of the interactive object displayed on the display device based on the pose parameter value. ,including. [Selection drawing] Fig. 2

Description

<Mutual citation of related applications>
The present invention claims the priority of a Chinese patent application with an application number of 202010247569 and a filing date of March 31, 2020, and all the contents of the Chinese patent application are incorporated herein by reference.
The present invention relates to the field of computer technology, specifically, to driving methods, devices, devices, and recording media for interactive objects.

Human-computer interactions are primarily input by keystrokes, touch, and voice, and respond by displaying images, text, or virtual characters on the display screen. Currently, virtual characters are being improved primarily based on voice assistants.

The embodiments of the present invention provide a technical solution for driving an interactive object.

According to one aspect of the present invention, a method for driving an interactive object exhibited in a display device is provided, in which the method obtains a phoneme sequence corresponding to the voice-driven data of the interactive object, and the phoneme sequence is used. It includes acquiring the appearance parameter value of the interactive object to be matched, and controlling the appearance of the interactive object displayed on the display device based on the appearance parameter value. Combined with any embodiment provided by the present invention, the method further comprises controlling the output speech and / or text of the display device based on the phoneme sequence.

Coupling with any embodiment provided by the present invention to obtain the figure parameter value of the interactive object matching the phoneme sequence is to perform feature encoding on the phoneme sequence to obtain the phoneme sequence. It includes obtaining the feature information and acquiring the figure parameter value of the interactive object corresponding to the feature information of the phoneme sequence.

In combination with any embodiment provided by the present invention, performing feature encoding on the phoneme sequence to obtain feature information of the phoneme sequence is a plurality of phoneme types included in the phoneme sequence. For each phoneme in the phoneme, the chord sequence for each phoneme is generated, the chord value of the chord sequence corresponding to each phoneme, and the time length corresponding to a plurality of types of phonemes in the phoneme sequence. Based on the above, the feature information of the chord sequence of each of the phonemes is obtained, and the feature information of the phoneme sequence is obtained based on the feature information of the chord sequence corresponding to each of the plurality of types of phonemes. ..

In combination with any embodiment provided by the present invention, it is possible to generate a chord sequence for each phoneme for each phoneme among a plurality of types of phonemes included in the phoneme sequence. Detecting whether or not the phoneme is supported at the time point, setting the code value at the time when the phoneme is supported as the first numerical value, and setting the code value at the time when the phoneme is not supported as the second numerical value. By setting it as a numerical value, it includes obtaining the chord sequence corresponding to the phoneme.

Combined with any embodiment provided by the present invention, the plurality is based on the chord value of the chord sequence corresponding to each of the plurality of phonemes and the time length corresponding to each of the plurality of phonemes. To obtain the feature information of the chord sequence corresponding to each type of phoneme, the phoneme is used for the chord sequence corresponding to the phoneme for each phoneme in the plurality of types of phonemes by using a Gaussian filter. Includes performing a Gaussian convolution operation on continuous values in time to obtain feature information of the chord sequence corresponding to the phoneme.

Combined with any embodiment provided by the present invention, the facial features parameter comprises a facial figure parameter, the facial figure parameter comprises a facial muscle control coefficient, and the facial muscle control coefficient comprises at least one facial muscle. Controlling the appearance of the interactive object displayed on the display device based on the appearance parameter values used to control the motor state of the is based on facial muscle control factor values matching the phonetic sequence. It includes driving the interactive object so that the interactive object performs a facial action matching each phone element in the element sequence.

Combined with any embodiment provided by the present invention, the method obtains body shape drive data related to the facial shape parameter value and exhibits it on the display device based on the shape parameter value. Controlling the shape of the interactive object and driving the interactive target so that the interactive target performs limb movements based on the driving data of the body shape related to the facial shape parameter value. And further include.

In combination with any embodiment provided by the present invention, acquiring the figure parameter value of the interactive object corresponding to the characteristic information of the phonetic sequence samples the feature information of the phonetic sequence at predetermined time intervals. Then, the sampling feature information corresponding to the first sampling time is obtained, and the sampling feature information corresponding to the first sampling time is input to the pre-trained neural network to correspond to the sampling feature information. Includes obtaining the figure parameter value of an interactive object.

Combined with any embodiment provided by the present invention, the neural network includes a long-term storage network and a fully connected network, and is pre-trained with the sampling feature information corresponding to the first sampling time. To obtain the shape parameter value of the interactive object corresponding to the sampling feature information by inputting to the neural network, the sampling feature information corresponding to the first sampling time is input to the long-term storage network and the first sampling is performed. Output related feature information based on sampling feature information before time, input the related feature information to the fully connected network, and correspond to the related feature information based on the classification result of the fully connected network. Each type in the classification result corresponds to a set of appearance parameter values, including determining the appearance parameter value.

Combined with any embodiment provided by the present invention, the neural network was obtained by training using a phonetic sequence sample, the method of which is a video segment of the voice emitted by the character. Acquiring, acquiring a plurality of first image frames including the character based on the video segment, and acquiring a plurality of audio frames corresponding to the plurality of the first image frames, and the first image frame. Is converted into a second image frame including the interactive object, and the figure parameter value corresponding to the second image frame is acquired, and the figure parameter value corresponding to the second image frame is obtained. Further comprising labeling the audio frame corresponding to the first image frame and obtaining the phonetic sequence sample based on the audio frame to which the appearance parameter values are leveled.

Combined with any embodiment provided by the present invention, the method performs sample feature encoding on the phonetic sequence to obtain feature information corresponding to the second sampling time and the feature information neural network. It is possible to obtain a feature information sample for the shape parameter value corresponding to the ring, train the initial neural network based on the feature information sample, and train the neural network after the change in network loss satisfies the convergence condition. And, further included, where the network loss includes the difference between the shape parameter value predicted and obtained by the initial neural network and the leveled shape parameter value.

Combined with any embodiment provided by the present invention, the network loss is the second norm of the difference between the shape parameter values predicted and obtained by the initial neural network and the leveled shape parameter values. The network loss further includes the first norm of the appearance parameter values predicted and obtained by the initial neural network.

According to one aspect of the present invention, a drive device for an interactive object exhibited in a display device is provided, and the device is a phoneme sequence acquisition unit for acquiring a phoneme sequence corresponding to the voice drive data of the interactive object. , A parameter acquisition unit for acquiring the figure parameter value of the interactive object matching the phoneme sequence, and a drive for controlling the figure of the interactive object displayed on the display device based on the figure parameter value. Equipped with a unit.

According to one aspect of the invention, an electronic device is provided, the device comprising a memory and a processor, the memory storing computer instructions operable on the processor, the processor executing the computer instructions. When done, the interactive object driving method described in any of the embodiments provided by the present invention is realized.

According to one aspect of the invention, a computer-readable recording medium in which a computer program is stored is provided, and when the program is executed by a processor, the interactive object according to any embodiment provided by the present invention. The drive method is realized.

According to the driving method, device, device, and computer-readable recording medium of the interactive object of one or more embodiments of the present invention, the phonetic sequence corresponding to the voice-driven data of the interactive object displayed on the display device is acquired. , The figure parameter value of the interactive object matching the phonetic sequence is acquired, and the figure of the interactive object displayed on the display device is controlled based on the figure parameter value of the interactive object matching the phonetic sequence. By doing so, the target object feels as if it is interacting with the interactive object so that the interactive object interacts with the target object or has a matching appearance in response to the target object. Improves the interactive experience with the target interactive target.

Hereinafter, in order to more clearly explain the technical solution in one or more embodiments or prior art of the present specification, the drawings that need to be used in the description of the embodiment or prior art are briefly introduced. .. Obviously, the drawings described below are only a few examples described in one or more embodiments herein, and those skilled in the art will be able to base on these drawings without any creative work. Can be obtained.
It is a schematic diagram of the display device in the driving method of an interactive object provided by at least one embodiment of the present invention. It is a flowchart of the driving method of an interactive object provided by at least one Embodiment of this invention. It is a schematic diagram of the process of performing feature encoding on the phoneme sequence provided by at least one embodiment of the present invention. FIG. 3 is a schematic diagram of a phoneme sequence sample provided by at least one embodiment of the present invention. It is a schematic diagram of the configuration of the drive device of an interactive object provided by at least one embodiment of the present invention. It is a schematic diagram of the configuration of the electronic device provided by at least one embodiment of the present invention.

Hereinafter, exemplary embodiments will be described in detail and examples will be shown in the drawings. Where the following description refers to drawings, the same numbers in different drawings indicate the same or similar elements, unless otherwise stated. The embodiments described in the following exemplary examples do not represent all embodiments consistent with the present invention. Conversely, they are merely examples of devices and methods consistent with some aspects of the invention described in the appended claims.

The term "and / or" as used herein merely describes the relationship of a related object and indicates that three relationships can exist, for example, A and / or B, where A is present alone. There are three relationships, such as the existence of A and B at the same time, and the existence of B alone. Further, the term "at least one kind" in the present specification refers to any one kind in a plurality of kinds or any combination of at least two kinds in a plurality of kinds, for example, in A, B, C. Inclusion of at least one indicates that it comprises any one or more elements selected from a set composed of A, B, and C.

At least one embodiment of the present invention provides a driving method for an interactive object, which driving method can be performed by an electronic device such as a terminal device or a server. The terminal device may be a fixed terminal or a mobile terminal such as a mobile phone, a tablet personal computer, a game machine, a desktop personal computer, an advertising machine, an all-in-one machine, and an in-vehicle terminal. The server includes a local server, a cloud server, and the like. The method can be realized by a method of calling a computer-readable instruction stored in a memory by a processor.

In an embodiment of the invention, the interactive object can be any virtual image capable of performing interactive with the target object. In one embodiment, the interactive object may be a virtual character, and may be another virtual image capable of realizing an interactive function, such as a virtual animal, a virtual article, or a cartoon image. The display format of the interactive object is 2D or 3D, but the present invention is not limited thereto. The target can be a user, a robot, or other smart device. The interactive method of the interactive object with the target object may be an active interactive method or a passive interactive method. In one example, depending on the target object, a request can be made by performing a gesture or a limb movement, and the interactive object can be triggered and interactively performed by an active interactive method. In another example, the interactive object can be made to interact with the interactive object by a passive method by actively greeting and prompting the target object to perform an action or the like.

The interactive object can be exhibited using a terminal device, and the terminal device includes a television, an all-in-one device having a display function, a projector, a virtual reality (VR) device, and an augmented reality (AR). ) The present invention may be a device or the like, and the present invention is not limited to a specific form of the terminal device.

FIG. 1 shows a display device provided by at least one embodiment of the present invention. As shown in FIG. 1, the display device has a transparent display screen, and by displaying a stereoscopic image on the transparent display screen, a virtual scene having a stereoscopic effect and an interactive object can be represented. For example, the interactive object displayed on the transparent screen of FIG. 1 includes a virtual cartoon person. In some embodiments, the terminal device described in the present invention may be a display device having the transparent display screen described above, wherein the display device is arranged with a memory and a processor, and the memory is stored on the processor. The processor stores an operable computer instruction, and when the computer instruction is executed, the interactive object displayed on the transparent display screen is realized by realizing the driving method of the interactive object provided by the present invention. It can be driven to interact or respond to the target.

In some embodiments, the interactive object can emit a designated voice to the target object in response to voice-driven data for driving the interactive object to output voice. The terminal device generates voice-driven data based on the movement, facial expression, status, preference, etc. of the target object around the terminal device so that the interactive target emits a specified voice to interact or respond. By driving, it is possible to provide anthropomorphic services to the target target. It should be explained that the voice-driven data may be generated by other methods, for example, by a server and transmitted to a terminal device.

In the process of the interactive object interacting with the target object, when the interactive object is driven to emit the specified voice based on the voice-driven data, the interactive target has a face synchronized with the specified voice. It cannot be driven to perform the movement of the part, and it becomes dull and unnatural when the interactive object emits a voice, which may affect the interactive experience with the interactive object of the target object. In view of this, at least one embodiment of the present invention submits an interactive object driving method to enhance the interactive experience with the interactive object of the target object.

FIG. 2 is a flowchart showing a driving method of an interactive object according to at least one embodiment of the present invention, and as shown in FIG. 2, the method includes steps 201 to 203.

In step 201, the phoneme sequence corresponding to the voice-driven data of the interactive target is acquired.

The voice-driven data may include audio data (voice data), text, and the like. In response to the audio-driven data being audio data, the audio data can be used directly to drive the interactive object to output audio, that is, the terminal device is directly based on the audio data. Audio can be output. In response to the voice-driven data being text, it is necessary to generate the corresponding phoneme based on the morpheme contained in the text and drive the interactive object to output voice based on the generated phoneme. be. The voice-driven data may be other forms of drive data, and the present invention is not limited thereto.

In the embodiment of the present invention, the voice-driven data may be drive data generated based on the movement, facial expression, status, preference, etc. of the target object that interacts with the interactive object by the server or the terminal device, or the terminal. It may be voice-driven data called from the internal memory by the device. The present invention is not limited to the method of acquiring the voice-driven data.

Formed by the audio data by dividing the audio data into a plurality of audio frames in response to the voice-driven data being audio data and combining the audio frames to form phonemes based on the state of the audio frames. A phoneme sequence can be formed based on each phoneme made. Here, a phoneme is the smallest phoneme unit divided based on the natural attributes of speech, and one pronunciation action of a real person can form one phoneme.

The corresponding phoneme sequence can be obtained by obtaining the phoneme corresponding to the morpheme based on the morpheme contained in the text in response to the voice-driven data being a text.

Those skilled in the art should further understand that the phoneme sequence corresponding to the voice driven data can be obtained by other methods, and the present invention is not limited thereto.

In step 202, the appearance parameter value of the interactive object that matches the phoneme sequence is acquired.

In the embodiment of the present invention, it is also possible to obtain the figure parameter value of the interactive object matching the phoneme sequence based on the acoustic feature of the phoneme sequence, and the feature encoding is executed on the phoneme sequence to perform the feature code. It is also possible to determine the appearance parameter value corresponding to the phoneme sequence by determining the appearance parameter value corresponding to.

The form parameters are used to control the form of the interactive object, and different form parameter values can be used to drive the interactive object to perform the appropriate form. The figure parameter may include a face figure parameter, and in some embodiments, the figure parameter may further include a limb figure parameter. Here, the facial appearance parameter is used to control the facial appearance of the interactive object, including facial expressions, mouth shapes, movements of the five officials, head appearance, etc., and the limb appearance parameter is the limb of the interactive object. It is used to control facial expressions, that is, to drive the interactive object to perform limb movements. In the embodiment of the present invention, by preliminarily constructing a correspondence relationship between a specific feature of the phoneme sequence and the appearance parameter value of the interactive object, the corresponding appearance parameter value can be obtained based on the phoneme sequence. .. A specific method for acquiring the shape parameter value of the interactive object that matches the phoneme sequence will be described in detail later. The specific shape of the figure parameter can be determined according to the type of interactive target model.

In step 203, the appearance of the interactive object displayed on the display device is controlled based on the appearance parameter value.

The appearance parameter value is matched with the phoneme sequence corresponding to the voice-driven data of the interactive object. By controlling the appearance of the interactive object based on the appearance parameter value, it is possible to match the appearance of the interactive object with the interaction or response that the interactive object performs with respect to the target object. For example, when the interactive object interacts with or responds to the target object by voice, the appearance to be performed and the output voice are synchronized, thus giving the target object the feeling that the interactive object is speaking.

In the embodiment of the present invention, the phonetic sequence corresponding to the voice-driven data of the interactive object displayed on the display device is acquired, the figure parameter value of the interactive object matching the phonetic sequence is acquired, and the phonetic sequence is used. Matching to interact with or respond to the target object by controlling the shape of the interactive object displayed on the display device based on the shape parameter value of the interactive object to be matched. Improve the interactive experience of the target by making it look like it is interacting with the interactive target.

In some embodiments, the method applies to a server, which includes a local server, a cloud server, and the like. The server executes processing on the voice-driven data of the interactive target, generates the shape parameter value of the interactive target, renders the image using a three-dimensional rendering engine based on the shape parameter value, and then renders the interactive target. Get an interactive video. The server can interact with or respond to the target target by transmitting the video to the terminal and displaying it, and further, by transmitting the video to the cloud, the terminal acquires the video from the cloud. Can be made to interact with or respond to the target. After the server generates the appearance parameter value of the interactive object, the server further sends the appearance parameter value to the terminal so that the terminal completes the process of performing rendering, generating a moving image, executing an exhibition, and the like. To.

In some embodiments, the method is applied to a terminal, which performs processing on the voice-driven data of the interactive object to generate a state parameter value for the interactive object, the state parameter value. Based on the above, rendering is performed using a three-dimensional rendering engine to obtain the moving object to be interactive. The terminal can interact with or respond to the target by displaying the moving image.

In some embodiments, the output audio and / or exhibition text of the display device can be controlled based on the phoneme sequence. Further, it is possible to control the output voice and / or the exhibition text of the display device based on the phoneme sequence, and at the same time, control the appearance of the interactive object displayed on the display device based on the appearance parameter value. ..

In the embodiment of the present invention, since the appearance parameter value and the phoneme sequence are matched, the voice and / or the text of the exhibition output based on the phoneme sequence and the interactive object controlled based on the appearance parameter value. When the appearance is synchronized, the appearance performed by the interactive object and the output voice and / or the exhibited text are synchronized, giving the target object the feeling of talking to the interactive object.

Since it is necessary to maintain continuity in the output of voice, in one embodiment, the time window is moved on the phoneme sequence, and the phonemes in the time window are output in each movement process. Here, a predetermined time length is set as the step size of each movement time window. For example, the length of the time window can be set to 1 second and the predetermined time length can be set to 0.1 second. At the same time as outputting the phoneme in the time window, the figure parameter value corresponding to the phoneme at a predetermined position in the time window or the characteristic information of the phoneme is acquired, and the figure of the interactive object is controlled by using the figure parameter value. The predetermined position is a position having a predetermined time length from the start position of the time window. For example, when the length of the time window is set to 1s, the predetermined position is from the start position of the time window. It can be in the 0.5s position. Each time the time window is moved, the phonetic elements in the time window are output, and at the same time, the shape of the interactive object and the output voice are controlled by controlling the shape of the interactive object with the shape parameter value corresponding to the predetermined position of the time window. And are synchronized to give the target object the feeling of talking to the interactive object.

By changing the predetermined time length, the time interval (frequency) for acquiring the appearance parameter value can be changed, and therefore, the frequency with which the interactive object performs the appearance can be changed. The predetermined time length can be set according to the actual interactive scene, and the change in the appearance of the interactive object can be made more natural.

In some embodiments, feature encoding can be performed on the phoneme sequence to obtain feature information of the phoneme sequence, and the figure parameter value of the interactive object can be determined based on the feature information.

According to the embodiment of the present invention, the feature encoding is performed on the phonetic sequence corresponding to the voice-driven data of the interactive target, and the corresponding figure parameter value is obtained based on the obtained feature information, based on the phonetic sequence. At the same time, the appearance of the interactive object is controlled based on the appearance parameter value corresponding to the feature information, and in particular, the interactive object performs a facial movement based on the facial appearance parameter value corresponding to the feature information. Driven to do, the facial expression of the interactive object and the emitted voice could be synchronized, giving the target object the sensation of talking to the interactive object and improving the interactive experience of the target object.

In some embodiments, the feature encoding can be performed on the phoneme sequence to obtain the feature information of the phoneme sequence by the following method.

First, for a plurality of types of phonemes included in the phoneme sequence, a chord sequence corresponding to each of the plurality of types of phonemes is generated.

In one example, it is detected whether or not the first phoneme corresponds to each time point, and the first phoneme is any one of the plurality of phonemes. By setting the code value at the time when the first phoneme is supported as the first numerical value and setting the code value at the time when the first phoneme is not supported as the second numerical value, the code value at each time point is set. After assigning a value to, the chord sequence corresponding to the first phoneme can be obtained. For example, the chord value at the time when the first phoneme is supported can be set to 1, and the chord value at the time when the first phoneme is not supported can be set to 0. That is, for each phoneme in the plurality of phonemes included in the phoneme sequence, it is detected whether or not the phoneme corresponds to each time point, and the code value at the time when the phoneme corresponds to the phoneme. Is set as the first numerical value, the chord value at the time when the phoneme is not supported is set as the second numerical value, the value is assigned to the chord value at each time point, and then the chord sequence corresponding to the phoneme is set. Obtainable. Those skilled in the art should understand that the above-mentioned setting of the code value is merely an example, and that the code value may be set to another value, and the present invention is not limited thereto.

After that, based on the chord value of the chord sequence corresponding to each phoneme and the time length of each phoneme in the phoneme sequence, the feature information of the chord sequence corresponding to each phoneme is obtained.

In one example, a Gaussian filter is used to execute a Gaussian convolution operation on a continuous value of the first phoneme for a chord sequence corresponding to the first phoneme, and the chord corresponding to the first phoneme is executed. The characteristic information of the sequence is obtained, and here, the first phoneme is any one of the plurality of phonemes.

Finally, the feature information of the phoneme sequence is obtained based on the set of feature information of each chord sequence.

FIG. 3 is a schematic diagram showing a process of performing feature encoding on a phoneme sequence. As shown in FIG. 3, the phoneme sequence 310 includes phonemes j, i, j, ie4 (for simplification, only some phonemes are shown) for each phoneme j, i, ie4, respectively. The chord sequences 321, 322, and 323 corresponding to each of the above-mentioned phonemes are obtained. In each chord sequence, the chord value corresponding to the time point corresponding to the phoneme is set as the first numerical value (for example, set to 1), and the chord value second numerical value corresponding to the time point corresponding to the time point corresponding to the phoneme is set. (For example, set to 0). Taking the example of the chord sequence 321, when the phoneme j is present in the phoneme sequence 310, the value of the chord sequence 321 is the first numerical value, and when there is no phoneme j, the value of the chord sequence 321 is the second numerical value. .. All chord sequences 321, 322, 323 make up the complete chord sequence 320.

Based on the chord values of the chord sequences 321, 322, and 323 corresponding to the chord sequences j, i, and ie4, and the time lengths of the chord sequences corresponding to the three chord sequences, that is, the time of j in the chord sequence 321. The feature information of the code sequence 321, 322, and 323 can be obtained based on the length, the time length of i1 in the code sequence 322, and the time length of ie4 in the code sequence 323.

For example, using a Gaussian filter and using the continuous values of the phonemes j, i, and ie4 in the chord sequences 321, 322, and 323, respectively, a Gaussian convolution operation is executed to obtain the feature information of the chord sequence. Obtainable. That is, by performing a Gaussian convolution operation on continuous values in time of a phonetic element using a Gaussian filter, the chord values in each chord sequence are changed from the second numerical value to the first numerical value or from the first numerical value to the second numerical value. Make the stage of change to smooth. A Gaussian convolution operation is executed for each code sequence 321, 322, and 323, respectively, and the feature value of each code sequence is obtained. Here, the feature value constitutes a parameter of the feature information, and the feature information 330 corresponding to the phoneme sequence 310 is obtained based on the set of the feature information of each chord sequence. Those skilled in the art should understand that other operations can be performed on each code sequence to obtain characteristic information of the code sequence, and the present invention is not limited thereto.

In the embodiment of the present invention, by obtaining the characteristic information of the chord sequence based on the time length of each phoneme in the phoneme sequence, the stage of change of the chord sequence is made smooth. For example, the value of the code sequence may be an intermediate state value in addition to 0 and 1, and may be, for example, 0.2, 0.3, and the like. The figure parameter values obtained based on these intermediate state values allow the change in the figure of the interactive person to be smoother and more natural, and in particular, the change in the facial expression of the interactive person to be smoother and more natural. Improved the targeted interactive experience.

In some embodiments, the facial appearance parameters may include facial muscle control factors.

Human facial movements, from an anatomical point of view, are the result of coordinated deformations of various facial muscles. Therefore, the facial muscles of the interactive target are divided to obtain a facial muscle model, and the movement of each muscle (region) obtained by dividing is controlled based on the corresponding facial muscle control coefficient, that is, each muscle. It is possible to perform contraction / expansion control on the face so that the face of the interactive person has various facial expressions. For each muscle of the facial muscle model, it is possible to set an exercise state corresponding to a different muscle control coefficient based on the facial position where the muscle is located and the motor characteristics of the muscle itself. For example, in the case of the upper lip muscle, the range of the numerical value of the control coefficient is 0 to 1, and different numerical values in the range correspond to different contraction / expansion states of the upper lip muscle, and by changing the numerical value, the numerical value is changed. It is possible to open and close the mouth in the vertical direction. In the case of the left corner of the mouth muscle, the range of the numerical value of the control coefficient is 0 to 1, and different numerical values in the range correspond to the contraction / expansion state of the left corner of the mouth muscle, and by changing the numerical value. , The lateral change of the mouth can be realized.

At the same time as outputting voice based on the phonetic sequence, the interactive object is driven to make a facial expression based on the facial muscle control coefficient value corresponding to the phonetic sequence, and when the display device outputs voice, it is interactive. By making the subject make a facial expression that emits the voice at the same time, the target subject is given the feeling that the interactive subject is speaking, and the interactive experience of the target subject is improved.

In some embodiments, the facial movement of the interactive object can be associated with body shape. That is, the facial figure parameter value corresponding to the facial motion can be associated with the body figure, and the body figure may include a limb motion, a gesture motion, a walking figure, and the like.

In the process of driving the interactive object, the drive data of the body shape related to the face shape parameter value is acquired, the voice is output based on the phonetic sequence, and at the same time, the drive data of the body shape related to the face shape parameter value is output. Based on, the interactive object is driven so that the interactive object performs a limb movement. That is, at the same time that the interactive target is driven to perform a face movement based on the voice driving data of the interactive target, at the same time, the driving data of the body shape associated with the interactive target is acquired based on the face shape parameter value corresponding to the face movement. Then, when outputting the voice, the interactive target is driven to perform the facial movement and the limb movement corresponding to the synchronization, so that the speech state of the interactive target becomes more vivid and natural, and the target interactive target is interactive. Improved the experience.

In some embodiments, the figure parameter value of the interactive object corresponding to the feature information of the phoneme sequence can be obtained by the following method.

First, the feature information of the phoneme sequence is sampled at predetermined time intervals to obtain sampling feature information corresponding to each first sampling time. For example, if the predetermined time interval is 0.1 s, each first sampling time may be 0.1 s, 0.2 s, 0.3 s, or the like.

Referring to FIG. 3, feature information 330 is time-based information. Therefore, when the feature information is sampled at a predetermined time interval, the sampling feature information corresponding to each first sampling time can be obtained.

Subsequently, the sampling feature information corresponding to the first sampling time can be input to the pre-trained neural network to obtain the figure parameter value of the interactive object corresponding to the sampling feature information. Based on the sampling feature information corresponding to each first sampling time, the appearance parameter value of the interactive object corresponding to each first sampling time can be obtained.

As described above, when the time window is moved on the phoneme sequence and the phoneme is output, the feature information at a predetermined position of the time window is acquired. That is, the interactive target is emitted by obtaining the feature information of the first sampling time corresponding to the predetermined position of the time window, acquiring the figure parameter value corresponding to the feature information, and controlling the figure of the interactive target. Make sure that the appearance matches the existing voice, and that the process by which the interactive object emits the voice becomes more vivid and natural.

In some embodiments, the neural network comprises a long short-term memory network (Long Short-Term Memory, LSTM) and a fully connected network. Here, the long-short-term memory network is a time-recurrent neural network, and the historical information of the input sampling feature information can be learned. In addition, the long-term memory network and the fully connected network are jointly trained.

When the neural network includes a long-short-term storage network and a fully connected network, first, sampling feature information corresponding to the first sampling time is input to the long-short-term storage network. The long-short-term storage network outputs related feature information based on the sampling feature information before the first sampling time. That is, the information output by the long-short-term storage network includes the influence of the historical feature information on the current feature information. Subsequently, the related feature information is input to the fully connected network, and the appearance parameter value corresponding to the related feature information is determined based on the classification result of the fully connected network. Here, each classification corresponds to a set of appearance parameter values, that is, to the distribution of one type of facial muscle control coefficient.

In the embodiment of the present invention, the appearance parameter values corresponding to the sampling feature information of the phoneme sequence are predicted based on the long-term memory network and the fully connected network, and the relevant historical feature information and the current feature information are fused. Allows the historical form parameter value to affect the change in the current form parameter value so that the change in the figure state parameter value of the interactive person becomes smoother and more natural.

In some embodiments, the neural network can be trained by the following methods.

First, get a phoneme sequence sample. The phoneme sequence sample includes the appearance parameter values of the interactive object leveled at a second sampling time at predetermined time intervals. Like the phoneme sequence sample shown in FIG. 4, the dotted line in the sample indicates the second sampling time, and the appearance parameter value of the interactive object is leveled at each second sampling time.

Subsequently, sample feature encoding is executed for the phoneme sequence to obtain feature information corresponding to each second sampling time, and feature information samples are provided for the appearance parameter values corresponding to the feature information leveling. obtain. That is, the feature information sample includes the appearance parameter values of the interactive object leveled during the second sampling time.

After obtaining the feature information sample, the neural network can be trained based on the feature information sample. If the network loss is less than the given loss value, the training is completed. Here, the network loss includes a difference between the appearance parameter value predicted and obtained by the neural network and the leveled appearance parameter value.

In one example, the equation of the network loss function is as shown in equation (1).

は、ニューラルネットワークが予測して得たi番目の姿態パラメータ値であり、

は、レーベリングしたi番目の姿態パラメータ値であり、つまり、実在の値であり、

は、ベクトルの２番目のノルムを示す。 here,

Is the i-th figure parameter value predicted and obtained by the neural network.

Is the leveled i-th form parameter value, that is, the real value,

Indicates the second norm of the vector.

When the network parameter value of the neural network is adjusted to minimize the network loss function and the change in network loss satisfies the convergence condition, for example, the amount of change in network loss is smaller than a predetermined threshold value, or the number of iterations is reduced. When the predetermined number of times is reached, the training is completed and the trained neural network is obtained.

In another example, the equation of the network loss function is as shown in equation (2).

は、ニューラルネットワークが予測して得たi番目の姿態パラメータ値であり、

は、レーベリングしたi番目の姿態パラメータ値であり、つまり、実在の値であり、

は、ベクトルの２番目のノルムを示し、

は、ベクトルの最初のノルムを示す。 here,

Is the i-th figure parameter value predicted and obtained by the neural network.

Is the leveled i-th form parameter value, that is, the real value,

Indicates the second norm of the vector,

Indicates the first norm of the vector.

By adding the first norm of the predicted shape parameter values to the network loss function, we increased the constraint on the sparsity of the face parameters.

In some examples, phoneme sequence samples can be obtained by the following methods.

First, the video segment of the voice emitted by the character is acquired. For example, you can get the video segment that a real person is talking about.

For the video segment, a plurality of first image frames including the character and a plurality of audio frames corresponding to the first image frame are acquired. That is, the video segment is divided into an image frame and an audio frame. Here, each image frame corresponds to each audio frame, that is, for one image frame, it is possible to determine an audio frame corresponding to the sound emitted when the character makes the facial expression of the image frame. can.

Subsequently, the image frame including the character, which is the first image frame, is converted into the second image frame including the interactive object, and the figure parameter value corresponding to the second image frame is acquired. do. Taking an example in which the first image frame is an image frame including a real person, the image frame of the real person can be converted into a second image frame including an image indicated by an interactive object. Further, since the figure parameter value of the real person corresponds to the figure parameter value of the interactive object, the figure parameter value of the interactive object in each second image frame can be acquired.

After that, the audio frame corresponding to the first image frame is leveled based on the appearance parameter value corresponding to the second image frame, and the phoneme sequence sample is obtained based on the audio frame leveled with the appearance parameter value. obtain.

In an embodiment of the present invention, a video segment of a character is divided into a corresponding image frame and an audio frame, and a first image frame containing a real person is converted into a second image frame containing an interactive object. By acquiring the appearance parameter value corresponding to the phoneme sequence, the correspondence between the phoneme and the appearance parameter value can be improved, and a higher quality phoneme sequence sample can be obtained.

FIG. 5 is a schematic diagram showing a configuration of a drive device for an interactive object according to at least one embodiment of the present invention. The interactive object is exhibited in a display device, and as shown in FIG. 5, the device is shown. A phoneme sequence acquisition unit 501 for acquiring a phoneme sequence corresponding to the voice-driven data of the interactive object, a parameter acquisition unit 502 for acquiring a figure parameter value of the interactive object matching with the phoneme sequence, and the like. A drive unit 503 for controlling the appearance of the interactive object displayed on the display device based on the appearance parameter value may be provided.

In some embodiments, the device further comprises an output unit for controlling the output audio and / or display text of the display device based on the phoneme sequence.

In some embodiments, the parameter acquisition unit specifically performs feature encoding on the phoneme sequence to obtain feature information on the phoneme sequence and the interactive object corresponding to the feature information on the phoneme sequence. Get the appearance parameter value of.

In some embodiments, when performing feature encoding on the phoneme sequence to obtain feature information of the phoneme sequence, the parameter acquisition unit specifically includes a plurality of types included in the phoneme sequence. For each phoneme in the phoneme, a chord sequence corresponding to each of a plurality of types of phonemes is generated, the chord value of the chord sequence corresponding to each of the plurality of types of phonemes, and a plurality of types in the phoneme sequence. Based on the time length corresponding to each phoneme, the characteristic information of the chord sequence corresponding to each of the plurality of types of phonemes is obtained, and the characteristic information of the chord sequence corresponding to each of the plurality of types of phonemes is used as the characteristic information of the said phoneme sequence. Get the feature information of.

In some embodiments, when the parameter acquisition unit specifically generates a chord sequence corresponding to a plurality of phonemes for a plurality of types of phonemes included in the phoneme sequence, the parameter acquisition unit is specified at each time point. Is detected whether or not the first phoneme is supported, the code value at the time when the first phoneme is supported is set as the first numerical value, and the code value at the time when the first phoneme is not supported is set. By setting it as a second numerical value, a chord sequence corresponding to the first phoneme is obtained, where the first phoneme is any one of the plurality of phonemes.

In some embodiments, the plurality of phonemes are each based on the chord value of the chord sequence corresponding to each of the plurality of phonemes and the time length corresponding to each of the plurality of phonemes in the phoneme sequence. When obtaining the feature information of the corresponding chord sequence, the parameter acquisition unit specifically, for the chord sequence corresponding to the first phoneme, uses a Gaussian filter to set the continuous value in the time of the first phoneme. On the other hand, a Gaussian convolution operation is executed to obtain feature information of a chord sequence corresponding to the first phoneme, where the first phoneme is any one of the plurality of phonemes.

In some embodiments, the facial features parameter comprises a facial profile parameter, said facial profile parameters include a facial muscle control factor, the facial muscle control factor to control the motor state of at least one facial muscle. Used, the drive unit is specifically such that the interactive subject performs a facial action that matches each phone in the phone sequence, based on a facial muscle control factor that matches the phone sequence. Drive the subject.

In some embodiments, the device acquires body shape drive data related to the face shape parameter, and the interactive subject moves a limb based on the body shape drive data related to the face shape parameter value. The operation drive unit for driving the interactive object is further provided.

In some embodiments, when acquiring the figure parameter value of the interactive object corresponding to the feature information of the phonetic sequence, the parameter acquisition unit specifically takes the feature information of the phonetic sequence at predetermined time intervals. To obtain sampling feature information corresponding to the first sampling time, input the sampling feature information corresponding to the first sampling time into a pre-trained neural network, and interactively correspond to the sampling feature information. Obtain the shape parameter value of the target.

In some embodiments, the neural network comprises a long-term storage network and a fully connected network, and the sampling feature information corresponding to the first sampling time is input to the pre-trained neural network to perform the sampling. When obtaining the figure parameter value of the interactive object corresponding to the feature information, the parameter acquisition unit specifically inputs the sampling feature information corresponding to the first sampling time into the long-term storage network, and the first The related feature information is output based on the sampling feature information before the sampling time, the related feature information is input to the fully connected network, and the state corresponding to the related feature information is based on the classification result of the fully connected network. The parameter values are determined, and each type in the classification result corresponds to a set of appearance parameter values.

In some examples, the neural network was obtained by training using a phoneme sequence sample. The device acquires a video segment of the voice emitted by the character, and based on the video segment, acquires a plurality of first image frames including the character and a plurality of audio frames corresponding to the first image frame. , The first image frame is converted into a second image frame including the interactive object, a figure parameter value corresponding to the second image frame is acquired, and the figure parameter value corresponding to the second image frame is obtained. Further, a sample acquisition unit for leveling the audio frame corresponding to the first image frame and obtaining a sound element sequence sample based on the audio frame whose appearance parameter value is leveled is further provided.

At least one embodiment of the present specification further provides an electronic device, which, as shown in FIG. 6, comprises a memory and a processor, the memory storing computer instructions that can be run on the processor. , The processor realizes the method of driving an interactive object according to any embodiment of the present invention when the computer instruction is executed.

At least one embodiment of the present specification further provides a computer-readable recording medium in which a computer program is stored and is an interactive subject according to any embodiment of the invention when the program is executed by a processor. Realize the driving method of.

Those skilled in the art should appreciate that one or more embodiments of the invention may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the invention may use the form of complete hardware embodiments, complete software embodiments, or a combination of software and hardware embodiments. Also, one or more embodiments of the present invention include, but include, one or more computer-usable recording media (disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. You can use the format of the computer program product implemented on (but not limited to).

Each of the embodiments in the present invention is described using a gradual method, the same or similar parts between the embodiments can be referred to each other, and the other embodiments can be referred to each other. The explanation focused on the differences from the examples. In particular, in the case of the example of the data processing device, since it is basically similar to the example of the method, it is described relatively briefly, but for the relevant part, refer to the explanation of a part of the example of the method. do it.

Specific embodiments of the invention have been described above. Other examples are within the scope of the attached "claims". In some cases, the actions or steps described in the claims may be performed in a different order than in the examples, and the expected result may still be achieved. Also, the process depicted in the drawings does not necessarily require the specific order or sequential order shown to achieve the expected result. In some embodiments, multitasking and parallel processing are also possible or may be beneficial.

Examples of the subject matter and functional operation in the present invention are digital electronic circuits, tangible computer software or firmware, computer hardware including the configurations and structural equivalents thereof disclosed in the present invention, or one or more thereof. It can be realized by combination. The embodiments of the subject in the present invention can be realized as one or more computer programs, i.e., encoded on a tangible non-temporary program carrier and executed by a data processing apparatus. , Can be implemented as one or more modules in a computer program instruction to control the operation of the data processing device. Alternatively or additionally, the program instruction can be encoded on a manually generated propagation signal, for example, on a machine-generated electrical, optical, or electromagnetic signal. can. The signal is generated to encode the information and transmit it to the appropriate receiver device for execution by the data processing device. The computer recording medium can be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or a combination thereof.

The processing and logical flow in the present invention can be performed by one or more programmable computers running one or more computer programs, performing operations based on input data to produce output. By doing so, the corresponding function is executed. The processing and logic flow can be further executed by a dedicated logic circuit such as FPGA (field programmable gate array) or ASIC (dedicated integrated circuit), and the device can also be realized as a dedicated logic circuit. Can be done.

Suitable computers for running computer programs include, for example, general purpose and / or dedicated microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. In general, a computer further includes or is operable with one or more mass storage devices for storing data, such as magnetic disks, magnetic optical disks, or optical disks. Combined with a device to receive data, transmit data, or both. However, computers do not necessarily have such devices. The computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or general purpose serial bus. It can be embedded in portable storage devices such as (USB) flash drives, and these devices are just a few examples.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, intermediaries, and memory devices, such as semiconductor memory devices (eg, EPROMs, EEPROMs, and flash devices). Includes magnetic discs (eg, internal hard disks or mobile discs), magnetic optical discs, and CD ROMs, and DVD-ROM discs. Processors and memory can be complemented by dedicated logic circuits or incorporated into dedicated logic circuits.

The present invention contains many specific implementation details, which should not be construed as limiting the scope of the invention or the scope of which it seeks to protect, primarily of some embodiments of the invention. Used to describe features. Specific features in a plurality of embodiments of the present invention may also be implemented in combination with a single embodiment. On the other hand, the various features in a single embodiment can be implemented separately in multiple embodiments or in any suitable sub-combination. It should be noted that the features play a role in a particular combination as described above and are claimed to be protected in this way from the beginning, but one or more features from the combination claimed to be protected may be excluded from the combination in some cases. And the combinations claimed to be protected can be sub-combined or directed to sub-combining variants.

Similarly, the drawings depict operations in a specific order, which either requires them to be performed in a specific order that indicates them, or requires them to be performed in sequence, or all the operations illustrated. It should not be understood that it is a requirement that the expected result be achieved by being carried out. In some cases, multitasking and parallel processing can be advantageous. It should not be understood that the separation of the various system modules and components in the above embodiments must be separated in this way in all embodiments, and the described program components and systems are: In general, it should be understood that they can be integrated together into a single software product or packaged into multiple software products.

Therefore, specific examples of the subject have already been described. Other examples are within the scope of the attached "Claims". In some cases, the actions described in the claims may be performed in different orders and still achieve the expected results. It should be noted that the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the expected results. Multitasking and parallel processing can be beneficial in some implementations.

The above are only a few embodiments of the invention and are not used to limit the invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention should be included within the scope of one or more embodiments of the invention.

Claims (20)

It is a driving method for interactive objects displayed on display devices.
Acquiring a phoneme sequence corresponding to the voice-driven data of the interactive target,
Acquiring the appearance parameter value of the interactive object that matches the phoneme sequence,
A method for driving an interactive object, which comprises controlling the appearance of the interactive object displayed on the display device based on the appearance parameter value. The driving method for an interactive object according to claim 1, further comprising controlling the output voice and / or exhibition text of the display device based on the phoneme sequence. Acquiring the appearance parameter value of the interactive object that matches the phoneme sequence is
To obtain the feature information of the phoneme sequence by executing the feature encoding on the phoneme sequence,
The driving method for an interactive object according to claim 1 or 2, wherein the figure parameter value of the interactive object corresponding to the characteristic information of the phoneme sequence is acquired, and the present invention includes. Performing feature encoding on the phoneme sequence to obtain feature information on the phoneme sequence
Generating a chord sequence corresponding to the phoneme for each phoneme among a plurality of types of phonemes included in the phoneme sequence.
Obtaining the feature information of the chord sequence corresponding to the phoneme based on the chord value of the chord sequence corresponding to the phoneme and the time length corresponding to the phoneme.
The driving method for an interactive object according to claim 3, wherein the feature information of the phoneme sequence is obtained based on the feature information of the chord sequence corresponding to each of the plurality of types of phonemes. Generating a chord sequence corresponding to a phoneme for each phoneme among a plurality of types of phonemes included in the phoneme sequence is not possible.
Detecting whether or not the phoneme is supported at each time point,
By setting the chord value at the time when the phoneme is supported as the first numerical value and setting the chord value at the time when the phoneme is not supported as the second numerical value, the chord sequence corresponding to the phoneme is obtained. The method for driving an interactive object according to claim 4, wherein the method comprises. It is possible to obtain the feature information of the chord sequence corresponding to each of the plurality of types of phonemes based on the chord value of the chord sequence corresponding to each of the plurality of phonemes and the time length corresponding to each of the plurality of types of phonemes. ,
For each phoneme in the plurality of types of phonemes, a Gaussian convolution operation is executed for a continuous value of the phonemes in time by using a Gaussian filter for the chord sequence corresponding to the phoneme. The driving method for an interactive object according to claim 4 or 5, wherein the feature information of the chord sequence corresponding to the phoneme is obtained. The appearance parameter includes a facial appearance parameter, the facial appearance parameter includes a facial muscle control coefficient, and the facial muscle control coefficient is used to control the motor state of at least one facial muscle.
Controlling the appearance of the interactive object displayed on the display device based on the appearance parameter value is
It is characterized by including driving the interactive object so that the interactive object performs a facial motion matching each phoneme in the phoneme sequence based on a facial muscle control coefficient value matching the phoneme sequence. The driving method for an interactive object according to any one of claims 1 to 6. Acquiring the drive data of the body shape related to the face shape parameter value and
7. The aspect of claim 7 is further comprising driving the interactive object so that the interactive object performs a limb movement based on the driving data of the body shape related to the facial appearance parameter value. Described how to drive an interactive object. Acquiring the appearance parameter value of the interactive object corresponding to the feature information of the phoneme sequence is
The feature information of the phoneme sequence is sampled at a predetermined time interval to obtain the sampling feature information corresponding to the first sampling time.
The feature is that the sampling feature information corresponding to the first sampling time is input to a pre-trained neural network to obtain the appearance parameter value of the interactive object corresponding to the sampling feature information. The method for driving an interactive object according to claim 3. The pre-trained neural network includes a long-term memory network and a fully connected network.
Inputting the sampling feature information corresponding to the first sampling time into a pre-trained neural network to obtain the appearance parameter value of the interactive object corresponding to the sampling feature information is possible.
The sampling feature information corresponding to the first sampling time is input to the long-short-term storage network, and the related feature information is output based on the sampling feature information before the first sampling time.
The related feature information is input to the fully connected network, and the appearance parameter value corresponding to the related feature information is determined based on the classification result of the fully connected network.
The driving method for an interactive object according to claim 9, wherein each type corresponds to a set of the appearance parameter values in the classification result. The neural network was obtained by training using a phoneme sequence sample.
The driving method of the interactive object is
To get the video segment of the audio emitted by the character,
Acquiring a plurality of first image frames including the character and a plurality of audio frames corresponding to the plurality of the first image frames based on the video segment.
The first image frame is converted into a second image frame including the interactive object, and the appearance parameter value corresponding to the second image frame is acquired.
Labeling the audio frame corresponding to the first image frame based on the appearance parameter value corresponding to the second image frame, and
The driving method for an interactive object according to claim 9 or 10, further comprising obtaining the phoneme sequence sample based on the audio frame to which the appearance parameter value is leveled. The sample feature encoding is executed for the phoneme sequence to obtain the feature information corresponding to the second sampling time, and the feature information sample is obtained for the figure parameter value corresponding to the feature information leveling.
Further comprising training the initial neural network based on the feature information sample and allowing the neural network to be trained after the change in network loss satisfies the convergence condition.
The driving method for an interactive object according to claim 11, wherein the network loss includes a difference between the appearance parameter value predicted and obtained by the initial neural network and the leveled appearance parameter value. .. The network loss includes a second norm of the difference between the figure parameter value predicted and obtained by the initial neural network and the leveled figure parameter value.
The driving method for an interactive object according to claim 12, wherein the network loss further includes the first norm of the appearance parameter value predicted and obtained by the initial neural network. An interactive drive device on display in a display device.
A phoneme sequence acquisition unit for acquiring a phoneme sequence corresponding to the voice-driven data of the interactive target, and a phoneme sequence acquisition unit.
A parameter acquisition unit for acquiring the figure parameter value of the interactive object that matches the phoneme sequence, and
A drive device for an interactive object, comprising: a drive unit for controlling the appearance of the interactive object displayed on the display device based on the appearance parameter value. The parameter acquisition unit is
For each phoneme among the plurality of types of phonemes included in the phoneme sequence, a chord sequence corresponding to the phoneme is generated.
Based on the encoding value of the chord sequence corresponding to the phoneme and the time length corresponding to the phoneme, the feature information of the chord sequence corresponding to the phoneme is obtained.
Based on the characteristic information of the chord sequence corresponding to each of the plurality of types of phonemes, the characteristic information of the phoneme sequence is obtained.
Here, it is possible to generate a chord sequence corresponding to the phoneme for each phoneme among a plurality of types of phonemes included in the phoneme sequence.
Detecting whether or not the phoneme is supported at each time point,
By setting the encoding value at the time when the phoneme is supported as the first numerical value and setting the encoding value at the time when the phoneme is not supported as the second numerical value, the chord sequence corresponding to the phoneme is obtained. The drive device for an interactive object according to claim 14, wherein the drive device is characterized by including the above. The appearance parameter includes a facial appearance parameter, the facial appearance parameter includes a facial muscle control coefficient, and the facial muscle control coefficient is used to control the motor state of at least one facial muscle.
The drive unit is
Based on the facial muscle control coefficient value that matches the phoneme sequence, the interactive object is driven so that the interactive object performs a facial motion that matches each phoneme in the phoneme sequence.
The drive device of the interactive object acquires the drive data of the body shape related to the face shape parameter value, and the interactive target moves the limbs based on the drive data of the body shape related to the face shape parameter value. The driving device for an interactive object according to claim 14 or 15, further comprising an operation driving unit for driving the interactive object. When acquiring the figure parameter value of the interactive object corresponding to the feature information of the phoneme sequence,
The parameter acquisition unit is
The feature information of the phoneme sequence is sampled at predetermined time intervals to obtain the sampling feature information corresponding to the first sampling time.
The sampling feature information corresponding to the first sampling time is input to the pre-trained neural network to obtain the appearance parameter value of the interactive object corresponding to the sampling feature information.
Here, the neural network includes a long-short period storage network and a fully connected network.
When the sampling feature information corresponding to the first sampling time is input to a pre-trained neural network to obtain the appearance parameter value of the interactive object corresponding to the sampling feature information.
The parameter acquisition unit is
The sampling feature information corresponding to the first sampling time is input to the long and short period storage network, and the related feature information is output based on the sampling feature information before the first sampling time.
The related feature information is input to the fully connected network, and the appearance parameter value corresponding to the related feature information is determined based on the classification result of the fully connected network.
Here, the drive device for an interactive object according to claim 15, wherein each type corresponds to a set of the appearance parameter values in the classification result. The neural network was obtained by training using a phoneme sequence sample.
The interactive object drive device further comprises a sample acquisition unit.
The sample acquisition unit is
A video segment of the sound emitted by the character is acquired, and a plurality of first image frames including the character and a plurality of audio frames corresponding to the plurality of the first image frames are acquired based on the video segment.
The first image frame is converted into a second image frame including the interactive object, and the appearance parameter value corresponding to the second image frame is acquired.
The audio frame corresponding to the first image frame is leveled based on the appearance parameter value corresponding to the second image frame.
The phoneme sequence sample was obtained based on the audio frame in which the appearance parameter values were leveled.
The interactive object drive is further equipped with a training unit.
The training unit
A sample feature code is executed for the phoneme sequence to obtain feature information corresponding to the second sampling time, and a feature information sample is obtained for the appearance parameter value corresponding to the feature information leveling.
The initial neural network can be trained based on the feature information sample, and the neural network can be trained after the change in network loss satisfies the convergence condition.
Here, the network loss includes a difference between the appearance parameter value predicted and obtained by the initial neural network and the leveled appearance parameter value.
The network loss includes a second norm of the difference between the figure parameter value predicted and obtained by the initial neural network and the leveled figure parameter value.
The drive device for an interactive object according to claim 17, wherein the network loss further includes the first norm of the appearance parameter value predicted and obtained by the initial neural network. An electronic device with memory and a processor
The memory stores computer instructions that can be operated on the processor.
An electronic device, wherein the processor realizes the method according to any one of claims 1 to 13 when the computer instruction is executed. A computer-readable recording medium that stores computer programs.
A computer-readable recording medium, wherein the method according to any one of claims 1 to 13 is realized when the computer program is executed by a processor.

Images