JP2022531072A - Interactive object drive methods, devices, devices, and storage media - Google Patents

Abstract

The present invention discloses a driving method, apparatus, device and storage medium for an interactive object, said interactive object being displayed on a display device. The method includes obtaining driving data of the interactive object, determining a driving mode of the driving data, and obtaining a control parameter value of the interactive object based on the driving data in response to the driving mode. and controlling the pose of the interactive object based on the control parameter value. [Selection drawing] Fig. 2

Description

The present invention relates to the field of computer technology, specifically to driving methods, devices, devices, and storage media for interactive objects.

<Mutual citation of related applications>
The present invention claims priority for a Chinese patent application with application number 2020102461120 and filing date March 31, 2020, the entire contents of which are incorporated herein by reference.

Human-computer interactions are primarily input by keystrokes, touch, and voice, and respond by displaying images, text, or virtual characters on the display screen. Currently, virtual characters have been improved primarily based on voice assistants and only output the device's voice.

The embodiments of the present invention provide a technical solution for driving an interactive object.

According to one aspect of the present invention, a method for driving an interactive object exhibited in a display device is provided, in which the method acquires driving data of the interactive object and determines a driving mode of the driving data. In response to the drive mode, the control parameter value of the interactive object is acquired based on the drive data, and the appearance of the interactive object is controlled based on the control parameter value.

Combined with any embodiment provided by the present invention, the method further comprises controlling the display device output audio and / or displaying text based on the driving data.

Combining with any embodiment provided by the present invention to determine the drive mode corresponding to the drive data obtains a voice data sequence corresponding to the drive data based on the type of drive data. That is, in response to the fact that the voice data sequence includes a plurality of voice data units and the detection of the target data contained in the voice data unit, the drive mode of the drive data is set to the first. Determining as a drive mode, wherein the target data corresponds to a predetermined control parameter value of the interactive object, and in response to the drive mode, the interactive object is based on the drive data. Acquiring the control parameter value includes using the predetermined control parameter value corresponding to the target data as the control parameter value of the interactive object in response to the first drive mode.

Combined with any embodiment provided by the present invention, the target data comprises a key word or key character, wherein the key word or key character is a predetermined control parameter value of a predetermined operation of the interactive object. Or, the target data includes a syllable, which corresponds to a predetermined control parameter value of a predetermined mouth shape movement of the interactive object.

Combining with any embodiment provided by the present invention to determine the drive mode corresponding to the drive data obtains a voice data sequence corresponding to the drive data based on the type of drive data. That is, if the voice data sequence includes a plurality of voice data units and the target data included in the voice data unit is not detected, the drive mode of the drive data is determined as the second drive mode. , The target data corresponds to a predetermined control parameter value of the interactive object. Acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode is a unit of at least one voice data in the voice data sequence in response to the second drive mode. It includes acquiring the feature information and acquiring the control parameter value of the interactive object corresponding to the feature information.

Combined with any embodiment provided by the present invention, the phoneme sequence includes a phoneme sequence, and acquiring feature information of at least one voice data unit in the voice data sequence is the phoneme sequence. The feature encoding is executed on the phoneme to obtain the first chord sequence corresponding to the phoneme sequence, the feature code corresponding to at least one phoneme is obtained based on the first chord sequence, and the feature code is obtained. To obtain characteristic information of at least one phoneme based on the above.

Combined with any embodiment provided by the present invention, said voice data sequence comprises a voice frame sequence, and acquiring feature information of at least one voice data unit in said voice data sequence is said voice. Acquiring the first acoustic feature sequence corresponding to the frame sequence, the first acoustic feature sequence includes the acoustic feature vector corresponding to each audio frame in the audio frame sequence, and the first acoustic feature. It includes acquiring an acoustic feature vector corresponding to at least one voice frame based on a sequence, and obtaining feature information corresponding to the at least one voice frame based on the acoustic feature vector.

Combined with any embodiment provided by the present invention, the control parameter of the interactive object includes a facial appearance parameter, the facial appearance parameter includes a facial muscle control coefficient, and the facial muscle control. The coefficient is used to control the motor state of at least one facial muscle, and acquiring the control parameter value of the interactive object based on the driving data is based on the driving data of the face of the interactive object. Controlling the appearance of the interactive object based on the control parameter value, including acquiring the part muscle control coefficient, means that the interactive object matches the driving data based on the acquired facial muscle control coefficient. It includes driving the interactive object to perform a partial motion.

Combined with any embodiment provided by the present invention, the method relates to obtaining body shape driving data associated with the facial shape parameter and in relation to the facial shape parameter value. Further including driving the interactive object to perform limb movements based on the driving data of the existing body shape.

Combined with any embodiment provided by the present invention, the control parameter value of the interactive object comprises a control vector of at least one partial region of the interactive object and is based on the driving data of the interactive object. Acquiring the control parameter value includes acquiring the control vector of at least one partial region of the interactive object based on the driving data, and controls the appearance of the interactive object based on the control parameter value. That includes controlling the facial and / or limb movements of the interactive object based on the acquired control vector of at least one partial region.

Capturing the control parameter values of the interactive object corresponding to the feature information in combination with any embodiment provided by the present invention inputs the feature information into a pre-trained recurrent neural network. , Includes obtaining control parameter values for the interactive object corresponding to the feature information.

According to one aspect of the present invention, there is provided a drive device for an interactive object exhibited in a display device, the device for acquiring drive data for the interactive object and determining a drive mode for the drive data. The 1 acquisition unit, the second acquisition unit for acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode, and the appearance of the interactive object based on the control parameter value. It is equipped with a drive unit for control.

According to one aspect of the invention, an electronic device is provided, the electronic device comprising the device memory and a processor, the memory storing computer instructions that can be run on the processor, the processor being the computer. Realization When the instruction is executed, the method of driving an interactive object according to any embodiment provided by the present invention is executed.

According to one aspect of the invention, an interactive object according to any embodiment provided by the present invention provides a computer-readable recording medium in which a computer program is stored and the computer program is executed by a processor. The driving method of is executed.

According to the driving method, apparatus, device, and computer readable storage medium of the interactive object of one or more embodiments of the present invention, the control parameter value of the interactive object is set based on the driving mode of the driving data of the interactive object. By acquiring, the appearance of the interactive object is controlled. Here, the control parameter values of the corresponding interactive objects are acquired according to different methods for different drive modes, and the appearance in which the interactive objects are matched with the contents of the drive data and / or the corresponding voices is exhibited. The target was given the feeling of interacting with the interactive target, and the interactive experience with the target was improved.

Hereinafter, in order to more clearly explain the technical solution in one or more embodiments or prior art of the present specification, the drawings that need to be used in the description of the embodiment or prior art are briefly introduced. .. Obviously, the drawings described below are only a few examples described in one or more embodiments herein, and those skilled in the art will be able to base on these drawings without any creative work. Can be obtained.
It is a schematic diagram of the display device in the driving method of an interactive object provided by at least one embodiment of the present invention. It is a flowchart of the driving method of an interactive object provided by at least one Embodiment of this invention. It is a schematic diagram of the process of performing feature encoding on the phoneme sequence provided by at least one embodiment of the present invention. It is a schematic diagram of the process of obtaining the control parameter value based on the phoneme sequence provided by at least one embodiment of the present invention. It is a schematic diagram of the process of obtaining the control parameter value based on the voice frame sequence provided by at least one embodiment of the present invention. It is a schematic diagram of the configuration of the drive device of an interactive object provided by at least one embodiment of the present invention. It is a schematic diagram of the configuration of the electronic device provided by at least one embodiment of the present invention.

Hereinafter, exemplary embodiments will be described in detail and examples will be shown in the drawings. Where the following description refers to drawings, the same numbers in different drawings indicate the same or similar elements, unless otherwise stated. The embodiments described in the following exemplary examples do not represent all embodiments consistent with the present invention. Conversely, they are merely examples of devices and methods consistent with some aspects of the invention described in the appended claims.

The term "and / or" as used herein merely describes the relationship of a related object and indicates that three relationships can exist, for example, A and / or B, where A is present alone. There are three relationships, such as the existence of A and B at the same time, and the existence of B alone. Further, the term "at least one kind" in the present specification refers to any one kind in a plurality of kinds or any combination of at least two kinds in a plurality of kinds, for example, in A, B, C. Inclusion of at least one indicates that it comprises any one or more elements selected from a set composed of A, B, and C.

At least one embodiment of the present invention provides a driving method for an interactive object, which driving method can be performed by an electronic device such as a terminal device or a server. The terminal device may be a fixed terminal or a mobile terminal such as a mobile phone, a tablet personal computer, a game machine, a desktop personal computer, an advertising machine, an all-in-one machine, and an in-vehicle terminal. The server includes a local server, a cloud server, and the like. The method can be realized by a method of calling a computer-readable instruction stored in a memory by a processor.

In an embodiment of the invention, the interactive object can be any virtual image capable of performing interactive with the target object. In one embodiment, the interactive object may be a virtual character, and may be another virtual image capable of realizing an interactive function, such as a virtual animal, a virtual article, or a cartoon image. The display format of the interactive object is 2D or 3D, but the present invention is not limited thereto. The target can be a user, a robot, or other smart device. The interactive method of the interactive object with the target object may be an active interactive method or a passive interactive method. In one example, depending on the target object, a request can be made by performing a gesture or a limb movement, and the interactive object can be triggered and interactively performed by an active interactive method. In another example, the interactive object can be made to interact with the interactive object by a passive method by actively greeting and prompting the target object to perform an action or the like.

The interactive object can be exhibited using a terminal device, and the terminal device includes a television, an all-in-one device having a display function, a projector, a virtual reality (VR) device, and an augmented reality (AR). ) The present invention may be a device or the like, and the present invention is not limited to a specific form of the terminal device.

FIG. 1 shows a display device provided by at least one embodiment of the present invention. As shown in FIG. 1, the display device has a transparent display screen, and by displaying a stereoscopic image on the transparent display screen, a virtual scene having a stereoscopic effect and an interactive object can be represented. For example, the interactive object displayed on the transparent screen of FIG. 1 includes a virtual cartoon person. In some embodiments, the terminal device described in the present invention may be a display device having the transparent display screen described above, wherein the display device is arranged with a memory and a processor, and the memory is stored on the processor. The processor stores an operable computer instruction, and when the computer instruction is executed, the interactive object displayed on the transparent display screen is realized by realizing the driving method of the interactive object provided by the present invention. It can be driven to interact or respond to the target.

In some embodiments, the interactive object can emit a designated voice to the target object in response to voice-driven data for driving the interactive object to output voice. The terminal device generates voice-driven data based on the movement, facial expression, status, preference, etc. of the target object around the terminal device so that the interactive target emits a specified voice to interact or respond. By driving, it is possible to provide anthropomorphic services to the target target. It should be explained that the voice-driven data may be generated by other methods, for example, by a server and transmitted to a terminal device.

In the process of the interactive object interacting with the target object, when the interactive object is driven to emit the specified voice based on the voice-driven data, the interactive target has a face synchronized with the specified voice. It cannot be driven to perform the movement of the part, and it becomes dull and unnatural when the interactive object emits a voice, which may affect the interactive experience with the interactive object of the target object. In view of this, at least one embodiment of the present invention submits an interactive object driving method to enhance the interactive experience with the interactive object of the target object.

FIG. 2 is a flowchart of a method of driving an interactive object according to at least one embodiment of the present invention, wherein the interactive object is exhibited in a display device, and as shown in FIG. 2, the method is step 201. -Including step 203.

In step 201, the driving data of the interactive target is acquired, and the driving mode of the driving data is determined.

In the embodiment of the present invention, the voice-driven data may include audio data (voice data), text, and the like. The voice-driven data may be drive data generated based on the movement, facial expression, status, preference, etc. of the target object that interacts with the interactive target by the server or the terminal device, or may be the internal memory directly acquired by the terminal device. It may be voice-driven data called from. The present invention is not limited to the method of acquiring the voice-driven data.

The drive mode of the drive data can be determined based on the type of the drive data and the information contained in the drive data.

In one example, a voice data sequence corresponding to the drive data can be obtained based on the type of drive data, where the voice data sequence includes a plurality of voice data units. Here, the voice data unit may be configured in units of characters or words, or may be configured in units of phonemes or syllables. Character sequences, word sequences, etc. corresponding to the drive data can be obtained corresponding to the text type drive data, and phoneme sequences, syllable sequences, etc. corresponding to the drive data can be obtained corresponding to the audio type drive data. You can get audio frame sequences and the like. In one embodiment, the audio data and the text data can be converted to each other. For example, audio data can be converted to text data and then divided into voice data units, or text data can be converted to audio data and then divided into voice data units. On the other hand, it is not limited.

When the target data included in the voice data unit is detected, the drive mode of the drive data can be determined as the first drive mode, where the target data is a predetermined control parameter of the interactive target. Corresponds to the value.

The target data may be a set key word or key character, and the key word or key character corresponds to a predetermined control parameter value of a predetermined operation of an interactive target.

In the embodiment of the present invention, a predetermined operation is matched with each target data in advance. Each predetermined operation is realized by being controlled by the corresponding control parameter value. Therefore, each target data and the control parameter value of the predetermined operation are matched. Taking the example where the key word is "hand gesture", if the voice data unit includes a text-shaped "hand gesture" and / or a voice-shaped "hand gesture", it is determined that the driving data includes the target data. be able to.

Illustratively, the target data includes a syllable, which corresponds to a predetermined control parameter value of a predetermined mouth shape motion of the interactive object.

The syllables corresponding to the target data belong to the pre-divided and different syllable types, and the different syllable types are matched to different predetermined mouth shapes. Here, the syllable is formed by combining at least one phoneme and includes a voice unit. The syllables may include pinyin language syllables and non-pinyin language (eg, Chinese) syllables. Different syllable types are syllables that match or essentially match the pronunciation action, and different syllable types correspond to different actions of the interactive object. In one embodiment, the different syllable types correspond to different predetermined mouth shapes as the interactive object speaks, i.e., to correspond to different pronunciation movements. In this case, the different syllable types match the different predetermined mouth shape control parameter values. For example, syllables of the type "ma", "man", "mang", etc., which are pin-ins, can be regarded as the same type because their pronunciation behaviors are basically the same, and all of them are spoken by an interactive object. Corresponds to the control parameter value of the mouth shape of "the mouth is open" at the time.

When the target data included in the voice data unit is not detected, the drive mode of the drive data can be determined as the second drive mode, where the target data is a predetermined control parameter of the interactive target. Corresponds to the value.

Those skilled in the art should understand that the first drive mode and the second drive mode described above are merely examples, and the embodiments of the present invention do not limit specific drive modes.

In step 202, in response to the drive mode, the control parameter value of the interactive object is acquired based on the drive data.

It is possible to acquire the control parameter value of the interactive target by adopting the corresponding method for various drive modes of the drive data.

In one example, in response to the first drive mode determined in step 201, the predetermined control parameter value corresponding to the target data can be used as the control parameter value of the interactive object. For example, for the first drive mode, a predetermined control parameter value corresponding to the target data (for example, “hand gesture”) included in the voice data sequence is used as the control parameter value of the interactive target. Can be done.

In one example, in response to the second drive mode determined in step 201, the feature information of at least one voice data unit in the voice data sequence is acquired, and the control parameter value of the interactive object corresponding to the feature information is acquired. Can be obtained. That is, if the target data included in the voice data sequence is not detected, the corresponding control parameter value can be acquired based on the feature information of the voice data unit. The feature information includes feature information of a voice data unit obtained by executing feature encoding on the voice data sequence, feature information of a voice data unit obtained based on the acoustic feature information of the voice data sequence, and the like. Can include.

In step 203, the appearance of the interactive object is controlled based on the control parameter value.

In some embodiments, the interactive subject control parameter comprises a facial appearance parameter, the facial appearance parameter comprises a facial muscle control factor, and the facial muscle control coefficient comprises at least one face. Used to control the state of muscle movement. In one embodiment, the facial muscle control coefficient of the interactive target is acquired based on the driving data, and the interactive target performs a facial motion matching the driving data based on the acquired facial muscle control coefficient. The interactive object can be driven.

In some embodiments, the control parameter value of the interactive object comprises a control vector of at least one partial region of the interactive object. In one embodiment, the control vector of at least one partial region of the interactive object can be acquired based on the driving data, and the interactive object can be acquired based on the acquired control vector of at least one partial region of the interactive object. Can control facial and / or limb movements.

The appearance of the interactive object is controlled by acquiring the control parameter value of the interactive object based on the drive mode of the drive data of the interactive object. Here, we will exhibit a mode in which the interactive target matches the content of the drive data and / or the corresponding voice by acquiring the control parameter values of the corresponding interactive target by different methods for different drive modes. In this way, the target was given the feeling of interacting with the interactive target, improving the interactive experience with the target's interactive target.

In some embodiments, the display device output audio can be further controlled and / or the text can be exhibited based on the drive data. Further, at the same time as the output of the voice and / or the display of the text, the appearance of the interactive object can be controlled based on the control parameter value.

In the embodiment of the present invention, since the control parameter value and the drive data are matched, the output of the voice and / or the display of the text based on the drive data can be seen as the appearance of the control interactive object based on the control parameter value. When synchronized, the appearance performed by the interactive object and the output voice and / or the exhibited text are also synchronized, so that the target object is given the feeling of interacting with the interactive object.

In some embodiments, the audio data sequence comprises a phoneme sequence. In response to the drive data including the audio data, the audio data can be divided into a plurality of audio frames, and the audio frames can be combined to form a phoneme based on the state of the audio frames. Each phoneme formed based on the audio data forms a phoneme sequence. Here, a phoneme is the smallest phoneme unit divided based on the natural attributes of speech, and one pronunciation action of a real person can form one phoneme. The corresponding phoneme sequence can be obtained by obtaining the phoneme corresponding to the morpheme based on the morpheme contained in the text in response to the driving data being a text.

In some embodiments, the following method can be used to obtain feature information for at least one voice data unit in the voice data sequence, wherein the method performs feature encoding on the phoneme sequence. To obtain the first chord sequence corresponding to the phoneme sequence, to acquire the feature code corresponding to at least one phoneme based on the first chord sequence, and to obtain the feature code corresponding to the at least one phoneme based on the feature code. Includes getting feature information.

FIG. 3 is a schematic diagram showing a process of performing feature encoding on a phoneme sequence. As shown in FIG. 3, the phoneme sequence 310 includes phonemes j, i1, j, ie4 (for simplification, only some phonemes are shown) for each phoneme j, i1, ie4, respectively. The corresponding chord sequences 321, 322, and 323 are obtained. In each chord sequence, the chord value corresponding to the time point corresponding to the phoneme is set as the first numerical value (for example, set to 1), and the chord value corresponding to the time point corresponding to the phoneme is second. Set as a number (for example, set to 0). Taking the example of the chord sequence 321, when the phoneme j is present in the phoneme sequence 310, the value of the chord sequence 321 is 1, which is the first numerical value, and when there is no phoneme j, the value of the chord sequence 321 is the second. It is set to 0, which is a numerical value. All chord sequences 321, 322, 323 make up the complete chord sequence 320.

Based on the chord values of the chord sequences 321, 322, and 323 corresponding to the chord sequences j, i1, and ie4, and the time lengths of the chord sequences corresponding to the three chord sequences, that is, the time of j in the chord sequence 321. The feature information of the code sequence 321, 322, and 323 can be obtained based on the length, the time length of i1 in the code sequence 322, and the time length of ie4 in the code sequence 323.

For example, using a Gaussian filter, a Gaussian convolution operation is executed for continuous values of phonemes j, i1, and ie4 in the chord sequences 321, 322, and 323, respectively, to obtain feature information of the chord sequence. Can be done. That is, by using a Gaussian filter to execute a Gaussian convolution operation on continuous values in time of phonemes, the chord values in each chord sequence are changed from the second numerical value to the first numerical value or from the first numerical value to the second numerical value. Make the stage of change smooth. By executing the Gaussian convolution operation for each code sequence 321, 322, and 323, the feature value of each code sequence is obtained. Here, the feature value constitutes a parameter in the feature information, and the feature information 330 corresponding to the phoneme sequence 310 is obtained based on the set of feature information of each chord sequence. Those skilled in the art should understand that other operations can be performed on each code sequence to obtain characteristic information of the code sequence, and the present invention is not limited thereto.

In the embodiment of the present invention, by obtaining the feature information of the chord sequence based on the time length of each phoneme in the phoneme sequence, the stage of change of the chord sequence is made smooth. For example, the value of the code sequence may be an intermediate state value in addition to 0 and 1, and may be, for example, 0.2, 0.3, and the like. The figure parameter values obtained based on these intermediate state values allow the change in the figure of the interactive person to be smoother and more natural, and in particular, the change in the facial expression of the interactive person to be smoother and more natural. Improved the targeted interactive experience.

In some embodiments, the facial appearance parameters may include facial muscle control factors.

Human facial movements, from an anatomical point of view, are the result of coordinated deformations of various facial muscles. Therefore, the facial muscles of the interactive target are divided to obtain a facial muscle model, and the movement of each muscle (region) obtained by dividing is controlled based on the corresponding facial muscle control coefficient, that is, each muscle. It is possible to execute contraction / expansion control on the face so that the face of the interactive person has various facial expressions. For each muscle of the facial muscle model, it is possible to set an exercise state corresponding to a different muscle control coefficient based on the facial position where the muscle is located and the motor characteristics of the muscle itself. For example, in the case of the upper lip muscle, the range of the numerical value of the control coefficient is 0 to 1, and different numerical values in the range correspond to different contraction / expansion states of the upper lip muscle, and by changing the numerical value, the numerical value is changed. It is possible to open and close the mouth in the vertical direction. In the case of the left corner of the mouth muscle, the range of the numerical value of the control coefficient is 0 to 1, and different numerical values in the range correspond to the contraction / expansion state of the left corner of the mouth muscle, and by changing the numerical value. , The lateral change of the mouth can be realized.

At the same time as outputting voice based on the phonetic sequence, the interactive target is driven to make a facial expression based on the facial muscle control coefficient corresponding to the phonetic sequence, and when the display device outputs voice, the interactive target is output. At the same time, by making the facial expression that emits the voice, the target is given the feeling that the interactive target is speaking, and the interactive experience of the target is improved.

In some embodiments, the facial movement of the interactive object can be associated with body shape. That is, the facial figure parameter value corresponding to the facial motion can be associated with the body figure, and the body figure may include a limb motion, a gesture motion, a walking figure, and the like.

In the process of driving the interactive object, the drive data of the body shape related to the face shape parameter value is acquired, the voice is output based on the phonetic sequence, and at the same time, the drive data of the body shape related to the face shape parameter value is output. Based on, the interactive object is driven so that the interactive object performs a limb movement. That is, at the same time that the interactive target is driven to perform a face movement based on the voice driving data of the interactive target, at the same time, the driving data of the body shape associated with the interactive target is acquired based on the face shape parameter value corresponding to the face movement. Then, when outputting the voice, the interactive target is driven to perform the facial movement and the limb movement corresponding to the synchronization, so that the spoken state of the interactive target becomes more vivid and natural, and the target target is Improved the interactive experience.

Since it is necessary to maintain continuity in the output of voice, in one embodiment, the time window is moved on the phoneme sequence, and the phonemes in the time window are output in each movement process. Here, a predetermined time length is set as the step size of each movement time window. For example, the length of the time window can be set to 1 second and the predetermined time length can be set to 0.1 seconds. At the same time as outputting the phoneme in the time window, the figure parameter value corresponding to the phoneme at a predetermined position in the time window or the characteristic information of the phoneme is acquired, and the figure of the interactive object is controlled by using the figure parameter value. The predetermined position is a position having a predetermined time length from the start position of the time window. For example, when the length of the time window is set to 1s, the predetermined position is from the start position of the time window. It can be in the position of 0.5s. Each time the time window is moved, the phonetic elements in the time window are output, and at the same time, the shape of the interactive object and the output voice are controlled by controlling the shape of the interactive object with the shape parameter value corresponding to the predetermined position of the time window. And are synchronized to give the target object the feeling of talking to the interactive object.

By changing the predetermined time length, the time interval (frequency) for acquiring the appearance parameter value can be changed, and therefore, the frequency with which the interactive object performs the appearance can be changed. The predetermined time length can be set according to the actual interactive scene, and the change in the appearance of the interactive object can be made more natural.

In some embodiments, the appearance of the interactive object can be controlled by obtaining a control vector for at least one partial region of the interactive object.

The local area is obtained by dividing the entire interactive object (including the face and / or the body). Control of one or more local areas of the face may correspond to a series of facial expressions or movements of interactive objects. For example, control of the eye area may correspond to facial movements such as eye opening, closing, winking, and visual angle conversion of an interactive object. Also, for example, control of the mouth region may correspond to facial movements such as closing the mouth of an interactive object, opening to different degrees. Control of one or more local areas within the body may correspond to a series of limb movements of an interactive subject. For example, control of the foot area may correspond to movements such as interactive target walking, jumping, and kicking.

The control parameter of the local region of the interactive target includes a shape control vector of the local region. The shape control vector of each local region is used to drive the operation of the local region of the interactive object. Different morphological control vector values correspond to different motions or motion amplitudes. For example, in the case of a shape control vector in the mouth region, one set of shape control vector values can allow the interactive object to slightly open the mouth, and another set of shape control vector values is said. The interactive object can be made to open its mouth wide. By driving the interactive object with different shape control vector values, the corresponding local region can be made to perform different movements or different amplitude movements.

The local area can be selected based on the movement of the interactive object that needs to be controlled, for example, if the face and limbs of the interactive object need to be controlled to perform simultaneous movements, the local area of all the local areas. The appearance control vector value can be acquired, and when it is necessary to control the facial expression of the interactive object, the appearance control vector value of the local region corresponding to the face can be acquired.

In some embodiments, a feature code corresponding to at least one phoneme can be obtained by performing window sliding on the first chord sequence. Here, the first chord sequence may be a chord sequence after passing through a Gaussian convolution operation.

A window sliding is performed on the chord sequence with a time window of a predetermined length and a predetermined step size, and the feature code in the time window is set as the feature code of at least one phoneme corresponding to the window sliding. After completion, a second code sequence can be obtained based on the resulting feature codes. As shown in FIG. 4, on the first code sequence 320 or the smoothed first code sequence 430, a time window of a predetermined length is slid, and the feature code 1, the feature code 2, and the feature code 3 are slid. Can be obtained respectively, and so on. After traversing the first code sequence, the second code sequence 440 is obtained by obtaining the feature code 1, the feature code 2, the feature code 3, ..., The feature code M. Here, M is a positive integer, the numerical value of which is determined according to the length of the first code sequence, the length of the time window, and the step size for sliding the time window.

Based on the feature code 1, the feature code 2, the feature code 3, ..., The feature code M, the corresponding appearance control vector 1, the appearance control vector 2, the appearance control vector 3, ..., The appearance control vector M can be obtained, respectively. Therefore, the sequence 450 of the shape control vector is obtained.

The sequence 450 of the shape control vector and the second code sequence 440 are time-aligned. Since each feature code in the second chord sequence is obtained based on at least one phoneme in the phoneme sequence, each control vector in the appearance control vector sequence 450 is also at least in the phoneme sequence. It was obtained based on one phoneme. At the same time as playing the phonetic sequence corresponding to the text data, when the interactive object is driven to perform an action based on the sequence of the state control vector, the driven interactive object emits a voice corresponding to the text content at the same time. Improved the interactive experience of the target, which can be made to perform voice-synchronized movements and give the target the feeling of talking to the interactive target.

Assuming that the feature code starts to be output at a predetermined timing in the first time window, the appearance control vector value before the predetermined timing can be set to the default value, that is, the phoneme sequence is played first. Occasionally, the interactive object is made to perform the default action, and the interactive object is made to perform the action by using the sequence of the shape control vector obtained based on the first code sequence after the predetermined timing. Start driving. Taking FIG. 4 as an example, the feature code 1 starts to be output at the timing of t0, and the default shape control vector corresponds to the timing before the timing of t0.

The length of the time window is related to the amount of information contained in the feature code. If the amount of information contained in the time window is larger, a more uniform result will be output through the recurrent neural network processing. If the time window is too long, the facial expressions when the interactive object speaks will not be able to accommodate some characters. If the length of the time window is too small, the interactive subject will look stiff when speaking. Therefore, the time length of the time window is determined by the minimum time that the phoneme corresponding to the text data lasts, so that the action performed by driving the interactive object has a stronger relationship with the voice.

The step size of sliding the time window is related to the time interval (frequency) of acquiring the shape control vector, that is, to the frequency with which the driven interactive object performs an action. By setting the length and step size of the time window according to the actual interactive scene, the facial expression and movement performed by the interactive object are more closely related to the voice, and become more vivid and natural. To do so.

In some embodiments, when the time interval between phonemes in the phoneme sequence is greater than a predetermined threshold, the interactive object is driven to perform an action based on a predetermined shape control vector in the local region. .. That is, if the interactive person's utterance is stopped for a longer period of time, the interactive object is driven to perform a predetermined action. For example, when the output audio has a longer stagnation, the interactive object makes a smiling expression or shakes a little, so that the interactive object stands upright without a facial expression when the stagnation is longer. Avoiding this, making the process of speaking by the interactive target more natural and smooth, and improving the interactive experience with the target's interactive target.

In some embodiments, the voice data sequence comprises a voice frame sequence, and acquiring feature information of at least one voice data unit in the voice data sequence is a first acoustic feature corresponding to the voice frame sequence. Acquiring a sequence, wherein the first acoustic feature sequence includes an acoustic feature vector corresponding to each audio frame in the audio frame sequence, and at least one audio based on the first acoustic feature sequence. It includes acquiring the acoustic feature vector corresponding to the frame and obtaining the feature information corresponding to the at least one voice frame based on the acoustic feature vector.

In an embodiment of the invention, the control parameters of at least one partial region of the interactive object may be determined based on the acoustic features of the voice frame sequence, or may be controlled based on the other features of the voice frame sequence. The parameters may be fixed.

First, the acoustic feature sequence corresponding to the voice frame sequence is acquired. Here, in order to distinguish it from the acoustic feature sequence referred to later, the acoustic feature sequence corresponding to the voice frame sequence is referred to as a first acoustic feature sequence.

In the embodiments of the present invention, the acoustic feature may be a feature related to voice emotion, such as a fundamental frequency feature, a common peak feature, a Mel Frequency Cofficient, MFCC, and the like.

The first acoustic feature sequence is obtained by executing processing on the entire voice frame sequence. Taking an example of the MFCC feature, a window is displayed for each voice frame in the voice frame sequence. , Fast Fourier Transform, Filtering, Logarithmic Processing, Discrete Cosine Transform, to Obtain MFCC Coefficients Corresponding to Each Speech Frame.

The first acoustic feature sequence was obtained by executing processing on the entire voice frame sequence, and reflected the entire acoustic feature of the voice data sequence.

In the embodiment of the present invention, the first acoustic feature sequence includes an acoustic feature vector corresponding to each voice frame in the voice frame sequence. Taking the example of MFCC, the first acoustic feature sequence includes the MFCC coefficient of each audio frame. The first acoustic feature sequence obtained based on the voice frame sequence is as shown in FIG.

Subsequently, the acoustic features corresponding to at least one voice frame are acquired based on the first acoustic feature sequence.

When the first acoustic feature sequence includes an acoustic feature vector corresponding to each voice frame in the voice frame sequence, the same number of feature vectors corresponding to the at least one voice frame are used as the acoustic feature of the voice frame. be able to. Here, the same number of feature vectors described above can form one feature matrix, which is the acoustic feature of the at least one audio frame.

Taking the example of FIG. 5, the acoustic features of the corresponding N voice frames are formed by the N feature vectors in the first acoustic feature sequence, where N is a positive integer. The first acoustic feature matrix may include a plurality of acoustic features, where some overlap between audio frames corresponding to the respective acoustic features.

Finally, the control vector of at least one partial region of the interactive object corresponding to the acoustic feature is acquired.

It is possible to acquire the control vector of at least one partial region for the acoustic feature corresponding to the obtained at least one audio frame. The partial area can be selected based on the behavior of the interactive object that needs to be controlled. For example, when it is necessary to control the face and limbs of the interactive object to perform simultaneous movements, it is possible to acquire a control vector of the entire partial region, and it is necessary to control the facial expression of the interactive object. If there is, the control vector of the partial region corresponding to the facial expression can be acquired.

At the same time that the voice data sequence is reproduced, the terminal device outputs voice at the same time by driving the interactive object to perform an operation based on the control vector corresponding to each acoustic feature obtained by the first acoustic feature sequence. , The interactive object can be made to perform an action matched to the output voice, where the action includes a facial action, a facial expression, a limb action, and the like. Therefore, it is possible to give the target object the feeling that the interactive object is speaking. Since the control vector is related to the acoustic characteristics of the output voice, by driving based on the control vector, an emotional element is added to the facial expression and limb movement of the interactive object, and the process in which the interactive object speaks. Improves the interactive experience of the target by making it more natural and vivid.

In some embodiments, the method of performing window sliding in the first acoustic feature sequence can be used to obtain acoustic features corresponding to the at least one audio frame.

Perform window sliding on the first acoustic feature sequence with a time window of a given length and a given step size to make the acoustic feature vectors in the time window into the acoustic features of the corresponding number of audio frames. By setting, it is possible to obtain acoustic features that are jointly supported by these audio frames. After completing window sliding, a second acoustic feature sequence can be obtained based on the plurality of acoustic features obtained.

Taking the example of the driving method of the interactive object shown in FIG. 5, the voice frame sequence includes 100 voice frames per second, the time window length is 1 s, and the step size is 0. It is 04s. Each feature vector in the first acoustic feature sequence corresponds to a voice frame, and accordingly, the first acoustic feature sequence also includes 100 feature vectors as in 1 second. In the process of performing window sliding in the first acoustic feature sequence, each time 100 feature vectors in the time window are obtained, the obtained 100 feature vectors are converted into the acoustic features of the corresponding 100 audio frames. Set. By moving the time window in the step size of 0.04 s in the first acoustic feature sequence, it corresponds to the acoustic feature 1 corresponding to the first to 100 voice frames and the fourth to 104 voice frames, respectively. After obtaining the acoustic feature 2 and completing the processing for the first acoustic feature by analogy, the acoustic feature 1, the acoustic feature 2, ..., The acoustic feature M is obtained, and thus the second acoustic feature sequence is obtained. Here, M is a positive integer, and the numerical value is determined by the number of frames of the voice frame sequence (the number of feature vectors in the first acoustic feature sequence), the length of the time window, and the step size. To.

Based on the acoustic feature 1, the acoustic feature 2, ..., The acoustic feature M, the corresponding control vector 1, control vector 2, ..., Control vector M can be obtained, and thus a sequence of control vectors can be obtained. ..

As shown in FIG. 5, the sequence of the control vector is aligned with the second acoustic feature sequence in time, and the acoustic feature 1, the acoustic feature 2, ..., The acoustic feature M in the second acoustic feature sequence are Each is obtained based on the N feature vectors in the first acoustic feature sequence, thus playing the audio frame and at the same time driving the interactive object to perform an action based on the sequence of control vectors. Can be done.

Assuming that the output of acoustic features starts at a given timing in the first time window, the control vector prior to that given timing can be set to the default value, that is, when the audio frame sequence has just been played. First, the interactive object is made to perform the default operation, and after the predetermined timing, the interactive object starts to be driven to perform the operation by using the sequence of the control vector obtained based on the first acoustic feature sequence.

Taking the example of FIG. 5, the output of the acoustic feature 1 is started at the t0 timing, the acoustic feature is output at an interval of 0.04 s corresponding to the step size, and the output of the acoustic feature 2 is started at the t1 timing. The output of the acoustic feature 3 is started at the t2 timing, and the acoustic feature M is output at the t (M-1) timing. On the other hand, the feature vector (i + 1) corresponds to the ti to t (i + 1) time zone, where i is an integer less than (M-1). However, before the t0 timing, the control vector is the default control vector.

In the embodiment of the present invention, the speech data sequence is reproduced, and at the same time, the interactive object is driven to perform an action based on the sequence of the control vector, so that the motion of the interactive object is synchronized with the output voice. To give the target object the feeling that the interactive object is talking, and to improve the interactive experience with the interactive object of the target object.

The length of the time window is related to the amount of information contained in the acoustic feature. The larger the length of the time window, the more information is contained, and the stronger the relationship between the action performed by driving the interactive object and the voice. The step size of the time window sliding is related to the time interval (frequency) of acquiring the control vector, that is, the frequency of driving the interactive object to perform an action. The length and step size of the time window can be set according to the actual interactive scene, allowing the interactive object to have a stronger relationship between facial expressions and movements and speech, making it more vivid and natural.

In some embodiments, the acoustic feature comprises L-dimensional Mel frequency coefficients MFCC, where L is a positive integer. MFCC is a distribution in the frequency range of the energy of an audio signal. A plurality of voice frame data in the voice frame sequence are converted into frequency domains, and a mel filter containing L sub-regions is used to obtain L-dimensional MFCCs. Emotional to the facial and limb movements of the interactive object by acquiring a control vector based on the MFCC of the audio data sequence and driving the interactive object to perform facial and limb movements based on the control vector. It adds elements to make the interactive object more natural and vibrant than the talking process, thus improving the interactive experience with the target's interactive object.

In some embodiments, the feature information of the voice data unit is input to a pre-trained recurrent neural network to obtain the control parameter value of the interactive object corresponding to the feature information. The recurrent neural network is a time recurrent neural network, and can learn historical information of input feature information and output control parameters based on a voice unit sequence. For example, the control parameter may be a facial appearance control parameter or a control vector of at least one partial region.

In an embodiment of the present invention, a pre-trained recurrent neural network is used to acquire control parameters corresponding to the feature information of the voice data unit, and the relevant historical feature information and the current feature information are fused. By allowing the history control parameters to influence changes in the current control parameters, the facial expression changes and limb movements of the interactive person are made smoother and more natural.

In some embodiments, the recurrent neural network can be trained by the following methods.

First, a feature information sample is acquired. For example, the feature information sample can be obtained by the following method.

The video segment of the voice that emitted the character is acquired, and the corresponding voice segment of the character is extracted from the video segment. For example, you can get the video segment that a real person is talking about. Sampling can be performed on the video segment to acquire a plurality of first image frames of the character. Further, sampling can be performed on the voice segment to obtain a plurality of voice frames.

The feature information corresponding to the voice frame can be acquired based on the voice data unit included in the voice frame corresponding to the first image frame.

The first image frame can be converted into a second image frame including the interactive object, and the control parameter value of the interactive object corresponding to the second image frame can be acquired.

Based on the control parameter value, the feature information corresponding to the first image frame can be labeled to obtain a feature information sample.

In some embodiments, the feature information includes a phoneme feature code and the control parameter includes a facial muscle control factor. Using the facial muscle control coefficient obtained by the method for acquiring the feature information sample described above, the feature encoding of the phoneme corresponding to the first image frame is labeled, and the feature information corresponding to the feature code of the phoneme is labeled. Samples can be obtained.

In some embodiments, the feature information comprises a phoneme feature code and the control parameter comprises at least one partial control vector of the interactive object. By using at least one partial control vector obtained by the above-mentioned method for acquiring the feature information sample, the phoneme feature code corresponding to the first image frame is labeled to correspond to the phoneme feature encoding. You can get a sample of feature information.

In some embodiments, the feature information includes acoustic features of the audio frame and the control parameters include at least one partial control vector of the interactive object. The acoustic feature of the audio frame corresponding to the first image frame is labeled by using at least one partial control vector obtained by the method of acquiring the feature information sample described above, and the acoustic feature of the audio frame is used. It is possible to obtain a feature information sample corresponding to.

Those skilled in the art should understand that the feature information sample is not limited to the above description, and that the corresponding feature information sample can be obtained corresponding to various features of each type of audio data unit. ..

After obtaining the feature information sample, an initial recurrent neural network is trained based on the feature information sample, and when the change in network loss satisfies the convergence condition, the recurrent neural network is trained. The network loss includes a difference between the control parameter value predicted and obtained by the recurrent neural network and the labeled control parameter value.

In an embodiment of the present invention, a video segment of a character is divided into a plurality of corresponding first image frames and a plurality of audio frames, and a first image frame containing a real person is divided into a second image including an interactive object. By converting to a frame and acquiring the control parameter value corresponding to the feature information of at least one voice frame, the correspondence between the feature information and the control parameter value can be improved, and a high-quality feature information sample can be obtained. Obtain, make the appearance of the interactive object closer to the actual appearance of the corresponding character.

FIG. 6 is a schematic diagram showing a configuration of a drive device for an interactive object according to at least one embodiment of the present invention. As shown in FIG. 6, the device acquires drive data for the interactive object, and the device is described. A first acquisition unit 601 for determining the drive mode of the drive data, and a second acquisition unit 602 for acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode. A drive unit 603 for controlling the appearance of the interactive object based on the control parameter value is provided.

In some embodiments, the device further comprises an output unit for controlling the display device output audio and / or displaying text based on the drive data.

In some embodiments, when determining the drive mode corresponding to the drive data, the first acquisition unit specifically has a voice data sequence corresponding to the drive data based on the type of drive data. Here, the voice data sequence includes a plurality of voice data units, and when the target data included in the voice data unit is detected, the drive mode of the drive data is changed to the first drive mode. Here, the target data corresponds to a predetermined control parameter value of the interactive target, and in response to the drive mode, the control parameter value of the interactive target is acquired based on the drive data. This includes using the predetermined control parameter value corresponding to the target data as the control parameter value of the interactive object in response to the first drive mode.

In some embodiments, the target data includes a key word or key character, the key word or the key character corresponds to a predetermined control parameter value of a predetermined operation of an interactive target, or the target data. Includes a syllable, which corresponds to a predetermined control parameter value for a predetermined mouth-shaped movement of the interactive object.

In some embodiments, when recognizing the drive mode of the drive data, the first acquisition unit acquires a voice data sequence corresponding to the drive data, specifically based on the type of drive data. Here, the voice data sequence includes a plurality of voice data units, and if the target data included in the voice data unit is not detected, the drive mode of the drive data is determined as the second drive mode. Here, the target data corresponds to a predetermined control parameter value of the interactive target, and it is possible to acquire the control parameter value of the interactive target based on the drive data in response to the drive mode. In response to the second drive mode, acquiring the feature information of at least one voice data unit in the voice data sequence, and acquiring the control parameter value of the interactive target corresponding to the feature information. including.

In some embodiments, the voice data sequence comprises a phoneme sequence, and when acquiring feature information of at least one voice data unit in the voice data sequence, the second acquisition unit specifically. Feature encoding is performed on the phoneme sequence to obtain a first chord sequence corresponding to the phoneme sequence, and a feature code corresponding to at least one phoneme is obtained based on the first chord sequence. Based on this, the characteristic information of at least one phoneme is obtained.

In some embodiments, the voice data sequence comprises a voice frame sequence, and when acquiring feature information of at least one voice data unit in the voice data sequence, the second acquisition unit specifically. , The first acoustic feature sequence corresponding to the voice frame sequence is acquired, wherein the first acoustic feature sequence includes an acoustic feature vector corresponding to each voice frame in the voice frame sequence, and the first acoustic feature sequence is also included. The acoustic feature vector corresponding to at least one voice frame is acquired based on the acoustic feature sequence, and the feature information corresponding to the at least one voice frame is obtained based on the acoustic feature vector.

In some embodiments, the interactive subject control parameter comprises a facial appearance parameter, the facial appearance parameter comprises a facial muscle control coefficient, and the facial muscle control coefficient comprises at least one face. Used to control the motor state of the muscle, when acquiring the control parameter value of the interactive object based on the driving data, the second acquisition unit specifically, the interactive based on the driving data. The facial muscle control coefficient of the target is acquired, and the drive unit specifically performs the interactive target so that the interactive target performs a facial motion matching the drive data based on the acquired facial muscle control coefficient. The device also acquires drive data of the body shape related to the face shape parameter, and the interactive is based on the drive data of the body shape related to the face shape parameter value. It further comprises a limb drive unit for driving the subject to perform limb movements.

In some embodiments, the control parameter of the interactive object comprises a control vector of at least one partial region of the interactive object, and when the control parameter value of the interactive object is acquired based on the driving data, the control parameter value of the interactive object is acquired. The second acquisition unit specifically acquires the control vector of at least one partial region of the interactive object based on the drive data, and the drive unit specifically acquires the at least one portion. The facial movement and / or limb movement of the interactive object is controlled based on the control vector of the target area.

According to one aspect of the invention, an electronic device is provided that comprises the device memory and a processor, the memory storing computer instructions operable on the processor, the processor executing the computer instructions. Occasionally, the interactive object driving method described in any of the embodiments provided by the present invention is realized.

According to one aspect of the invention, a computer-readable recording medium in which a computer program is stored is provided, and when the program is executed by a processor, the interactive object according to any embodiment provided by the present invention. The drive method is realized.

At least one embodiment of the present specification further provides an electronic device, which, as shown in FIG. 7, comprises a memory and a processor, the memory storing computer instructions that can be run on the processor. , The processor realizes the method of driving an interactive object according to any embodiment of the present invention when the computer instruction is executed.

At least one embodiment of the present invention further provides a computer-readable recording medium in which a computer program is stored and is an interactive subject according to any embodiment of the invention when the program is executed by a processor. The drive method is realized.

Those skilled in the art should appreciate that one or more embodiments of the invention may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the invention may use the form of complete hardware embodiments, complete software embodiments, or a combination of software and hardware embodiments. Also, one or more embodiments of the present invention include, but include, one or more computer-usable recording media (disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. You can use the format of the computer program product implemented on (but not limited to).

Each of the embodiments in the present invention is described using a gradual method, the same or similar parts between the embodiments can be referred to each other, and the other embodiments can be referred to each other. The explanation focused on the differences from the examples. In particular, in the case of the example of the data processing device, since it is basically similar to the example of the method, it is described relatively briefly, but for the relevant part, refer to the explanation of a part of the example of the method. do it.

Specific embodiments of the invention have been described above. Other examples are within the scope of the attached "claims". In some cases, the actions or steps described in the claims may be performed in a different order than in the examples, and the expected result may still be achieved. Also, the process depicted in the drawings does not necessarily require the specific order or sequential order shown to achieve the expected result. In some embodiments, multitasking and parallel processing are also possible or may be beneficial.

Examples of the subject matter and functional operation in the present invention are digital electronic circuits, tangible computer software or firmware, computer hardware including the configurations and structural equivalents thereof disclosed in the present invention, or one or more thereof. It can be realized by combination. The embodiments of the subject in the present invention can be realized as one or more computer programs, i.e., encoded on a tangible non-temporary program carrier and executed by a data processing apparatus. , Can be implemented as one or more modules in a computer program instruction to control the operation of the data processing device. Alternatively or additionally, the program instruction can be encoded on a manually generated propagation signal, for example, on a machine-generated electrical, optical, or electromagnetic signal. can. The signal is generated to encode the information and transmit it to the appropriate receiver device for execution by the data processing device. The computer recording medium can be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or a combination thereof.

The processing and logical flow in the present invention can be performed by one or more programmable computers running one or more computer programs, performing operations based on input data to produce output. By doing so, the corresponding function is executed. The processing and logic flow can be further executed by a dedicated logic circuit such as FPGA (field programmable gate array) or ASIC (dedicated integrated circuit), and the device can also be realized as a dedicated logic circuit. Can be done.

Suitable computers for running computer programs include, for example, general purpose and / or dedicated microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. In general, a computer further includes or is operable with one or more mass storage devices for storing data, such as magnetic disks, magnetic optical disks, or optical disks. Combined with a device to receive data, transmit data, or both. However, computers do not necessarily have such devices. The computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or general purpose serial bus. It can be embedded in portable storage devices such as (USB) flash drives, and these devices are just a few examples.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, intermediaries, and memory devices, such as semiconductor memory devices (eg, EPROMs, EEPROMs, and flash devices). Includes magnetic discs (eg, internal hard disks or mobile discs), magnetic optical discs, and CD ROMs, and DVD-ROM discs. Processors and memory can be complemented by dedicated logic circuits or incorporated into dedicated logic circuits.

The present invention contains many specific implementation details, which should not be construed as limiting the scope of the invention or the scope of which it seeks to protect, primarily of some embodiments of the invention. Used to describe features. Specific features in a plurality of embodiments of the present invention may also be implemented in combination with a single embodiment. On the other hand, the various features in a single embodiment can be implemented separately in multiple embodiments or in any suitable sub-combination. It should be noted that the features play a role in a particular combination as described above and are claimed to be protected in this way from the beginning, but one or more features from the combination claimed to be protected may be excluded from the combination in some cases. And the combinations claimed to be protected can be directed to sub-combinations or variants from sub-combinations.

Similarly, the drawings depict operations in a particular order, which either requires them to be performed in a specific order or requires them to be performed in sequence, or all the operations illustrated. It should not be understood that it is a requirement that the expected result be achieved by being carried out. In some cases, multitasking and parallel processing can be advantageous. It should not be understood that the separation of the various system modules and components in the above embodiments must be separated in this way in all embodiments, and the described program components and systems are: In general, it should be understood that they can be integrated together into a single software product or packaged into multiple software products.

Therefore, specific examples of the subject have already been described. Other examples are within the scope of the attached "Claims". In some cases, the actions described in the claims may be performed in different orders and still achieve the expected results. It should be noted that the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the expected results. Multitasking and parallel processing can be beneficial in some implementations.

The above are only a few embodiments of the invention and are not used to limit the invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention should be included within the scope of one or more embodiments of the invention.

Claims (20)

It is a driving method for interactive objects displayed on display devices.
Acquiring the drive data of the interactive target and determining the drive mode of the drive data,
Acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode,
A method for driving an interactive object, which comprises controlling the appearance of the interactive object based on the control parameter value. The driving method for an interactive object according to claim 1, further comprising controlling the display device output voice and / or displaying text based on the driving data. Determining the drive mode corresponding to the drive data is
Acquiring a voice data sequence corresponding to the drive data based on the type of drive data, wherein the voice data sequence includes a plurality of voice data units.
In response to the detection of the target data included in the voice data unit, the drive mode of the drive data is determined as the first drive mode, and the target data is a predetermined object of the interactive object. Corresponding to control parameter values, including
Acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode
The interactive according to claim 1 or 2, wherein the predetermined control parameter value corresponding to the target data is used as the control parameter value of the interactive target in response to the first drive mode. The driving method of the target. The target data includes a key word or a key character, and the key word or the key character corresponds to a predetermined control parameter value of a predetermined operation of the interactive target, or.
The driving method for an interactive object according to claim 3, wherein the target data includes a syllable, and the syllable corresponds to a predetermined control parameter value of a predetermined mouth shape operation of the interactive object. Determining the drive mode of the drive data is
Acquiring a voice data sequence corresponding to the drive data based on the type of drive data, wherein the voice data sequence includes a plurality of voice data units.
In response to the fact that the target data included in the voice data unit is not detected, the drive mode of the drive data is determined as the second drive mode, and the target data is the interactive target. Corresponding to a given control parameter value, including
Acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode
Acquiring the feature information of at least one voice data unit in the voice data sequence in response to the second drive mode,
The driving method for an interactive object according to any one of claims 1 to 4, wherein the control parameter value of the interactive object corresponding to the feature information is acquired, and the control parameter value of the interactive object is acquired. The audio data sequence includes a phoneme sequence.
Acquiring the feature information of at least one voice data unit in the voice data sequence is
Performing feature encoding on the phoneme sequence to obtain a first chord sequence corresponding to the phoneme sequence,
Acquiring a feature code corresponding to at least one phoneme based on the first chord sequence,
The driving method for an interactive object according to claim 5, wherein the feature information of at least one phoneme is obtained based on the feature code, and the feature information is included. The audio data sequence includes an audio frame sequence.
Acquiring the feature information of at least one voice data unit in the voice data sequence is
Acquiring the first acoustic feature sequence corresponding to the voice frame sequence, the first acoustic feature sequence includes the acoustic feature vector corresponding to each voice frame in the voice frame sequence.
Acquiring an acoustic feature vector corresponding to at least one audio frame based on the first acoustic feature sequence,
The driving method for an interactive object according to claim 5, wherein the feature information corresponding to the at least one voice frame is obtained based on the acoustic feature vector. The control parameter of the interactive object includes a facial appearance parameter, the facial appearance parameter includes a facial muscle control coefficient, and the facial muscle control coefficient controls the motor state of at least one facial muscle. Used for
Acquiring the control parameter value of the interactive object based on the driving data is
Including acquiring the facial muscle control coefficient of the interactive object based on the driving data.
Controlling the appearance of the interactive object based on the control parameter value
One of claims 1 to 7, wherein the interactive object drives the interactive object so as to perform a facial motion matching the driving data based on the acquired facial muscle control coefficient. The method of driving the interactive object described in the section. Acquiring the drive data of the body shape related to the face shape parameter, and
The interactive object according to claim 8, further comprising driving the interactive object to perform a limb movement based on the driving data of the body condition related to the facial appearance parameter value. Driving method. The control parameters of the interactive object include a control vector of at least one partial region of the interactive object.
Acquiring the control parameter value of the interactive object based on the driving data is
Including acquiring the control vector of at least one partial region of the interactive object based on the driving data.
Controlling the appearance of the interactive object based on the control parameter value
6. How to drive an interactive object. Acquiring the control parameter value of the interactive object corresponding to the feature information is
The driving of an interactive object according to claim 5, wherein the feature information is input to a pre-trained recurrent neural network to obtain a control parameter value of the interactive object corresponding to the feature information. Method. An interactive drive device on display in a display device.
A first acquisition unit for acquiring the driving data of the interactive target and determining the driving mode of the driving data, and
A second acquisition unit for acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode.
A drive device for an interactive object, comprising: a drive unit for controlling the appearance of the interactive object based on the control parameter value. 12. The interactive drive device according to claim 12, further comprising an output unit for controlling the display device output voice and / or displaying text based on the drive data. When determining the drive mode corresponding to the drive data,
The first acquisition unit is
Based on the type of drive data, a voice data sequence corresponding to the drive data is acquired, wherein the voice data sequence comprises a plurality of voice data units.
Further, in response to the detection of the target data included in the voice data unit, the drive mode of the drive data is determined as the first drive mode, where the target data is a predetermined interactive target. Corresponds to the control parameter value of
Acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode
In response to the first drive mode, the predetermined control parameter value corresponding to the target data is used as the control parameter value of the interactive object.
Here, the target data includes a key word or a key character, and the key word or the key character corresponds to a predetermined control parameter value of a predetermined operation of the interactive target, or.
The driving device for an interactive object according to claim 12, wherein the target data includes a syllable, and the syllable corresponds to a predetermined control parameter value of a predetermined mouth shape operation of the interactive object. When determining the drive mode of the drive data,
The first acquisition unit is
Based on the type of drive data, a voice data sequence corresponding to the drive data is acquired, wherein the voice data sequence comprises a plurality of voice data units.
Further, in response to the fact that the target data included in the voice data unit is not detected, the drive mode of the drive data is determined as the second drive mode, and the target data is a predetermined object of the interactive target. Corresponds to the control parameter value,
Acquiring the control parameter value of the interactive object based on the drive data in response to the drive mode
Acquiring the feature information of at least one voice data unit in the voice data sequence in response to the second drive mode,
The driving device for an interactive object according to any one of claims 12 to 14, further comprising acquiring a control parameter value of the interactive object corresponding to the feature information. The audio data sequence includes a phoneme sequence.
When acquiring the feature information of at least one voice data unit in the voice data sequence,
The second acquisition unit is
Feature encoding is performed on the phoneme sequence to obtain a first chord sequence corresponding to the phoneme sequence.
The feature code corresponding to at least one phoneme is acquired based on the first chord sequence, and the feature code is obtained.
Based on the feature code, the feature information of the at least one phoneme is obtained.
or,
The audio data sequence includes an audio frame sequence.
When acquiring the feature information of at least one voice data unit in the voice data sequence,
The second acquisition unit is
The first acoustic feature sequence corresponding to the voice frame sequence is acquired, wherein the first acoustic feature sequence includes an acoustic feature vector corresponding to each voice frame in the voice frame sequence.
Further, an acoustic feature vector corresponding to at least one voice frame is acquired based on the first acoustic feature sequence, and the sound feature vector is acquired.
The driving device for an interactive object according to claim 15, wherein the feature information corresponding to the at least one voice frame is obtained based on the acoustic feature vector. The control parameter of the interactive object includes a facial appearance parameter, the facial appearance parameter includes a facial muscle control coefficient, and the facial muscle control coefficient controls the motor state of at least one facial muscle. Used for
When acquiring the control parameter value of the interactive object based on the driving data,
The second acquisition unit is
The facial muscle control coefficient of the interactive object is acquired based on the driving data, and the face muscle control coefficient is acquired.
The drive unit is
The interactive object is driven so that the interactive object performs a facial motion matching the driving data based on the acquired facial muscle control coefficient.
The drive device of the interactive object acquires the drive data of the body shape related to the face shape parameter, and the interactive target is based on the drive data of the body shape related to the face shape parameter value. The interactive object driving device according to any one of claims 12 to 16, further comprising a limb driving unit for driving to perform limb movements. The control parameters of the interactive object include a control vector of at least one partial region of the interactive object.
When acquiring the control parameter value of the interactive object based on the driving data,
The second acquisition unit is
Based on the driving data, the control vector of at least one partial region of the interactive object is acquired, and the control vector is acquired.
The drive unit is
The interactive object according to any one of claims 12 to 16, wherein the face movement and / or the limb movement of the interactive object is controlled based on the acquired control vector of the at least one partial region. Drive device. It ’s an electronic device,
Equipped with memory and processor,
The memory stores computer instructions that can be operated on the processor.
An electronic device, wherein the processor performs the method according to any one of claims 1 to 11 when the computer instruction is executed. A computer-readable recording medium that stores computer programs.
A computer-readable recording medium, wherein the method according to any one of claims 1 to 11 is executed when the computer program is executed by a processor.

Images