JP2022531056A - Interactive target drive methods, devices, devices, and recording media - Google Patents

Abstract

Disclosed is an interactive object driving method, apparatus, device, and recording medium, the method comprising: obtaining audio-driven data for an interactive object displayed on a display device; Obtaining a control parameter sequence of a predetermined action of an interactive object that matches the target data based on the data, and controlling the interactive object to perform the predetermined action based on the obtained control parameter sequence. including doing and [Selection drawing] Fig. 2

Description

The present invention relates to the field of computer technology, specifically, to driving methods, devices, devices, and recording media for interactive objects.

Most of the methods of human-computer interaction are that the user inputs by keystroke, touch, or voice, and the device responds by displaying an image, text, or virtual character on the screen. Currently, virtual characters are being improved primarily based on voice assistants. User-virtual character interactivity is still on the surface.

The embodiments of the present invention provide a technical solution for driving an interactive object.

According to one aspect of the invention, a method of driving an interactive object is provided, wherein the method obtains the voice-driven data of the interactive object displayed on the display device and the target included in the voice-driven data. Based on the data, the control parameter sequence of the predetermined operation of the interactive object matching the target data is acquired, and the interactive object is controlled to perform the predetermined operation based on the obtained control parameter sequence. Including to do.

Combined with any embodiment provided by the present invention, the method controls the display device to output audio based on the audio information corresponding to the audio driven data, and / or said. It further includes displaying texts based on the textual information corresponding to the voice-driven data.

In combination with any embodiment provided by the present invention, controlling the interactive object to perform the predetermined operation based on the obtained control parameter sequence is a voice corresponding to the target data. Based on the determination of information, the acquisition of time information for outputting the voice information, the determination of the execution time of a predetermined operation corresponding to the target data based on the time information, and the execution time. The control parameter sequence corresponding to the target data is used to control the interactive object to perform the predetermined operation.

Combined with any embodiment provided by the present invention, the control parameter sequence comprises one or more sets of control parameters, and based on the execution time, by the control parameter sequence corresponding to the target data. Controlling the interactive object to perform the predetermined operation calls the control parameters of each set in the control parameter sequence at a predetermined speed, and the interactive target corresponds to the control parameters of each set. Includes having to exhibit.

Combined with any embodiment provided by the present invention, the control parameter sequence comprises one or more sets of control parameters, and based on the execution time, by the control parameter sequence corresponding to the target data. Controlling the interactive object to perform the predetermined operation determines the calling speed of the control parameter sequence based on the execution time, and controls each set of control parameters in the control parameter sequence. It includes calling at the calling speed so that the interactive object outputs a state corresponding to each set of control parameters.

Combined with any embodiment provided by the present invention, it is possible to control the interactive object to perform the predetermined operation by a control parameter sequence corresponding to the target data based on the execution time. , Including starting to call the control parameter sequence corresponding to the target data at a predetermined time point before outputting the voice information corresponding to the target data so that the interactive object starts to perform the predetermined operation. ..

Combined with any embodiment provided by the present invention, the voice driven data comprises a plurality of target data so that the interactive object performs the predetermined operation based on the obtained control parameter sequence. Controlling to is a control parameter sequence corresponding to the previously arranged target data according to word order in response to the detection that adjacent target data in the plurality of target data overlap. To control the interactive object to perform the predetermined operation based on the above.

Combined with any embodiment provided by the present invention, the voice driven data includes a plurality of target data, and the interactive object performs the predetermined operation based on a control parameter sequence corresponding to the target data. Controlling to do so corresponds to the adjacent target in response to the detection that the execution times of the control parameter sequences corresponding to the adjacent target data in the plurality of target data overlap. Includes fusing overlapping parts of the control parameter sequence corresponding to the data.

Combined with any embodiment provided by the present invention, based on the target data contained in the voice-driven data, obtaining a control parameter sequence of a predetermined operation of the interactive object matching the target data. Executes voice recognition for the audio data in response to the voice-driven data including audio data, and determines the target data included in the audio data based on the recognized voice content. In response to the voice-driven data including the text data, the target data contained in the text data is determined based on the text content contained in the text data. include.

Combined with any embodiment provided by the present invention, the voice-driven data includes syllable data and is an interactive object that matches the target data based on the target data contained in the voice-driven data. Acquiring a control parameter sequence of a predetermined operation is to determine whether or not the syllable data included in the voice-driven data is matched with the target tuned data, and the target tuned data is obtained in advance. It belongs to the divided different tune types, the different tune types correspond to the different predetermined mouth shapes, the control parameter sequences corresponding to the different predetermined mouth shapes are set, and the tune data. In response to being matched to the target syllable data, a predetermined mouth shape control parameter sequence corresponding to the matched target syllable data based on the syllable type to which the matched target syllable data belongs. To get and include.

Combined with any embodiment provided by the present invention, the method is to acquire first data other than the target data in the voice driven data and to acquire the acoustic features of the first data. Further, acquisition of a shape control parameter matching the acoustic feature, and control of the shape of the interactive object based on the shape control parameter are further included.

According to one aspect of the present invention, a drive device for an interactive object is submitted, and the device is used for a first acquisition unit for acquiring voice drive data for an interactive object displayed on a display device, and the voice drive data. A second acquisition unit for acquiring a control parameter sequence of a predetermined operation of an interactive object matching the target data based on the included target data, and the interactive object based on the obtained control parameter sequence. A drive unit for controlling to perform the predetermined operation is provided.

Combined with any embodiment provided by the present invention, the device controls the display device to output audio based on the audio information corresponding to the audio driven data, and / or the audio. It is further equipped with an output unit for displaying text based on the text information corresponding to the drive data.

Combined with any embodiment provided by the present invention, the drive unit specifically determines audio information corresponding to the target data, acquires time information for outputting the audio information, and obtains the time information. Based on the information, the execution time of the predetermined operation corresponding to the target data is determined, and based on the execution time, the interactive target executes the predetermined operation by the control parameter sequence corresponding to the target data. To control.

Combined with any embodiment provided by the present invention, the control parameter sequence comprises one or more sets of control parameters, and the drive unit corresponds to the target data based on the execution time. When the interactive object is controlled to perform the predetermined operation by the control parameter sequence, specifically, each set of control parameters in the control parameter sequence is called at a predetermined speed, and the interactive object causes the interactive object. The appearance corresponding to each set of control parameters will be exhibited.

Combined with any embodiment provided by the present invention, the control parameter sequence comprises one or more sets of control parameters, and the drive unit corresponds to the target data based on the execution time. When the interactive object is controlled to perform the predetermined operation by the control parameter sequence, specifically, the calling speed of the control parameter sequence is determined based on the execution time, and the control parameter sequence is in the control parameter sequence. The control parameters of each set are called at the calling speed so that the interactive target outputs the state corresponding to the control parameters of each set.

Combined with any embodiment provided by the present invention, the control parameter sequence comprises one or more sets of control parameters, and the drive unit corresponds to the target data based on the execution time. When the interactive object is controlled to perform the predetermined operation by the control parameter sequence, specifically, the target data corresponds to the target data at a predetermined time before outputting the voice information corresponding to the target data. Start calling the control parameter sequence to make the interactive object start performing the predetermined action.

Combined with any embodiment provided by the present invention, the voice drive data may include a plurality of target data, and the drive unit may specifically include adjacent target data in the plurality of target data. In response to the detection of overlap, the interactive object is controlled to perform the predetermined action based on the control parameter sequence corresponding to the target data previously arranged according to the word order. ..

Combined with any embodiment provided by the present invention, the voice-driven data comprises a plurality of target data, and the drive unit specifically comprises adjacent target data in the plurality of target data. In response to the detection that the execution times of the corresponding control parameter sequences overlap, the overlapping parts of the control parameter sequences corresponding to the adjacent target data are fused.

Combined with any embodiment provided by the present invention, the second acquisition unit specifically performs voice recognition on the audio data in response to the voice driven data including audio data. Execute, determine the target data included in the audio data based on the audio content contained in the audio data, and respond to the audio driven data including the text data to the text data. Based on the contained text content, the target data included in the text data is determined.

Combined with any embodiment provided by the present invention, the voice-driven data includes syllable data, and the second acquisition unit specifically targets the syllable data contained in the voice-driven data. It is determined whether or not it is matched with the syllable data, where the target syllable data belongs to the pre-divided and different syllable types, and the different syllable types correspond to different predetermined mouth shapes and are different from each other. The corresponding control parameter sequence is set for the predetermined mouth shape, and the syllable to which the matched target syllable data belongs in response to the matching of the syllable data with the target syllable data. Based on the type, a predetermined mouth shape control parameter sequence corresponding to the matched target syllable data is acquired.

Combined with any embodiment provided by the present invention, the apparatus acquires first data other than the target data in the voice-driven data, acquires the acoustic features of the first data, and the first. 1 Further, a shape control unit for acquiring a shape control parameter matching the acoustic characteristics of the data and controlling the shape of the interactive target based on the shape control parameter is further provided.

According to one aspect of the invention, an electronic device is provided, the device comprising a memory and a processor, the memory storing computer instructions operable on the processor, the processor executing the computer instructions. When done, the interactive object driving method described in any of the embodiments provided by the present invention is realized.

According to one aspect of the invention, a computer-readable recording medium in which a computer program is stored is provided, and when the program is executed by a processor, the interactive object according to any embodiment provided by the present invention. The drive method is realized.

According to the method, apparatus, device, and computer-readable recording medium of the interactive object of one or more embodiments of the present invention, at least one of the voice-driven data of the interactive object displayed on the display device. Based on the target data, the interactive target is voice-driven data by acquiring control parameters of a predetermined motion of the interactive target matching the target data and controlling the motion of the interactive target displayed on the display device. The action corresponding to the target data contained in is made to perform the action corresponding to the target data, the speaking state of the interactive target is made natural and vivid, and the interactive experience of the target target is improved.

It is a schematic diagram of the display device in the driving method of an interactive object provided by the Embodiment of this invention. It is a flowchart of the driving method of an interactive object provided by the Example of this invention. It is a flowchart of the driving method of an interactive object provided by the Example of this invention. It is a flowchart of the driving method of an interactive object provided by the Example of this invention. It is a schematic diagram of the structure of the drive device of an interactive object provided by the Example of this invention. It is a schematic diagram of the structure of the electronic device provided by the Example of this invention.

Hereinafter, exemplary embodiments will be described in detail and examples will be shown in the drawings. Where the following description refers to drawings, the same numbers in different drawings indicate the same or similar elements, unless otherwise stated. The embodiments described in the following exemplary examples do not represent all embodiments consistent with the present invention. Conversely, they are merely examples of devices and methods consistent with some aspects of the invention described in the appended claims.

The term "and / or" as used herein merely describes the relationship of a related object and indicates that three relationships can exist, for example, A and / or B, where A is present alone. There are three relationships, such as the existence of A and B at the same time, and the existence of B alone. Further, the term "at least one kind" in the present specification refers to any one kind in a plurality of kinds or any combination of at least two kinds in a plurality of kinds, for example, in A, B, C. Inclusion of at least one indicates that it comprises any one or more elements selected from a set composed of A, B, and C.

At least one embodiment of the present invention provides a driving method for an interactive object, which driving method can be performed by an electronic device such as a terminal device or a server. The terminal device may be a fixed terminal or a mobile terminal such as a mobile phone, a tablet personal computer, a game machine, a desktop personal computer, an advertising machine, an all-in-one machine, and an in-vehicle terminal. The server includes a local server, a cloud server, and the like. The method can be realized by a method of calling a computer-readable instruction stored in a memory by a processor.

In an embodiment of the invention, the interactive object can be any virtual image capable of performing interactive with the target object. In one embodiment, the interactive object may be a virtual character, and may be another virtual image capable of realizing an interactive function, such as a virtual animal, a virtual article, or a cartoon image. The display format of the interactive object is 2D or 3D, but the present invention is not limited thereto. The target can be a user, a robot, or other smart device. The interactive method of the interactive object with the target object may be an active interactive method or a passive interactive method. In one example, depending on the target object, a request can be made by performing a gesture or a limb movement, and the interactive object can be triggered and interactively performed by an active interactive method. In another example, the interactive object can be made to interact with the interactive object by a passive method by actively greeting and prompting the target object to perform an action or the like.

The interactive object can be exhibited using a terminal device, and the terminal device includes a television, an all-in-one device having a display function, a projector, a virtual reality (VR) device, and an augmented reality (AR). ) The present invention may be a device or the like, and the present invention is not limited to a specific form of the terminal device.

FIG. 1 shows a display device according to an embodiment of the present invention. As shown in FIG. 1, the display device has a display screen, and by displaying a stereoscopic image on the display screen, a virtual scene having a stereoscopic effect and an interactive object can be represented. For example, the interactive object displayed on the display screen of FIG. 1 includes a virtual cartoon person.

The electronic device according to the present invention includes a built-in display, which can be used to display a stereoscopic image to represent a virtual scene and an interactive object. In some other embodiments, the electronic device described in the present invention may not include a built-in display, and the content that needs to be displayed can be displayed on an external display via a wired or wireless connection for virtual scenes and interactive objects. Can be notified to display.

In some embodiments, in response to the electronic device receiving voice-driven data to drive the interactive object to output audio, the interactive object delivers the audio specified for the target object. Can be emitted. The terminal device generates voice-driven data based on the movement, facial expression, status, preference, etc. of the target object around the terminal device so that the interactive target emits a specified voice to interact or respond. By driving, it is possible to provide anthropomorphic services to the target target. In the process of interacting with the target object, the interactive object is driven to emit a specified voice based on the voice-driven data, and at the same time, the interactive target performs a face motion in synchronization with the specified voice. It may not be able to be driven by, and it becomes dull and unnatural when the interactive object emits sound, which affects the target object and the interactive experience. In view of this, in at least one embodiment of the invention, the interactive object driving method enhances the interactive experience with the interactive object of the target object.

FIG. 2 is a flowchart of a driving method of an interactive object according to an embodiment of the present invention, and as shown in FIG. 2, the method includes steps 201 to 203.

In step 201, the voice-driven data of the interactive object displayed on the display device is acquired.

In the embodiment of the present invention, the voice-driven data may include audio data (voice data), text data, and the like. The voice-driven data may be drive data generated based on the movement, facial expression, status, preference, etc. of the target object that interacts with the interactive object by the terminal device, or may be voice-driven data called from the internal memory by the electronic device. It may be directly acquired drive data such as data. The present invention is not limited to the method of acquiring the voice-driven data.

In step 202, based on the target data included in the voice-driven data, a control parameter sequence of a predetermined operation of the interactive target matching with the target data is acquired, and here, the control parameter sequence is a set. Or it contains multiple sets of control parameters.

In the embodiment of the present invention, the target data is data that is pre-matched with a predetermined operation, and the predetermined operation is controlled and realized based on the corresponding control parameter sequence. Therefore, the target data is described above. Matches to a control parameter sequence of predetermined operation. The target data may be a predetermined keyword, word, sentence, or the like. Taking the example where the keyword is "hand gesture", when the voice-driven data includes text data, the target data corresponding to the "hand gesture" is the text data "hand gesture" and / or the voice. When the drive data includes audio or syllable data, the target data corresponding to the "hand gesture" can be the voice data "hand gesture". When the voice-driven data is matched with the target data described above, it can be determined that the voice-driven data includes the target data.

The predetermined operation can be realized by using a general-purpose unit moving image, the unit moving image may include an image frame sequence, and each image frame in the sequence corresponds to one form of the interactive object. , Corresponding changes in appearance between image frames can allow the interactive object to achieve a given action. Here, the appearance of the interactive object in one image frame can be realized by using a set of control parameters, for example, using a set of control parameters formed by the displacement of a plurality of bone points. Can be realized. Therefore, it is possible to control the change in the appearance of the interactive object by using the control parameter sequence formed by a plurality of sets of control parameters, and it is possible to control the interactive object so as to realize a predetermined operation.

In some embodiments, the target data may include target syllable data, the target syllable data corresponds to a predetermined mouth shape control parameter, and one type of target syllable data is pre-divided and different syllables. The syllable types that belong to the type and are different from each other correspond to the predetermined mouth shapes that are different from each other, and the control parameter sequences corresponding to the predetermined mouth shapes that are different from each other are set.

The syllable data may be a voice unit formed by combining at least one phoneme, and the syllable data may be syllable data in a pin-in language and syllable data in a non-pin-in language (for example, Chinese). Different syllable types can be syllable data that match or essentially match the pronunciation action, and different syllable types correspond to one type of motion of the interactive object. Specifically, the different syllable types correspond to different predetermined mouth shapes as the interactive object speaks, i.e., to correspond to one type of pronunciation action. In this case, the same type of syllable data is matched to a predetermined same type of mouth shape control parameter sequence. For example, pinyin types of syllable data such as "ma", "man", and "mang" can be regarded as the same type because the pronunciation behavior of this type of syllable data basically matches. Corresponds to the mouth-shaped control parameter sequence of "open mouth" when the interactive object speaks. In this case, if it is detected that this type of target syllable data is present in the voice-driven data, the interactive target will perform the corresponding mouth shape based on the mouth shape control parameter sequence that matches the target syllable data. Can be controlled to. Further, it is possible to match a plurality of different types of mouth shape control parameter sequences based on a plurality of types of syllable data, and control changes in the mouth shape of an interactive target by using the plurality of control parameter sequences. And it is possible to control the realization of anthropomorphic speech states of interactive objects.

In step 203, the interactive object is controlled to perform the predetermined operation based on the obtained control parameter sequence.

For one or more target data included in the voice-driven data, it is possible to obtain a control parameter sequence of the corresponding predetermined operation. By controlling the operation of the interactive object based on the obtained control parameter sequence, it is possible to realize a predetermined operation corresponding to each target data in the voice-driven data.

In an embodiment of the present invention, a control parameter sequence of a predetermined operation of an interactive object matching the target data is acquired based on the target data included in the voice-driven data of the interactive object displayed on the display device. By controlling the motion of the interactive object displayed on the display device, the interactive object is made to perform the motion corresponding to the target data included in the voice-driven data, and the talking state of the interactive object is naturally made. Brilliant and improved the targeted interactive experience.

FIG. 3 is a flowchart of a driving method for an interactive object according to an embodiment of the present invention, and as shown in FIG. 3, the method includes the following steps.

In step 204, the display device is controlled to output voice based on the voice information corresponding to the voice-driven data, and / or the text is displayed based on the text information corresponding to the voice-driven data. To control.

The display device controls to output the voice corresponding to the voice-driven data, and at the same time, the interactive object performs the corresponding action based on a control parameter sequence matching each target data in the voice-driven data. By controlling in order, the interactive target can output the voice and at the same time perform the action based on the content contained in the voice, making the spoken state of the interactive target natural and vivid, and the target target. Improved the interactive experience.

Further, at the same time as controlling the display device to output the voice corresponding to the voice-driven data, the text corresponding to the voice-driven data can be displayed on the display device, and each of the texts in the voice-driven data can be displayed. By sequentially controlling the interactive object to perform the corresponding action based on a control parameter sequence that matches the target data, the interactive object outputs the voice and displays the text, and at the same time, it is included in the voice and the text. You can make the action based on what you are doing, make the state expressed by the interactive object natural and vivid, and improve the interactive experience of the target object.

In the embodiment of the present invention, the image frame sequence corresponding to the variable content can be configured only by setting the control parameter sequence for the specified operation, and the driving efficiency of the interactive object is improved. By increasing or modifying the target data as needed, it is possible to respond to changing contents, and maintenance and updating of the drive system becomes easy.

In some embodiments, the method applies to a server, which includes a local server, a cloud server, and the like. The server executes processing on the voice-driven data of the interactive target to generate the appearance parameter value of the interactive target, and utilizes a three-dimensional or two-dimensional rendering engine based on the appearance parameter value. It can be rendered to obtain the response video of the interactive object. The server can respond to the target target by transmitting the response video to the terminal device and displaying it, and further, by transmitting the response video to the cloud, the terminal device acquires the response video from the cloud. And respond to the target. After the server generates the appearance parameter value of the interactive object, the server further sends the appearance parameter value to the terminal so that the terminal completes the process of performing rendering, generating a moving image, executing an exhibition, and the like. To.

In some embodiments, the method is applied to a terminal device, which performs a process on the voice-driven data of the interactive object to generate a figure parameter value of the interactive object, said the figure. Rendering is performed using a three-dimensional or two-dimensional rendering engine based on the parameter value to obtain the response video of the interactive target, and the terminal can respond to the target target by displaying the response video. ..

In response to the voice-driven data including audio data, voice recognition is performed on the voice-driven data to obtain the voice content contained in the audio data, and the target data included in the audio data is obtained. Can be confirmed. By matching the voice content with the target data, the target data included in the voice drive data can be determined.

In response to the voice-driven data including the text data, the target data included in the text data can be determined based on the text content included in the text data.

In some embodiments, when the voice-driven data includes syllable data, the voice-driven data can be divided to obtain at least one syllable data. Those skilled in the art can obtain different combinations of syllable data by different division methods, and the method of dividing voice-driven data is not limited to one method, and by setting priorities for different division methods. It should be understood that the combination of syllable data obtained by the high priority division method can be used as the division result.

The syllable data obtained by division is matched with the target syllable data, and the syllable data is matched with the target syllable data in response to the matching of the syllable data with the target syllable data of any syllable type. And further, it can be determined that the voice-driven data includes the target data. For example, the target syllable data may include syllable data of types such as "ma", "man", "mang", and the voice driven data may be any of "ma", "man", "mang". In response to including one matching syllable data, it can be determined that the voice driven data includes the target syllable data.

When the voice-driven data includes the target syllable data, a predetermined mouth shape control parameter sequence corresponding to the target syllable data is acquired based on the syllable type to which the target syllable data belongs, and the interactive target corresponds to the target syllable data. It can be controlled to form a mouth shape. By the method described above, it is possible to control the change in the mouth shape of the interactive object based on the control parameter sequence of the mouth shape corresponding to the voice-driven data, so that the interactive object realizes an anthropomorphic utterance state. Can be done.

The syllable data obtained by dividing can be a plurality of syllable data. For each syllable data in a plurality of syllable data, it is searched whether or not the syllable data is matched with a specific target syllable data, and when the syllable data is matched with a specific target syllable data, the said It is possible to acquire a control parameter sequence of a predetermined mouth shape corresponding to the target syllable data.

In some embodiments, as shown in FIG. 4, step 203 further comprises the following steps:

In step 2031, the voice information corresponding to the target data is determined.

In step 2032, the time information for outputting the voice information is acquired.

In step 2033, the execution time of a predetermined operation corresponding to the target data is determined based on the time information.

In step 2034, based on the execution time, the interactive target is controlled to perform the predetermined operation in the control parameter sequence corresponding to the target data.

When the display device is controlled to output voice based on the voice information corresponding to the voice-driven data, the time information for outputting the voice information corresponding to the target data can be determined. For example, the time for starting the output of the voice information corresponding to the target data, the time for ending the output, and the time length can be determined. The execution time of a predetermined operation corresponding to the target data can be determined based on the time information, and within the execution time or within a specific range of the execution time, by the control parameter sequence corresponding to the target data. The interactive object can be controlled to perform the predetermined operation.

In the embodiment of the present invention, the time length for outputting the voice based on the voice-driven data coincides with the time length for controlling the interactive object to perform a predetermined continuous operation based on a plurality of control parameter sequences. Or similar. Also, the length of time to output the corresponding audio for each target data matches or is similar to the length of time to control the interactive object to perform a given action based on the corresponding control parameter sequence. be. Therefore, the time during which the interactive object speaks and the time during which the action is performed are matched, and the voice of the interactive object and the action are synchronized and adjusted.

In some embodiments, the control parameters of each set in the control parameter sequence can be called at a predetermined speed so that the interactive object exhibits the appearance corresponding to each set of control parameters. That is, the control parameter sequence corresponding to each target data is always executed at a constant speed.

If the number of phonemes corresponding to the target data is smaller, but the control parameter sequence of a given action that matches the target data is longer, that is, the interactive target spends less time speaking the target data, but the time to perform the action. If is longer, the call to the control parameter sequence can be stopped at the same time that the output of the voice is finished, and the execution of the predetermined operation can be stopped. In addition, the appearance of the end of the predetermined movement and the appearance of the start of the next specified movement are smoothly transitioned so that the movement of the interactive target becomes smooth and natural, and the interactive experience of the target target is improved. did.

In some embodiments, for each target data, the call speed of the control parameter sequence corresponding to the target data is determined based on the execution time of the predetermined operation corresponding to the target data, and the target data is determined. Containing that the control parameters of each set in the corresponding control parameter sequence are called at the calling speed so that the interactive object exhibits the appearance corresponding to each set of control parameters.

If the execution time is shorter, the call speed of the control parameter sequence is relatively high, and vice versa. The call speed of the control parameter sequence determines the speed at which the interactive target operation is performed. For example, when the control parameter sequence is called at a higher speed, the rate of change of the shape of the interactive object is correspondingly faster, and a predetermined operation can be completed in a shorter time.

In some embodiments, the time to perform a predetermined action can be adjusted based on the time of the voice that outputs the target data. For example, the audio and motion of an interactive object are synchronized and coordinated so that the time that the interactive object performs a given action, which can be compressed or expanded, is matched to the time of the audio that outputs the target data. ..

In one example, the control parameter sequence corresponding to the target data can be started to be called based on a predetermined time point before the phoneme voice corresponding to the target data is output, and the interactive target corresponds to the control parameter sequence. It is possible to start performing a predetermined operation.

For example, the interactive object performs a given action by starting to call the control parameter sequence corresponding to the target data in a very short time, for example 0.1 seconds, before the interactive object starts outputting the voice corresponding to the target data. We improved the interactive experience of the target by making the speech of the interactive target more natural and vivid, more closely matched to the state of speech of the real person, as if to start.

In some embodiments, if it is detected that adjacent target data in a plurality of target data overlap, it is according to the word order (that is, the natural arrangement order of the received voice-driven data). ) Targets arranged after overlapping with the target data by controlling the interactive object to perform the corresponding predetermined action based on the control parameter sequence corresponding to the previously arranged target data. Ignore the data.

Each target data contained in the voice-driven data can be stored in the form of an array, and each target data is an element in the array. Note that since different target data can be obtained by combining morphemes according to different methods, it is possible that there may be overlapping parts between two adjacent target data in multiple target data. Should. For example, if the text corresponding to the voice-driven data is "weather is really good", the corresponding target data are "1, heaven", "2, weather", "3, really good", respectively. In the case of adjacent target data 1 and 2, the joint morpheme "heaven" is included in it, and target data 1 and 2 can be matched to the same specified action, for example a finger. Can be matched to an upward pointing motion.

Priority can be set for each target data, and it is possible to determine which of the overlapping target data to be executed based on the priority.

In one example, the priority of the target data that appears first can be set higher than the target data that appears after that. In the case of an example such as "weather is really good", the priority of "heaven" is higher than that of "weather", so that the interactive target is determined based on the control parameter sequence of the predetermined operation corresponding to "heaven". You can control to perform the action and ignore the remaining morpheme "Qi" (ignoring the target data "Weather" that overlaps with the target data "Heaven"), and so on. And directly match "really good".

In the embodiment of the present invention, when adjacent target data overlap, it is possible to set a matching rule to prevent the interactive target from repeatedly executing a predetermined operation.

In some embodiments, when it is detected that the execution times of the control parameter sequences corresponding to the adjacent target data in the plurality of target data overlap, the control corresponding to the adjacent target data is detected. Overlapping parts of the parameter sequence can be fused.

In one embodiment, overlapping control parameter sequences can be fused by performing averaging or weighted averaging on overlapping parts of the control parameter sequences.

In another embodiment, the interpolation method is used to perform a specific frame of the previous operation (for example, the Nth set of control parameters n of the first control parameter sequence corresponding to the operation) according to the transition time. Interpolation transition to the next operation. When it starts to overlap with the first frame in the next operation (for example, whether to search for the first set of control parameters 1 in the second control parameter sequence corresponding to the next operation having the same control parameter n). Or, by interpolating the next operation into the specific frame, the total execution time of the two operations after the interpolation transition and the playback or display time of the corresponding voice data / text data are made the same. ), By ignoring all frames after a particular frame during the previous operation and directly performing the next operation, fusion of overlapping control parameter sequences can be achieved.

By fusing the overlapping parts of the control parameter sequences corresponding to the adjacent target data, it is possible to make a smooth transition between the motions of the interactive object, and the motion of the interactive object is smooth and natural. And improved the interactive experience of the target.

In some embodiments, data other than the respective target data in the voice-driven data can be referred to as, for example, first data. The appearance of the interactive object can be controlled based on the appearance control parameter matching the acoustic feature of the first data.

In response to the voice-driven data including audio data, the voice frame sequence contained in the first data is acquired, the acoustic feature corresponding to at least one voice frame is acquired, and the sound feature corresponds to the sound feature. The appearance of the interactive object can be controlled based on the appearance control parameters of the interactive object, for example, the appearance control vector.

In response to the voice-driven data including text data, the acoustic features corresponding to the phonemes are acquired based on the phonemes corresponding to the morphemes in the text data, and the appearance of the interactive object corresponding to the acoustic features. The shape of the interactive object can be controlled based on a control parameter, for example, a shape control vector.

In the embodiments of the present invention, the acoustic feature may be a feature related to voice emotion, such as a fundamental frequency feature, a common peak feature, a Mel Frequency Cofficient, MFCC, and the like.

Since the shape control parameter value and the voice frame sequence of the voice segment are matched, the voice output based on the first data and / or the exhibited text and the interactive object controlled based on the shape parameter value. When the appearance is synchronized, the appearance performed by the interactive object and the output voice and / or text are synchronized, giving the target object the feeling of talking to the interactive object. In addition, since the shape control vector is related to the acoustic feature that outputs voice, by driving based on the shape control vector, an emotional element is added to the facial expression and limb movement of the interactive target, and the interactive target becomes. The process of speaking has been made more natural and vivid, and the interactive experience of the target has been improved.

In some embodiments, the voice-driven data includes at least one target data and first data other than the target data. For the first data, the appearance control parameters are determined based on the acoustic characteristics of the first data, and the appearance of the interactive object is controlled. With respect to the target data, the interactive target is controlled to perform the predetermined operation based on a control parameter sequence of a predetermined operation matching the target data.

FIG. 5 is a schematic diagram showing a configuration of an interactive object driving device according to at least one embodiment of the present invention, and as shown in FIG. 5, the device is an interactive object audio display displayed on a display device. Based on the first acquisition unit 301 for acquiring drive data and the target data included in the voice drive data, for acquiring a control parameter sequence of a predetermined operation of an interactive target matching the target data. It includes a second acquisition unit 302 and a drive unit 303 for controlling the interactive object to perform the predetermined operation based on the obtained control parameter sequence.

In some embodiments, the device controls the display device to output voice based on the voice information corresponding to the voice driven data, and / or text information corresponding to the voice driven data. It also has an output unit for displaying texts based on it.

In some embodiments, the drive unit specifically determines audio information corresponding to the target data, acquires time information for outputting the audio information, and converts the target data into the target data based on the time information. The execution time of the corresponding predetermined operation is determined, and based on the execution time, the interactive target is controlled to execute the predetermined operation by the control parameter sequence corresponding to the target data.

In some embodiments, the control parameter sequence comprises one or more sets of control parameters, wherein the drive unit is the interactive object with the control parameter sequence corresponding to the target data based on the execution time. Specifically, when controlling to perform the predetermined operation, the control parameters of each set in the control parameter sequence are called at a predetermined speed, and the interactive object corresponds to the control parameters of each set. Try to display the figure.

In some embodiments, the control parameter sequence comprises one or more sets of control parameters, and the drive unit is the interactive object with the control parameter sequence corresponding to the target data based on the execution time. When controlling to execute the predetermined operation, specifically, the calling speed of the control parameter sequence is determined based on the execution time, and each set of control parameters in the control parameter sequence is set as described above. It is called at the calling speed so that the interactive target outputs the state corresponding to each set of control parameters.

In some embodiments, the control parameter sequence comprises one or more sets of control parameters, wherein the drive unit is the interactive object with the control parameter sequence corresponding to the target data based on the execution time. When controlling to perform the predetermined operation, specifically, at a predetermined time before outputting the voice information corresponding to the target data, the control parameter sequence corresponding to the target data is started to be called. Make the interactive object begin to perform the predetermined action.

In some embodiments, the voice-driven data includes a plurality of target data, and the drive unit specifically detects that adjacent target data in the plurality of target data overlap. In response to this, the interactive target is controlled to perform the predetermined operation based on the control parameter sequence corresponding to the target data previously arranged according to the word order.

In some embodiments, the voice-driven data comprises a plurality of target data, the drive unit specifically, execution time of a control parameter sequence corresponding to adjacent target data in the plurality of target data. In response to the detection of overlapping, the overlapping parts of the control parameter sequence corresponding to the adjacent target data are fused.

In some embodiments, the second acquisition unit specifically performs voice recognition on the audio data in response to the voice driving data including audio data, and the recognized voice content. Based on, the target data included in the audio data is determined, and in response to the voice-driven data including the text data, the text data is based on the text content contained in the text data. Establish the included target data.

In some embodiments, the target data includes target syllable data, and the second acquisition unit specifically determines whether or not the syllable data contained in the voice-driven data is matched with the target syllable data. Here, the target syllable data belongs to the pre-divided and different syllable types, and the different syllable types correspond to the different predetermined mouth shapes and the different predetermined mouth shapes. The control parameter sequence to be matched is set, and in response to the matching of the syllable data with the target syllable data, the matching is performed based on the syllable type to which the matching target syllable data belongs. Acquire a control parameter sequence of a predetermined mouth shape corresponding to the target syllable data.

In some embodiments, the apparatus acquires first data other than the target data in the voice-driven data, acquires the acoustic features of the first data, and sets a shape control parameter that matches the acoustic features. Further, a shape control unit for acquiring and controlling the shape of the interactive object based on the shape control parameter is provided.

At least one embodiment of the present specification further provides an electronic device, which, as shown in FIG. 6, comprises a memory and a processor, the memory storing computer instructions that can be run on the processor. , The processor realizes the method of driving an interactive object according to any embodiment of the present invention when the computer instruction is executed. At least one embodiment of the present specification further provides a computer-readable recording medium in which a computer program is stored and is an interactive subject according to any embodiment of the invention when the program is executed by a processor. Realize the driving method of.

Those skilled in the art should appreciate that one or more embodiments of the invention may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the invention may use the form of complete hardware embodiments, complete software embodiments, or a combination of software and hardware embodiments. Also, one or more embodiments of the present invention include, but include, one or more computer-usable recording media (disk memory, CD-ROM, optical memory, etc.) containing computer-usable program code. You can use the format of the computer program product implemented on (but not limited to).

Each of the embodiments in the present invention is described using a gradual method, the same or similar parts between the embodiments can be referred to each other, and the other embodiments can be referred to each other. The explanation focused on the differences from the examples. In particular, in the case of the example of the data processing device, since it is basically similar to the example of the method, it is described relatively briefly, but for the relevant part, refer to the explanation of a part of the example of the method. do it.

Specific embodiments of the invention have been described above. Other examples are within the scope of the attached "claims". In some cases, the actions or steps described in the claims may be performed in a different order than in the examples, and the expected result may still be achieved. Also, the process depicted in the drawings does not necessarily require the specific order or sequential order shown to achieve the expected result. In some embodiments, multitasking and parallel processing are also possible or may be beneficial.

Examples of the subject matter and functional operation in the present invention are digital electronic circuits, tangible computer software or firmware, computer hardware including the configurations and structural equivalents thereof disclosed in the present invention, or one or more thereof. It can be realized by combination. The embodiments of the subject in the present invention can be realized as one or more computer programs, i.e., encoded on a tangible non-temporary program carrier and executed by a data processing apparatus. , Can be implemented as one or more modules in a computer program instruction to control the operation of the data processing device. Alternatively or additionally, the program instruction can be encoded on a manually generated propagation signal, for example, on a machine-generated electrical, optical, or electromagnetic signal. can. The signal is generated to encode the information and transmit it to the appropriate receiver device for execution by the data processing device. The computer recording medium can be a machine-readable storage device, a machine-readable storage board, a random or serial access memory device, or a combination thereof.

The processing and logical flow in the present invention can be performed by one or more programmable computers running one or more computer programs, performing operations based on input data to produce output. By doing so, the corresponding function is executed. The processing and logic flow can be further executed by a dedicated logic circuit such as FPGA (field programmable gate array) or ASIC (dedicated integrated circuit), and the device can also be realized as a dedicated logic circuit. Can be done.

Suitable computers for running computer programs include, for example, general purpose and / or dedicated microprocessors, or any other type of central processing unit. Generally, the central processing unit will receive instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for executing or executing instructions, and one or more memory devices for storing instructions and data. In general, a computer further includes or is operable with one or more mass storage devices for storing data, such as magnetic disks, magnetic optical disks, or optical disks. Combined with a device to receive data, transmit data, or both. However, computers do not necessarily have such devices. The computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or general purpose serial bus. It can be embedded in portable storage devices such as (USB) flash drives, and these devices are just a few examples.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, intermediaries, and memory devices, such as semiconductor memory devices (eg, EPROMs, EEPROMs, and flash devices). Includes magnetic discs (eg, internal hard disks or mobile discs), magnetic optical discs, and CD ROMs, and DVD-ROM discs. Processors and memory can be complemented by dedicated logic circuits or incorporated into dedicated logic circuits.

The present invention contains many specific implementation details, which should not be construed as limiting the scope of the invention or the scope of which it seeks to protect, primarily of some embodiments of the invention. Used to describe features. Specific features in a plurality of embodiments of the present invention may also be implemented in combination with a single embodiment. On the other hand, the various features in a single embodiment can be implemented separately in multiple embodiments or in any suitable sub-combination. It should be noted that the features play a role in a particular combination as described above and are claimed to be protected in this way from the beginning, but one or more features from the combination claimed to be protected may be excluded from the combination in some cases. And the combinations claimed to be protected can be sub-combined or directed to sub-combination variants.

Similarly, the drawings depict operations in a specific order, which either requires them to be performed in a specific order that indicates them, or requires them to be performed in sequence, or all the operations illustrated. It should not be understood that it is a requirement that the expected result be achieved by being carried out. In some cases, multitasking and parallel processing can be advantageous. It should not be understood that the separation of the various system modules and components in the above embodiments must be separated in this way in all embodiments, and the described program components and systems are: In general, it should be understood that they can be integrated together into a single software product or packaged into multiple software products.

Therefore, specific examples of the subject have already been described. Other examples are within the scope of the attached "Claims". In some cases, the actions described in the claims may be performed in different orders and still achieve the expected results. It should be noted that the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the expected results. Multitasking and parallel processing can be beneficial in some implementations.

The above are only a few embodiments of the invention and are not used to limit the invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the invention should be included within the scope of one or more embodiments of the invention.

Claims (20)

It is a driving method for interactive objects.
Acquiring the audio-driven data of the interactive object displayed on the display device,
Acquiring a control parameter sequence of a predetermined operation of an interactive target matching the target data based on the target data included in the voice-driven data, and
A method of driving an interactive object, comprising controlling the interactive object to perform the predetermined operation based on the obtained control parameter sequence. Further, controlling the display device to output voice based on the voice information corresponding to the voice-driven data, and / or displaying the text based on the text information corresponding to the voice-driven data. The driving method for an interactive object according to claim 1, wherein the method includes. Controlling the interactive object to perform the predetermined operation based on the obtained control parameter sequence can be performed.
Determining the voice information corresponding to the target data and
Acquiring the time information to output the voice information and
Determining the execution time of a predetermined operation corresponding to the target data based on the time information.
The invention according to claim 1 or 2, wherein the interactive object is controlled to perform the predetermined operation by a control parameter sequence corresponding to the target data based on the execution time. How to drive an interactive object. The control parameter sequence comprises one or more sets of control parameters.
Controlling the interactive object to perform the predetermined operation by the control parameter sequence corresponding to the target data based on the execution time is possible.
3. The third aspect of the present invention comprises calling each set of control parameters in the control parameter sequence at a predetermined speed so that the interactive object exhibits a state corresponding to each set of control parameters. Described how to drive an interactive object. The control parameter sequence comprises one or more sets of control parameters.
Controlling the interactive object to perform the predetermined operation by the control parameter sequence corresponding to the target data based on the execution time is possible.
Determining the call speed of the control parameter sequence based on the execution time,
A claim comprising: calling each set of control parameters in the control parameter sequence at the calling speed so that the interactive object outputs a form corresponding to each set of control parameters. 3. The method for driving an interactive object according to 3. Controlling the interactive object to perform the predetermined operation by the control parameter sequence corresponding to the target data based on the execution time is possible.
Including starting to call the control parameter sequence corresponding to the target data at a predetermined time point before outputting the voice information corresponding to the target data so that the interactive target starts to perform the predetermined operation. 3. The driving method for an interactive object according to claim 3. The voice-driven data includes a plurality of target data, and includes a plurality of target data.
Controlling the interactive object to perform the predetermined operation based on the obtained control parameter sequence can be performed.
The interactive based on the control parameter sequence corresponding to the previously arranged target data according to word order in response to the detection that adjacent target data in the plurality of target data overlap. The method for driving an interactive object according to any one of claims 1 to 6, wherein the object is controlled to perform the predetermined operation, and includes. The voice-driven data includes a plurality of target data, and includes a plurality of target data.
Controlling the interactive object to perform the predetermined operation based on the control parameter sequence corresponding to the target data can be performed.
In response to the detection that the execution times of the control parameter sequences corresponding to the adjacent target data in the plurality of target data overlap each other, the control parameter sequence corresponding to the adjacent target data The driving method for an interactive object according to any one of claims 1 to 6, wherein the overlapping portions are fused. Acquiring a control parameter sequence of a predetermined operation of an interactive object matching the target data based on the target data included in the voice-driven data is possible.
In response to the voice-driven data including audio data, voice recognition is performed on the audio data, and the target data contained in the audio data is determined based on the recognized voice content. When,
It is characterized by including determining a target data included in the text data based on the text content contained in the text data in response to the voice-driven data including the text data. The driving method for an interactive object according to any one of claims 1 to 8. The voice-driven data includes syllable data, and the voice-driven data includes syllable data.
Acquiring a control parameter sequence of a predetermined operation of an interactive object matching the target data based on the target data included in the voice-driven data is possible.
It is to determine whether or not the syllable data included in the voice-driven data is matched with the target syllable data, and the target syllable data belongs to the pre-divided different syllable types and different syllables. The types correspond to different predetermined mouth shapes, and the corresponding control parameter sequences are set for the different predetermined mouth shapes.
In response to the matching of the syllable data to the target syllable data, a predetermined mouth shape corresponding to the matched target syllable data is based on the syllable type to which the matched target syllable data belongs. The driving method for an interactive object according to any one of claims 1 to 9, wherein the control parameter sequence is acquired and includes. Acquiring the first data other than the target data in the voice-driven data,
Acquiring the acoustic characteristics of the first data and
Acquiring the shape control parameters that match the acoustic features, and
The driving method for an interactive object according to any one of claims 1 to 10, further comprising controlling the appearance of the interactive object based on the appearance control parameter. An interactive drive device
The first acquisition unit for acquiring the voice-driven data of the interactive target displayed on the display device, and
A second acquisition unit for acquiring a control parameter sequence of a predetermined operation of an interactive target matching the target data based on the target data included in the voice-driven data, and a second acquisition unit.
A drive device for an interactive object, comprising: a drive unit for controlling the interactive object to perform the predetermined operation based on the obtained control parameter sequence. An output unit for controlling the display device to output voice based on the voice information corresponding to the voice-driven data and / or displaying text based on the text information corresponding to the voice-driven data. The interactive object drive device according to claim 12, further comprising. Specifically, the drive unit is
Confirm the voice information corresponding to the target data,
Acquire the time information to output the voice information, and
Based on the time information, the execution time of the predetermined operation corresponding to the target data is determined.
The drive of an interactive object according to claim 12 or 13, wherein the interactive object is controlled to perform the predetermined operation by a control parameter sequence corresponding to the target data based on the execution time. Device. The control parameter sequence comprises one or more sets of control parameters.
Specifically, when the drive unit controls the interactive object to perform the predetermined operation by the control parameter sequence corresponding to the target data based on the execution time.
Call each set of control parameters in the control parameter sequence at a predetermined rate so that the interactive object exhibits the appearance corresponding to each set of control parameters, or
Based on the execution time, the call rate of the control parameter sequence is determined.
The control parameters of each set in the control parameter sequence are called at the call rate so that the interactive object outputs the state corresponding to the control parameters of each set, or
It is characterized in that the control parameter sequence corresponding to the target data is started to be called at a predetermined time before the voice information corresponding to the target data is output so that the interactive target starts to perform the predetermined operation. 14. The drive device for interactive objects according to claim 14. The voice-driven data includes a plurality of target data, and includes a plurality of target data.
Specifically, the drive unit is
In response to the detection that adjacent target data in the plurality of target data overlap, the interactive according to the control parameter sequence corresponding to the previously arranged target data based on the word order. Controlling the subject to perform the predetermined action, or
In response to the detection that the execution times of the control parameter sequences corresponding to the adjacent target data in the plurality of target data overlap each other, the control parameter sequence corresponding to the adjacent target data The drive device for an interactive object according to any one of claims 12 to 15, wherein the overlapping portions are fused. The voice-driven data includes syllable data, and the voice-driven data includes syllable data.
Specifically, the second acquisition unit is
It is determined whether or not the syllable data included in the voice-driven data is matched with the target syllable data, and here, the target syllable data belongs to one kind of pre-divided syllable type and one kind. The syllable type corresponds to one type of predetermined mouth shape, and the corresponding control parameter sequence is set for one type of predetermined mouth shape.
In response to the matching of the syllable data to the target syllable data, a predetermined mouth shape corresponding to the matched target syllable data is based on the syllable type to which the matched target syllable data belongs. The driving device for an interactive object according to any one of claims 12 to 16, wherein a control parameter sequence is acquired. The device further comprises a shape control unit.
The shape control unit is
Acquire the first data other than the target data in the voice-driven data,
Acquire the acoustic characteristics of the first data,
Acquire the shape control parameters that match the acoustic features,
The driving device for an interactive object according to any one of claims 12 to 17, wherein the appearance of the interactive object is controlled based on the appearance control parameter. It ’s an electronic device,
Equipped with memory and processor,
The memory stores computer instructions that can be operated on the processor.
An electronic device, wherein the processor performs the method according to any one of claims 1 to 11 when the computer instruction is executed. A computer-readable recording medium that stores computer programs.
A computer-readable recording medium, wherein the method according to any one of claims 1 to 11 is realized when the program is executed by a processor.

Images