JP2018001404A - Method, system and robot body for synchronizing voice and virtual operation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance man-machine interaction experience.SOLUTION: A method for synchronizing voices and virtual motions comprises: acquiring user's multi-mode information; creating interaction contents including at least voice information and motion information on the basis of the user's multi-mode information and a daily time axis; and performing synchronization adjustment between time length of the voice information and time length of the motion information. Consequently, the interaction contents including at least the voice information and the motion information can be created through one or more kinds of multi-mode information including user's voices, expressions and motions, furthermore, in order to synchronize the voice information and the motion information, the invention similarly adjusts the time length of the voice information and the time length of the motion information, thereby, a robot is enabled to synchronously match the voices and the motions during playback thereof, the robot can be further personified, as well as user's interaction experiences with the robot can be enhanced.SELECTED DRAWING: Figure 1

Description

  The present invention relates to the field of robot interaction technology, and more particularly, to a method, system, and robot body for synchronizing voice and virtual motion.

  Robots are increasingly used as a tool for interacting with humankind. For example, when an elderly person or a child feels lonely, he can interact with the robot and interact with it. Conventional man-machine interaction technology can generally support only one type of man-machine interaction mode, for example, it can only give a mechanical response to the user, and at best only gives a limited expression according to the response. Commercially available child companion robots can only have four or five types that are set, and these simple expressions do not need to be synchronized with the output voice.

  In addition, the user is increasingly demanding for the experience of using the robot, and the robot can respond to the human voice in multiple modes, for example, by responding with a voice and at the same time being able to perform facial expressions and actions more suitable for anthropomorphism. It has become necessary to have the ability to In order to interact with humankind with two or more output methods at the same time, the robot needs to synchronize the output methods, for example, say “yes” and simultaneously say “nod” and “no” Shake "or sharpen your eyes wide open when you are angry. Only then can human beings get an immersive experience through interaction with robots and feel that the objects they face are able to interact.

  However, if voice and actions cannot be matched, it will have a profound effect on the user's interaction experience. However, whether to synchronize the voice and facial expressions such as facial expressions included in the contents returned by the virtual robot is a very complex issue, and it relates to multiple departments such as robot engineering, psychology, and social science. Then, the solution of this problem is a troublesome one that is imminent, and no system has solved the above problem relatively well.

  An object of the present invention is to improve a man-machine interaction experience by providing a method, a system, and a robot body that synchronize voice and virtual motion.

The objects of the present invention are realized in the following technical aspects:
To synchronize voice and virtual motion,
Obtaining user multi-mode information,
Depending on the user's multi-mode information and life time axis, generation of interaction contents including at least voice information and motion information,
It includes synchronization adjustment for the time length of audio information and the time length of motion information.

Preferably, the step of adjusting the time length of the voice information and the time length of the motion information in the same manner is specifically,
If the difference between the time length of the voice information and the time length of the motion information is less than or equal to the threshold, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby Including making the time length of the information the same as the time length of the voice information.

  Preferably, when the time length of the voice information is larger than the time length of the motion information, the playback speed of the voice information is increased or / and the playback speed of the motion information is decreased, whereby the time length of the motion information is set to the voice information. Is equal to the length of time.

Preferably, the step of adjusting the time length of the voice information and the time length of the motion information in the same manner is specifically,
When the difference between the time length of the voice information and the time length of the motion information is greater than the threshold, if the time length of the voice information is greater than the time length of the motion information, at least two sets of motion information are combined in order , Thereby making the time length of the motion information equal to the time length of the audio information.

  Preferably, when the time length of the voice information is smaller than the time length of the motion information, select some motions in the motion information so that the time length of these motions is equal to the time length of the voice information. .

Preferably, also including generation of a robot life time axis,
Expansion of robot self-awareness,
Get life time axis parameters,
Match the robot's self-recognition parameters with the parameters of the life time axis, thereby generating the robot's life time axis.

  Preferably, the step of expanding the self-recognition of the robot specifically includes combining a life scene with the self-recognition of the robot to generate a self-recognition curve based on a life time axis.

  Preferably, the step of matching the self-recognition parameter of the robot with the parameter of the life time axis specifically uses a probability algorithm, and after the scene parameter of the time axis is changed, each parameter of the robot on the life time axis is changed. Calculating the probability of changing and forming a matching curve.

  Preferably, here, the life time axis indicates a time axis including 24 hours of a day, and the parameters on the life time axis are at least daily activities performed by the user on the time axis and values of parameters representing the actions. Including.

The system that synchronizes voice and virtual motion
An acquisition module that acquires the user's multi-mode information,
An artificial intelligence module that generates interaction content including at least voice information and motion information based on the user's multi-mode information and life time axis;
It includes a control module that adjusts the time length of the voice information and the time length of the operation information in the same manner.

Preferably, the control module is specifically:
If the difference between the time length of the voice information and the time length of the motion information is less than or equal to the threshold, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby Including making the time length of the information the same as the time length of the voice information.

  Preferably, when the time length of the voice information is larger than the time length of the motion information, the playback speed of the voice information is increased or / and the playback speed of the motion information is decreased, whereby the time length of the motion information is set to the voice information. Same as the time length of.

Preferably, the control module is specifically:
If the difference between the time length of the voice information and the time length of the motion information is greater than the threshold, if the time length of the voice information is greater than the time length of the motion information, combine at least two sets of motion information; Thus, the time length of the operation information is made the same as the time length of the voice information.

  Preferably, when the time length of the audio information is smaller than the time length of the operation information, select some operations in the operation information so that the time length of these operations is the same as the time length of the audio information. To do.

Preferably, the system comprises
Expansion of robot self-awareness,
Acquisition of parameters of life time axis,
This includes matching the robot's self-recognition parameters with the parameters of the life time axis to generate the robot's life time axis.

  Preferably, the processing module is specifically used to generate a self-recognition curve based on a life time axis by combining the life scene with the self-recognition of the robot.

  Preferably, the processing module uses a probability algorithm to calculate a probability that each parameter of the robot on the life time axis changes after the scene parameter on the time axis changes, and is used to form a matching curve.

  Preferably, here, the life time axis indicates a time axis including 24 hours of a day, and the parameters on the life time axis are at least daily activities performed by the user on the time axis and values of parameters representing the actions. Including.

  The present invention discloses a robot and includes a system for synchronizing voice and virtual motion as described above.

  A system that synchronizes speech and virtual motion, including a microphone, an analog-digital converter, a speech identification processor, an image acquisition device, a face recognition processor, a speech synthesizer, a power amplifier, a speaker, an imaging system, an interaction content processor, and a memory.

The microphone, the analog-digital converter, the voice identification processor and the interaction content processor are sequentially connected, the image acquisition device, the face recognition processor and the interaction content processor are sequentially connected, and the interaction content processor is connected to the memory, An interaction content processor, a speech synthesizer, a power amplifier and a speaker are sequentially connected, and the imaging system is connected to an interaction content processor;
The microphone is used to acquire a user's voice signal when the user and the robot interact, the analog-digital converter is used to convert the voice signal into voice digital information, and the voice identification processor uses the voice digital information as character information. Used to input to the intent identification processor
The image acquisition device is used to acquire an image of a user, and the face recognition processor identifies and acquires facial expression information of the user from an image of the user and uses it for input to the intention identification processor.
The interaction content processor generates interaction content including at least voice information and action information based on a user's multi-mode information including at least the character information and facial expression information, and a life time axis stored in the memory, Used to adjust the time length of voice information and the time length of motion information in the same way,
The method of synchronizing audio and virtual motion, wherein the imaging system generates a virtual 3D image according to the motion information, and the speaker reproduces the audio information simultaneously.

Preferably, in the interaction content processor, the step of adjusting the time length of the voice information and the time length of the motion information in the same manner is specifically,
If the difference between the time length of the voice information and the time length of the motion information is less than or equal to the threshold, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby Including making the time length of the information the same as the time length of the voice information.

  Preferably, in the interaction content processor, when the time length of the voice information is larger than the time length of the motion information, the playback speed of the voice information is increased or / and the playback speed of the motion information is decreased, thereby The time length is made the same as the time length of the voice information.

Preferably, the step of adjusting the time length of the voice information and the time length of the motion information in the interaction content processor is specifically,
When the difference between the time length of the voice information and the time length of the motion information is greater than the threshold, if the time length of the voice information is greater than the time length of the motion information, at least two sets of motion information are combined in order , Thereby making the time length of the operation information the same as the time length of the voice information.

  Preferably, in the interaction content processor, when the time length of the voice information is smaller than the time length of the motion information, a part of motions in the motion information is selected, and the time length of these motions is the time length of the voice information. Same as time length.

Preferably, the system also includes an artificial intelligence cloud processor, the artificial intelligence cloud processor used to generate a robot time-of-life parameter, specifically,
Expansion of robot self-awareness,
Obtaining the user's life time axis parameters from the memory;
13. The system of claim 12, comprising matching robot self-recognition parameters with parameters on a user's life time axis, thereby generating a robot's life time axis.

  Preferably, in the artificial intelligence cloud processor, the step of expanding the self-recognition of the robot specifically generates a self-recognition curve based on a life time axis by combining the life scene with the self-recognition of the robot. Including.

  Preferably, in the artificial intelligence cloud processor, the step of matching the self-recognition parameter of the robot with the parameter of the life time axis specifically uses a probability algorithm, and the life time axis is changed after the scene parameter of the time axis is changed. And calculating the probability that each parameter of the robot is changed to form a matching curve.

Preferably, the operation of adjusting the time length of the voice information and the time length of the operation information in the same manner is as follows:
Acquisition of the time position where the robot is currently located on the life time axis,
The difference between the time length of the voice information and the time length of the motion information is compared, and it is determined whether the time length of the voice information is adjusted based on the time position where the robot is located or whether the time length of the motion information is adjusted. ,
The system according to claim 17, comprising adjusting the time length of the voice information and the time length of the motion information in the same manner based on the determination result.

Conventional man-machine interaction technology usually supports only one type of man-machine interaction mode, or can only output limited movements and facial expressions. For example, commercially available child companion robots have only four or five types set. Can not. From the prior art, the present invention is a method for synchronizing the voice and the virtual motion, the content of the interaction including at least the voice information and the motion information based on the acquisition of the user's multi-mode information and the user's multi-mode information and the life time axis. And the synchronization adjustment for the time length of the voice information and the time length of the motion information. By doing so, it is possible to generate interaction contents including at least audio information and operation information according to a kind or multipurpose of multi-mode information such as user's voice, facial expression, operation, etc. In order to synchronize audio information and operation information, The time length of the voice information and the time length of the motion information are adjusted in the same way, so that it is possible to match at the same time when the robot plays back the voice and motion, and when the robot interacts with humankind, only the voice expression In addition, various expression formats such as motion can be used, the robot expression format is further diversified, and the generated robot motion and facial expressions are not limited to several or a dozen types, and the motion grip in the motion library The robot can interact more anthropomorphically and can improve the user interaction experience

FIG. 1 is a flowchart showing a flow of a method for synchronizing voice and virtual operation according to the first embodiment of the present invention. FIG. 2 is a diagram showing a system for synchronizing voice and virtual motion according to the second embodiment of the present invention. FIG. 3 is a circuit block diagram of a system for synchronizing voice and virtual operation according to the third embodiment of the present invention. FIG. 4 is a preferred circuit block diagram of a system for synchronizing voice and virtual operation according to the third embodiment of the present invention. FIG. 5 is a diagram showing that a system for synchronizing voice and virtual operation according to the third embodiment of the present invention is combined with a wearable device. FIG. 6 is a diagram showing that a system for synchronizing voice and virtual operation according to the third embodiment of the present invention is combined with a mobile terminal. FIG. 7 is a diagram illustrating an application scene in which the system for synchronizing the voice and the virtual motion according to the third embodiment of the present invention is combined with the robot.

  Although the flow diagram has been described as processing each operation in sequence, many of the operations can be performed in parallel, merged or simultaneously. You may arrange the order of each operation anew. When the operation is completed, the process can be stopped, but it may not be included in the drawing and may have additional steps. The processing can correspond to a method, a function, a rule, a subroutine, a subprogram, and the like.

  Computer devices include user devices and network devices. Here, user devices and clients include, but are not limited to, computers, smartphones, PDAs, and the like, and network devices include a large number of computers and network servers based on a single network server, a server group consisting of multiple network servers, or cloud computing. Including, but not limited to, a cloud comprised of The computer device may implement the present invention by independent operation, and may implement the present invention by accessing a network and interacting with other computer devices there. The network in which the computing device is located includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, and the like.

  Here, each unit may be described in terms such as “first”, “second”, etc., but these units are not limited to these terms, and the use of these terms is only one unit. Only to distinguish it from another unit. As used herein, the term “and / or” includes any and all combinations of one or more of the associated projects. When one unit is defined as “connected” or “coupled” to another unit, it may be directly connected or coupled to the other unit, but may be present in an intermediate unit.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the illustrative embodiments. Except as otherwise specified in the text, the singular forms “one” and “one term” used herein are intended to include the plural as well. It should be understood that the terms “include” and / or “contain” as used herein define the presence of features, integers, steps, operations, units and / or modules described by one or more. The presence or addition of other features, integers, steps, operations, units, modules and / or combinations is not excluded.
In the following, the invention will be described in more detail in conjunction with the drawings and the exemplary embodiments.

Embodiment 1
As shown in FIG. 1, the present embodiment discloses a method for synchronizing voice and virtual motion,
Step S101 for acquiring user multi-mode information,
Step S102 for generating an interaction content including at least voice information and motion information based on the user's multi-mode information and life time axis,
Step S103 is included which adjusts the time length of the voice information and the time length of the motion information in the same manner.

  The method of synchronizing voice and virtual motion according to the present invention includes obtaining multi-mode information of a user, generating interaction contents including at least voice information and motion information based on the multi-mode information and life time axis of the user, Includes synchronization adjustment for time length and time length of motion information. By doing so, it is possible to generate interaction contents including at least voice information and action information by one or many kinds of multi-mode information such as user's voice, facial expression, action, etc., and in order to synchronize voice information and action information The time length of the voice information and the time length of the motion information are adjusted in the same way, so that the robot can match at the same time when playing back the voice and motion, so that the robot is not only voice expression but also motion It became possible to interact in various expression formats such as the above, diversifying the robot expression method, making the robot more anthropomorphic and improving the user's interaction experience with the robot.

  For humanity, everyday life has a certain degree of regularity, and in order to make the interaction between the robot and humanity more anthropomorphic, in 24 hours of the day, the robot sleeps, exercises, eats, Give them the ability to dance, read a book, make up, and sleep. As a result, the present invention adds a certain life time axis to the generation of the interaction content of the robot, so that the robot can interact with humanity more anthropomorphically, and the robot has a human life style on the life time axis. The method can improve the anthropomorphism of the generated robot interaction content and the man-machine interaction experience, and enhance the intelligent. The interaction content may be one type of facial expressions, characters, voices and actions, or a combination of various types. The robot's life time axis is aligned and arranged in advance. Specifically, the robot's life time axis is a series of parameter collections, and these parameters are transmitted to the system to generate interaction contents. .

  The multi-mode information in this embodiment may be one kind or many kinds of user facial expression information, voice information, hand gesture information, scene information, image information, video information, face information, iris information, light sensing information, fingerprint information, and the like.

  In this embodiment, based on the life time axis, specifically, the self-recognition value on the daily life time axis of the robot body is matched based on the human life style based on the human life time axis. The operation is performed by matching, that is, the action of the robot main body is obtained in one day, and the robot makes its own action such as generating interaction contents and interacting with humanity based on the life time axis. When the robot was awake, it acted on this time axis, and the robot's self-recognition was changed according to this time axis. The life time axis and variable parameters can change attributes in self-recognition such as feeling value, fatigue value, etc., and new self-recognition information can be added automatically.For example, there is no anger value so far, The scene based on the variable elements simulates the scene of humankind's self-recognition based on the previous information, and adds the robot's self-recognition. The life time axis includes not only voice information but also information such as actions.

  For example, if the user says “sleepy” to the robot, the robot will understand the word that the user is sleepy and then connect the robot's life time axis, for example, if the current time is 9 am The robot knows that the master has just woken up, should greet the master, respond with a voice like "Good morning", or sing a song and add a corresponding dance action. If the user says “sleepy” to the robot, the robot understands the word that the user is sleepy and then combines the robot's life time axis, eg if the current time is 9pm , You can see that the robot is sleeping time of the master, you can reply with similar words such as “Good night” and add a suitable rest, sleep action, etc. Such a situation is closer to human life than a simple voice and facial response, and the movement is more anthropomorphic.

In the present embodiment, the step of adjusting the time length of the voice information and the time length of the motion information in the same way is specifically,
If the difference between the time length of the voice information and the time length of the motion information is less than or equal to the threshold, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby Including making the time length of the information the same as the time length of the voice information.

  If the time length of the voice information is greater than the time length of the motion information, the playback speed of the voice information is increased or / and the playback speed of the motion information is decreased, whereby the time length of the motion information is set to the time length of the voice information. To do the same.

  Therefore, when the difference between the time length of the voice information and the time length of the motion information is equal to or less than the threshold, the specific meaning of the adjustment is to compress or extend the time length of the voice information and / or the time length of the motion information. It may also indicate that the playback speed is to be increased or decreased. For example, the reproduction speed of audio information is multiplied by 2, or the reproduction time of operation information is multiplied by 0.8.

  For example, the threshold between the time length of the voice information and the time length of the motion information is 1 minute, and in the interaction content generated by the robot based on the user's multi-mode information, the time of the voice information is 1 minute, If the time is 2 minutes, the playback speed of the motion information may be increased to twice the original speed, so that the playback time after the motion information is adjusted becomes 1 minute, thereby synchronizing with the audio information. Realize. Of course, the reproduction speed of the sound information may be adjusted to 0.5 times the original speed, so that the reproduction time after the sound information is adjusted becomes 2 minutes, thereby realizing synchronization with the operation information. It is also possible to adjust both voice information and motion information. For example, the speed of voice information can be reduced and the speed of motion information can be increased, and both can be adjusted to 1 minute 30 seconds. it can.

In addition, in the present embodiment, the step of adjusting the time length of the audio information and the time length of the operation information in the same manner is specifically,
When the difference between the time length of the voice information and the time length of the motion information is greater than the threshold, if the time length of the voice information is greater than the time length of the motion information, at least two sets of motion information are combined in order Thereby, the time length of the operation information is made the same as the time length of the voice information.

  When the time length of the audio information is smaller than the time length of the operation information, some operations in the operation information are selected so that the time length of these operations is the same as the time length of the audio information.

  Therefore, when the difference between the time length of the voice information and the time length of the motion information is larger than the threshold, the meaning of the adjustment is to add or delete some motion information, so that the time length of the motion information and the voice Synchronous adjustment with the time length of information can be realized.

  For example, when the threshold value of the time length of the voice information and the time length of the motion information is 30 seconds, the robot has the time length of the voice information of 3 minutes in the interaction content generated based on the user's multi-mode information. If the time length of the motion information is 1 minute, it is necessary to add other motion information to the original motion information. For example, after finding motion information having a time length of 2 minutes and combining the above two sets of motion information Matched in the same way as the time of the voice information. Of course, if the operation information of 2 minutes and a half is found instead of the operation information having a time length of 2 minutes, the operation of this 2 minutes and a half is performed so that the time length of the selected operation information is 2 minutes. You can select some actions (some frames) in the information and then match them as well as the time length of the voice information.

  In this embodiment, it is possible to select the operation information closest to the time length of the sound information depending on the time length of the sound information, and further to select the sound information closest to the time length of the operation information.

  Then, selecting according to the time length of the audio information at the time of selection makes it easier for the control module to adjust the time length of the audio information and the operation information, can be adjusted until they match more easily, and the adjusted playback is more natural. , Became smooth.

  According to an embodiment of the present invention, the audio information and the operation information after the adjustment are output to the virtual image after the step of adjusting the time length of the audio information and the operation information in the same manner. Including displaying.

  Then, it can be output after adjusting until it matches, and output in virtual video is possible, so the virtual robot became more anthropomorphic and improved the user experience.

In one embodiment, the method for generating a time axis parameter for the robot includes:
Expansion of robot self-awareness,
Acquisition of life time axis parameters,
This includes matching the robot's self-recognition parameters with the parameters of the life time axis to generate the robot's life time axis.

  Thus, by adding the life time axis to the self-recognition of the robot body, the robot began to have an anthropomorphic life. For example, adding recognition to the robot to have lunch.

  According to one embodiment, the step of expanding the self-recognition of the robot specifically includes combining a life scene and the self-recognition of the robot to form a self-recognition curve based on a life time axis. .

  Then, the life time axis can be specifically added to the parameters of the robot body.

  In one embodiment, the step of matching the self-recognition parameter of the robot with the parameter of the time-of-life axis may be performed by using a probability algorithm. After changing, calculating the probability that each parameter will change to form a matching curve. Then, the self-recognition parameter of the robot and the parameter of the life time axis can be specifically matched. Here, the probability algorithm may be a Bayes probability algorithm.

  For example, in 24 hours a day, the robot is allowed to have actions such as sleeping, exercising, eating, dancing, reading a book, eating, making up, and sleeping. Each movement affects the self-recognition of the robot body, and after matching the parameters in the life time axis by combining the self-recognition of the robot body, that is, the robot's self-recognition, feeling, fatigue value, familiarity, favorable sensitivity , The number of dialogues, 3D recognition of robots, age, height, weight, intimacy, game scene values, game target values, place scene values, place target values, etc. Robots can now identify places where they are located, such as cafes and bedrooms.

  Robots behave differently within the time axis of the day, such as sleeping at night, eating in the day, exercising in the day, all of these scenes on the time axis of life affect self-awareness. The change of these numerical values is a dynamic matching method based on a probabilistic model, and finishes matching the probability that these actions occur on the time axis. Scene identification: This location scene identification may change the geographic scene value in self-recognition.

Embodiment 2
As shown in FIG.
An acquisition module 201 for acquiring user multi-mode information;
An artificial intelligence module 202 that generates interaction content including at least voice information and motion information based on the user's multi-mode information and the life time axis generated by the life time axis module 205;
A system for synchronizing voice and virtual motion, comprising a control module 203 for adjusting the time length of voice information and the time length of motion information in the same way.

  By doing so, it is possible to generate interaction contents including at least voice information and action information by one or many kinds of multi-mode information such as user's voice, facial expression, action, etc., and in order to synchronize voice information and action information The time length of the voice information and the time length of the motion information are adjusted in the same way, so that the robot can match at the same time when playing back the voice and motion, so that the robot is not only voice expression but also motion As a result, the robot's expression method has become more anthropomorphic and the user's interaction experience with the robot has also been improved.

  For human beings, everyday life has a certain degree of regularity, and in order to make the interaction between the robot and humanity more anthropomorphic, the robot sleeps, exercises, eats, dances in 24 hours a day Doing, reading books, eating, making up, sleeping, etc. As a result, the present invention adds a certain life time axis to the generation of the interaction content of the robot, so that the robot can interact with humanity more anthropomorphically, and the robot has a human life style on the life time axis. It is possible to improve the anthropomorphism of the generated robot interaction content and the human interaction experience, and enhance the intelligent. The interaction content may be a kind of expression, character, voice, action, or a combination of many kinds. The robot's life time axis is aligned and arranged in advance. Specifically, the robot's life time axis is a series of parameter collections, and these parameters are transmitted to the system to generate interaction contents. .

  The multi-mode information in this embodiment may be one kind or many kinds of user facial expression information, voice information, hand gesture information, scene information, image information, video information, face information, iris information, light sensing information, fingerprint information, and the like.

  In this embodiment, based on the life time axis, specifically, the self-recognition value on the daily life time axis of the robot body is matched based on the human life style based on the human life time axis. The operation is performed by matching, that is, the action of the robot main body is obtained in one day, and the robot makes its own action such as generating interaction contents and interacting with humanity based on the life time axis. When the robot was awake, it acted on this time axis, and the robot's self-recognition was changed according to this time axis. The life time axis and variable parameters can change attributes in self-recognition such as feeling value, fatigue value, etc., and new self-recognition information can be added automatically.For example, there is no anger value so far, The scene based on the variable elements simulates the scene of humankind's self-recognition based on the previous information, and adds the robot's self-recognition. The life time axis includes not only voice information but also information such as actions.

  For example, if the user says “sleepy” to the robot, the robot will understand the word that the user is sleepy and then connect the robot's life time axis, for example, if the current time is 9 am The robot knows that the master has just woken up, should greet the master, respond with a voice like "Good morning", or sing a song and add a corresponding dance action. If the user says “sleepy” to the robot, the robot understands the word that the user is sleepy and then combines the robot's life time axis, eg if the current time is 9pm , You can see that the robot is sleeping time of the master, you can reply with similar words such as “Good night” and add a suitable rest, sleep action, etc. Such a situation is closer to human life than a simple voice and facial response, and the movement is more anthropomorphic.

In this embodiment, the control module is specifically:
If the difference between the time length of the voice information and the time length of the motion information is less than or equal to the threshold, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby The time length of the information is made the same as the time length of the voice information.

  If the time length of the voice information is greater than the time length of the motion information, the playback speed of the voice information is increased or / and the playback speed of the motion information is decreased, whereby the time length of the motion information is set to the time length of the voice information. To do the same.

  Therefore, when the difference between the time length of the voice information and the time length of the motion information is equal to or less than the threshold, the specific meaning of the adjustment is to compress or extend the time length of the voice information and / or the time length of the motion information. It may also indicate that the playback speed is to be increased or decreased. For example, the reproduction speed of audio information is multiplied by 2, or the reproduction time of operation information is multiplied by 0.8.

  For example, the threshold between the time length of the voice information and the time length of the motion information is 1 minute, and in the interaction content generated by the robot based on the user's multi-mode information, the time of the voice information is 1 minute, If the time is 2 minutes, the playback speed of the motion information may be increased to twice the original speed, so that the playback time after the motion information is adjusted becomes 1 minute, thereby synchronizing with the audio information. Realize. Of course, the reproduction speed of the sound information may be adjusted to 0.5 times the original speed, so that the reproduction time after the sound information is adjusted becomes 2 minutes, thereby realizing synchronization with the operation information. Note that both voice information and motion information may be adjusted. For example, the speed of voice information can be reduced and the speed of motion information can be increased, and both can be adjusted to 1 minute 30 seconds. it can.

In addition, in this embodiment, the control module is specifically:
If the difference between the time length of the voice information and the time length of the motion information is greater than the threshold, if the time length of the voice information is greater than the time length of the motion information, combine at least two sets of motion information; Thus, the time length of the operation information is used to be the same as the time length of the voice information.

  When the time length of the audio information is smaller than the time length of the operation information, some operations in the operation information are selected so that the time length of these operations is the same as the time length of the audio information.

  Therefore, when the difference between the time length of the voice information and the time length of the motion information is larger than the threshold, the meaning of the adjustment is to add or delete some motion information, so that the time length of the motion information and the voice Synchronous adjustment with the time length of information can be realized.

  For example, when the threshold value of the time length of the voice information and the time length of the motion information is 30 seconds, the robot has the time length of the voice information of 3 minutes in the interaction content generated based on the user's multi-mode information. If the time length of the motion information is 1 minute, it is necessary to add other motion information to the original motion information. For example, find the motion information whose time length is 2 minutes and combine the above two sets of motion information After the same match as the time of the voice information. Of course, if the operation information of 2 minutes and a half is found instead of the operation information having a time length of 2 minutes, the operation of this 2 minutes and a half is performed so that the time length of the selected operation information is 2 minutes. You can select some actions (some frames) in the information and then match them as well as the time length of the voice information.

  In this embodiment, the artificial intelligence module is specifically used for selecting the operation information closest to the time length of the sound information according to the time length of the sound information, and further selecting the sound information closest to the time length of the operation information.

  Then, selecting according to the time length of the audio information at the time of selection makes it easier for the control module to adjust the time length of the audio information and the operation information, can be adjusted until they match more easily, and the adjusted playback is more natural. , Became smooth.

  According to one embodiment, the system also includes an output module 204 that outputs the adjusted audio information and motion information to a virtual image for display.

  Then, it can be output after adjusting until it matches, and it can be output in virtual video, so the virtual robot became more anthropomorphic and improved the user experience.

According to one embodiment, the system includes a time axis based artificial intelligence cloud processor module,
Expansion of robot self-awareness,
Acquisition of life time axis parameters,
The robot's self-recognition parameter is matched with the life time axis parameter to generate the robot life time axis.

  Thus, by adding the life time axis to the self-recognition of the robot body, the robot began to have an anthropomorphic life. For example, adding recognition to the robot to have lunch.

  According to one embodiment, the system is a time-based artificial intelligence cloud processor module that combines a life scene and robot self-recognition to form a self-recognition curve based on the life time axis. Used for. Then, the life time axis can be specifically added to the parameters of the robot body.

  According to one of the embodiments, the artificial intelligence cloud processing module based on the time axis specifically uses the probability algorithm to change each parameter of the robot on the life time axis after the scene parameter on the time axis changes. Calculating the probability of changing to form a matching curve. Then, the self-recognition parameter of the robot and the parameter of the life time axis can be specifically matched. Here, the probability algorithm may be a Bayes probability algorithm.

  For example, in 24 hours a day, the robot is allowed to have actions such as sleeping, exercising, eating, dancing, reading a book, eating, making up, and sleeping. Each movement affects the self-recognition of the robot body, and after matching the parameters in the life time axis by combining the self-recognition of the robot body, that is, the robot's self-recognition, feeling, fatigue value, familiarity, favorable sensitivity , The number of dialogues, 3D recognition of robots, age, height, weight, intimacy, game scene values, game target values, place scene values, place target values, etc. Robots can now identify places where they are located, such as cafes and bedrooms.

  Robots behave differently within the time axis of the day, such as sleeping at night, eating in the day, exercising in the day, all of these scenes on the time axis of life affect self-awareness. The change of these numerical values is a dynamic matching method based on a probabilistic model, and finishes matching the probability that these actions occur on the time axis. Scene identification: This location scene identification may change the geographic scene value in self-recognition.

  The present invention discloses a robot including the system for synchronizing the voice and the virtual motion described in any of the above.

Embodiment 3
As shown in FIG. 3, this embodiment discloses a system 300 that synchronizes voice and virtual motion, and includes a microphone 301, an analog-to-digital converter 302, a voice identification processor 303, an image acquisition device 304, a face recognition processor 305, and an interaction content processor. 306, a speech synthesizer 307, a power amplifier 308, a speaker 309, an imaging system 310, and a memory 311.

  The microphone 301, the analog-digital converter 302, the voice identification processor 303 and the intention identification processor 306 are sequentially connected, and the image acquisition device 304, the face recognition processor 305 and the intention identification processor 306 are sequentially connected, and the interaction content processor 306 is connected. Is connected to the memory 311, the interaction content processor 306, the speech synthesizer 307, the power amplifier 308 and the speaker 309 are sequentially connected, and the imaging system 310 is connected to the interaction content processor 306.

  The microphone 301 is used to acquire a user's voice signal when the user and the robot interact, the analog-digital converter 302 is used to convert the voice signal into voice digital information, and the voice identification processor 303 is used to convert the voice digital information. Is converted into character information and used for input to the interaction content processor 306.

  The image acquisition device 304 is used to acquire an image of the user, and the face recognition processor 305 identifies and acquires the facial expression information of the user from the image of the user and uses it for input to the interaction content processor 306. The image acquisition device 304 may be a video camera, a camera, etc., and can identify not only the user's facial expression information but also the environment where the user is, the user's operation information, etc., and these information are input to the interaction content processor 306. The interaction content generated thereby more closely matches the user's current demand.

  The interaction content processor 306 generates an interaction content including at least voice information and action information based on the user's multi-mode information including at least the character information and facial expression information, and the life time axis stored in the memory 311. Then, it is used to adjust the time length of the voice information and the time length of the operation information in the same way. Here, first, the voice information of the interaction content is generated based on the user's multi-mode information and the life time axis, thereby selecting an appropriate operation grip from the operation library of the memory 311 and adding an appropriate transition operation to complete the operation. Operation information is generated. Here, the life time axis indicates a time axis including 24 hours in a day, and the parameters on the life time axis include at least daily life activities performed by the user on the time axis and values of parameters representing the actions.

  Here, in the interaction content processor 306, the step of adjusting the time length of the voice information and the time length of the motion information in the same way is specifically the difference between the time length of the voice information and the time length of the motion information being a threshold value. If it is below, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby making the time length of the motion information the same as the time length of the voice information, If the time length of the voice information is greater than the time length of the motion information, the playback speed of the voice information is increased or / and the playback speed of the motion information is decreased, whereby the time length of the motion information is set to the time length of the voice information. To do the same.

  Therefore, when the difference between the time length of the voice information and the time length of the motion information is equal to or less than the threshold, the specific meaning of the adjustment is to compress or extend the time length of the voice information and / or the time length of the motion information. It may also indicate that the playback speed is to be increased or decreased. For example, the reproduction speed of audio information is multiplied by 2, or the reproduction time of operation information is multiplied by 0.8.

  For example, the threshold between the time length of the voice information and the time length of the motion information is 1 minute, and in the interaction content generated by the robot based on the user's multi-mode information, the time of the voice information is 1 minute, If the time is 2 minutes, the playback speed of the motion information may be increased to twice the original speed, so that the playback time after the motion information is adjusted becomes 1 minute, thereby synchronizing with the audio information. Realize. Of course, the reproduction speed of the audio information may be reduced to 0.5 times the original speed, so that the reproduction time after the audio information is adjusted becomes 2 minutes, thereby realizing synchronization with the operation information. Note that both voice information and motion information may be adjusted. For example, the speed of voice information can be reduced and the speed of motion information can be increased, and both can be adjusted to 1 minute 30 seconds. it can.

  Here, in the interaction content processor 306, the step of adjusting the time length of the voice information and the time length of the motion information in the same way is specifically the difference between the time length of the voice information and the time length of the motion information being based on the threshold value. If so, if the time length of the voice information is greater than the time length of the motion information, at least two sets of motion information are combined in order, whereby the time length of the motion information is the same as the time length of the voice information If the time length of the audio information is smaller than the time length of the operation information, select some operations in the operation information so that the time length of these operations is the same as the time of the audio information. .

  As a result, when the difference between the time length of the voice information and the time length of the motion information is larger than the threshold, the meaning of the adjustment is to add or delete some motion information, Synchronous adjustment with the time length of information can be realized.

  For example, when the threshold value of the time length of the voice information and the time length of the motion information is 30 seconds, the robot has the time length of the voice information of 3 minutes in the interaction content generated based on the user's multi-mode information. If the time length of the motion information is 1 minute, it is necessary to add other motion information to the original motion information. For example, find the motion information whose time length is 2 minutes and combine the above two sets of motion information After the same match as the time of the voice information. Of course, if the operation information of 2 minutes and a half is found instead of the operation information having a time length of 2 minutes, the operation of this 2 minutes and a half is performed so that the time length of the selected operation information is 2 minutes. You can select some actions (some frames) in the information and then match them as well as the time length of the voice information.

  The imaging system 310 generates a virtual 3D image based on the operation information, and the speaker 309 reproduces the audio signal simultaneously. Here, the imaging system 310 may be a normal display or a holographic projection device, thereby increasing the stereoscopic effect and authenticity of the displayed robot and enhancing the user experience.

  The memory 311 can also be used to store data used when the interaction content processor 306 operates. You can select an interaction content processor 306: CPU (Central Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), CPLD (Complex Programmable Logic Device) Complex programmable logic device).

  Depending on the system 300 that synchronizes voice and virtual motion according to the present embodiment, the user's multi-mode information can be acquired, and based on the user's multi-mode information and the life time axis, generation of interaction content including at least voice information and motion information The time length of the voice information and the time length of the operation information are adjusted in the same way. By doing so, it is possible to generate interaction contents including at least voice information and action information by one or many kinds of multi-mode information such as user's voice, facial expression, action, etc., and in order to synchronize voice information and action information The time length of the voice information and the time length of the motion information are adjusted in the same way, so that the robot can match at the same time when playing back the voice and motion, so that the robot is not only voice expression but also motion It became possible to interact in various expression formats such as the above, diversifying the robot expression method, making the robot more anthropomorphic and improving the user's interaction experience with the robot.

  For example, if the user says “sleepy” to the robot, the robot will understand the word that the user is sleepy and then connect the robot's life time axis, for example, if the current time is 9 am The robot knows that the master has just woken up, should greet the master, respond with a voice like "Good morning", or sing a song and add a corresponding dance action. If the user says “sleepy” to the robot, the robot understands the word that the user is sleepy and then combines the robot's life time axis, eg if the current time is 9pm , You can see that the robot is sleeping time of the master, you can reply with similar words such as “Good night” and add a suitable rest, sleep action, etc. Such a situation is closer to human life than a simple voice and facial response, and the movement is more anthropomorphic.

  As shown in FIG. 4, the system 300 for synchronizing voice and virtual motion disclosed in the present embodiment also acquires some physiological signals of the user, and after the physiological signals are preprocessed by the signal preprocessor 314, the physiological parameters are obtained. And a plurality of sensors 313 used to transmit the physiological parameters to the interaction content processor 306. Correspondingly, the interaction content processor 306 generates an interaction content based on the character information, facial expression information and physiological parameters, transmits the interaction content to the imaging system 310, and transmits audio information in the interaction content to a speaker. To 309.

  The sensor 313 in the system 300 for synchronizing voice and virtual motion includes, but is not limited to, a light sensor, an iris recognition sensor, a fingerprint acquisition sensor, a temperature sensor, a heart rate sensor, and the like. It enriches multi-mode information including one kind and many kinds of physiological signals such as fingerprint information, body temperature information, heart rate information.

  In the following, information acquired and output by the microphone 301, the image acquisition device 304, and the sensor 313 is collectively referred to as multi-mode information.

  As shown in FIG. 5, some sensors 313 can be integrated with a system 300 that synchronizes voice and virtual motion, and some centers 313 can be integrated into a wearable device 400, for example, temperature sensors. The heart rate sensor is integrated into the smart wristband and the information acquired by the wireless communication device is transmitted to the interaction content processor 306 in the system 300 that synchronizes voice and virtual motion. FIG. 5 merely displays the connection relationship between the wireless communication apparatus and the interaction content processor 306 in the system 300 that synchronizes the voice and the virtual operation, and the other connection relationship in the system 300 that synchronizes the voice and the virtual operation is as shown in FIG. Reference may be made to FIG.

  The system 300 also includes an artificial intelligence cloud processor, which is used to generate a robot's life time axis parameter, specifically expanding the robot's self-awareness, and obtaining the user's life time axis parameter from the memory. Including matching the robot's self-recognition parameters with the parameters on the user's life time axis, thereby generating the robot's life time axis. Here, the step of expanding the self-recognition of the robot specifically includes generating a self-recognition curve based on the life time axis by combining the life scene with the self-recognition of the robot. Specifically, the step of matching the robot's self-recognition parameters with the parameters on the life time axis uses a probability algorithm to determine the probability that each parameter of the robot on the life time axis will change after the scene parameter on the time axis changes. Calculating to form a matching curve. Then, the self-recognition parameter of the robot and the parameter of the life time axis can be specifically matched. Here, the probability algorithm may be a Bayes probability algorithm.

  For example, in 24 hours a day, the robot is allowed to have actions such as sleeping, exercising, eating, dancing, reading a book, eating, making up, and sleeping. Each movement affects the self-recognition of the robot body, and after matching the parameters in the life time axis by combining the self-recognition of the robot body, that is, the robot's self-recognition, feeling, fatigue value, familiarity, favorable sensitivity , The number of dialogues, 3D recognition of robots, age, height, weight, intimacy, game scene values, game target values, place scene values, place target values, etc. Robots can now identify places where they are located, such as cafes and bedrooms.

  The system of the present embodiment can also guide the synchronization of the robot's voice and motion using the robot's life time axis. Specifically, the step of adjusting the time length of the voice information and the time length of the motion information in the same way specifically includes obtaining a time position on the current life time axis of the robot, and the time length of the voice information. Compare the difference with the time length of the motion information, determine whether the robot adjusts the time length of the voice information based on a certain time position or the time length of the motion information, and the voice information according to the judgment result And adjusting the time length of the motion information in the same manner. For example, man-machine interaction occurs when the robot is resting or just getting up, and if the operation time> the length of the language, the length of the language is extended, which is relatively low because the robot is in a lazy state. It has become possible for humans to better align expressions in daily life with both slow movements and voice.

  When adjusting the time length of voice information and motion information, in addition to the time position where the robot is currently on the time axis, the robot's current feeling (long-time driving-corresponding to human fatigue), the robot's current state ( The amount of electricity remaining-corresponding to human hunger) should be considered.

  Robots behave differently within the time axis of the day, such as sleeping at night, eating in the day, exercising in the day, all of these scenes on the time axis of life affect self-awareness. The change of these numerical values is a dynamic matching method based on a probabilistic model, and finishes matching the probability that these actions occur on the time axis. Scene identification: This location scene identification may change the geographic scene value in self-recognition.

  The system 300 disclosed in this embodiment also includes a wireless communication device 314, which is connected to an interaction content processor 306, as shown in FIG. 6, which also communicates to the mobile terminal 500 for interaction content. The mobile terminal 500 generates a virtual 3D video based on the operation information, and simultaneously reproduces the audio information using the speaker of the mobile terminal 500. FIG. 6 merely displays the connection relationship between the wireless communication apparatus and the interaction content processor 311 in the system 300 that synchronizes voice and virtual operation, and the other connection relationship in the system 300 that synchronizes voice and virtual operation is as shown in FIG. Reference may be made to FIG.

  The system 300 that synchronizes the voice and virtual motion of the robot disclosed in the present embodiment can be rich in various forms of interacting with the user of the robot in various aspects, and the robot can interact with humanity more anthropomorphically. The system is expected to improve the anthropomorphism of the generated robot interaction content and the man-machine interaction experience, and enhance the intelligent.

  As shown in FIG. 7, a system 300 for synchronizing voice and virtual motion may also be integrated inside the robot 600, and includes a voice acquisition device 612, a video camera 611, and various types of sensors (see FIG. The user's multi-mode information can be obtained and transmitted to the interaction content processor 306, for example by a GPS navigation device (not shown in FIG. 4), for example when the user joins the robot to a location The position information of the user can be obtained with the navigation device, so that the variable parameter can be obtained by combining with the life time axis, and the self-recognition of the robot body can be expanded, and the application scene parameter in the self-recognition parameter and the variable parameter can be matched. As a result, the effect of anthropomorphism is produced.

  The interaction content processor 306 is used to read the program stored in the memory 311 and obtains the user's multi-mode information and generates the interaction content including at least voice information and operation information based on the user's multi-mode information and the life time axis. Then, the synchronization adjustment of the time length of the voice information and the time length of the motion information, and the process of outputting the voice information and the motion information whose time length are similarly adjusted are executed. The audio information output from the interaction content processor 306 is reproduced by the audio system 613 of the robot 600, and the motion information output from the interaction content processor 306 is converted into a control signal for each joint of the robot by the host controller of the robot 600. Control the movement of each joint 614 of the robot, thereby causing the robot 600 to synchronize with the voice, for example, the joint inside the head of the robot 600 swings the head sideways, swings back and forth, and nods. The specific method of controlling and controlling the motion of the robot is prior art and will not be described in detail here. The data processed by the interaction content processor 306 is transmitted by a wireless medium via the wireless communication device 314. Further, the wireless communication device 314 receives the data and then transfers the data to the interaction content processor 306. In addition to accessing the Internet via the wireless communication device 314, it is also possible to acquire and upload various data of the user via the Internet, access the user's mobile terminal via the wireless communication device 314, and to interact with the robot and various settings for the robot. realizable.

  A system that synchronizes voice and virtual motion can also be realized via a software program using an electronic device terminal as a carrier, exemplified by using a smartphone as a carrier, and the information acquisition device is a voice acquisition device, video camera that already exists in the smartphone After reusing each type of sensor, GPS navigation device, etc. to acquire the user's multi-mode information and sending it to the processor built in the smartphone, the processor reads the program stored in the memory, and the user's multi-mode information Based on information and variable parameters, the process of generating the interaction content including at least voice information and motion information, the process of adjusting the time length of voice information and the time length of motion information in the same way, and the time length are adjusted in the same way Output audio information and motion information To run the process. The operation of the virtual robot is displayed on the screen of the smartphone, and the sound is played back simultaneously with the speaker. It is connected to external devices and networks by the wireless communication module of the smartphone, completing the data interaction.

  The system that synchronizes the voice and the virtual motion of the present embodiment can generate an interaction content including at least the voice information and the motion information according to one type or various benefits of the multi-mode information such as the user's voice, facial expression, motion, etc. In order to synchronize the motion information, the time length of the voice information and the time length of the motion information are adjusted in the same way, so that when the robot plays back the voice and motion, it can be matched at the same time. In addition to voice expression, various expression formats such as motion can be used when interacting with the robot, making the robot expression format more diverse, allowing the robot to interact more anthropomorphically and improving the user interaction experience.

  The above description is a more detailed description of the present invention after combining specific preferred embodiments, but the specific embodiments of the present invention are not limited to these descriptions. For those skilled in the art, various modifications and improvements can be made to the above-described embodiment on the premise that they do not depart from the spirit of the present invention, and should fall within the protection scope of the present invention.

Claims (20)

A method of synchronizing voice and virtual motion,
Obtaining user multi-mode information,
Depending on the user's multi-mode information and life time axis, generation of interaction contents including at least voice information and motion information,
A method of synchronizing a voice and a virtual motion, comprising a synchronization adjustment for the time length of the voice information and the time length of the motion information. Specifically, the step of adjusting the time length of the voice information and the time length of the motion information in the same manner,
If the difference between the time length of the voice information and the time length of the motion information is less than or equal to the threshold, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby The method according to claim 1, comprising making the time length of information the same as the time length of the voice information. If the time length of the audio information is larger than the time length of the motion information, the playback speed of the speech information is increased or / and the playback speed of the motion information is decreased, whereby the time length of the motion information is set to the time of the voice information. The method of claim 2, wherein the method is the same as the length. Specifically, the step of adjusting the time length of the voice information and the time length of the motion information in the same manner,
When the difference between the time length of the voice information and the time length of the motion information is greater than the threshold, if the time length of the voice information is greater than the time length of the motion information, at least two sets of motion information are combined in order The method according to claim 1, further comprising making the time length of the operation information the same as the time length of the voice information. When the time length of the audio information is smaller than the time length of the operation information, some operations in the operation information are selected so that the time length of these operations is the same as the time of the audio information. The method according to claim 4. It also includes the generation of the robot's life time axis.
Expansion of robot self-awareness,
Get life time axis parameters,
The method of claim 1, comprising matching the robot's self-recognition parameters to the parameters of the life time axis, thereby generating the robot's life time axis. The step of expanding the robot's self-awareness is specifically:
7. The method of claim 6, comprising combining the life scene with the robot's self-recognition, thereby generating a self-recognition curve based on the life time axis. The step of matching the self-recognition parameter of the robot with the parameter of the life time axis specifically includes:
7. The method according to claim 6, further comprising: calculating a probability that each parameter of the robot on the life time axis changes after the scene parameter on the time axis is changed through a probability algorithm, and forming a matching curve. The method described. Here, the life time axis indicates a time axis of 24 hours in a day, and the parameters on the life time axis include at least daily activities performed by the user on the time axis, and values of parameters representing the actions. The method of claim 1, characterized in that: A system that synchronizes voice and virtual motion,
An acquisition module for acquiring user multi-mode information;
An artificial intelligence module that generates interaction content including at least voice information and motion information based on the user's multi-mode information and life time axis;
A system that synchronizes voice and virtual motion, including a control module that adjusts the time length of voice information and the time length of motion information in the same manner. A robot comprising the system for synchronizing voice and virtual motion according to claim 10. A system for synchronizing speech and virtual operation, including a microphone, an analog-digital converter, a speech identification processor, an image acquisition device, a face recognition processor, a speech synthesizer, a power amplifier, a speaker, an imaging system, an interaction content processor, and a memory
The microphone, the analog-digital converter, the voice identification processor, and the interaction content processor are sequentially connected, the image acquisition device, the face recognition processor, and the interaction content processor are sequentially connected, and the interaction content processor is the memory. The interaction content processor, the speech synthesizer, the power amplifier and the speaker are sequentially connected, and the imaging system is connected to the interaction content processor,
The microphone is used to acquire a user's voice signal when the user and the robot interact, the analog-digital converter is used to convert the voice signal into voice digital information, and the voice identification processor uses the voice digital information as character information. Used to input to the intent identification processor
The image acquisition device is used to acquire an image of a user, and the face recognition processor identifies and acquires user facial expression information from an image of the user and uses it for input to the interaction content processor.
The interaction content processor generates interaction content including at least voice information and action information based on a user's multi-mode information including at least the character information and the facial expression information and a life time axis stored in the memory. , Used to adjust the time length of voice information and the time length of motion information in the same way,
The imaging system generates a virtual 3D image according to the motion information, and the speaker reproduces the audio information at the same time. A method of synchronizing sound and virtual motion. In the interaction content processor, the step of adjusting the time length of the voice information and the time length of the motion information in the same manner is specifically,
If the difference between the time length of the voice information and the time length of the motion information is less than or equal to the threshold, if the time length of the voice information is smaller than the time length of the motion information, the playback speed of the motion information is increased, thereby The method according to claim 1, comprising making the time length of information the same as the time length of the voice information. In the interaction content processor, when the time length of the voice information is larger than the time length of the motion information, the playback speed of the voice information is increased or / and the playback speed of the motion information is decreased, thereby reducing the time length of the motion information. The method according to claim 13, wherein the time length of the voice information is the same. In the interaction content processor, the step of adjusting the time length of the voice information and the time length of the motion information in the same manner is specifically,
If the difference between the time length of the voice information and the time length of the motion information is greater than the threshold, if the time length of the voice information is greater than the time length of the motion information, at least two sets of motion information are combined in order, The method according to claim 1, further comprising making the time length of the motion information the same as the time length of the voice information. In the interaction content processor, when the time length of the voice information is smaller than the time length of the motion information, a part of the motion information is selected, and the time length of these motions is the same as the time of the voice information The method system according to claim 15, characterized in that: The system also includes an artificial intelligence cloud processor used to generate the robot time-of-life parameters, specifically,
Expansion of robot self-awareness,
Obtaining the user's life time axis parameters from the memory;
13. The system of claim 12, including matching the robot's self-recognition parameters to parameters on the user's life time axis, thereby generating the robot's life time axis. In the artificial intelligence cloud processor, the step of expanding the self-recognition of the robot is specifically,
The system according to claim 17, further comprising: generating a self-recognition curve based on a life time axis by combining the life scene with the self-recognition of the robot. In the artificial intelligence cloud processor, the step of matching the self-recognition parameter of the robot with the parameter in the life time axis specifically includes:
18. The robot according to claim 17, wherein in the probability algorithm, the robot on the life time axis calculates the probability that each parameter changes after the scene parameter on the time axis changes, thereby forming a matching curve. System. The operation to adjust the time length of the voice information and the time length of the motion information in the same way,
The acquisition of the time position where the robot is currently located on the life time axis,
Performing a determination of whether to adjust the time length of the voice information or the time length of the motion information according to the time position where the robot is located, by comparing the time difference between the voice information and the motion information;
The system according to claim 17, comprising adjusting the time length of the voice information and the time length of the motion information in the same manner based on the determination result.

Images