CN106933345A - For the multi-modal exchange method and device of intelligent robot - Google Patents
For the multi-modal exchange method and device of intelligent robot Download PDFInfo
- Publication number
- CN106933345A CN106933345A CN201710034038.4A CN201710034038A CN106933345A CN 106933345 A CN106933345 A CN 106933345A CN 201710034038 A CN201710034038 A CN 201710034038A CN 106933345 A CN106933345 A CN 106933345A
- Authority
- CN
- China
- Prior art keywords
- modal
- information
- user
- brain wave
- topic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Neurosurgery (AREA)
- General Health & Medical Sciences (AREA)
- Neurology (AREA)
- Health & Medical Sciences (AREA)
- Dermatology (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- Manipulator (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The present invention provides a kind of multi-modal exchange method and device for intelligent robot.Wherein, the method is comprised the following steps：Obtain the multi-modal input information of user；Parse involved object or topic in multi-modal input information；The brain wave information that the user produces to the object or the topic is obtained, multi-modal man-machine interaction strategy is determined, corresponding multi-modal output data is exported, the brain wave information carries user emotion information.The present invention is interacted by the man-machine interaction strategy determined based on brain wave with reference to the strategy, can increase interactive topic, improves user's unhealthy emotion, improves the intelligent and class human nature of intelligent robot, enhancing user's viscosity.
Description
Technical field
The present invention relates to field in intelligent robotics, specifically, it is related to a kind of multi-modal exchange method of intelligent robot
And device.
Background technology
With continuing to develop for science and technology, the introducing of information technology, computer technology and artificial intelligence technology, machine
Industrial circle is progressively walked out in the research of people, gradually extend to the neck such as medical treatment, health care, family, amusement and service industry
Domain.And people for the requirement of robot also conform to the principle of simplicity the multiple mechanical action of substance be promoted to anthropomorphic question and answer, independence and with
The intelligent robot that other robot is interacted, man-machine interaction also just turns into the key factor for determining intelligent robot development,
Therefore, the interactive capability of intelligent robot is improved, intelligent, the weight as current urgent need to resolve of intelligent robot is lifted
Want problem.
The content of the invention
Above mentioned problem it is an object of the invention to solve prior art, there is provided a kind of for the multi-modal of intelligent robot
Exchange method.The described method comprises the following steps：
Obtain the multi-modal input information of user；
Parse involved object information or topic information in multi-modal input information；
The brain wave information that the user produces to the object or the topic is obtained, multi-modal man-machine interaction strategy is determined, it is defeated
Go out corresponding multi-modal output data, the brain wave information carries user emotion information.
Multi-modal exchange method for intelligent robot of the invention, it is preferred that also including by following step
It is rapid to preset multi-modal man-machine interaction strategy,
The advance visual information for gathering user in training environment, auditory information and brain wave information；
The brain wave information is parsed, the mood of user is determined；
Object or topic to being related in visual information and auditory information carry out the mark based on mood, with generate on
Object or the policy information of the multi-modal output of topic.
Multi-modal exchange method for intelligent robot of the invention, it is preferred that also include：
After multi-modal output data is exported, brain wave letter of the collection user to the multi-modal output data feedback
Breath, to obtain the feedback mood of the user；
The mood of user feedback is redefined into multi-modal man-machine interaction strategy, the multi-modal output number that output matches
According to.
Multi-modal exchange method for intelligent robot of the invention, it is preferred that also include：
Obtain user's current emotional states and the default multi-modal man-machine interaction strategy is carried out according to emotional state
Adjustment；
The emotional state is obtained by gathering brain wave information mode.
Multi-modal exchange method for intelligent robot of the invention, it is preferred that described brain wave information
Gathered using EEG equipment.
According to another aspect of the present invention, a kind of multi-modal interactive device for intelligent robot is additionally provided, it is described
Device is included with lower module：
Multi-modal information input module, its multi-modal input information for being used to obtain user；
Multi-modal information parsing module, it is used to parse involved object information or topic letter in multi-modal input information
Breath；
Multi-modal information output module, it is used to obtain the brain wave information that the user produces the object or the topic,
Determine multi-modal man-machine interaction strategy, export corresponding multi-modal output data, the brain wave information carries user emotion letter
Breath.
Multi-modal interactive device for intelligent robot of the invention, it is preferred that also including the default mould of strategy
Block, it presets multi-modal man-machine interaction strategy, and the module is further included such as lower unit：
Information acquisition unit, it is used to gather in advance visual information of the user in training environment, auditory information and brain electricity
Ripple information；
Mood determining unit, it is used to parse the brain wave information, determines the mood of user；
Policy information generation unit, its object or topic for being used to being related in visual information and auditory information carry out base
In the mark of mood, to generate the policy information on object or the multi-modal output of topic.
Multi-modal interactive device for intelligent robot of the invention, it is preferred that the multi-modal information is defeated
Go out module, it further after multi-modal output data is exported, gathers brain of the user to the multi-modal output data feedback
Wave information, to obtain the feedback mood of the user；The mood of user feedback is redefined into multi-modal man-machine interaction strategy, it is defeated
Go out the multi-modal output data for matching.
Multi-modal interactive device for intelligent robot of the invention, it is preferred that also include：
Policy information adjusting module, it is used to obtain user's current emotional states and according to emotional state to described default
Multi-modal man-machine interaction strategy is adjusted, and the emotional state is obtained by gathering brain wave information mode.
Multi-modal interactive device for intelligent robot of the invention, it is preferred that described information collecting unit
Including EEG equipment, its described brain wave information of collection.
The present invention is brought to be beneficial in that, method according to embodiments of the present invention, and robot is obtaining user's
After multi-modal input information, involved object or topic in multi-modal input information are parsed, then, obtain the user to the object
Or the brain wave information that the topic is produced, determine multi-modal man-machine interaction strategy, corresponding multi-modal output data is exported, it is described
Brain wave information carries user emotion information.Man-machine interaction strategy by being determined based on brain wave of the invention, with reference to the strategy
Interact, interactive topic can be increased, improve user's unhealthy emotion, improve the intelligent and class human nature of intelligent robot, increase
Strong user's viscosity.
Other features and advantages of the present invention will be illustrated in the following description, also, the partly change from specification
Obtain it is clear that or being understood by implementing the present invention.The purpose of the present invention and other advantages can be by specification, rights
Specifically noted structure is realized and obtained in claim and accompanying drawing.
Brief description of the drawings
Accompanying drawing is used for providing a further understanding of the present invention, and constitutes a part for specification, with reality of the invention
Apply example to be provided commonly for explaining the present invention, be not construed as limiting the invention.In the accompanying drawings：
Fig. 1 is that the flow for being related to the multi-modal exchange method for intelligent robot of first embodiment of the invention is illustrated
Figure.
Fig. 2 is the idiographic flow schematic diagram for being related to the default multi-modal man-machine interaction strategy of first embodiment of the invention.
Fig. 3 is that the flow for being related to the multi-modal exchange method for intelligent robot of second embodiment of the invention is illustrated
Figure.
Fig. 4 is that the flow for being related to the multi-modal exchange method for intelligent robot of third embodiment of the invention is illustrated
Figure.
Fig. 5 is the structural frames for being related to the multi-modal interactive device 400 for intelligent robot of fourth embodiment of the invention
Figure.
Specific embodiment
Describe embodiments of the present invention in detail below with reference to drawings and Examples, how the present invention is applied whereby
Technological means solves technical problem, and reaches the implementation process of relevant art effect and can fully understand and implement according to this.This Shen
Each feature that please be in embodiment and embodiment, can be combined with each other under the premise of not colliding, the technical scheme for being formed
Within protection scope of the present invention.
In addition, the flow of accompanying drawing can be in the such as one group computer system of computer executable instructions the step of illustrating
Middle execution.And, although show logical order in flow charts, but in some cases, can be with different from herein
Order performs shown or described step.
In the prior art, when user talks with robot, language, facial expression that robot can be sent by user
With action etc. it is multi-modal enter data to determine active user heartbeat conditions, then according to heartbeat conditions come to user output compared with
It is suitable multi-modal information.But, due to the multi-modal informations such as the language of user, facial expression and action have it is uncertain
Property, such as user can cover up the true emotional of oneself by way of pretending facial expression, therefore, robot is to these
When data are analyzed, parsing precision is relatively low, and the reliability of Emotion identification is low, so that the multi-modal data of correspondence output is not yet
Match somebody with somebody.Therefore, the intelligent and personification of existing robot is poor, it is impossible to analyze the real heartbeat conditions of user, drop exactly
Low user experience.Therefore, being accomplished by one kind can exactly analyze the real heartbeat conditions of user, increase interaction topic,
Improve user's unhealthy emotion, improve the intelligent and class human nature of intelligent robot, the solution of enhancing user's viscosity.
The multi-modal exchange method for intelligent robot of the embodiment of the present invention can make robot with user mutual
During, the real heartbeat conditions of user are analyzed exactly, increase interaction topic, improve intelligent robot and class human nature.Tool
For body, intelligent robot obtains multi-modal input information, parses object or topic, Yi Jiyong involved by multi-modal input information
To the object or the brain wave information of the topic, being then based on default multi-modal man-machine interaction strategy (can referred to as " man-machine friendship at family
Mutual strategy ") the corresponding multi-modal output data of output, i.e. according to the user plane to the object or the topic when brain wave that produces
The emotional information that information is shown finds corresponding multi-modal output data and is exported from man-machine interaction strategy.It is default
Multi-modal man-machine interaction strategy can be directed to the Personalized Policies of the individual customization of a certain user, the user plane pair can be based on
Brain wave information that different objects or theme are produced generates policy information, for example, in robot according to the corresponding brain of object
When wave information obtains user's mood now for negative feeling, by combining multi-modal man-machine interaction strategy, exported to user
One can make user's easily multi-modal data, song that such as user is liked etc., can so improve the bad of user
Mood, enhancing user uses the viscosity of robot.
In addition, the preset procedures on multi-modal man-machine interaction strategy, robot collect in advance user plane to different things,
Brain wave delta data during topic scene, and the thing observed and hear according to brain wave change, corresponding emotional state pair
Thing, content, scene are classified, and user is liked according to fancy grade, disagreeable degree etc./nuisance thing, topic etc. set
Determine priority, then deep learning is carried out based on above- mentioned information, generate man-machine interaction strategy.Because the man-machine interaction strategy is to be based on
Brain wave information carries out mood parsing, therefore the degree of accuracy is higher, and accurate multimode can be provided when being easy to be used below
State data are to user.
It should be noted that because default multi-modal interactive strategy is directed to hobby of the user within certain time
Set policy information, with the change of user's subjective factor, the policy information in default policy library can exist certain inclined
Difference, it is therefore desirable to which adjustment carrys out optimisation strategy information.The embodiment of the present invention obtains user to multi-modal defeated during interacting
Go out the emotional state of data response, man-machine interaction strategy is adjusted according to emotional state, so as in next time interactive, can be defeated
Go out and give user more accurate multi-modal information.
In addition, after multi-modal output data is exported to user, it is possible that user is to multi-modal output data institute
Emotional information and unmatched phenomenon in the mood and man-machine interaction strategy of feedback, therefore, it is also desirable to further gather user
For the brain wave information that a certain multi-modal output data is fed back, and then feedback mood is obtained, then again by feedback data weight
It is new to combine man-machine interaction strategy, the content more matched to user is exported again, and it is possible to reference to feedback data, to man-machine
Interactive strategy is adjusted, so that man-machine interaction strategy is closer to user's request.So can further adjust the bad of user
Mood.Certainly, it is active mood in the mood for monitoring the brain wave message reflection of user feedback in order to increase topic content
When, it is also possible to recombine man-machine interaction strategy, output and more abundant, detailed topic content.
When brain wave information is gathered, because EEG equipment costs are relatively low, information gathering is simple and easy to do and ensure that collection
During user comfortableness, it is preferred to use EEG equipment.
First embodiment
Fig. 1 is the schematic flow sheet of the example one for being related to the multi-modal exchange method for intelligent robot of the invention,
The method of the embodiment is mainly included the following steps that.
In step s 110, robot obtains the multi-modal input information of user.
In this step, the multi-modal input information of acquisition mainly include coming from the text data of user, voice data,
Video data, view data and/or the program for allowing the robot to export certain action or execution software or hardware
The information of instruction.In specific example, during robot and user mutual, robot is called and starts text input and sets
Standby (such as touch-screen), sound collection equipment (such as microphone) and image capture device (such as camera), are set by these
It is standby come persistently monitor and catch text message, voice messaging and user's face expression information and/or limb action information.
In the step s 120, involved object information or topic information in multi-modal input information are parsed.
Then, the multi-modal input information for receiving is parsed, obtains the thing included in these multi-modal datas
Body information or topic information, specifically, object information is mainly the title of object, and subject information is mainly subject name and general
Summary/particular content, such as film, music, dancing etc.
In this example, robot is mainly and parses the relevant thing of extraction according to keyword from text message or acoustic information
The object information or subject information of body.By taking acoustic information as an example, robot obtain multi-modal data after, by the multi-modal data
It is submitted to ASR and VPR (vocal prints that the ASR or local and cloud server of local ASR or cloud server mix
Identification, Voiceprint Recognition) engine.These engines convert voice data to text message using ASR technology.
The specific pretreatment that such as denoising etc is first carried out to multi-modal input data, then carries out language by pretreated voice messaging
The comprehensive analysis of sound identification, generates text message corresponding with voice messaging.Furthermore, it is understood that according to voice in identification process
The model of identification, the sound template that will be prestored is compared with the feature of the voice signal of input, according to certain search
And matching strategy, find out a series of optimal templates with input voice match.Then according to the definition of this template, by tabling look-up
Recognition result can be just given.Then, the keyword of noun part-of-speech is extracted from recognition result, then confirm object information or
Subject information.If not parsing object information or subject information from text message or voice messaging, robot is further
View data is parsed, the object information in image is identified by object recognition technique.
In step s 130, the brain wave information that the user produces to the object or the topic is obtained, multi-modal people is determined
Machine interactive strategy, exports corresponding multi-modal output data, wherein, brain wave information carries user emotion information.
Specifically, the brain wave information that user produces to the object or topic is obtained, according to brain wave information correspondence
Emotional information search corresponding multi-modal output data, be then output to user.
In default multi-modal man-machine interaction strategy is set up, be may include steps of for overview：Design induces each
The stimulus of mood (active mood and negative feeling) is planted, is then obtained with the EEG signal for inducing mood using EEG equipment,
EEG signals are obtained into temporal signatures, frequency domain character or time-frequency characteristics by mathematic(al) manipulation, is ground according to related physiology, psychology
Study carefully achievement and extract EEG signal feature, by Data Dimensionality Reduction and feature selecting, so as to obtain being adapted to the EEG feature sets of mood classification,
Feature based collection carries out mood classification.Fig. 2 is the tool for being related to the default multi-modal man-machine interaction strategy of first embodiment of the invention
Body schematic flow sheet.Illustrate how to preset multi-modal man-machine interaction strategy below with reference to Fig. 2.
In sub-step S1010, the advance visual information for gathering user in training environment, auditory information and brain wave letter
Breath.
During emotional training is carried out to a certain user, the stimulus material for inducing user's difference mood is first designed.
The stimulus material of conventional induction mood has visual stimulus material, acoustic stimuli material and multimedia stimulus material etc. at present.Depending on
Feel that stimulus material mainly has words and phrases sentence and picture etc., acoustic stimuli material mainly has the non-karst areas sound with different emotions
Section, word, words and phrases or phrase etc., multimedia stimulus material often combine the dynamic materials such as vision, the sense of hearing.Mood induces
Refer to directly to manipulate emotional state, then record and observe positive or negative feeling.Because viewing picture or film can be quantified
Ground induces tested related emotional, and it turns into mood and induces a kind of the most frequently used method.Using picture as visual stimulus source, in order to
Ensure that stimulus material can preferably induce specific mood, when selected picture is showed to user, every width picture is in
The existing time may remain in five seconds.In addition, having various objects or subject information in these stimulus materials, mood is being carried out
Before training, these information can be in advance obtained.
During being trained, user can produce different eeg signals, brain electric information collection according to stimulus material
Equipment records the brain electric information of subject according to the sampling period of setting.At the same time, in order to preferably accurately identify user
State, user can also select corresponding emotional state according to the impression of oneself, and the emotional state is synchronous with brain electric information
Record.
In sub-step S1020, brain wave information is parsed, determine the mood of user.
Because EEG signals are very faint, therefore the interference of other noise signals is highly susceptible in gatherer process, needed
The EEG signals for collecting are pre-processed to remove the artefact of doping.Spy then is carried out to the EEG signals for removing artefact
Extraction is levied, these features can be temporal signatures, frequency domain character or time-frequency characteristics.According to these features and before according to training
Sample obtained by different moods (such as tranquil, glad, sad, frightened) corresponding brain electrical feature determine the feelings of user
Thread.In order to ensure the accuracy of mood classification, it is also possible to reference to the emotional state that user selects in training.In addition, except upper
Outside the common attribute of three kinds of face, many other features can also be extracted from EEG signals, such as entropy, fractal dimension and self-defined
Feature etc..
It should be noted that, although coming etc. to realize using in this example using single physiological signal, i.e. brain wave mode
Trained for the Emotion identification of different objects/topic at family.It is also possible to be based on facial expression, voice, posture, text and brain telecommunications
Number etc. the mode of Multi-source Information Fusion realize that this example is not specifically limited.
In sub-step S1030, object or topic to being related in visual information and auditory information are carried out based on mood
Mark, to generate the policy information on object or the multi-modal output of topic.
Because different brain wave information is that the different object of correspondence or topic are produced, therefore according to the user after determination
Mood is classified to different objects, topic, at the same time it can also excellent to object or topic setting according to hobby/disagreeable degree
First level, being then based on above- mentioned information carries out deep learning (such as algorithm of support vector machine) to generate man-machine interactive strategy.
In this example, man-machine interaction strategy is also wrapped both comprising brain electric information, the emotional state for different objects/topic
Multi-modal output data containing the different emotional states of correspondence.Represented with three base tables：Eeg data table, storage user is not for
The brain electric information of same object/topic；Emotional state table, for the emotional state of different brain electric informations；Multi-modal output policy table,
For the exportable different other multi-modal output datas of priority of different emotional states.
After robot obtains the multi-modal data from user, can be by searching eeg data table, emotional state table
With multi-modal output policy table, the multi-modal data for most matching is exported according to priority.For example, speaking of one in user makes it
During offending theme, the emotional state that robot passes through the current brain electricity display of user should according to man-machine interaction policy selection
The favorite music of user or film are exported to the user, and then improve the unhappy mood of user.
To sum up, according to the present embodiment, robot can be according to using brain wave information during being interacted with user
The multi-modal output data of the man-machine interaction strategy output matching of generation, more can accurately provide the multi-modal output number of matching
According to, and the topic that interacts with user can be increased, and improve user's unhealthy emotion, improve the intelligent and class people of intelligent robot
Property, enhancing user's viscosity.
Second embodiment
Fig. 3 is the schematic flow sheet of the example two for being related to the multi-modal exchange method for intelligent robot of the invention,
The method of the embodiment mainly includes the following steps that, wherein, the step similar to first embodiment is marked with identical label,
And its particular content is repeated no more, only difference step is specifically described.
In step s 110, robot obtains the multi-modal input information of user.
In the step s 120, involved object information or topic information in multi-modal input information are parsed.
In step s 130, the brain wave information that the user produces to the object or the topic is obtained, multi-modal people is determined
Machine interactive strategy, exports corresponding multi-modal output data, and the brain wave information carries user emotion information.
In step S210, after multi-modal output data is exported, collection user is anti-to the multi-modal output data
The brain wave information of feedback, to obtain the feedback mood of the user.
In other words, the step is the validity of the multi-modal output policy before being judged by brain wave information.In user
After obtaining multi-modal output data, the data can be responded and feedback output is carried out, the brain wave using EEG equipment collection user is believed
Breath, by being pre-processed to brain wave information and the conversion process such as time domain, frequency domain obtains brain electrical characteristic data, further according to these
Different moods (such as tranquil, glad, sad, frightened) obtained by characteristic and the before sample according to training are corresponding
Brain electrical feature determines the mood of user.
It should be noted that the brain wave information except gathering user, can also simultaneously gather facial expression, the language of user
The data such as sound, further the analysis result according to these multi-modal datas judge the emotional state of user.
In step S220, the mood of user feedback is redefined into multi-modal man-machine interaction strategy, what output matched
Multi-modal output data.
When strategy is reselected, on the one hand, in the case that the emotional state before user is negative feeling, if with
The emotional state of family current feedback is active mood, then according to man-machine interaction strategy, select and the multi-modal output for sending before
The more detailed content of object information or the subject information correlation in strategy further improves the interactive entertaining of user to user
Property；If the emotional state of user's current feedback is negative feeling, according to man-machine interaction strategy, selection is more with what is sent before
The content of object information or the low one-level of subject information priority level in mode output policy helps relax use to user, further
The negative feeling at family.On the other hand, in the case that the emotional state before user is active mood, if user's current feedback
Emotional state be active mood, then according to man-machine interaction strategy, can select with the multi-modal output policy for sending before
Object information or the related more detailed content of subject information be shared with user；If the emotional state of user's current feedback
It is negative feeling, then according to man-machine interaction strategy, highest-ranking object information or theme on fancy grade can be selected to believe
Breath issues user as multi-modal output data, and help relaxes the negative feeling of user.
3rd embodiment
Fig. 4 is the schematic flow sheet of the example three for being related to the multi-modal exchange method for intelligent robot of the invention,
The method of the embodiment mainly includes the following steps that, wherein, the step similar to second embodiment is marked with identical label,
And its particular content is repeated no more, only difference step is specifically described.
In step s 110, robot obtains the multi-modal input information of user.
In the step s 120, involved object information or topic information in multi-modal input information are parsed.
In step s 130, the brain wave information that the user produces to the object or the topic is obtained, multi-modal people is determined
Machine interactive strategy, exports corresponding multi-modal output data, and the brain wave information carries user emotion information.
In step S210, after multi-modal output data is exported, collection user is anti-to the multi-modal output data
The brain wave information of feedback, to obtain the feedback mood of the user.
In step S220, the mood of user feedback is redefined into multi-modal man-machine interaction strategy, what output matched
Multi-modal output data.
In step S310, user's current emotional states are obtained and according to emotional state to default multi-modal man-machine interaction
Strategy is adjusted, and emotional state is obtained by gathering brain wave information mode.
After the multi-modal output data newly matched to user is exported again, the brain wave of user is gathered by EEG equipment
Information, the analysis result according to brain wave information determines the emotional state of user, and the specific method for obtaining emotional state can be adopted
Operated with step S210 identicals, man-machine interaction strategy is then adjusted according to emotional state.If for example, preceding once export multimode
After state output data, the emotional state of user feedback is negative feeling, and after this exports new multi-modal output data again,
The emotional state of user feedback is active mood, then be adjusted the priority level of this multi-modal output data for exporting.
By so adjustment, can export and give user more accurate multi-modal information in next time interactive.
Fourth embodiment
Fig. 5 is the structured flowchart of the multi-modal interactive device 400 for intelligent robot of the embodiment of the present application.Such as Fig. 5
Shown, the multi-modal interactive device 400 of the embodiment of the present application mainly includes：Multi-modal information input module 410, multi-modal information
Parsing module 420 and multi-modal information output module 430.
Wherein, multi-modal information input module 410, its multi-modal input information for being used to obtain user.
Multi-modal information parsing module 420, it is used to parse involved object or topic in multi-modal input information；
Multi-modal information output module 430, it is used to obtain the brain wave letter that the user produces the object or the topic
Breath, determines multi-modal man-machine interaction strategy, exports corresponding multi-modal output data, and the brain wave information carries user emotion
Information.Multi-modal information output module 430, further after multi-modal output data is exported, user is to described more for collection for it
The brain wave information of mode output data feedback, to obtain the feedback mood of the user；The mood of user feedback is redefined
Multi-modal man-machine interaction strategy, the multi-modal output data that output matches.
In addition, the device 400 also includes tactful presetting module 440, it presets multi-modal man-machine interaction strategy, the module
440 further include such as lower unit：Information acquisition unit 4410, mood determining unit 4420 and policy information generation unit
4430。
Information acquisition unit 4410, its be used to gather in advance visual information of the user in training environment, auditory information and
Brain wave information.Information acquisition unit 4410 includes EGG equipment, its described brain wave information of collection.Mood determining unit 4420,
It is used to parse the brain wave information, determines the mood of user.Policy information generation unit 4430, it is used to regarding
The object or topic being related in feel information and auditory information carry out the mark based on mood, many on object or topic to generate
The policy information of mode output.
In addition, the device 400 also includes policy information adjusting module 450, it is used to obtain user's current emotional states simultaneously
The default multi-modal man-machine interaction strategy is adjusted according to emotional state, the emotional state is by gathering brain wave
Information mode is obtained.
By rationally setting, the multi-modal interactive device 400 of the present embodiment can perform first embodiment, second embodiment
With each step of 3rd embodiment, here is omitted.
Because the method for the present invention describes what is realized in computer systems.The computer system can for example be set
In the control core processor of robot.For example, method described herein can be implemented as what can be performed with control logic
Software, it is performed by the CPU in robot operating system.Function as herein described can be implemented as storage to be had in non-transitory
Programmed instruction set in shape computer-readable medium.When implemented in this fashion, the computer program includes one group of instruction,
When group instruction is run by computer, it promotes computer to perform the method that can implement above-mentioned functions.FPGA can be temporary
When or be permanently mounted in non-transitory tangible computer computer-readable recording medium, for example ROM chip, computer storage,
Disk or other storage mediums.In addition to being realized with software, logic as herein described can utilize discrete parts, integrated electricity
What road and programmable logic device (such as, field programmable gate array (FPGA) or microprocessor) were used in combination programmable patrols
Volume, or embodied including any other equipment that they are combined.All such embodiments are intended to fall under model of the invention
Within enclosing.
It should be understood that disclosed embodiment of this invention is not limited to ad hoc structure disclosed herein, process step
Or material, and the equivalent substitute of these features that those of ordinary skill in the related art are understood should be extended to.Should also manage
Solution, term as used herein is only used for describing the purpose of specific embodiment, and is not intended to limit.
" one embodiment " or " embodiment " mentioned in specification means special characteristic, the structure for describing in conjunction with the embodiments
Or characteristic is included at least one embodiment of the present invention.Therefore, the phrase " reality that specification various places throughout occurs
Apply example " or " embodiment " same embodiment might not be referred both to.
While it is disclosed that implementation method as above, but described content is only to facilitate understanding the present invention and adopting
Implementation method, is not limited to the present invention.Any those skilled in the art to which this invention pertains, are not departing from this
On the premise of the disclosed spirit and scope of invention, any modification and change can be made in the formal and details implemented,
But scope of patent protection of the invention, must be still defined by the scope of which is defined in the appended claims.
Claims (10)
1. a kind of multi-modal exchange method for intelligent robot, it is characterised in that the described method comprises the following steps：
Obtain the multi-modal input information of user；
Parse involved object information or topic information in multi-modal input information；
The brain wave information that the user produces to the object or the topic is obtained, multi-modal man-machine interaction strategy is determined, it is right to export
The multi-modal output data answered, the brain wave information carries user emotion information.
2. the multi-modal exchange method of intelligent robot is used for as claimed in claim 1, it is characterised in that also including by such as
Lower step presets multi-modal man-machine interaction strategy,
The advance visual information for gathering user in training environment, auditory information and brain wave information；
The brain wave information is parsed, the mood of user is determined；
Object or topic to being related in visual information and auditory information carry out the mark based on mood, to generate on object
Or the policy information of the multi-modal output of topic.
3. the multi-modal exchange method of intelligent robot is used for as claimed in claim 1 or 2, it is characterised in that also included：
After multi-modal output data is exported, the brain wave information that collection user feeds back to the multi-modal output data, with
Obtain the feedback mood of the user；
The mood of user feedback is redefined into multi-modal man-machine interaction strategy, the multi-modal output data that output matches.
4. the multi-modal exchange method of intelligent robot is used for as claimed in claim 3, it is characterised in that also included：
Obtain user's current emotional states and the default multi-modal man-machine interaction strategy is adjusted according to emotional state；
The emotional state is obtained by gathering brain wave information mode.
5. the multi-modal exchange method for intelligent robot as described in claim any one of 1-4, it is characterised in that described
Brain wave information using EEG equipment gather.
6. a kind of multi-modal interactive device for intelligent robot, it is characterised in that described device is included with lower module：
Multi-modal information input module, its multi-modal input information for being used to obtain user；
Multi-modal information parsing module, it is used to parse involved object information or topic information in multi-modal input information；
Multi-modal information output module, it is used to obtain the brain wave information that the user produces the object or the topic, it is determined that
Multi-modal man-machine interaction strategy, exports corresponding multi-modal output data, and the brain wave information carries user emotion information.
7. the multi-modal interactive device of intelligent robot is used for as claimed in claim 6, it is characterised in that also including tactful pre-
If module, it presets multi-modal man-machine interaction strategy, and the module is further included such as lower unit：
Information acquisition unit, it is used to gather in advance visual information of the user in training environment, auditory information and brain wave letter
Breath；
Mood determining unit, it is used to parse the brain wave information, determines the mood of user；
Policy information generation unit, its object or topic for being used to being related in visual information and auditory information are carried out based on feelings
The mark of thread, to generate the policy information on object or the multi-modal output of topic.
8. the multi-modal interactive device for intelligent robot as claimed in claims 6 or 7, it is characterised in that
The multi-modal information output module, further after multi-modal output data is exported, user is to described more for collection for it
The brain wave information of mode output data feedback, to obtain the feedback mood of the user；The mood of user feedback is redefined
Multi-modal man-machine interaction strategy, the multi-modal output data that output matches.
9. the multi-modal interactive device of intelligent robot is used for as claimed in claim 8, it is characterised in that also included：
Policy information adjusting module, it is used to obtain user's current emotional states and according to emotional state to the default multimode
State man-machine interaction strategy is adjusted, and the emotional state is obtained by gathering brain wave information mode.
10. the multi-modal interactive device for intelligent robot as described in claim any one of 7-9, it is characterised in that institute
Stating information acquisition unit includes EEG equipment, its described brain wave information of collection.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710034038.4A CN106933345B (en) | 2017-01-18 | 2017-01-18 | Multi-modal interaction method and device for intelligent robot |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710034038.4A CN106933345B (en) | 2017-01-18 | 2017-01-18 | Multi-modal interaction method and device for intelligent robot |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106933345A true CN106933345A (en) | 2017-07-07 |
| CN106933345B CN106933345B (en) | 2020-02-07 |
Family
ID=59422826
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710034038.4A Active CN106933345B (en) | 2017-01-18 | 2017-01-18 | Multi-modal interaction method and device for intelligent robot |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106933345B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108037905A (en) * | 2017-11-21 | 2018-05-15 | 北京光年无限科技有限公司 | A kind of interaction output method and intelligent robot for intelligent robot |
| CN108279777A (en) * | 2018-02-11 | 2018-07-13 | 广东欧珀移动通信有限公司 | Brain wave control method and relevant device |
| CN108392201A (en) * | 2018-02-26 | 2018-08-14 | 广东欧珀移动通信有限公司 | Brain training method and relevant device |
| CN108494952A (en) * | 2018-03-05 | 2018-09-04 | 广东欧珀移动通信有限公司 | Voice communication processing method and relevant device |
| CN108594989A (en) * | 2018-03-26 | 2018-09-28 | 广东欧珀移动通信有限公司 | Acquiring brain waves method and relevant device |
| CN109278051A (en) * | 2018-08-09 | 2019-01-29 | 北京光年无限科技有限公司 | Exchange method and system based on intelligent robot |
| CN110109541A (en) * | 2019-04-25 | 2019-08-09 | 广州智伴人工智能科技有限公司 | A kind of method of multi-modal interaction |
| CN110147729A (en) * | 2019-04-16 | 2019-08-20 | 深圳壹账通智能科技有限公司 | User emotion recognition methods, device, computer equipment and storage medium |
| CN110287305A (en) * | 2019-07-03 | 2019-09-27 | 浪潮云信息技术有限公司 | A kind of intelligent answer management system based on natural language processing |
| CN110742559A (en) * | 2019-10-24 | 2020-02-04 | 佛山市云米电器科技有限公司 | Floor sweeping robot control method and system based on brain wave detection |
| CN111046148A (en) * | 2018-10-11 | 2020-04-21 | 上海智臻智能网络科技股份有限公司 | Intelligent interaction system and intelligent customer service robot |
| US20200201435A1 (en) * | 2018-12-20 | 2020-06-25 | Massachusetts Institute Of Technology | End-To-End Deep Neural Network For Auditory Attention Decoding |
| CN112106381A (en) * | 2018-05-17 | 2020-12-18 | 高通股份有限公司 | User experience assessment |
| CN112518743A (en) * | 2020-11-20 | 2021-03-19 | 南京航空航天大学 | Multi-mode neural decoding control system and method for on-orbit operation of space manipulator |
| CN116872222A (en) * | 2023-06-21 | 2023-10-13 | 广州市恒缘健康信息科技有限公司 | Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140114588A (en) * | 2013-03-19 | 2014-09-29 | 세종대학교산학협력단 | Method and Apparatus for Emotion recognition based on Complex physiological signals |
| CN104407699A (en) * | 2014-11-24 | 2015-03-11 | 深圳信息职业技术学院 | Human-computer interaction method, device and system |
| CN105867633A (en) * | 2016-04-26 | 2016-08-17 | 北京光年无限科技有限公司 | Intelligent robot oriented information processing method and system |
-
2017
- 2017-01-18 CN CN201710034038.4A patent/CN106933345B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20140114588A (en) * | 2013-03-19 | 2014-09-29 | 세종대학교산학협력단 | Method and Apparatus for Emotion recognition based on Complex physiological signals |
| CN104407699A (en) * | 2014-11-24 | 2015-03-11 | 深圳信息职业技术学院 | Human-computer interaction method, device and system |
| CN105867633A (en) * | 2016-04-26 | 2016-08-17 | 北京光年无限科技有限公司 | Intelligent robot oriented information processing method and system |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108037905A (en) * | 2017-11-21 | 2018-05-15 | 北京光年无限科技有限公司 | A kind of interaction output method and intelligent robot for intelligent robot |
| CN108279777A (en) * | 2018-02-11 | 2018-07-13 | 广东欧珀移动通信有限公司 | Brain wave control method and relevant device |
| CN108392201A (en) * | 2018-02-26 | 2018-08-14 | 广东欧珀移动通信有限公司 | Brain training method and relevant device |
| CN108392201B (en) * | 2018-02-26 | 2022-03-22 | Oppo广东移动通信有限公司 | Brain training method and related equipment |
| CN108494952A (en) * | 2018-03-05 | 2018-09-04 | 广东欧珀移动通信有限公司 | Voice communication processing method and relevant device |
| CN108494952B (en) * | 2018-03-05 | 2021-07-09 | Oppo广东移动通信有限公司 | Voice call processing method and related equipment |
| CN108594989A (en) * | 2018-03-26 | 2018-09-28 | 广东欧珀移动通信有限公司 | Acquiring brain waves method and relevant device |
| CN108594989B (en) * | 2018-03-26 | 2022-03-22 | Oppo广东移动通信有限公司 | Brain wave acquisition method and related equipment |
| CN112106381A (en) * | 2018-05-17 | 2020-12-18 | 高通股份有限公司 | User experience assessment |
| CN109278051A (en) * | 2018-08-09 | 2019-01-29 | 北京光年无限科技有限公司 | Exchange method and system based on intelligent robot |
| CN111046148A (en) * | 2018-10-11 | 2020-04-21 | 上海智臻智能网络科技股份有限公司 | Intelligent interaction system and intelligent customer service robot |
| US20200201435A1 (en) * | 2018-12-20 | 2020-06-25 | Massachusetts Institute Of Technology | End-To-End Deep Neural Network For Auditory Attention Decoding |
| US11630513B2 (en) * | 2018-12-20 | 2023-04-18 | Massachusetts Institute Of Technology | End-to-end deep neural network for auditory attention decoding |
| CN110147729A (en) * | 2019-04-16 | 2019-08-20 | 深圳壹账通智能科技有限公司 | User emotion recognition methods, device, computer equipment and storage medium |
| CN110109541A (en) * | 2019-04-25 | 2019-08-09 | 广州智伴人工智能科技有限公司 | A kind of method of multi-modal interaction |
| CN110109541B (en) * | 2019-04-25 | 2022-04-05 | 广州智伴人工智能科技有限公司 | Multi-modal interaction method |
| CN110287305A (en) * | 2019-07-03 | 2019-09-27 | 浪潮云信息技术有限公司 | A kind of intelligent answer management system based on natural language processing |
| CN110742559A (en) * | 2019-10-24 | 2020-02-04 | 佛山市云米电器科技有限公司 | Floor sweeping robot control method and system based on brain wave detection |
| CN110742559B (en) * | 2019-10-24 | 2021-09-14 | 佛山市云米电器科技有限公司 | Floor sweeping robot control method and system based on brain wave detection |
| CN112518743A (en) * | 2020-11-20 | 2021-03-19 | 南京航空航天大学 | Multi-mode neural decoding control system and method for on-orbit operation of space manipulator |
| CN116872222A (en) * | 2023-06-21 | 2023-10-13 | 广州市恒缘健康信息科技有限公司 | Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium |
| CN116872222B (en) * | 2023-06-21 | 2024-02-06 | 广州市恒缘健康信息科技有限公司 | Psychological consultation robot, psychological consultation system, psychological consultation control method and psychological consultation storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106933345B (en) | 2020-02-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106933345A (en) | For the multi-modal exchange method and device of intelligent robot | |
| Cavallo et al. | Emotion modelling for social robotics applications: a review | |
| Feine et al. | A taxonomy of social cues for conversational agents | |
| Hu et al. | Attention recognition in EEG-based affective learning research using CFS+ KNN algorithm | |
| Zhang et al. | Intelligent facial emotion recognition and semantic-based topic detection for a humanoid robot | |
| US20210191506A1 (en) | Affective interaction systems, devices, and methods based on affective computing user interface | |
| Pell et al. | Emotional speech processing: Disentangling the effects of prosody and semantic cues | |
| Ma et al. | ElderReact: a multimodal dataset for recognizing emotional response in aging adults | |
| Wang et al. | Multi-modal emotion recognition using EEG and speech signals | |
| CN105760362B (en) | A kind of question and answer evaluation method and device towards intelligent robot | |
| CN103412646A (en) | Emotional music recommendation method based on brain-computer interaction | |
| CN105912530A (en) | Intelligent robot-oriented information processing method and system | |
| Guediche et al. | Temporal cortex reflects effects of sentence context on phonetic processing | |
| CN107016046A (en) | The intelligent robot dialogue method and system of view-based access control model displaying | |
| CN108186030A (en) | A kind of stimulus information provides the cognition index analysis method of device and latent energy value test | |
| CN106502382A (en) | Active exchange method and system for intelligent robot | |
| Heraz et al. | Recognition of emotions conveyed by touch through force-sensitive screens: Observational study of humans and machine learning techniques | |
| Ramzan et al. | Emotion recognition by physiological signals | |
| CN113749656B (en) | Emotion recognition method and device based on multidimensional physiological signals | |
| Le Maitre et al. | Self-talk discrimination in human–robot interaction situations for supporting social awareness | |
| CN113053492B (en) | Self-adaptive virtual reality intervention system and method based on user background and emotion | |
| Marcos et al. | Emotional AI in Healthcare: a pilot architecture proposal to merge emotion recognition tools | |
| CN112466435B (en) | Psychological coaching scheme determination method and device, storage medium and electronic device | |
| Esposito et al. | On the recognition of emotional vocal expressions: motivations for a holistic approach | |
| Jelec | Are Abstract Concepts Like Dinosaur Feathers?: Conceptual Metaphor Theory and the conceptualisation strategies in gesture of blind and visually impaired children |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |