CN111221406A - Information interaction method and device - Google Patents
Information interaction method and device Download PDFInfo
- Publication number
- CN111221406A CN111221406A CN201811409375.8A CN201811409375A CN111221406A CN 111221406 A CN111221406 A CN 111221406A CN 201811409375 A CN201811409375 A CN 201811409375A CN 111221406 A CN111221406 A CN 111221406A
- Authority
- CN
- China
- Prior art keywords
- image
- gesture information
- trigger event
- preset
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003993 interaction Effects 0.000 title claims abstract description 67
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000004891 communication Methods 0.000 claims description 29
- 230000002452 interceptive effect Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 230000000875 corresponding effect Effects 0.000 description 76
- 230000008569 process Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 230000009471 action Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The embodiment of the invention provides an information interaction method, which comprises the following steps: acquiring a first to-be-identified image acquired by a user; identifying the first image to be identified to obtain first gesture information in the first image to be identified; and determining and executing a trigger event corresponding to the first gesture information so as to interact with the user. Therefore, the electronic equipment can determine and execute the corresponding trigger event by recognizing the gesture of the user, namely, the user can perform information interaction with the electronic equipment through the gesture, and therefore, the electronic equipment can provide richer interaction experience for the user.
Description
Technical Field
The present invention relates to the field of communications technologies, and in particular, to an information interaction method and apparatus.
Background
In some scenarios, the electronic device needs to perform information interaction with a user, for example, the user may send a voice instruction to the smart sound box, and the smart sound box may play a corresponding audio file after receiving the voice instruction of the user; alternatively, the user may send an instruction to the robot through the touch interface, the robot may perform a corresponding action after receiving the instruction of the user, and so on.
Generally, information interaction between the electronic device and the user is based on a voice signal sent by the user or an operation instruction sent by the user to the electronic device, and an application scene is single, so that richer interaction experience cannot be provided.
Disclosure of Invention
The embodiment of the invention aims to provide an information interaction method to provide richer interaction experience. The specific technical scheme is as follows:
the embodiment of the invention provides an information interaction method, which is applied to electronic equipment and comprises the following steps:
acquiring a first to-be-identified image acquired by a user;
identifying the first image to be identified to obtain first gesture information in the first image to be identified;
and determining and executing a trigger event corresponding to the first gesture information so as to interact with the user.
Optionally, the identifying the first image to be identified to obtain first gesture information in the first image to be identified includes:
performing template matching on the first image to be recognized and a plurality of preset images, and determining the preset image matched with the image to be recognized as a matched image;
and taking the gesture information corresponding to the matched image as the first gesture information of the first image to be recognized.
Optionally, the template matching the first image to be recognized with a plurality of preset images, and determining the preset image matched with the first image to be recognized as a matching image includes:
determining a plurality of regions to be detected from the first image to be recognized according to a preset traversal rule;
calculating a difference value between each region to be detected and each preset image according to each preset image, and determining the region to be detected corresponding to each preset image as a target region according to the difference value;
and taking the preset image with the minimum difference value of the target area as a preset image matched with the first image to be recognized.
Optionally, after determining and executing a trigger event corresponding to the first gesture information to interact with the user, the method further includes:
acquiring a second image to be identified, which is acquired by aiming at a user;
identifying the second image to be identified to obtain second gesture information in the second image to be identified;
judging whether the second gesture information meets a preset condition or not;
if yes, determining and executing a trigger event corresponding to the second gesture information; if not, acquiring a control instruction, and determining and executing a trigger event corresponding to the control instruction;
and returning to the step of acquiring the second image to be identified acquired aiming at the user.
Optionally, the determining whether the second gesture information meets a preset condition includes:
judging whether the confidence of the trigger event corresponding to the second gesture information is greater than a preset threshold value;
if yes, judging that the second gesture information meets a preset condition; if not, judging that the second gesture information does not meet the preset condition.
Optionally, the following steps are adopted to determine the confidence of the trigger event corresponding to the second gesture information:
acquiring a plurality of candidate trigger events and initial confidence coefficients; wherein the initial confidence of each candidate trigger event is equal;
randomly selecting a target trigger event from the candidate trigger events, using the target trigger event as a trigger event corresponding to the gesture information, and executing the trigger event;
if the interactive instruction of the user is received after the trigger event is executed, increasing the confidence coefficient of the target trigger event according to a preset updating rule;
and if the interactive instruction of the user is not received after the trigger event is executed, reducing the confidence coefficient of the target trigger event according to the preset updating rule.
Optionally, the determining whether the second gesture information meets a preset condition includes:
judging whether a time difference value between the moment of identifying the second gesture information and the moment of executing any trigger event at the last time is greater than a preset threshold value or not;
if not, judging that the second gesture information meets a preset condition; and if so, judging that the second gesture information does not meet the preset condition.
The embodiment of the invention also provides an information interaction device, which is applied to electronic equipment, and the device comprises:
the image acquisition module is used for acquiring a first to-be-identified image acquired by a user;
the processor is used for identifying the first image to be identified to obtain first gesture information in the first image to be identified and determining and executing a trigger event corresponding to the first gesture information;
and the communication module is used for interacting with the user.
Optionally, the processor is specifically configured to perform template matching on the first image to be recognized and a plurality of preset images, and determine a preset image matched with the image to be recognized as a matching image; and taking the gesture information corresponding to the matched image as the first gesture information of the first image to be recognized.
Optionally, the processor is specifically configured to:
determining a plurality of regions to be detected from the first image to be recognized according to a preset traversal rule;
calculating a difference value between each region to be detected and each preset image according to each preset image, and determining the region to be detected corresponding to each preset image as a target region according to the difference value;
and taking the preset image with the minimum difference value of the target area as a preset image matched with the first image to be recognized.
Optionally, the image acquisition module is further configured to acquire a second image to be identified, which is acquired by a user;
the processor is further configured to identify the second image to be identified to obtain second gesture information in the second image to be identified; judging whether the second gesture information meets a preset condition or not; if yes, determining and executing a trigger event corresponding to the second gesture information; if not, triggering the communication module;
the communication module is used for acquiring a control instruction;
the processor is further used for determining and executing a trigger event corresponding to the control instruction; and triggering the image acquisition module.
Optionally, the processor is further configured to determine whether a confidence of the trigger event corresponding to the second gesture information is greater than a preset threshold; if yes, judging that the second gesture information meets a preset condition; if not, judging that the second gesture information does not meet the preset condition.
Optionally, the following steps are adopted to determine the confidence of the trigger event corresponding to the second gesture information:
acquiring a plurality of candidate trigger events and initial confidence coefficients; wherein the initial confidence of each candidate trigger event is equal;
randomly selecting a target trigger event from the candidate trigger events, using the target trigger event as a trigger event corresponding to the gesture information, and executing the trigger event;
if the interactive instruction of the user is received after the trigger event is executed, increasing the confidence coefficient of the target trigger event according to a preset updating rule;
and if the interactive instruction of the user is not received after the trigger event is executed, reducing the confidence coefficient of the target trigger event according to the preset updating rule.
Optionally, the processor is further configured to determine whether a time difference between the time when the second gesture information is identified and the time when any trigger event is executed last time is greater than a preset threshold; if not, judging that the second gesture information meets a preset condition; and if so, judging that the second gesture information does not meet the preset condition.
The embodiment of the invention also provides electronic equipment which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus;
a memory for storing a computer program;
and the processor is used for realizing any one of the information interaction methods when executing the program stored in the memory.
The embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the computer program implements any of the above information interaction methods.
According to the information interaction method and device provided by the embodiment of the invention, the first to-be-identified image acquired by a user is acquired and identified to obtain the first gesture information in the first to-be-identified image, and then the trigger event corresponding to the first gesture information is determined and executed to interact with the user; therefore, the electronic equipment can determine and execute the corresponding trigger event by recognizing the gesture of the user, namely, the user can perform information interaction with the electronic equipment through the gesture, and therefore, the electronic equipment can provide richer interaction experience for the user. Of course, not all of the advantages described above need to be achieved at the same time in the practice of any one product or method of the invention.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart of an information interaction method according to an embodiment of the present invention;
FIG. 2 is a diagram illustrating a preset image;
fig. 3 is a schematic diagram of determining a plurality of regions to be detected from a first image to be recognized according to a preset traversal rule under one condition;
fig. 4 is a schematic flowchart of another information interaction method according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of an information interaction apparatus according to an embodiment of the present invention;
fig. 6 is a schematic diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In some scenarios, the electronic device needs to perform information interaction with a user, for example, the user may send a voice instruction to the smart sound box, and the smart sound box may play a corresponding audio file after receiving the voice instruction of the user; alternatively, the user may send an instruction to the robot through the touch interface, the robot may perform a corresponding action after receiving the instruction of the user, and so on.
Generally, information interaction between the electronic device and the user is based on a voice signal sent by the user or an operation instruction sent by the user to the electronic device, an application scene is single, and particularly for infants with weak language capability or persons with limited language capability, rich body gesture actions of the infants and the persons with limited language capability cannot be recognized, and more comprehensive and rich interaction experience cannot be provided.
In order to solve the above technical problem, the present invention provides an information interaction method, which may be applied to electronic devices, such as a smart camera, a mobile terminal, a robot, and the like, and is not limited in this embodiment of the present invention.
The following generally describes the information interaction method provided by the embodiment of the present invention.
In one implementation, the information interaction method includes:
acquiring a first to-be-identified image acquired by a user;
identifying the first image to be identified to obtain first gesture information in the first image to be identified;
and determining and executing a trigger event corresponding to the first gesture information so as to interact with the user.
As can be seen from the above, with the information interaction method provided by the embodiment of the present invention, the electronic device can determine and execute the corresponding trigger event by recognizing the gesture of the user, that is, the user can perform information interaction with the electronic device through the gesture.
The following describes in detail the information interaction method provided by the embodiment of the present invention by using a specific embodiment.
As shown in fig. 1, a schematic flow chart of an information interaction method provided in an embodiment of the present invention includes the following steps:
s101: acquiring a first image to be identified acquired aiming at a user.
The first image to be recognized is an image captured for the user, that is, the first image to be recognized includes image information of the user.
For example, the first image to be recognized may be acquired by the electronic device (execution subject) in real time, or may be acquired by the electronic device after receiving a trigger instruction from a user; the image may be a single image captured by the electronic device (execution subject), or may be a certain frame in a video captured by the electronic device (execution subject), which is not limited specifically.
S102: and identifying the first image to be identified to obtain first gesture information in the first image to be identified.
By recognizing the first image to be recognized, the first gesture information of the user in the first image to be recognized can be determined, and it can be understood that the first gesture information is a gesture performed by the user at the acquisition moment of the first image to be recognized, that is, an interactive instruction of the user to the electronic device (execution main body).
For example, the first to-be-recognized image may be recognized in a template matching manner to obtain first gesture information in the first to-be-recognized image: the method comprises the steps of firstly carrying out template matching on a first image to be recognized and a plurality of preset images, determining the preset images matched with the image to be recognized as matching images, and then taking gesture information corresponding to the matching images as first gesture information of the first image to be recognized. For example, as shown in fig. 2, the gesture information in the preset image is a number "3".
The first image to be recognized and the plurality of preset images can be subjected to template matching in the following mode:
in the first step, a plurality of regions to be detected can be determined from the first image to be recognized according to a preset traversal rule.
For example, as shown in fig. 3, the schematic diagram is a schematic diagram of determining a plurality of regions to be detected from a first image to be recognized according to a preset traversal rule in one case. The first image to be recognized is a sliding window with a size of 1920 × 1080 pixels, the gray area is a sliding window with a size of 128 × 128 pixels, the area to be detected is located in the sliding window, and the first image to be recognized can be traversed from the upper left corner, and the first image to be recognized is moved by one pixel in each traversing.
And secondly, calculating a difference value between each region to be detected and each preset image according to each preset image.
For example, the difference value between each region to be detected and the preset image can be calculated by the following formula:
wherein template (i, j) represents the first image to be recognized, current (i, j) denotes a preset image, xx denotes an abscissa of the first image to be recognized, yy denotes an ordinate of the first image to be recognized, mxn denotes a size of the preset image,
and expressing the difference value between each region to be detected and the preset image.
And thirdly, determining a to-be-detected region corresponding to the preset image as a target region according to the difference value.
For example, the region to be detected with the lowest difference value may be directly used as the target region, so that each preset image has the corresponding target region; or, the region to be detected whose difference value satisfies the preset value interval may be used as the target region. For example, in one implementation, the region to be detected with the lowest difference value and less than 50 may be used as the target region.
And fourthly, taking the preset image with the minimum difference value of the target area as the preset image matched with the first image to be recognized.
Or, the first to-be-recognized image may be recognized by inputting the first to-be-recognized image into a pre-trained neural network model for extracting gesture information, so as to obtain the first gesture information in the first to-be-recognized image, and the like, which is not limited specifically.
S103: and determining and executing a trigger event corresponding to the first gesture information so as to interact with the user.
For example, if the first gesture information is a number "3", the triggering event corresponding to the first gesture information may be playing a song numbered 3 or a story related to the number 3; alternatively, if the first gesture information is an avatar of a bird, the triggering event corresponding to the first gesture information may be playing audio content related to the bird, such as an audio introduction of the bird, or a cry of the bird.
That is to say, the electronic device (executing body) feeds back the gesture of the user by executing the trigger event corresponding to the first gesture information, so that information interaction with the user is realized.
After the trigger event corresponding to the first gesture information is determined and executed, the information interaction process may be ended, or the interaction with the user may be continued on the basis of the information interaction.
For example, continuing the above example, if the first gesture information is a number "3" and the triggering event corresponding to the first gesture information is the playing of a song numbered 3, in this case, the user may be further interacted with to communicate content related to the song numbered 3.
In this step, the process of further interacting with the user may be:
firstly, acquiring a second image to be recognized acquired by a user, recognizing the second image to be recognized to obtain second gesture information in the second image to be recognized, then judging whether the second gesture information meets a preset condition, if so, determining and executing a trigger event corresponding to the second gesture information, if not, acquiring a control instruction, determining and executing the trigger event corresponding to the control instruction, finally, returning to the step of acquiring the second image to be recognized acquired by the user, and continuing to perform information interaction with the user.
Under the condition that the second gesture information is judged not to meet the preset condition, the control instruction can be obtained by sending help seeking information to a preset user, and the preset user can be a user who is currently interacting with the electronic equipment (execution main body) or a credible third party user such as an administrator, a guardian and the like. For example, if the user currently interacting with the electronic device (executive) is a child, the predetermined user may be the guardian of the child. In this way, the whole interaction process can be adjusted in real time by the trusted third party user, and the continuity of the interaction can be maintained.
It can be understood that the manner of obtaining the second gesture information in the second image to be recognized by recognizing the second image to be recognized may be the same as the manner of obtaining the first gesture information in the first image to be recognized by recognizing the first image to be recognized.
In one case, whether the second gesture information meets the preset condition or not can be judged through the confidence of the trigger event corresponding to the second gesture information, if the confidence of the trigger event corresponding to the second gesture information is larger than a preset threshold, it is judged that the second gesture information meets the preset condition, and if the confidence of the trigger event corresponding to the second gesture information is not larger than the preset threshold, it is judged that the second gesture information does not meet the preset condition.
The following steps may be adopted to determine the confidence of the trigger event corresponding to the second gesture information:
the method comprises the steps of firstly, obtaining a plurality of candidate trigger events and initial confidence degrees, wherein the initial confidence degree of each candidate trigger event is equal, then randomly selecting a target trigger event from the candidate trigger events to serve as the trigger event corresponding to gesture information, executing the trigger event, increasing the confidence degree of the target trigger event according to a preset updating rule if an interaction instruction of a user is received after the trigger event is executed, and reducing the confidence degree of the target trigger event according to the preset updating rule if the interaction instruction of the user is not received after the trigger event is executed.
In this way, the electronic device (executing subject) may automatically analyze whether the electronic device is capable of feeding back the second gesture information of the user, and when the confidence of the trigger event corresponding to the second gesture information is lower than a certain value, it indicates that the trigger event has a low possibility of effectively feeding back the second gesture information of the user, and therefore, it is determined that the second gesture information does not satisfy the preset condition. Meanwhile, the learning capability of the electronic device (execution subject) can be improved by adjusting the confidence of the trigger event corresponding to the second gesture information.
Or, in another case, whether the second gesture information satisfies the preset condition may be determined by recognizing a time difference between a time when the second gesture information is recognized and a time when any trigger event is executed last time, if the time difference between the time when the second gesture information is recognized and the time when any trigger event is executed last time is not greater than a preset threshold, it is determined that the second gesture information satisfies the preset condition, and if the time difference between the time when the second gesture information is recognized and the time when any trigger event is executed last time is greater than the preset threshold, it is determined that the second gesture information does not satisfy the preset condition. Therefore, errors caused by interaction timeout can be reduced, and the interaction experience of the user is further improved.
Or, in this step, the process of further interacting with the user may also be through a voice instruction of the user or an operation instruction on the touch screen, and the like, which is not limited specifically.
As can be seen from the above, in the information interaction method provided in the embodiment of the present invention, the first to-be-identified image acquired by the user is acquired, the first to-be-identified image is identified, so as to obtain the first gesture information in the first to-be-identified image, and then, the trigger event corresponding to the first gesture information is determined and executed, so as to interact with the user; therefore, the electronic equipment can determine and execute the corresponding trigger event by recognizing the gesture of the user, namely, the user can perform information interaction with the electronic equipment through the gesture, and therefore, the electronic equipment can provide richer interaction experience for the user.
As shown in fig. 4, a schematic flow chart of another information interaction method provided in the embodiment of the present invention includes the following steps:
s401: acquiring a first image to be identified acquired aiming at a user.
The first image to be recognized is an image captured for the user, that is, the first image to be recognized includes image information of the user.
For example, the first image to be recognized may be acquired by the electronic device (execution subject) in real time, or may be acquired by the electronic device after receiving a trigger instruction from a user; the image may be a single image captured by the electronic device (execution subject), or may be a certain frame in a video captured by the electronic device (execution subject), which is not limited specifically.
S402: and identifying the first image to be identified to obtain first gesture information in the first image to be identified.
By recognizing the first image to be recognized, the first gesture information of the user in the first image to be recognized can be determined, and it can be understood that the first gesture information is a gesture performed by the user at the acquisition moment of the first image to be recognized, that is, an interactive instruction of the user to the electronic device (execution main body).
For example, the first to-be-recognized image may be recognized in a template matching manner to obtain first gesture information in the first to-be-recognized image: the method comprises the steps of firstly carrying out template matching on a first image to be recognized and a plurality of preset images, determining the preset images matched with the image to be recognized as matching images, and then taking gesture information corresponding to the matching images as first gesture information of the first image to be recognized.
The first image to be recognized and the plurality of preset images can be subjected to template matching in the following mode:
in the first step, a plurality of regions to be detected can be determined from the first image to be recognized according to a preset traversal rule.
And secondly, calculating a difference value between each region to be detected and each preset image according to each preset image.
For example, the difference value between each region to be detected and the preset image can be calculated by the following formula:
wherein template (i, j) represents a first image to be recognized, current (i, j) represents a preset image, xx represents an abscissa of the first image to be recognized, yy represents an ordinate of the first image to be recognized, M × N represents a size of the preset image,
and expressing the difference value between each region to be detected and the preset image.
And thirdly, determining a to-be-detected region corresponding to the preset image as a target region according to the difference value.
For example, the region to be detected with the lowest difference value may be directly used as the target region, so that each preset image has the corresponding target region; or, the region to be detected whose difference value satisfies the preset value interval may be used as the target region. For example, in one implementation, the region to be detected with the lowest difference value and less than 50 may be used as the target region.
And fourthly, taking the preset image with the minimum difference value of the target area as the preset image matched with the first image to be recognized.
S403: and determining and executing a trigger event corresponding to the first gesture information so as to interact with the user.
For example, if the first gesture information is a number "3", the triggering event corresponding to the first gesture information may be playing a song numbered 3 or a story related to the number 3; alternatively, if the first gesture information is an avatar of a bird, the triggering event corresponding to the first gesture information may be playing audio content related to the bird, such as an audio introduction of the bird, or a cry of the bird.
That is to say, the electronic device (executing body) feeds back the gesture of the user by executing the trigger event corresponding to the first gesture information, so that information interaction with the user is realized.
S404: and acquiring a second image to be identified acquired by the user, identifying the second image to be identified, and obtaining second gesture information in the second image to be identified.
For example, continuing the above example, if the first gesture information is a number "3", and the trigger event corresponding to the first gesture information is a song with a play number of 3, in this case, the second gesture information in the second image to be recognized may be obtained by acquiring a second image to be recognized collected for the user, recognizing the second image to be recognized, and further interacting with the user to communicate content related to the song with a play number of 3.
It can be understood that, in S404, a second image to be recognized collected by the user is obtained, and the second image to be recognized is recognized to obtain second gesture information in the second image to be recognized, which may be the same as the way of recognizing the first image to be recognized to obtain the first gesture information in the first image to be recognized in S401 and S402.
S405: judging whether the second gesture information meets a preset condition or not; if yes, determining and executing a trigger event corresponding to the second gesture information; if not, acquiring a control instruction, and determining and executing a trigger event corresponding to the control instruction; returning to S404.
For example, in one case, whether the second gesture information meets the preset condition may be determined according to the confidence level of the trigger event corresponding to the second gesture information, if the confidence level of the trigger event corresponding to the second gesture information is greater than a preset threshold, it is determined that the second gesture information meets the preset condition, and if the confidence level of the trigger event corresponding to the second gesture information is not greater than the preset threshold, it is determined that the second gesture information does not meet the preset condition.
The following steps may be adopted to determine the confidence of the trigger event corresponding to the second gesture information:
the method comprises the steps of firstly, obtaining a plurality of candidate trigger events and initial confidence degrees, wherein the initial confidence degree of each candidate trigger event is equal, then randomly selecting a target trigger event from the candidate trigger events to serve as the trigger event corresponding to gesture information, executing the trigger event, increasing the confidence degree of the target trigger event according to a preset updating rule if an interaction instruction of a user is received after the trigger event is executed, and reducing the confidence degree of the target trigger event according to the preset updating rule if the interaction instruction of the user is not received after the trigger event is executed.
For example, the initial confidence of all candidate trigger events [ a1, a2, A3, … An ] may be set to 0.5, and if the answer a5 adopts An answer and the user is willing to communicate further, the initial confidence of the answer a5 is increased by 10%, that is, the confidence of the answer a5 is 0.5 × 1.1 — 5.5; if answer a5 takes the answer and the user does not want to communicate further, the initial confidence of answer a5 is reduced by 10%, i.e., answer a5 has a confidence of 0.5 × 0.9 — 4.0.
In this way, the electronic device (executing subject) may automatically analyze whether the electronic device is capable of feeding back the second gesture information of the user, and when the confidence of the trigger event corresponding to the second gesture information is lower than a certain value, it indicates that the trigger event has a low possibility of effectively feeding back the second gesture information of the user, and therefore, it is determined that the second gesture information does not satisfy the preset condition. Meanwhile, the learning capability of the electronic device (execution subject) can be improved by adjusting the confidence of the trigger event corresponding to the second gesture information.
Or, in another case, whether the second gesture information satisfies the preset condition may be determined by recognizing a time difference between a time when the second gesture information is recognized and a time when any trigger event is executed last time, if the time difference between the time when the second gesture information is recognized and the time when any trigger event is executed last time is not greater than a preset threshold, it is determined that the second gesture information satisfies the preset condition, and if the time difference between the time when the second gesture information is recognized and the time when any trigger event is executed last time is greater than the preset threshold, it is determined that the second gesture information does not satisfy the preset condition. Therefore, errors caused by interaction timeout can be reduced, and the interaction experience of the user is further improved.
Under the condition that the second gesture information is judged not to meet the preset condition, the control instruction can be obtained by sending help seeking information to a preset user, and the preset user can be a user who is currently interacting with the electronic equipment (execution main body) or a credible third party user such as an administrator, a guardian and the like. For example, if the user currently interacting with the electronic device (executive) is a child, the predetermined user may be the guardian of the child. In this way, the whole interaction process can be adjusted in real time by the trusted third party user, and the continuity of the interaction can be maintained.
As can be seen from the above, in the information interaction method provided in the embodiment of the present invention, the first to-be-identified image acquired by the user is acquired, the first to-be-identified image is identified, so as to obtain the first gesture information in the first to-be-identified image, and then, the trigger event corresponding to the first gesture information is determined and executed, so as to interact with the user; therefore, the electronic equipment can determine and execute the corresponding trigger event by recognizing the gesture of the user, namely, the user can perform information interaction with the electronic equipment through the gesture, and therefore, the electronic equipment can provide richer interaction experience for the user.
Corresponding to the information interaction method, the embodiment of the invention also provides an information interaction device.
As shown in fig. 5, a schematic structural diagram of an information interaction apparatus provided in an embodiment of the present invention is applied to an electronic device, and the apparatus includes:
an image acquisition module 501, configured to acquire a first image to be identified, which is acquired for a user;
the processor 502 is configured to identify the first image to be identified, obtain first gesture information in the first image to be identified, and determine and execute a trigger event corresponding to the first gesture information;
a communication module 503, configured to interact with the user.
In one implementation, the processor 502 is specifically configured to perform template matching on the first image to be recognized and a plurality of preset images, and determine a preset image matched with the image to be recognized as a matching image; and taking the gesture information corresponding to the matched image as the first gesture information of the first image to be recognized.
In one implementation, the processor 502 is specifically configured to:
determining a plurality of regions to be detected from the first image to be recognized according to a preset traversal rule;
calculating a difference value between each region to be detected and each preset image according to each preset image, and determining the region to be detected corresponding to each preset image as a target region according to the difference value;
and taking the preset image with the minimum difference value of the target area as a preset image matched with the first image to be recognized.
In one implementation, the image acquisition module 501 is further configured to acquire a second image to be identified, which is acquired by a user;
the processor 502 is further configured to identify the second image to be identified, so as to obtain second gesture information in the second image to be identified; judging whether the second gesture information meets a preset condition or not; if yes, determining and executing a trigger event corresponding to the second gesture information; if not, triggering the communication module;
the communication module is used for acquiring a control instruction;
the processor is further used for determining and executing a trigger event corresponding to the control instruction; and triggering the image acquisition module.
In an implementation manner, the processor 502 is further configured to determine whether a confidence of a trigger event corresponding to the second gesture information is greater than a preset threshold; if yes, judging that the second gesture information meets a preset condition; if not, judging that the second gesture information does not meet the preset condition.
In one implementation, the following steps are adopted to determine the confidence level of the trigger event corresponding to the second gesture information:
acquiring a plurality of candidate trigger events and initial confidence coefficients; wherein the initial confidence of each candidate trigger event is equal;
randomly selecting a target trigger event from the candidate trigger events, using the target trigger event as a trigger event corresponding to the gesture information, and executing the trigger event;
if the interactive instruction of the user is received after the trigger event is executed, increasing the confidence coefficient of the target trigger event according to a preset updating rule;
and if the interactive instruction of the user is not received after the trigger event is executed, reducing the confidence coefficient of the target trigger event according to the preset updating rule.
In one implementation, the processor 502 is further configured to determine whether a time difference between the time when the second gesture information is recognized and the time when any trigger event is executed last time is greater than a preset threshold; if not, judging that the second gesture information meets a preset condition; and if so, judging that the second gesture information does not meet the preset condition.
As can be seen from the above, the information interaction apparatus provided in the embodiment of the present invention enables the electronic device to determine and execute the corresponding trigger event by recognizing the gesture of the user, that is, the user can perform information interaction with the electronic device through the gesture.
An embodiment of the present invention further provides an electronic device, as shown in fig. 6, including a processor 601, a communication interface 602, a memory 603, and a communication bus 604, where the processor 601, the communication interface 602, and the memory 603 complete mutual communication through the communication bus 604,
a memory 603 for storing a computer program;
the processor 601 is configured to implement the following steps when executing the program stored in the memory 603:
acquiring a first to-be-identified image acquired by a user;
identifying the first image to be identified to obtain first gesture information in the first image to be identified;
and determining and executing a trigger event corresponding to the first gesture information so as to interact with the user.
The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the electronic equipment and other equipment.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic device, discrete hardware component.
As can be seen from the above, the information interaction method provided in the embodiment of the present invention enables the electronic device to determine and execute the corresponding trigger event by recognizing the gesture of the user, that is, the user can perform information interaction with the electronic device through the gesture.
In another embodiment of the present invention, a computer-readable storage medium is further provided, which has instructions stored therein, and when the computer-readable storage medium runs on a computer, the computer is caused to execute the information interaction method described in any one of the above embodiments.
In yet another embodiment, the present invention further provides a computer program product containing instructions, which when run on a computer, causes the computer to execute the information interaction method described in any of the above embodiments.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the apparatus embodiment, the electronic device embodiment and the storage medium embodiment, since they are substantially similar to the method embodiment, the description is relatively simple, and in relation to the description, reference may be made to some portions of the description of the method embodiment.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (16)
1. An information interaction method is applied to electronic equipment, and is characterized in that the method comprises the following steps:
acquiring a first to-be-identified image acquired by a user;
identifying the first image to be identified to obtain first gesture information in the first image to be identified;
and determining and executing a trigger event corresponding to the first gesture information so as to interact with the user.
2. The method according to claim 1, wherein the recognizing the first image to be recognized to obtain first gesture information in the first image to be recognized comprises:
performing template matching on the first image to be recognized and a plurality of preset images, and determining the preset image matched with the image to be recognized as a matched image;
and taking the gesture information corresponding to the matched image as the first gesture information of the first image to be recognized.
3. The method according to claim 2, wherein the template matching the first image to be recognized with a plurality of preset images, and determining the preset image matched with the first image to be recognized as the matched image comprises:
determining a plurality of regions to be detected from the first image to be recognized according to a preset traversal rule;
calculating a difference value between each region to be detected and each preset image according to each preset image, and determining the region to be detected corresponding to each preset image as a target region according to the difference value;
and taking the preset image with the minimum difference value of the target area as a preset image matched with the first image to be recognized.
4. The method of claim 1, wherein after determining and executing a triggering event corresponding to the first gesture information to interact with the user, the method further comprises:
acquiring a second image to be identified, which is acquired by aiming at a user;
identifying the second image to be identified to obtain second gesture information in the second image to be identified;
judging whether the second gesture information meets a preset condition or not;
if yes, determining and executing a trigger event corresponding to the second gesture information; if not, acquiring a control instruction, and determining and executing a trigger event corresponding to the control instruction;
and returning to the step of acquiring the second image to be identified acquired aiming at the user.
5. The method according to claim 4, wherein the determining whether the second gesture information satisfies a preset condition comprises:
judging whether the confidence of the trigger event corresponding to the second gesture information is greater than a preset threshold value;
if yes, judging that the second gesture information meets a preset condition; if not, judging that the second gesture information does not meet the preset condition.
6. The method of claim 5, wherein the confidence level of the trigger event corresponding to the second gesture information is determined by the following steps:
acquiring a plurality of candidate trigger events and initial confidence coefficients; wherein the initial confidence of each candidate trigger event is equal;
randomly selecting a target trigger event from the candidate trigger events, using the target trigger event as a trigger event corresponding to the gesture information, and executing the trigger event;
if the interactive instruction of the user is received after the trigger event is executed, increasing the confidence coefficient of the target trigger event according to a preset updating rule;
and if the interactive instruction of the user is not received after the trigger event is executed, reducing the confidence coefficient of the target trigger event according to the preset updating rule.
7. The method according to claim 4, wherein the determining whether the second gesture information satisfies a preset condition comprises:
judging whether a time difference value between the moment of identifying the second gesture information and the moment of executing any trigger event at the last time is greater than a preset threshold value or not;
if not, judging that the second gesture information meets a preset condition; and if so, judging that the second gesture information does not meet the preset condition.
8. An information interaction device applied to electronic equipment, the device comprising:
the image acquisition module is used for acquiring a first to-be-identified image acquired by a user;
the processor is used for identifying the first image to be identified to obtain first gesture information in the first image to be identified and determining and executing a trigger event corresponding to the first gesture information;
and the communication module is used for interacting with the user.
9. The apparatus of claim 8,
the processor is specifically configured to perform template matching on the first image to be recognized and a plurality of preset images, and determine a preset image matched with the image to be recognized as a matched image; and taking the gesture information corresponding to the matched image as the first gesture information of the first image to be recognized.
10. The apparatus of claim 9, wherein the processor is specifically configured to:
determining a plurality of regions to be detected from the first image to be recognized according to a preset traversal rule;
calculating a difference value between each region to be detected and each preset image according to each preset image, and determining the region to be detected corresponding to each preset image as a target region according to the difference value;
and taking the preset image with the minimum difference value of the target area as a preset image matched with the first image to be recognized.
11. The apparatus of claim 8,
the image acquisition module is also used for acquiring a second image to be identified acquired by a user;
the processor is further configured to identify the second image to be identified to obtain second gesture information in the second image to be identified; judging whether the second gesture information meets a preset condition or not; if yes, determining and executing a trigger event corresponding to the second gesture information; if not, triggering the communication module;
the communication module is used for acquiring a control instruction;
the processor is further used for determining and executing a trigger event corresponding to the control instruction; and triggering the image acquisition module.
12. The apparatus of claim 11,
the processor is further configured to determine whether a confidence of a trigger event corresponding to the second gesture information is greater than a preset threshold; if yes, judging that the second gesture information meets a preset condition; if not, judging that the second gesture information does not meet the preset condition.
13. The apparatus of claim 12, wherein the confidence level of the trigger event corresponding to the second gesture information is determined by:
acquiring a plurality of candidate trigger events and initial confidence coefficients; wherein the initial confidence of each candidate trigger event is equal;
randomly selecting a target trigger event from the candidate trigger events, using the target trigger event as a trigger event corresponding to the gesture information, and executing the trigger event;
if the interactive instruction of the user is received after the trigger event is executed, increasing the confidence coefficient of the target trigger event according to a preset updating rule;
and if the interactive instruction of the user is not received after the trigger event is executed, reducing the confidence coefficient of the target trigger event according to the preset updating rule.
14. The apparatus of claim 11,
the processor is further configured to determine whether a time difference between the time when the second gesture information is identified and the time when any trigger event is executed last time is greater than a preset threshold; if not, judging that the second gesture information meets a preset condition; and if so, judging that the second gesture information does not meet the preset condition.
15. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor and the communication interface are used for realizing mutual communication by the memory through the communication bus;
a memory for storing a computer program;
a processor for implementing the method steps of any of claims 1 to 7 when executing a program stored in the memory.
16. A computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, which computer program, when being executed by a processor, carries out the method steps of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811409375.8A CN111221406B (en) | 2018-11-23 | 2018-11-23 | Information interaction method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811409375.8A CN111221406B (en) | 2018-11-23 | 2018-11-23 | Information interaction method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111221406A true CN111221406A (en) | 2020-06-02 |
| CN111221406B CN111221406B (en) | 2023-10-13 |
Family
ID=70808521
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811409375.8A Active CN111221406B (en) | 2018-11-23 | 2018-11-23 | Information interaction method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111221406B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113869296A (en) * | 2020-06-30 | 2021-12-31 | 杭州九阳小家电有限公司 | Terminal equipment and automatic control method thereof |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090324008A1 (en) * | 2008-06-27 | 2009-12-31 | Wang Kongqiao | Method, appartaus and computer program product for providing gesture analysis |
| CN104020843A (en) * | 2013-03-01 | 2014-09-03 | 联想(北京)有限公司 | Information processing method and electronic device |
| US20150131855A1 (en) * | 2013-11-13 | 2015-05-14 | Omron Corporation | Gesture recognition device and control method for the same |
| CN106339067A (en) * | 2015-07-06 | 2017-01-18 | 联想(北京)有限公司 | Control method and electronic equipment |
| CN106527674A (en) * | 2015-09-14 | 2017-03-22 | 上海羽视澄蓝信息科技有限公司 | Human-computer interaction method, equipment and system for vehicle-mounted monocular camera |
| US20170168582A1 (en) * | 2015-12-09 | 2017-06-15 | Le Holdings (Beijing) Co., Ltd. | Click response processing method, electronic device and system for motion sensing control |
| WO2018033154A1 (en) * | 2016-08-19 | 2018-02-22 | 北京市商汤科技开发有限公司 | Gesture control method, device, and electronic apparatus |
| CN107831995A (en) * | 2017-09-28 | 2018-03-23 | 努比亚技术有限公司 | A kind of terminal operation control method, terminal and computer-readable recording medium |
| CN107831987A (en) * | 2017-11-22 | 2018-03-23 | 出门问问信息科技有限公司 | The error touch control method and device of anti-gesture operation |
| CN107967061A (en) * | 2017-12-21 | 2018-04-27 | 北京华捷艾米科技有限公司 | Man-machine interaction method and device |
| CN108446073A (en) * | 2018-03-12 | 2018-08-24 | 阿里巴巴集团控股有限公司 | A kind of method, apparatus and terminal for simulating mouse action using gesture |
| CN108594995A (en) * | 2018-04-13 | 2018-09-28 | 广东小天才科技有限公司 | Electronic equipment operation method based on gesture recognition and electronic equipment |
-
2018
- 2018-11-23 CN CN201811409375.8A patent/CN111221406B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090324008A1 (en) * | 2008-06-27 | 2009-12-31 | Wang Kongqiao | Method, appartaus and computer program product for providing gesture analysis |
| CN104020843A (en) * | 2013-03-01 | 2014-09-03 | 联想(北京)有限公司 | Information processing method and electronic device |
| US20150131855A1 (en) * | 2013-11-13 | 2015-05-14 | Omron Corporation | Gesture recognition device and control method for the same |
| CN106339067A (en) * | 2015-07-06 | 2017-01-18 | 联想(北京)有限公司 | Control method and electronic equipment |
| CN106527674A (en) * | 2015-09-14 | 2017-03-22 | 上海羽视澄蓝信息科技有限公司 | Human-computer interaction method, equipment and system for vehicle-mounted monocular camera |
| US20170168582A1 (en) * | 2015-12-09 | 2017-06-15 | Le Holdings (Beijing) Co., Ltd. | Click response processing method, electronic device and system for motion sensing control |
| WO2018033154A1 (en) * | 2016-08-19 | 2018-02-22 | 北京市商汤科技开发有限公司 | Gesture control method, device, and electronic apparatus |
| CN107831995A (en) * | 2017-09-28 | 2018-03-23 | 努比亚技术有限公司 | A kind of terminal operation control method, terminal and computer-readable recording medium |
| CN107831987A (en) * | 2017-11-22 | 2018-03-23 | 出门问问信息科技有限公司 | The error touch control method and device of anti-gesture operation |
| CN107967061A (en) * | 2017-12-21 | 2018-04-27 | 北京华捷艾米科技有限公司 | Man-machine interaction method and device |
| CN108446073A (en) * | 2018-03-12 | 2018-08-24 | 阿里巴巴集团控股有限公司 | A kind of method, apparatus and terminal for simulating mouse action using gesture |
| CN108594995A (en) * | 2018-04-13 | 2018-09-28 | 广东小天才科技有限公司 | Electronic equipment operation method based on gesture recognition and electronic equipment |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113869296A (en) * | 2020-06-30 | 2021-12-31 | 杭州九阳小家电有限公司 | Terminal equipment and automatic control method thereof |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111221406B (en) | 2023-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109240576B (en) | Image processing method and device in game, electronic device and storage medium | |
| CN110321845B (en) | Method and device for extracting emotion packets from video and electronic equipment | |
| CN110837758B (en) | Keyword input method and device and electronic equipment | |
| CN109345553B (en) | Palm and key point detection method and device thereof, and terminal equipment | |
| WO2020125229A1 (en) | Feature fusion method and apparatus, and electronic device and storage medium | |
| US11017253B2 (en) | Liveness detection method and apparatus, and storage medium | |
| CN111144344B (en) | Method, device, equipment and storage medium for determining person age | |
| US20210343042A1 (en) | Audio acquisition device positioning method and apparatus, and speaker recognition method and system | |
| US20170169062A1 (en) | Method and electronic device for recommending video | |
| CN113326821B (en) | Face driving method and device for video frame image | |
| CN112232506A (en) | Network model training method, image target recognition method, device and electronic equipment | |
| CN109670109A (en) | Information acquisition method, device, server, terminal and medium | |
| CN110287361B (en) | Figure picture screening method and device | |
| CN112434717A (en) | Model training method and device | |
| CN111027643A (en) | Deep neural network model training method, man-machine interaction recognition method and device, electronic equipment and storage medium | |
| CN112884801B (en) | High-altitude parabolic detection method, device, equipment and storage medium | |
| CN111221406B (en) | Information interaction method and device | |
| CN113342170A (en) | Gesture control method, device, terminal and storage medium | |
| CN112307260A (en) | Video identification method, video identification device, electronic equipment and computer readable storage medium | |
| CN115859964B (en) | Educational resource sharing method and system based on educational cloud platform | |
| CN111966920A (en) | Public opinion propagation stable condition prediction method, device and equipment | |
| CN116342940A (en) | Image approval method, device, medium and equipment | |
| CN111275921A (en) | Behavior monitoring method and device and electronic equipment | |
| CN115661564A (en) | Training method and device of image processing model, electronic equipment and storage medium | |
| CN110415688B (en) | Information interaction method and robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |