TW201942714A - Information output method, apparatus and system - Google Patents

Abstract

The application discloses an information output method, apparatus and system. The method includes the steps as follows: acquiring an eye image of an object by an interactive device; determining the position of a fixation point of the object according to the eye image by the interactive device; and outputting an interactive information according to the position of the fixation point and an object information of the object by the interactive device. According to the information output method, apparatus and system, the technical problem that the fixation point cannot be accurately positioned in man-machine interaction of the interactive device is solved.

Description

Method, device and system for outputting information

The present invention relates to the field of human-computer interaction, and in particular, to a method, a device, and a system for outputting information.

At present, robots have been widely used in various industries. For example, in the catering industry, robots can complete the actions of ordering and delivering meals for users through human-computer interaction. In the prior art, when a robot interacts with people, it usually uses face recognition technology to estimate the position of the eyes. For example, by default, one third of the position on the face of the human face is the position of the eyes, and finally the position is silent. The position of the eye's actual fixation point is considered to adjust the robot's eyeball so that the robot's eyeball and the human eye face each other.

However, due to the large individual differences between people, the use of face recognition to predict the position of the human eye and defaulting it to the actual gaze point position of the eye will cause a large error. For example, it often happens that the human eye does not actually look at the robot, but the human face falls into the scope of the robot's face recognition and triggers the interaction between the robot and the person accordingly, resulting in misinterpretation of the interaction. In addition, when a person and a robot have an intention to interact, but the person and the robot are not actually facing each other, the robot cannot confirm the position of the person's gaze through the method of face recognition, and thus cannot complete the interaction between the robot and the person, thereby generating interaction. Missing.

Aiming at the problem that in the human-machine interaction of the above-mentioned interactive device, the position fixation point cannot be accurately determined, the existing face recognition technology cannot accurately identify the position of the fixation point of the human eye, and no effective solution has been proposed at present.

In view of this, our inventors are concentrating on further research and proceeding with research and development and improvement, with a better design to solve the above problems, and the invention came out after continuous testing and modification.

The embodiments of the present invention provide a method, an apparatus, and a system for outputting information, so as to at least solve the technical problem that the gaze point cannot be accurately positioned in the human-computer interaction of the interactive device.

According to an aspect of the embodiment of the present invention, a method for outputting information is provided, which includes: an interactive device acquires an eye image of an object; the interactive device determines a position of a fixation point of the object according to the eye image; The object information of the object outputs interactive information.

According to another aspect of the embodiments of the present invention, a system for outputting information is further provided, including: an image acquisition device for acquiring an eye image of an object; and a processor connected to the image acquisition device for The external image determines the position of the fixation point of the object, and outputs interactive information according to the position of the fixation point and the object information of the object.

According to another aspect of the embodiments of the present invention, a device for outputting information is further provided, which includes: an acquisition module for an interactive device to acquire an eye image of an object; and a determination module for an interactive device to determine an object based on the eye image The position of the gaze point; an output module for the interactive device to output the interaction information according to the position of the gaze point and the object information of the object.

According to another aspect of the embodiments of the present invention, a storage medium is also provided. The storage medium includes a stored program, where the program executes a method of outputting information.

According to another aspect of the embodiments of the present invention, there is also provided a processor for running a program, wherein the program runs a method of outputting information.

In the embodiment of the present invention, a combination of eye tracking technology and object information is used to obtain an eye image of the object, then determine the position of the gaze point of the object according to the eye image, and finally according to the position of the gaze point and the object of the object The output of interactive information achieves the purpose of accurately identifying the position of the gaze point of the object, thereby realizing the technical effect of the accurate interaction between the interactive device and the object, and further solving the technical problem that the gaze point cannot be accurately positioned in the interactive device-human interaction.

Regarding the technical means of our inventors, several preferred embodiments are described in detail below in conjunction with the drawings, for the purpose of understanding and agreeing with the present invention.

In order to enable those with ordinary knowledge in the technical field to better understand the solution of the present invention, the technical solution in the embodiment of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. The described embodiments are only a part of embodiments of the present invention, but not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons with ordinary knowledge in the technical field to which the present invention belongs without making progressive labor should fall within the protection scope of the present invention.

It should be noted that the terms “first” and “second” in the scope of the specification and patent application of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data so used may be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in an order other than those illustrated or described herein. Furthermore, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units need not be limited to those explicitly listed Those steps or units may instead include other steps or units not explicitly listed or inherent to these processes, methods, products or equipment.

Examples 1

According to an embodiment of the present invention, an embodiment of a method for outputting information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. The logical order is shown in the flowchart, but in some cases, the steps shown or described may be performed in a different order than here.

FIG. 1 is a flowchart of a method for outputting information according to an embodiment of the present invention. As shown in FIG. 1, the method includes the following steps:

Step S102: The interactive device acquires an eye image of the subject.

It should be noted that the above-mentioned interactive device includes a robot, wherein the above-mentioned interactive device is any device capable of human-computer interaction, for example, a service robot in a hotel, a robot for cleaning and sanitation at home, and the like. The robot includes one or more image acquisition units, and the image acquisition unit can acquire an eye image of the object. The image acquisition unit may be, but is not limited to, a camera, and there are infrared light sources around the image acquisition unit. The number, brightness, and installation position of the infrared light sources may be set according to actual conditions. In addition, when there are multiple image acquisition units, the multiple image acquisition units are located at different positions of the robot, for example, the multiple image acquisition units are respectively located at the head, waist, and feet of the robot so that the object While looking at the robot, the robot can capture images of the subject's eyes.

Step S104: The interactive device determines the position of the gaze point of the object according to the eye image.

In an optional embodiment, the eye of the robot has an image acquisition unit. When the object faces the robot, the image acquisition unit of the robot's eye can acquire the image of the eye of the object, and the robot simultaneously The image is processed to obtain information such as the center position of the pupil of the subject, the center position of the spot formed by the infrared light source, the position of the upper and lower eyelids, and the iris data.

In another optional embodiment, the robot has multiple image acquisition units. When the object is not facing the robot, the image acquisition units located on different parts of the robot acquire multiple eye images from multiple angles. Based on the position of the image acquisition unit, the collected multiple eye images are analyzed to determine the subject's pupil center position, the center position of the spot formed by the infrared light source, the position of the upper and lower eyelids, and iris data. The position of the gaze point of the object is obtained by analyzing the above information.

In step S106, the interactive device outputs the interactive information according to the position of the gaze point and the object information of the object.

It should be noted that the object information of the object includes at least one of the following: the expression information of the object, the motion information of the object, and the voice information of the object.

In an optional embodiment, when the object looks at the robot and talks with the robot, the robot determines the position of the gaze point of the object by collecting an image of the eye of the object, and then fixes the object according to the position of the gaze point of the object, so Subject feels respected. When the object and the robot are not looking at each other, the robot determines the position of the gaze point of the object through the collected eye image of the object, and then outputs the corresponding interactive information in combination with the object's expression, language, etc. The subject looks at the robot's shoes and says, "The shoes you wear are very beautiful." The robot can determine the position of the fixation point of the object as the lower part of the robot through the eye image, and determine the position of the fixation point of the object as shoes based on the voice information of the object. At this time, the robot outputs interactive information "You can also try this shoe , I believe it will also look good. "

Based on the solutions defined in the above steps S102 to S106, it can be known that, by acquiring the eye image of the object, then determining the position of the fixation point of the object according to the eye image, and finally outputting interactive information according to the position of the fixation point and the object information of the object.

It is easy to notice that by using the method of analyzing the eye image, an accurate fixation point position of the object can be obtained. In addition, after the position of the fixation point is obtained, the interactive device can output accurate interactive information according to the object information of the object, thereby realizing the interaction between the object and the interactive device, and improving the human-computer interaction experience.

It can be known from the foregoing that the method for outputting information provided by the present invention can achieve the purpose of accurately identifying the position of a gaze point of an object, thereby realizing the technical effect of accurate interaction between the interactive device and the object, and furthermore, the human-computer interaction of the interactive device cannot be solved. Technical issues in accurately locating gaze points.

In an optional embodiment, as shown in the flowchart of a method for outputting information in FIG. 2, the method for outputting information provided by the present invention mainly includes image acquisition, image analysis, gaze point estimation, and fusion of gaze points. Five steps of content processing and data output. The image acquisition step is mainly to project a light source to an object's eyes through an infrared light source located on the robot, and then the image acquisition unit in the robot acquires images of the object's eyes and the surrounding parts of the eyes to obtain the object's eye image. . Among them, FIG. 3 shows an optional process of acquiring images.

It should be noted that after obtaining the eye image of the object, the eye image is analyzed through related algorithms, for example, algorithm modeling or deep learning neural network training algorithm, to obtain the eye feature information of the object. Then, the gaze point position of the object is determined according to the eye feature information of the object, that is, the gaze point is estimated. The specific steps are as follows:

Step S202: The interactive device analyzes eye feature information of the object according to the eye image;

Step S204: The interactive device determines the position of the gaze point of the object according to the eye feature information.

It should be noted that the above-mentioned eye feature information may include, but is not limited to, the pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, and spot position (that is, the Purchin spot position) of the subject.

Specifically, after obtaining the eye feature information of the object, the robot establishes a geometric space model of the eyeball of the object based on the eye feature information, and analyzes the eye image according to the eyeball geometric space model to obtain the position of the gaze point of the object.

It should be noted that, because the process of human-computer interaction is relatively complicated, after the robot recognizes the position of the gaze point of the object, the robot re-identifies the behavior of the object by recognizing the object's expression, motion, and speech, and further Make the robot make accurate feedback, that is, complete the content processing of the fused fixation point. In addition, after the content processing of the fused fixation point is completed, the robot outputs interactive information according to the position of the fixation point and the object information of the object. The specific steps are as follows:

Step S1060: The interactive device processes the position of the gaze point and the object information of the object to obtain a processing result;

Step S1062, the interactive device determines an operation instruction according to the processing result;

Step S1064: The interactive device sends an operation instruction to a corresponding component of the interactive device, where the operation instruction is used to control the corresponding component to output the interaction information.

In an optional embodiment, the processing center of the robot may process the position of the gaze point and the object information of the object, determine the feedback that each part of the robot should make, and then determine the operation corresponding to the feedback that the robot will make Instructions and send operation instructions to the corresponding components. For example, the processing center of the robot determines that the eyes of the robot need to perform a blinking action, and the hands need to perform a lifting action. At this time, the processing center of the robot will generate a blinking instruction corresponding to the blink and a rotation instruction corresponding to the lifting. A blink command is sent to the part that controls the eye, and a rotation command is sent to the part that controls the hand.

In an optional embodiment, the method for outputting information provided by the present invention can be applied to the restaurant industry, for example, robots receive customers in restaurants, order food for customers, serve meals, check out customers, and send customers out Scenes. Specifically, the customer walks on the street to find a suitable restaurant to eat. When the customer looks at the restaurant's store name, the table in the store, and the passing pedestrians in the store, the robot can determine through the image acquisition unit or the eye tracking module with an eye tracker function. The customer's gaze direction and gaze point position. If the customer's gaze direction is the restaurant, the robot will approach the customer and issue a voice prompt "Welcome to taste". After the customer enters the restaurant, the robot leads the customer to the free dining table. When a restaurant has multiple robots, if a customer looks at other robots, the eye movement module on the other robots can also recognize the other customers' gaze directions and issue a "welcome" voice to the customers. When the customer selects a seat, the robot estimates the seat that the customer is watching through the eye movement module, and determines the time that the customer is watching, or analyzes the customer's preference for different seats according to the change of the customer's pupil, and then sends a voice to the customer to recommend the seat, and Lead customers to the appropriate seats. When the customer selects a dish, the robot displays the display on the body to the customer to make the customer select the dish. Among them, when the customer looks at the display screen, the robot can determine the length and number of times the customer looks at the dish through the eye movement module, and analyzes the data such as the length and number of times the gaze is taken to determine the dish and the customer's intention. Potential preferences and recommend dishes to customers via the display or place orders directly. In the process of serving customers, the robot uses an eye movement module to determine which customer is watching which dish, and the staring time is relatively long, so that the corresponding dish is placed next to the corresponding customer. During the customer's meal, the robot detects whether the customer is looking at the robot through the eye movement module. If it is detected that the customer is looking at the robot, it indicates that the customer has a strong demand for the robot to help. At this time, the robot immediately responds to the facing customer. And smile to the customer. Finally, when the customer finishes the meal and the robot is watching by multiple customers at the same time, the robot's eye movement module determines whether the customer wants to check out by analyzing the customer's meal time and the remaining amount of food on the table. If it is determined that the customer needs to checkout, the robot completes the checkout work. When the customer leaves the restaurant, if the customer looks at the robot in the store, the robot's eye movement module can recognize that the customer is staring at it. At this time, the robot faces the customer with the customer's gaze direction to show etiquette. Better human-computer interaction.

Examples 2

According to the embodiment of the present invention, an embodiment of a system for outputting information is also provided. The system can execute the method for outputting information in Embodiment 1. FIG. 4 is a schematic structural diagram of a system for outputting information according to an embodiment of the present invention. As shown in FIG. 4, the system includes: an image acquisition device 401 and a processor 403.

Among them, an image acquisition device 401 is used to acquire an eye image of an object; a processor 403 is connected to the image acquisition device and used to determine the position of the gaze point of the object according to the eye image, and according to the position of the gaze point and the object The object information outputs interactive information.

It should be noted that the interactive device includes: a robot; the object information of the object includes at least one of the following: the expression information of the object, the motion information of the object, and the voice information of the object.

It can be known from the above that the eye image of the object is obtained through the image acquisition device, and the processor connected to the image acquisition device determines the position of the gaze point of the object according to the eye image, and outputs interaction based on the position of the gaze point and the object information of the object information.

It is easy to notice that by using the method of analyzing the eye image, an accurate fixation point position of the object can be obtained. In addition, after the position of the fixation point is obtained, the interactive device can output accurate interactive information according to the object information of the object, thereby realizing the interaction between the object and the interactive device, and improving the human-computer interaction experience.

It can be known from the foregoing that the method for outputting information provided by the present invention can achieve the purpose of accurately identifying the position of a gaze point of an object, thereby realizing the technical effect of accurate interaction between the interactive device and the object, and furthermore, the human-computer interaction of the interactive device cannot be solved. Technical issues in accurately locating gaze points.

In an optional embodiment, the processor is further configured to analyze the eye feature information of the object according to the eye image, and determine the position of the fixation point of the object according to the eye feature information. The eye feature information may include but not Limited to subject's pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, light spot position (that is, Purchin spot position).

In an optional embodiment, the processor is further configured to process the position of the gaze point and the object information of the object to obtain a processing result, then determine an operation instruction according to the processing result, and finally send the operation instruction to the corresponding component of the interactive device. The operation instruction is used to control the corresponding component to output interactive information.

Examples 3

According to an embodiment of the present invention, there is also provided an embodiment of an apparatus for outputting information. FIG. 5 is a schematic structural diagram of an apparatus for outputting information according to an embodiment of the present invention. As shown in FIG. 5, the apparatus includes: an acquiring module 501, Module 503 and output module 505.

The obtaining module 501 is used by the interactive device to obtain the eye image of the object; the determining module 503 is used by the interactive device to determine the position of the fixation point of the object based on the eye image; the output module 505 is used by the interactive device to determine the position of the fixation point of the object And the object information of the object outputs interactive information.

It should be noted that the acquisition module 501, the determination module 503, and the output module 505 correspond to steps S102 to S106 in Embodiment 1. The three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the above. What was disclosed in Example 1.

It should be noted that the interactive device includes: a robot; the object information of the object includes at least one of the following: the expression information of the object, the motion information of the object, and the voice information of the object.

In an optional embodiment, the determining module includes a first determining module, a second determining module, and a third determining module. The first determining module is used by the interactive device to analyze the eye feature information of the object according to the eye image; the third determining module is used by the interactive device to determine the position of the gaze point of the object according to the eye feature information, wherein the eye feature The information may include, but is not limited to, the pupil's pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, light spot position (ie, Pulcin spot position).

It should be noted that the first determination module, the second determination module, and the third determination module correspond to steps S202 to S204 in Embodiment 1. The two modules have the same examples and application scenarios as the corresponding steps, but It is not limited to the content disclosed in the first embodiment.

In an optional embodiment, the output module includes a processing module, a sixth determining module, and a control module. The processing module is used by the interactive device to process the position of the gaze point and the object information of the object to obtain a processing result; the sixth determination module is used by the interactive device to determine the operation instruction according to the processing result; the control module is used by the interactive device to operate The instruction is sent to the corresponding component of the interactive device, and the operation instruction is used to control the corresponding component to output the interactive information.

It should be noted that the processing module, the sixth determination module, and the control module correspond to steps S1060 to S1064 in Embodiment 1. The three modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the above implementation. What was disclosed in Example 1.

Examples 4

According to another aspect of the embodiments of the present invention, a storage medium is also provided, and the storage medium includes a stored program, where the program executes the method for outputting information in Embodiment 1.

Examples 5

According to another aspect of the embodiments of the present invention, there is also provided a processor for running a program, where the program runs the method for outputting information in Embodiment 1.

The sequence numbers of the foregoing embodiments of the present invention are only for description, and do not represent the superiority or inferiority of the embodiments.

In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For a part that is not described in detail in an embodiment, reference may be made to the description of other embodiments.

In the several embodiments provided by the present invention, it should be understood that the disclosed technical content may be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the unit may be a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or integrated To another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, units or modules, and may be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist separately physically, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention essentially or part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, Several instructions are included to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in various embodiments of the present invention. The foregoing storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk, or an optical disk, and other media that can store program codes.

The above is only a preferred embodiment of the present invention. It should be noted that for those who have ordinary knowledge in the technical field to which the present invention pertains, without departing from the principles of the present invention, several improvements and retouches can be made. These improvements And retouching should also be regarded as the protection scope of the present invention.

In summary, the technical means disclosed in the present invention can effectively solve problems such as knowledge, and achieve the intended purpose and effect. It has not been published in publications before application, has not been publicly used, and has long-term progress. The invention referred to in the Patent Law is correct, and he filed an application in accordance with the law. He earnestly hopes that Jun will give him a detailed review and grant a patent for the invention.

However, the above are only a few preferred embodiments of the present invention. When the scope of implementation of the present invention cannot be limited in this way, that is, equivalent changes and modifications made according to the scope of the patent application and the content of the invention specification of the present invention are all It should still fall within the scope of the invention patent.

〔this invention〕

401‧‧‧Image Acquisition Equipment

403‧‧‧Processor

501‧‧‧Get Module

503‧‧‧ Determine the module

505‧‧‧output module

S102-S106‧‧‧step

FIG. 1 is a flowchart of a method for outputting information according to an embodiment of the present invention.

FIG. 2 is a flowchart of an optional method for outputting information according to an embodiment of the present invention.

FIG. 3 is a schematic diagram of an optional process of acquiring an image according to an embodiment of the present invention.

FIG. 4 is a schematic structural diagram of a system for outputting information according to an embodiment of the present invention.

FIG. 5 is a schematic structural diagram of a device for outputting information according to an embodiment of the present invention.

Claims (15)

A method for outputting information includes: The interactive device acquires the eye image of the subject; Determining, by the interactive device, a gaze point position of the object according to the eye image; The interaction device outputs interaction information according to the gaze point position and the object information of the object. The method according to item 1 of the scope of patent application, wherein the interactive device comprises: a robot. The method according to item 1 of the scope of patent application, wherein the object information of the object includes at least one of the following: expression information of the object, motion information of the object, and voice information of the object. The method according to item 1 of the scope of patent application, wherein the determining, by the interactive device, the position of the fixation point of the object according to the eye image includes: Analyzing, by the interactive device, eye feature information of the subject according to the eye image; The interaction device determines a gaze point position of the object according to the eye feature information. The method according to item 4 of the scope of patent application, wherein the eye feature information includes at least one of the following: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, and spot position. The method according to item 1 of the scope of patent application, wherein the interactive device outputs interactive information according to the position of the gaze point and the object information of the object, including: The interactive device processes the position of the gaze point and the object information of the object to obtain a processing result; Determining, by the interactive device, an operation instruction according to the processing result; The interaction device sends the operation instruction to a corresponding component of the interaction device, wherein the operation instruction is used to control the corresponding component to output interaction information. A system for outputting information includes: An image acquisition device for acquiring an eye image of a subject; The processor is connected to the image acquisition device, and is configured to determine a gaze point position of the object according to the eye image, and output interactive information according to the gaze point position and object information of the object. A device for outputting information includes: An acquisition module for acquiring an eye image of an object by an interactive device; A determining module, configured to determine, by the interactive device, a gaze point position of the object according to the eye image; An output module is used by the interaction device to output interaction information according to the position of the gaze point and the object information of the object. The device according to item 8 of the scope of patent application, wherein the interactive device includes: a robot. The device according to item 8 of the scope of patent application, wherein the object information of the object includes at least one of the following: expression information of the object, motion information of the object, and voice information of the object. The device according to item 8 of the scope of patent application, wherein the determining module includes: A first determining module, configured to analyze, by the interaction device, eye feature information of the object according to the eye image; A third determining module, configured to determine, by the interaction device, a gaze point position of the object according to the eye feature information. The device according to item 11 of the scope of patent application, wherein the eye feature information includes at least one of the following: pupil position, pupil shape, iris position, iris shape, eyelid position, eye corner position, and spot position. The device according to item 8 of the scope of patent application, wherein the output module includes: A processing module, configured to process, by the interactive device, the position of the gaze point and the object information of the object to obtain a processing result; A sixth determining module, configured to determine an operation instruction by the interactive device according to the processing result; A control module is configured for the interaction device to send the operation instruction to a corresponding component of the interaction device, wherein the operation instruction is used to control the corresponding component to output interaction information. A storage medium, characterized in that the storage medium includes a stored program, wherein the program executes the method for outputting information according to any one of claims 1 to 6 of the scope of patent application. A processor is characterized in that the processor is used to run a program, wherein the program runs the method for outputting information as described in any one of claims 1 to 6 of the scope of patent application.