WO2020063021A1

WO2020063021A1 - Eye tracking method, apparatus and system applied to terminal device

Info

Publication number: WO2020063021A1
Application number: PCT/CN2019/095264
Authority: WO
Inventors: 孔祥晖; 严海; 黄通兵
Original assignee: 北京七鑫易维信息技术有限公司
Priority date: 2018-09-30
Filing date: 2019-07-09
Publication date: 2020-04-02
Also published as: CN109522789A

Abstract

Disclosed in the present application are an eye tracking method, apparatus and system applied to a terminal device. The method comprises: acquiring image information; extracting eye feature information from the image information to form an eye feature information set, wherein the image information comprises the eye feature information; sending the eye feature information set to a server; and receiving and parsing gaze information fed back from the server.

Description

Eye tracking method, device and system applied to terminal equipment

Technical field

The present application relates to the technical field of eye tracking, and in particular, to an eye tracking method, device, and system applied to a terminal device.

Background technique

Eye tracking technology, as an innovative interactive method, is more and more well known to the public, and has been widely used in people's work and learning. The process of eye tracking technology mainly includes the following three steps:

Step S1, the eye-tracking device acquires a user's face image using a collection device, where the collection device may be an optical device, an electrical device, etc .;

Step S2: The eye-tracking device processes the face image of the user, and extracts eye feature information of the user;

In step S3, the eye tracking device processes the eye feature information of the user, and obtains the user's gaze direction and gaze point position.

At present, the above three steps are implemented on the device side. However, in the process of implementing eye tracking for the user on the device side, the system running capacity, storage space and other resources are consumed on the device side. In addition, if the eye tracking program is run for a long time, the device side may become hot and hot. Problems such as increased power consumption, and the above problems reduce the user experience.

Aiming at the problem that the aforementioned eye tracking algorithms of the prior art all run on the device side, causing a large resource consumption on the device side, no effective solution has been proposed at present.

Summary of the Invention

Embodiments of the present application provide an eye tracking method, device, and system applied to a terminal device, so as to at least solve a technical problem that the prior art eye tracking algorithms are all run on the device side, which causes a large resource consumption on the device side.

According to an aspect of the embodiment of the present application, an eye tracking method applied to a terminal device is provided, which includes: collecting image information; sending image information to a cloud processor; and receiving and analyzing gaze information fed back by the cloud processor.

Further, the eye tracking method applied to the terminal device further includes: acquiring transmission information; determining an image identifier corresponding to the transmission information, wherein the image identifier is used to identify a transmission sequence of the first frame image; and transmitting the transmission information in a first regular manner To cloud processor.

Further, the eye tracking method applied to the terminal device further includes: determining a time stamp corresponding to the first frame image of the transmission information; and generating an image identifier corresponding to the first frame image according to the time stamp.

Further, the sending information includes: a first frame header, a first control word, a first check code, and cloud information.

Further, the eye tracking method applied to the terminal device further includes: receiving reception information sent in a second regular manner; analyzing the reception information, the reception information includes a second frame header, a second control word, a second check code, and gaze information Complete the verification of the second check code; if the verification result of the second check code satisfies a preset condition, determine the fixation information as the fixation information of the target object.

According to another aspect of the embodiments of the present application, an eye tracking method applied to a terminal device is further provided. The method includes: acquiring image information sent by the device end; extracting eye feature information from the image information to form an eye feature information set. , Wherein the image information includes eye feature information; determining gaze information according to the eye feature information set; and sending gaze information to the device.

According to another aspect of the embodiments of the present application, an eye tracking method applied to a terminal device is further provided, including: collecting image information; extracting eye feature information from the image information to form an eye feature information set, wherein the image The information includes eye feature information; sends the eye feature information set to the server; receives and analyzes the gaze information fed back by the server.

Further, before forming the eye feature information set, the eye tracking method applied to the terminal device further includes: converting the image information into corresponding eye feature information; configuring the corresponding eye feature information with a data label; The corresponding eye feature information of the configured data tags is sorted.

Further, the corresponding eye feature information includes one or more of pupil information, corneal information, spot information, iris information, and / or eyelid information, eye image information, and gaze depth information of the target object.

Further, the eye feature information set is configured as one or more groups of eye feature information with data labels.

Further, the eye tracking method applied to the terminal device further includes: acquiring transmission information; determining an image identifier corresponding to the transmission information, and acquiring a set of eye feature information corresponding to the series of image identifiers, wherein the image identifier is used to identify the first frame of the image. Sending order; sending the sending information to the server in a first regular manner.

Further, the sending information further includes: a first frame header, a first control word, and a first check code.

According to another aspect of the embodiments of the present application, an eye tracking device applied to a terminal device is further provided, including: a first acquisition module configured to acquire image information; and a first sending module configured to send image information to cloud processing A first receiving module configured to receive and analyze gaze information fed back by a cloud processor.

According to another aspect of the embodiments of the present application, an eye tracking device applied to a terminal device is further provided, including: a second acquisition module configured to acquire image information; and a second extraction module configured to extract eyes from the image information Eye feature information, forming eye feature information set, wherein the image information includes eye feature information; a second sending module is configured to send the eye feature information set to the server; a second receiving module is configured to receive and analyze the server feedback Gaze information.

According to another aspect of the embodiments of the present application, an eye tracking system applied to a terminal device is further provided, including: a device end configured to collect image information, and extract eye feature information from the image information to form eye features Information collection, and then send the eye feature information to the server to receive and analyze the gaze information returned by the server; the server is configured to receive the eye feature information sent by the device, and process the eye feature information to obtain the gaze of the target object information.

According to another aspect of the embodiments of the present application, a storage medium is also provided. The storage medium includes a stored program, where the program executes an eye tracking method applied to a terminal device.

According to another aspect of the embodiments of the present application, a processor is further provided. The processor is configured to run a program, wherein the program executes an eye tracking method applied to a terminal device when the program is run.

In the embodiment of the present application, the method of combining the device side with the server is used to collect image information on the device side, and extract eye feature information from the image information to form an eye feature information set, and then send the eye feature information set. To the server, the server processes according to the eye feature information set to obtain the fixation information of the target object, and sends the fixation information of the target object to the device side, and then the device side receives and analyzes the fixation information fed back by the server.

In the above process, the collection of image information of the target object and the extraction of the eye feature information set run on the device side, and the processing of the eye feature information set is implemented in the server. Because the gaze point algorithm that processes gaze information to obtain gaze information consumes more resources, processing it on the server effectively reduces the resource consumption on the device side. In addition, if the image is directly uploaded to the server, it will consume more bandwidth and cause resource consumption. Therefore, in this application, the device can extract the eye feature information set from the image information, and set the eye feature information set. Sending to the server instead of directly sending the image information to the server can reduce the bandwidth occupied by directly uploading the image information to the server, and further reduce the resource consumption on the device side.

It can be seen that the solution provided in the present application can solve the technical problem that the eye tracking algorithms in the prior art all run on the device side, which causes a large resource consumption on the device side.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described here are used to provide a further understanding of the present application and constitute a part of the present application. The schematic embodiments of the present application and the description thereof are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

1 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application;

2 is a schematic diagram of an optional frame format according to an embodiment of the present application;

3 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application;

4 is a schematic structural diagram of an eye tracking device applied to a terminal device according to an embodiment of the present application;

5 is a schematic structural diagram of an eye tracking device applied to a terminal device according to an embodiment of the present application;

6 is a schematic structural diagram of an eye tracking system applied to a terminal device according to an embodiment of the present application;

7 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application;

8 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application; and

FIG. 9 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application.

detailed description

In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only Examples are part of this application, but not all examples. Based on the embodiments in this application, all other embodiments obtained by a person of ordinary skill in the art without creative efforts should fall within the protection scope of this application.

It should be noted that the terms “first” and “second” in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data so used may be interchanged where appropriate so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. Furthermore, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units need not be limited to those explicitly listed Those steps or units may instead include other steps or units not explicitly listed or inherent to these processes, methods, products or equipment.

Example 1

According to the embodiment of the present application, an embodiment of an eye tracking method applied to a terminal device is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And, although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than here.

In addition, it should also be noted that the device end can execute the eye tracking method applied to the terminal device provided in this embodiment. Wherein, the device end includes a mobile device and a non-mobile device, and the mobile device may be, but is not limited to, a mobile phone, a tablet, and the like.

FIG. 1 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application. As shown in FIG. 1, the method includes the following steps:

In step S102, image information is collected.

It should be noted that the device has an image collector, which can collect image information of the target object, wherein the image information of the target object includes an eye image of the target object. Optionally, when the device is a mobile phone, the image collector may be a camera of the mobile phone. Generally, the resolution of the rear camera of the mobile phone is higher than the resolution of the front camera of the mobile phone, but the resolution of the front camera of the mobile phone is 10 million pixels, and the image information it collects meets the needs of eye tracking technology. In the case where the mobile phone has a front camera, the image collector in this embodiment uses the front camera of the mobile phone.

Step S104: Send the image information to the cloud processor.

It should be noted that because the cloud server has strong processing capabilities and can run more complex algorithms, after collecting the image information of the target object, the device sends the image information of the target object to the cloud processor for processing. To achieve the purpose of reducing equipment-side resource consumption. The cloud processor is a processor in a cloud server.

In addition, the device side and the cloud processor may communicate through a wireless communication method. The wireless communication method may include, but is not limited to, WIFI, 3G, 4G, and GPRS.

In an optional solution, after receiving the image information, the cloud processor extracts at least one of the following information in the image information based on the data extraction algorithm: pupil information, corneal information, iris information and / or eyelid information, eyes Image information, gaze depth information, and at least one piece of information is used as eye feature information. After obtaining the eye feature information, the cloud processor calculates the fixation information of the target object according to the eye tracking algorithm. The fixation information includes at least fixation direction information and fixation point information. The fixation point information may be the fixation point of the target object on the device. Coordinate information on the screen. Then, the cloud processor sends the calculated gaze information to the device side through wireless communication, and the device side can receive the gaze information.

Step S106: Receive and analyze gaze information fed back by the cloud processor.

It should be noted that the device side and the cloud processor can communicate through wireless communication. The wireless communication methods may include, but are not limited to, WIFI, 3G, 4G, and GPRS.

In addition, it should be noted that the cloud processor does not need to send the image information to the device side, and only sends the gaze information to the device side. The communication rate between the cloud processor and the device can meet the real-time requirements, and the throughput is small. Even if the GPRS with a slower communication rate is used, it can meet the requirements.

Based on the solutions defined in the above steps S102 to S106, it can be learned that the device-side and cloud processing methods are used to collect the image information of the target object on the device side and send the image information to the cloud processor. After receiving the image information, extract the eye feature information from the image information to form the eye feature information, and then process the eye feature information to obtain the fixation information of the target object, and send the fixation information of the target object to the device. The device side then receives and analyzes the gaze information fed back by the cloud processor.

It is easy to notice that because the gaze point algorithm that processes gaze information based on eye feature information consumes a lot of resources, processing it on a cloud processor effectively reduces resource consumption on the device side. In addition, because the cloud processor has strong processing power and can run more complex algorithms, the cloud processor can directly process the image information of the target object to obtain eye feature information, and extract gaze information based on the eye feature information. , And finally send the gaze information to the device side to further reduce the resource consumption of the device side.

It should be noted that after the image information of the target object is collected, the device end can extract a series of eye feature information from the image information. Specifically, the device side extracts at least one of the following information in the image information based on the data extraction algorithm: pupil information, corneal information, spot information, iris information and / or eyelid information, eye image information, gaze depth information, and at least One piece of information is used as the eye feature information.

In addition, it should be noted that the image information collected by the device may include image information in multiple frames of images. In order to ensure pupil information, corneal information, light spot information, iris information, and / or eyelid information, eye image information, Gaze depth information is a set of eye feature information extracted from the same frame image. During the process of sending the image information to the cloud processor, the current frame image is image-identified. The specific process may include the following steps;

Step S1060, obtaining the sending information;

Step S1062, determining an image identifier corresponding to the transmission information, where the image identifier is used to identify a transmission sequence of the transmission information;

Step S1064: Send the sending information to the processor in a first regular manner.

It should be noted that the image identifier may be, but is not limited to, a time stamp, a name, or a random assigned identifier. Where the image identification is a timestamp, the timestamp is a timestamp corresponding to the time when the image information was collected. Optionally, the foregoing sending information may be information corresponding to the current frame image. The device end may determine a timestamp corresponding to the first frame image of the sending information, and generate an image identifier corresponding to the first frame image according to the timestamp.

It is easy to notice that in the solutions defined in the above steps S1060 to S1064, the image identification is also sent to the cloud processor during the process of sending image information to the cloud processor. The cloud processor can receive the image identification in sequence according to the image identification of each frame of the image, which ensures that the cloud processor correctly analyzes the image information after the identification, and then processes the correct eye feature information to obtain accurate gaze information of the target object.

In an optional solution, before sending the image information corresponding to the image identifier, the device needs to configure the sending information corresponding to the image information, and then sends the sending information to the cloud processor in a first regular manner. The sending information includes: a first frame header, a first control word, a first check code, and image information. Optionally, the first rule mode may be a frame format. FIG. 2 shows a schematic diagram of an optional frame format. As can be seen from FIG. 2, the frame format includes at least four parts, that is, a frame header, a control word, and data. (Ie, image information) and a check code, where the check code may be, but is not limited to, a number obtained by CRC (Cyclic Redundancy Check).

It should be noted that the use of a CRC check can ensure that after the device communicates with the cloud processor, the cloud processor can parse the correct data and ensure the correctness of the data. At the same time, it can prevent the interference signal from affecting the analysis result.

In addition, it should also be noted that after receiving the eye feature information, the processor of the server analyzes the eye feature information and processes the parsed data to obtain the fixation information of the target object, and according to the second rule The gaze information is sent to the device. At this time, the data in the frame format is the gaze information. The device end receives the received information sent in the second regular mode, parses the received information, and then completes the verification of the second check code. Wherein, if the verification result of the second verification code satisfies a preset condition, the device end determines the fixation information as the fixation information of the target object. Optionally, the received information includes a second frame header, a second control word, a second check code, and gaze information.

It should be noted that, after receiving the data in the second regular mode, the device end uses the check code to verify the parsed data to ensure the correctness of the data, and at the same time, it can prevent the interference signal from affecting the analysis result. Get the correct gaze information for your target.

It can be known from the foregoing that the solution provided by the present application can effectively reduce the requirement of the computing capability of the device tracking of the eye tracking technology. In addition, the cloud processor updates the eye tracking algorithm in real time to complete the extraction of gaze information, which can improve the eye tracking processing rate on the device side. Finally, the solution provided in this application runs the image processing algorithm on the device side and the gaze point algorithm is run by the cloud processor, which can greatly reduce the throughput of communication data and increase the communication rate.

Example 2

According to the embodiment of the present application, an embodiment of an eye tracking method applied to a terminal device is provided. It should be noted that the server may execute the eye tracking method applied to a terminal device provided by this embodiment. FIG. 9 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application. As shown in FIG. 9, the method includes the following steps:

In step S902, image information is collected.

It should be noted that the device has an image collector, which can collect image information of the target object, wherein the image information of the target object includes an eye image of the target object. Optionally, when the device is a mobile phone, the image collector may be a camera of the mobile phone. Generally, the resolution of the rear camera of the mobile phone is higher than the resolution of the front camera of the mobile phone, but the resolution of the front camera of the mobile phone is 10 million pixels, and the image information it collects meets the needs of eye tracking technology. In the case where the mobile phone has a front camera, the image collector in this embodiment selects the front camera of the mobile phone.

Step S904, extracting eye feature information from the image information to form an eye feature information set, where the image information includes eye feature information.

It should be noted that the eye feature information set is configured as one or more groups of eye feature information with data labels. In addition, before forming the eye feature information set, the device side also converts the image information of the target object into corresponding eye feature information, configures the corresponding eye feature information with data tags, and then configures the configured data tags in a specified order. The corresponding eye feature information is sorted. The corresponding eye characteristic information includes one or more of pupil information, corneal information, spot information, iris information and / or eyelid information, eye image information, and gaze depth information of the target object. The pupil information is at least Including: pupil center position, pupil diameter; corneal information includes at least: corneal spot reflection information; iris information includes at least: iris edge information.

Step S906: Send the eye feature information set to the server.

It should be noted that the server in step S906 may be a cloud processor, wherein the device side and the server may communicate through wireless communication, and the wireless communication method may include, but is not limited to, WIFI, 3G, 4G, GPRS, etc. .

In addition, it should also be noted that during the process of sending the eye feature information set by the device to the server, the device does not need to send the entire collected image information to the server, and only sends the eye feature information to the server. Optionally, the device end may send the eye feature information to the server in the form of a frame format. The frame format may be composed of a frame header, a control word, human eye data, and a check code. The device end sends the eye feature on the device side. When the information is sent to the server, it can be based on a 1-byte frame header, a 1-byte control word, 3 to 6 bytes of eye characteristic information (that is, human eye data), and a 2-byte CRC check (that is, a check code). Sent to the server, where the total amount sent is about 10 bytes. In addition, in the case of processing 90 frames of image data per second, the amount of data sent by the device to the server is 90 * 10 * 8 = 7.2kbps. It can be seen that the communication rate between the device and the server can meet the real-time requirements, and the throughput is small, even if the GPRS with a slower communication rate can meet the requirements.

In an optional solution, the device side collects a two-dimensional face image of the target object, and the resolution of the two-dimensional face image is 1920 * 1080. The device side processes the two-dimensional face image to obtain a solid circle of 6 pixels, where the center coordinates of the solid circle are (800 * 800, 840 * 800). At this time, the device side sends the center coordinates of the solid circle to the server through wireless communication.

Step S908: Receive and analyze the gaze information fed back by the server.

In an optional solution, after the server receives the eye feature information set, the server calculates the fixation information of the target object according to the eye tracking algorithm, wherein the fixation information includes at least fixation direction information and fixation point information, wherein the fixation point The information may be coordinate information of the gaze point of the target object on the screen of the device. Then, the server sends the calculated gaze information to the device side through wireless communication, and the device side receives the gaze information. It should be noted that, in the above process, the server does not need to send the image information to the device side, and only sends the gaze information to the device side. The communication rate between the server and the device can meet the real-time requirements, and the throughput is small, even if the GPRS with a slower communication rate can meet the requirements.

In addition, it should also be noted that, because the eye tracking algorithm requires high computing power on the device side, the eye tracking algorithm has strict requirements on the device core and the main frequency, and it is difficult for the mid-level device side to meet the hardware index. In addition, if the eye tracking algorithm is always running on the device side, the resource consumption on the device side is large, and even the device side generates heat and freezes. The solution provided in this application runs the eye tracking algorithm on the server, which can effectively avoid the above problems.

In addition, the battery capacity on the device side is limited, and the eye tracking algorithm is a technology based on video streaming, which consumes high power. Therefore, running the eye tracking algorithm on the server can reduce energy consumption on the device side.

Based on the solutions defined in the above steps S902 to S908, it can be learned that the method of combining the device side with the server collects image information of the target object on the device side, and extracts eye feature information from the image information to form eye features. Information set, and then send the eye feature information set to the server, and the server processes the eye feature information set to obtain the fixation information of the target object, and sends the fixation information of the target object to the device side, and then the device side receives and analyzes Gaze information returned by the server.

It is easy to notice that the collection of image information of the target object and the extraction of the eye feature information set run on the device side, and the processing of the eye feature information set is implemented in the server. Because the gaze point algorithm that processes gaze information to obtain gaze information consumes more resources, processing it on the server effectively reduces the resource consumption on the device side. In addition, uploading the image directly to the server will consume more bandwidth and cause resource consumption. Therefore, because the server has a strong processing capability, it can run more complex algorithms. Therefore, in this application, the device side can also download The eye feature information set is extracted from the image information and sent to the server instead of directly sending the image information to the server, which can reduce the bandwidth occupied by directly uploading the image information to the server, further reducing the device-side LF.

It should be noted that after the image information of the target object is collected, the device end can extract the eye feature information from the image information. Specifically, the device side extracts at least one of the following information in the image information based on the data extraction algorithm: pupil information, corneal information, spot information, iris information and / or eyelid information, eye image information, gaze depth information, and at least One piece of information is used as the eye feature information.

In addition, it should be noted that the eye feature information set extracted from the image information may contain eye feature information in multiple frames of images. In order to ensure pupil information, corneal information, light spot information, iris information, and / or eyelid information The eye image information and gaze depth information are eye feature information extracted from the same frame image. In the process of sending the eye feature information set to the server, image identification is performed on the current frame image. The specific process may include the following steps;

Step S2060, obtaining the sending information;

Step S2062: Determine an image identifier corresponding to the transmission information, and obtain an eye feature information set corresponding to the series of image identifiers, where the image identifier is used to identify a transmission sequence of the transmission information;

Step S2064: Send the sending information to the server in a first regular manner.

It is easy to notice that in the solutions defined in the above steps S2060 to S2064, the image identification is also sent to the server during the process of sending the eye feature information set to the server. The server can receive the image identification in order according to the image identification of each frame of the image, which ensures that the server correctly analyzes the identified eye feature information, and then processes the correct eye information to obtain accurate gaze information of the target object.

In an optional solution, the device end needs to configure the sending information corresponding to the eye characteristic information before sending the set of eye characteristic information corresponding to the image identification series, and then sends the sending information to the server in a first regular manner. The sending information includes: a first frame header, a first control word, a first check code, and an eye feature information set. Optionally, the first rule mode may be a frame format. FIG. 2 shows a schematic diagram of an optional frame format. As can be seen from FIG. 2, the frame format includes at least four parts, that is, a frame header, a control word, and data. (Ie, eye feature information) and a check code, where the check code may be, but is not limited to, a number obtained by CRC (Cyclic Redundancy Check). It should be noted that the use of the CRC check can ensure that after the device communicates with the server, the server can parse the correct data and ensure the correctness of the data. At the same time, it can prevent the interference signal from affecting the analysis result.

In addition, it should be noted that after receiving the eye feature information, the server parses the eye feature information and processes the parsed data to obtain the fixation information of the target object, and fixes the fixation according to the second rule. The information is sent to the device. At this time, the data in the frame format is gaze information. The device end receives the received information sent in the second regular mode, parses the received information, and then completes the verification of the second check code. Wherein, if the verification result of the second verification code satisfies a preset condition, the device end determines the fixation information as the fixation information of the target object. Optionally, the received information includes a second frame header, a second control word, a second check code, and gaze information. It should be noted that after receiving the data in the second regular mode, the device end uses the check code to verify the parsed data to ensure the correctness of the data, and at the same time, it can prevent the interference signal from affecting the analysis result. Get the correct gaze information for your target.

It can be known from the foregoing that the solution provided by the present application can effectively reduce the requirement of the computing capability of the device tracking of the eye tracking technology. In addition, the eye tracking algorithm is updated in real time by the server to complete the extraction of gaze information, which can improve the processing rate of eye tracking on the device side. Finally, the solution provided in this application runs the image processing algorithm on the device side and the gaze point algorithm is run by the server, which can greatly reduce the throughput of communication data and increase the communication rate.

Example 3

According to the embodiment of the present application, an embodiment of an eye tracking method applied to a terminal device is provided. It should be noted that the server can execute the eye tracking method applied to a terminal device provided in this embodiment, and the server may be a cloud server. FIG. 3 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application. As shown in FIG. 3, the method includes the following steps:

Step S302: Receive eye feature information of the target object sent by the device.

It should be noted that the device end includes a mobile device and a non-mobile device, and the mobile device may be, but is not limited to, a mobile phone, a tablet, and the like. The device end has an image collector, which can collect image information of the target object, wherein the image information of the target object includes an eye image of the target object. Optionally, when the device is a mobile phone, the image collector may be a camera of the mobile phone. Generally, the resolution of the rear camera of the mobile phone is higher than the resolution of the front camera of the mobile phone, but the resolution of the front camera of the mobile phone is 10 million pixels, and the image information it collects meets the needs of eye tracking technology. In the case where the mobile phone has a front camera, the image collector of this embodiment selects the front camera of the mobile phone.

In addition, the eye feature information of the target object includes at least one of the following: pupil information, corneal information, spot information, iris information, eyelid information, eye image information, and gaze depth information, where the pupil information includes at least the pupil center position, Pupil diameter; corneal information includes at least: corneal spot reflection information; iris information includes at least: iris edge information.

Step S304: Determine gaze information of the target object according to the eye feature information.

In an optional solution, after the server receives the eye feature information, the server calculates the fixation information of the target object according to the eye tracking algorithm, where the fixation information may be coordinate information of the fixation point of the target object on the device-side screen .

Step S306: Send gaze information to the device.

It should be noted that the device and the server can communicate through wireless communication. The wireless communication methods may include, but are not limited to, WIFI, 3G, 4G, and GPRS. In addition, it should be noted that the server does not need to send the image information to the device side, it only needs to send the gaze information to the device side. The communication rate between the server and the device can meet the real-time requirements, and the throughput is small, even if the GPRS with a slower communication rate can meet the requirements.

Optionally, after receiving the eye feature information, the server calculates the fixation information of the target object according to the eye tracking algorithm, where the fixation information may be coordinate information of the fixation point of the target object on the screen of the device. Then, the server sends the calculated gaze information to the device side through wireless communication, and the device side receives the gaze information.

Based on the solutions defined in the above steps S302 to S306, it can be learned that the device side and the server are used to collect the image information of the target object on the device side, and extract the eye feature information of the target object from the image information, and then The eye feature information is sent to the server, and the server processes the eye feature information to obtain the fixation information of the target object, and sends the fixation information of the target object to the device side, so that the fixation information of the target object can be displayed on the device side.

It is easy to notice that the collection of image information of the target object and the extraction of eye feature information are run on the device side, while the processing of eye feature information is implemented on the server. Because the gaze point algorithm that processes gaze information based on eye feature information consumes a lot of resources, processing it on the server effectively reduces resource consumption on the device side. In addition, if the image is directly uploaded to the server, it will consume more bandwidth and cause resource consumption. Therefore, in this application, the device can extract the eye feature information from the image information and send the eye feature information to The server, instead of sending the image information directly to the server, can reduce the bandwidth occupied by directly uploading the image information to the server, further reducing the resource consumption on the device side.

In an optional solution, the device sends the eye feature information of the target object in a frame format, and determining the fixation information of the target object according to the eye feature information may include the following steps:

Step S3020: Parse the data in the frame format to obtain an analysis result, where the data in the frame format includes eye feature information;

Step S3022, verify the analysis result to obtain a verification result;

In step S3024, if the verification result meets a preset condition, fixation information of the target object is determined according to the eye feature information.

It should be noted that the foregoing frame format may be the frame format shown in FIG. 2. Optionally, after receiving the eye feature information, the server parses the eye feature information and checks the check code in the frame format. When the check code passes the check, the server processes the parsed data to obtain the fixation information of the target object.

Further, after obtaining the gaze information, the server sends the gaze information to the device side, and the specific steps may include:

Step S3060, determining an image identifier corresponding to the eye feature information;

Step S3062, obtaining gaze information corresponding to the image identification;

Step S3064, sending the gaze information and the image identification format to the device.

Optionally, the server may send the eye feature information and the image identifier corresponding to the current frame image according to the following frame format: a frame header, a control word, human eye data, and a check code, where the human eye data includes gaze information. It should be noted that the image identifier may be, but is not limited to, a time stamp, a name, or a random assigned identifier. Where the image identification is a timestamp, the timestamp is a timestamp corresponding to the time when the image information was collected.

Optionally, after parsing the eye feature information, the server processes the eye feature information based on the eye tracking algorithm to obtain fixation information, and feeds the fixation information to the device according to the frame format. At this time, the data in the frame format For gaze information. After receiving the gaze information, the device side parses the gaze information and passes the gaze information to the application side (for example, a display device). It should be noted that the predetermined format includes at least: a frame header, a control word, human eye data, and a check code. The check code may be, but is not limited to, obtained by CRC (Cyclic Redundancy Check). Numbers. The use of the CRC check can ensure that after the device communicates with the server, the server can parse the correct data and ensure the correctness of the data. At the same time, it can prevent the interference signal from affecting the analysis result.

In an optional solution, after receiving the eye feature information, the server analyzes the eye feature information and processes the parsed data to obtain the fixation information of the target object, and converts the fixation information according to the frame format. Sent to the device. At this time, the data in the frame format is gaze information. After receiving the gaze information, the device side parses the gaze information and passes the gaze information to the application side (for example, a display device).

It can be known from the foregoing that the solution provided by the present application can effectively reduce the requirement of the computing capability of the device tracking of the eye tracking technology. In addition, the eye tracking algorithm is updated in real time by the server to complete the extraction of gaze information, which can improve the processing rate of eye tracking on the device side. Finally, the solution provided in this application runs the image processing algorithm on the device side and the gaze point algorithm on the server, which can greatly reduce the throughput of communication data and increase the communication rate.

Example 4

According to an embodiment of the present application, an embodiment of an eye tracking method applied to a terminal device is provided, wherein FIG. 7 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application, as shown in FIG. 7, The method includes the following steps:

In step S702, the eye feature information of the target object is extracted from the image information of the target object.

It should be noted that the eye feature information of the target object includes at least one of the following: pupil information, corneal information, iris information, eyelid information, eye image information, and gaze depth information, where the pupil information includes at least the pupil center position, Pupil diameter; corneal spot includes at least: corneal spot reflection information; iris information includes at least: iris edge information.

In an optional solution, the device has an image collector, and the image collector can collect image information of the target object. After collecting the image information of the target object, the device uses the pupil-corneal reflection method to implement eye tracking technology to extract the eye feature information of the target object. Optionally, taking the pupil-corneal reflection method as an example, after the image acquisition device collects the image information of the target object, the device side uses an image processing algorithm to extract the eye characteristic information of the target object, where the eye characteristic information is expressed as The coordinate data in the image information includes, for example, coordinates of the center position of the pupil, coordinates of the center position of the corneal spot, and the like.

Step S704: Send eye feature information to the server, where the server processes the eye feature information to obtain gaze information of the target object.

It should be noted that the device and the server can communicate through wireless communication. The wireless communication methods may include, but are not limited to, WIFI, 3G, 4G, and GPRS. In addition, it should also be noted that during the process of sending eye feature information from the device to the server, the device does not need to send the entire collected image information to the server, only the pupil information, corneal information, iris information, and eyelid information, The eye feature information such as eye image information and gaze depth information may be transmitted to the server. When it is sent, it can be sent to the server according to the 1-byte frame header, 1-byte control word, 3 to 6 bytes of eye feature information, and 2-byte CRC check. Among them, the total number sent is about 10 words Section. In addition, in the case of processing 90 frames of image data per second, the amount of data sent by the device to the server is 90 * 10 * 8 = 7.2kbps. It can be seen that the communication rate between the device and the server can meet the real-time requirements, and the throughput is small, even if the GPRS with a slower communication rate can meet the requirements.

In an optional solution, after the server receives the eye feature information, the server calculates the fixation information of the target object according to the eye tracking algorithm, where the fixation information may be coordinate information of the fixation point of the target object on the device-side screen . Then, the server sends the calculated gaze information to the device side through wireless communication, and the device side receives the gaze information.

It should be noted that, in the above process, the server does not need to send the image information to the device side, and only sends the gaze information to the device side. The communication rate between the server and the device can meet the real-time requirements, and the throughput is small, even if the GPRS with a slower communication rate can meet the requirements.

Based on the solutions defined in the above steps S702 to S704, it can be learned that the device side and the server are used to collect the image information of the target object on the device side, and extract the eye feature information of the target object from the image information, and then The eye feature information is sent to the server, and the server processes the eye feature information to obtain the fixation information of the target object, and sends the fixation information of the target object to the device side, so that the fixation information of the target object can be displayed on the device side.

It is easy to notice that the collection of image information of the target object and the extraction of eye feature information are run on the device side, while the processing of eye feature information is implemented on the server. Because the gaze point algorithm that processes gaze information based on eye feature information consumes more resources, processing it on the server effectively reduces resource consumption on the device side.

Example 5

According to an embodiment of the present application, an embodiment of an eye tracking method applied to a terminal device is provided, wherein FIG. 8 is a flowchart of an eye tracking method applied to a terminal device according to an embodiment of the present application, as shown in FIG. 8, The method includes the following steps:

Step S802: Acquire image information sent by the device.

It should be noted that the device has an image collector, which can collect image information of the target object, wherein the image information of the target object includes an eye image of the target object. Optionally, when the device is a mobile phone, the image collector may be a camera of the mobile phone. Generally, the resolution of the rear camera of the mobile phone is higher than the resolution of the front camera of the mobile phone, but the resolution of the front camera of the mobile phone is 10 million pixels, and the image information it collects meets the needs of eye tracking technology. In the case where the mobile phone has a front camera, in this embodiment, the image collector uses the front camera of the mobile phone.

In addition, it should be noted that because the server has strong processing power and can run more complex algorithms, after collecting the image information of the target object, the device sends the image information of the target object to the server for processing in order to To achieve the purpose of reducing equipment-side resource consumption.

Step S804, extracting eye feature information from the image information to form an eye feature information set, where the image information includes eye feature information.

In an optional solution, after receiving the image information, the server extracts at least one of the following information in the image information based on the data extraction algorithm: pupil information, corneal information, iris information and / or eyelid information, and eye image information 2. Gaze depth information, and use at least one piece of information as eye feature information.

Step S806: Determine the visual information according to the eye feature information set.

In an optional solution, after obtaining the eye feature information, the server calculates the fixation information of the target object according to the eye tracking algorithm. The fixation information includes at least fixation direction information and fixation point information. The fixation point information may be The coordinate information of the target object's fixation point on the screen of the device. Then, the server sends the calculated gaze information to the device side through wireless communication, and the device side receives the gaze information, that is, step S808 is performed.

It should be noted that, because the eye tracking algorithm requires high computing power on the device side, the eye tracking algorithm has strict requirements on the device core and the main frequency, and it is difficult for the mid-level device side to meet the hardware index. In addition, if the eye tracking algorithm is always running on the device side, the resource consumption on the device side is large, and even the device side generates heat and freezes. The solution provided in this application runs the eye tracking algorithm on the server, which can effectively avoid the above problems.

Step S808: Send gaze information to the device.

It should be noted that the device and the server can communicate through wireless communication. The wireless communication methods may include, but are not limited to, WIFI, 3G, 4G, and GPRS.

In addition, it should be noted that the server does not need to send the image information to the device side, it only needs to send the gaze information to the device side. The communication rate between the server and the device can meet the real-time requirements, and the throughput is small, even if the GPRS with a slower communication rate can meet the requirements.

Based on the solutions defined in the above steps S802 to S808, it can be learned that, in a manner that the device side and the server are combined, after acquiring the image information of the target object sent by the device side, the server extracts the eyes of the target object from the image information. Feature information, and determine gaze information of the target object based on the eye feature information, and then send the gaze information to the device.

It is easy to notice that because the gaze point algorithm that processes gaze information based on eye feature information consumes a lot of resources, processing it on the server effectively reduces resource consumption on the device side. In addition, because the server has strong processing power and can run more complex algorithms, the server can directly process the image information of the target object to obtain eye feature information, extract gaze information based on the eye feature information, and finally fixate The information is sent to the device side, which further reduces the resource consumption of the device side.

Example 6

According to the embodiment of the present application, an embodiment of an eye tracking device applied to a terminal device is provided, and the device can execute the eye tracking method applied to a terminal device provided in Embodiment 1. 4 is a schematic structural diagram of an eyeball tracking device applied to a terminal device according to an embodiment of the present application. As shown in FIG. 4, the device includes a first acquisition module 401, a first sending module 403, and a first receiving module 405. .

The first collection module 401 is configured to collect image information; the first sending module 403 is configured to send an eye feature information set to a processor; and the first receiving module 405 is configured to receive and analyze gaze information fed back by the processor.

It should be noted that the above-mentioned first acquisition module 401, first sending module 403, and first receiving module 405 correspond to steps S102 to S104 in Embodiment 1. Examples and application scenarios implemented by the three modules and corresponding steps The same, but not limited to the content disclosed in the first embodiment.

In an optional solution, the first sending module includes a first obtaining module, a first determining module, and a second sending module. The first obtaining module is configured to obtain the sending information; the first determining module is configured to determine the image identifier corresponding to the sending information, wherein the image identifier is used to identify the sending order of the sending information; the second sending module is configured to send the sending information; The information is sent to the cloud processor in a first regular manner.

It should be noted that the first obtaining module, the first determining module, and the second sending module correspond to steps S1060 to S1064 in Embodiment 1. The three modules and the corresponding steps implement the same examples and application scenarios, but It is not limited to the content disclosed in the first embodiment.

In an optional solution, the first determining module includes a second determining module and a first generating module. The second determining module is configured to determine a timestamp corresponding to the first frame image of the transmitted information; the first generating module is configured to generate an image identifier corresponding to the first frame image according to the timestamp. The sending information includes: a first frame header, a first control word, a first check code, and image information.

In an optional solution, the first receiving module includes a second receiving module, a parsing module, a verifying module, and a third determining module. The second receiving module is configured to receive the received information sent in a second regular manner; the parsing module is configured to analyze the received information, and the received information includes a second frame header, a second control word, a second check code, and gaze information. A verification module configured to complete the verification of the second verification code; a third determination module configured to determine the fixation information as the fixation of the target object if the verification result of the second verification code meets a preset condition information.

Example 7

According to an embodiment of the present application, an embodiment of an eye tracking device applied to a terminal device is provided. The device can execute the eye tracking method applied to a terminal device provided in Embodiment 2. 5 is a schematic structural diagram of an eye tracking device applied to a terminal device according to an embodiment of the present application. As shown in FIG. 5, the device includes a second acquisition module 501, a second extraction module 503, and a second sending module 505. And the second receiving module 507.

The second acquisition module 501 is configured to collect image information; the second extraction module 503 is configured to extract eye characteristic information from the image information to form an eye characteristic information set, where the image information includes eye characteristic information; The second sending module 505 is configured to send the eye feature information set to the server; the second receiving module 507 is configured to receive and analyze the gaze information fed back by the server.

It should be noted that the above-mentioned second acquisition module 501, second extraction module 503, second sending module 505, and second receiving module 507 correspond to steps S902 to S908 in Embodiment 2, and the four modules correspond to the corresponding steps. The implementation example is the same as the application scenario, but is not limited to the content disclosed in the above embodiment 2.

In an optional solution, the second sending module includes an obtaining module, a determining module, and a sending module. The obtaining module is configured to obtain the sending information; the determining module is configured to determine the image identifier corresponding to the sending information and obtain the eye feature information set corresponding to the image identifier, wherein the image identifier is used to identify the sending order of the sending information; Set to send messages to the server. The sending information further includes: a first frame header, a first control word, a first check code, and an eye feature information set.

It should be noted that the foregoing obtaining module, determining module, and sending module correspond to steps S2060 to S2064 in Embodiment 2. The three modules are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the above embodiment 2. What's public.

Example 8

According to an embodiment of the present application, an embodiment of an eye tracking system applied to a terminal device is provided, wherein FIG. 6 is a schematic structural diagram of an eye tracking system applied to a terminal device according to an embodiment of the present application, as shown in FIG. 6, The system includes: a device end 601 and a server 603. Wherein, the device end 601 is configured to collect image information of the target object, complete the sorting of the eye feature information of the target object, form a series of eye feature information, and then send the series of eye feature information to the server to receive and analyze The gaze information fed back by the server; the server 603 is configured to receive the eye feature information sent by the device, and process the eye feature information to obtain the gaze information of the target object.

As can be seen from the above, the device side and the server are used to collect the image information of the target object on the device side, and extract the eye feature information of the target object from the image information, and then send the eye feature information to the server and the server Process according to the eye feature information to obtain the fixation information of the target object, and send the fixation information of the target object to the device side, and then the fixation information of the target object can be displayed on the device side.

It is easy to notice that the collection of image information of the target object and the extraction of eye feature information are run on the device side, while the processing of eye feature information is implemented on the server. Because the gaze point algorithm that processes gaze information based on eye feature information consumes a lot of resources, processing it on the server effectively reduces resource consumption on the device side. In addition, in this application, the device side extracts the eye feature information from the image information and sends the eye feature information to the server instead of directly sending the image information to the server, which can reduce the direct upload of the image information to the server. The occupied bandwidth further reduces resource consumption on the device side.

It should be noted that the device side 601 may perform the eye tracking method applied to the terminal device provided in Embodiments 1, 2, and 4; the server 603 may perform the eye tracking method applied to the terminal device provided in Embodiments 3 and 5. method. The specific content has been described in detail in Embodiments 1-5, and details are not described herein again.

Example 9

According to another aspect of the embodiments of the present application, a storage medium is also provided. The storage medium includes a stored program, where the program executes the eye tracking method applied to the terminal device provided in Embodiments 1 to 5.

Example 10

According to another aspect of the embodiments of the present application, a processor is further provided for running a program, wherein the program executes the eye tracking method applied to the terminal device provided in the embodiments 1 to 5 when the program is run.

The above-mentioned serial numbers of the embodiments of the present application are merely for description, and do not represent the superiority or inferiority of the embodiments.

In the above embodiments of the present application, the description of each embodiment has its own emphasis. For a part that is not described in detail in an embodiment, reference may be made to the related description of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the unit may be a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or may be combined. Integration into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, units or modules, and may be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional unit.

When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially a part that contributes to the existing technology or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium. , Including a number of instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application. The foregoing storage media include: U disks, Read-Only Memory (ROM), Random Access Memory (RAM), mobile hard disks, magnetic disks, or optical disks, and other media that can store program codes .

The above is only an optional implementation of the present application. It should be noted that, for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and retouches can be made. These improvements and Retouching should also be considered as the protection scope of this application.

Industrial applicability

The solution provided by the embodiment of the present application can be applied to the eye tracking technology. The method of processing the image by combining the device side with the server solves the problem that the eye tracking algorithms in the prior art are all run on the device side and cause the device side resources Expensive technical problems reduce the bandwidth occupied by image information transmission and reduce resource consumption on the device side.

Claims

An eye tracking method applied to a terminal device includes:

Collect image information;

Sending the image information to a cloud processor;

Receiving and analyzing gaze information fed back by the cloud processor.
The eye tracking method according to claim 1, wherein sending the image information to a cloud processor comprises:

Get sending information;

Determining an image identifier corresponding to the transmission information, where the image identifier is used to identify a transmission sequence of the transmission information;

Sending the sending information to the cloud processor in a first regular manner.
The eye tracking method according to claim 2, wherein determining an image identifier corresponding to the transmission information comprises:

Determining a timestamp corresponding to the first frame image of the sending information;

Generating an image identifier corresponding to the first frame image according to the timestamp.
The eye tracking method according to claim 3, wherein the transmission information comprises: a first frame header, a first control word, a first check code, and the image information.
The method according to claim 1, wherein receiving and analyzing the gaze information fed back by the cloud processor comprises:

Receiving reception information sent in a second rule manner;

Parse the received information, the received information includes a second frame header, a second control word, a second check code, and the gaze information;

Completing verification of the second check code;

If the verification result of the second verification code satisfies a preset condition, the fixation information is determined as fixation information of the target object.
An eye tracking method applied to a terminal device includes:

Obtain image information sent by the device;

Extracting eye feature information from the image information to form an eye feature information set, wherein the image information includes the eye feature information;

Determining gaze information according to the eye feature information set;

Sending the gaze information to the device.
An eye tracking method applied to a terminal device includes:

Collect image information;

Extracting eye feature information from the image information to form an eye feature information set, wherein the image information includes the eye feature information;

Sending the eye feature information set to the server;

Receive and parse gaze information fed back by the server.
The eye tracking method according to claim 7, before the forming of the eye feature information set, further comprising:

Converting the image information into corresponding eye feature information;

Configuring the corresponding eye feature information with a data label;

Sort the corresponding eye feature information of the configured data tags in a specified order.
The eye tracking method according to claim 8, wherein the corresponding eye feature information includes pupil information, corneal information, spot information, iris information and / or eyelid information, eye image information, and gaze depth information of the target One or more of them.
The eyeball tracking method according to claim 7, wherein the eye feature information set is configured as one or more groups of eye feature information with a data label.
The eye tracking method according to claim 7, wherein sending the eye feature information set to a server comprises:

Get sending information;

Determining an image identifier corresponding to the sending information, and acquiring the eye feature information set corresponding to the image identifier, where the image identifier is used to identify a sending order of the sending information;

And sending the sending information to the server in a first regular manner.
The eye tracking method according to claim 11, wherein determining an image identifier corresponding to the transmission information comprises:

Determining a timestamp corresponding to the first frame image of the sending information;

Generating an image identifier corresponding to the first frame image according to the timestamp.
The eyeball tracking method according to claim 12, wherein the transmission information comprises: a first frame header, a first control word, a first check code, and the eye feature information set.
The method according to claim 7, wherein receiving and analyzing the gaze information fed back by the server specifically comprises:

Receiving reception information sent in a second rule manner;

Parse the received information, the received information includes a second frame header, a second control word, a second check code, and the gaze information;

Completing verification of the second check code;

If the verification result of the second verification code satisfies a preset condition, the fixation information is determined as fixation information of the target object.
An eye tracking device applied to terminal equipment includes:

A first acquisition module, configured to acquire image information;

A first sending module configured to send the image information to a cloud processor;

The first receiving module is configured to receive and analyze gaze information fed back by the cloud processor.
An eye tracking device applied to terminal equipment includes:

A second acquisition module, configured to acquire image information;

A second extraction module configured to extract eye feature information from the image information to form an eye feature information set, wherein the image information includes the eye feature information;

A second sending module, configured to send the eye feature information set to a server;

The second receiving module is configured to receive and analyze gaze information fed back by the server.
An eye tracking system applied to a terminal device includes:

The device end is configured to collect image information and extract eye feature information from the image information to form a set of eye feature information, and then send the eye feature information to a server to receive and analyze gaze feedback from the server information;

The server is configured to receive eye feature information sent by the device, and process the eye feature information to obtain gaze information of a target object.
A storage medium includes a stored program, wherein the program executes the eye tracking method applied to a terminal device according to any one of claims 1 to 14.
A processor configured to run a program, wherein when the program runs, the eye tracking method applied to a terminal device according to any one of claims 1 to 14 is executed.