CN116434317A

CN116434317A - Eye image processing method and device, eye movement tracking system and electronic device

Info

Publication number: CN116434317A
Application number: CN202111635001.XA
Authority: CN
Inventors: 谢森栋; 田文军; 徐诚
Original assignee: Sunny Optical Zhejiang Research Institute Co Ltd
Current assignee: Sunny Optical Zhejiang Research Institute Co Ltd
Priority date: 2021-12-29
Filing date: 2021-12-29
Publication date: 2023-07-14

Abstract

The application relates to an eye image processing method, an eye image processing device, an eye movement tracking system and an electronic device, wherein the eye image processing method comprises the following steps: acquiring eye image information; performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a pupil rough positioning image; performing edge detection processing on the first preprocessing image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the pupil rough positioning image; and generating an eye movement characteristic extraction result aiming at the eye image information at least according to the target pupil positioning result. Through the method and the device, the problem of low accuracy and efficiency of eye image processing is solved, and efficient and accurate eye image processing is achieved.

Description

Eye image processing method and device, eye movement tracking system and electronic device

Technical Field

The present disclosure relates to the field of image processing technologies, and in particular, to an eye image processing method and apparatus, an eye movement tracking system, and an electronic device.

Background

Eye tracking is also called eye movement tracking, and is a technology for estimating the direction of a line of sight and coordinates of a falling point of the line of sight by a certain method by utilizing camera equipment to observe the movement condition of eyeballs of a person in real time. The core data of the eye tracking algorithm is the position information of the pupil and the light spot, for example, the quantization information of the eye gaze movement is estimated by calculating the relative offset of the pupil.

However, the difficulty of pupil centering is increased due to the inherent physiological mechanisms of the human eye and the non-linearity, randomness and complexity of eye movements. In the related art, pupil areas are usually positioned rapidly based on methods such as template matching, but positioning accuracy is low, and template parameters cannot adapt to differences of individual pupil sizes, so that pupil positioning accuracy is affected. Or the pupil is positioned by detection through a target detection method of deep learning, but accurate pupil edge information is obtained, and the refinement treatment is needed on the basis of a target detection result; if the pupil positioning task is regarded as target segmentation, a large amount of sample labeling is needed, the labeling difficulty is high, the efficiency is low, and therefore the accuracy and the efficiency of eye image processing are low.

At present, no effective solution is proposed for the problem of low accuracy and efficiency of eye image processing in the related art.

Disclosure of Invention

The embodiment of the application provides an eye image processing method, an eye image processing device, an eye tracking system and an electronic device, which are used for at least solving the problems of low accuracy and efficiency of eye image processing in the related technology.

In a first aspect, an embodiment of the present application provides an eye image processing method, including:

Acquiring eye image information;

performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a pupil rough positioning image;

performing edge detection processing on the first preprocessing image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the pupil rough positioning image;

and generating an eye movement characteristic extraction result aiming at the eye image information at least according to the target pupil positioning result.

In some embodiments, the performing inverse binarization processing on the eye image information to obtain a first preprocessed image includes:

calculating an integral graph according to the eye image information, calculating Haar features according to the integral graph, acquiring a Haar response maximum value and a Haar response radius according to the Haar features, and acquiring a first region-of-interest image according to the Haar response maximum value and the Haar response radius;

and performing inverse binarization processing on the first region-of-interest image to obtain the first preprocessed image.

In some embodiments, the performing contour extraction processing on the first preprocessed image to obtain a pupil rough positioning image includes:

Performing contour extraction processing on the first preprocessing image to obtain a first optimal contour, and determining an eyelid positioning sub-image of the eye image information based on the first optimal contour; calculating to obtain a pixel mean image according to the eyelid positioning sub-image, and calculating gradient information of the pixel mean image;

clustering the pixel mean images by using a clustering algorithm to obtain a clustering result, and acquiring a skin color brightness threshold according to the clustering result;

respectively comparing the pixel value obtained by traversing the pixel mean image with the skin color brightness threshold value, and determining a target eyelid positioning result according to the comparison result and the gradient information;

obtaining the pupil coarse positioning image based on the eyelid positioning sub-image under the condition that the eyelid distance in the target eyelid positioning result is larger than a preset eyelid distance threshold value; and obtaining a blink judgment result under the condition that the eyelid interval is smaller than or equal to the preset eyelid interval threshold value.

In some of these embodiments, the obtaining the pupil coarse positioning image based on the eyelid positioning sub-image comprises:

performing inverse binarization processing on the eyelid positioning sub-image to obtain a second preprocessed image, and performing contour extraction processing on the second preprocessed image to obtain a second optimal contour;

Acquiring pupil center position and pupil radius information based on the first optimal contour, and screening the second optimal contour according to the pupil center position and the pupil radius information to obtain a first contour point set;

under the condition that the number of the first contour point sets is larger than the number of preset contour points, performing iterative fitting on the first contour point sets to obtain a fitted circle result; obtaining a pupil contour point result according to the first contour point set and the fitted circle result, and obtaining the pupil rough positioning image according to the pupil contour point result;

and obtaining the blink judgment result under the condition that the number of the contour point sets is smaller than or equal to the preset contour point number.

In some embodiments, the obtaining the pupil contour point result according to the first contour point set and the fitted circle result includes:

performing proportional amplification and proportional reduction on the fitted circle result respectively to obtain a circular template result;

and obtaining the pupil contour point result according to the first contour point set and the circular template result.

In some embodiments, the obtaining the target pupil positioning result from the edge image and the pupil coarse positioning image includes:

Performing equal-proportion amplification and AND operation on the edge image and the pupil rough positioning image to obtain a first overlapping point set, and performing fitting treatment on the first overlapping point set to obtain a first pupil fitting result;

performing AND operation on the edge image and the first pupil fitting result to obtain a second overlapping point set, performing fitting treatment on the second overlapping point set to obtain a second pupil fitting result, and performing highlight filtering treatment on the second pupil fitting result to obtain a second contour point set;

and sequentially calculating gradient direction results of the traversed second contour point set, screening the second contour point set based on the gradient direction results to obtain a third contour point set, and generating the target pupil positioning result according to the third contour point set.

In some embodiments, the generating an eye movement feature extraction result for the eye image information based at least on the target pupil positioning result comprises:

the method comprises the steps of obtaining topological vector information of a light supplementing light source, wherein the light supplementing light source is used for supplementing light in the process of collecting eye image information;

extracting a second region of interest image according to the pupil rough positioning image, performing contour extraction processing on the second region of interest image to obtain a contour sequence, and obtaining a bright spot contour sequence based on the contour sequence;

And obtaining a target light spot positioning result according to the topological vector information and the bright spot contour sequence, and generating the eye movement characteristic extraction result according to the target pupil positioning result and the target light spot positioning result.

In some embodiments, the obtaining the target spot positioning result according to the topology vector information and the bright spot contour sequence includes:

sequentially taking the traversed bright spot contour points in the bright spot contour sequence as reference points, and calculating to obtain vector information between other bright spot contour points in the bright spot contour sequence and the reference points;

obtaining a template vector according to the topology vector information and the vector information, combining according to the topology vector information and the bright spot contour sequence to obtain a light source light spot sequence, and matching according to the light source light spot sequence and the template vector to obtain a cornea reflection light spot sequence;

traversing the cornea reflection light spot sequence to sequentially obtain third interested area images corresponding to each cornea reflection light spot in the cornea reflection light spot sequence, and carrying out contour extraction processing on each third interested area image to obtain a target light spot;

and obtaining the target light spot positioning result based on the target light spot.

In some embodiments, the performing edge detection processing on the eye image information to obtain an edge image includes:

and performing image segmentation processing on the eye image information by using a maximum inter-class variance method (OTSU) algorithm to obtain a segmentation threshold value, and performing edge detection on the eye image according to the segmentation threshold value to obtain the edge image.

In a second aspect, embodiments of the present application provide an eye image processing apparatus, the apparatus including: the device comprises an acquisition module, a coarse positioning module, a target module and a generation module;

the acquisition module is used for acquiring eye image information;

the coarse positioning module is used for performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a pupil coarse positioning image;

the target module is used for carrying out edge detection processing on the first preprocessing image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the pupil rough positioning image;

the generating module is used for generating an eye movement characteristic extraction result aiming at the eye image information at least according to the target pupil positioning result.

In a third aspect, embodiments of the present application provide an eye tracking system, including an image capturing device and a control apparatus;

the image acquisition equipment is used for acquiring eye image information;

the control device is configured to execute the eye image processing method according to the first aspect, with respect to the eye image information, to obtain the eye movement feature extraction result, and generate an eye movement tracking result based on the eye movement feature extraction result.

In a fourth aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the method for processing an eye image according to the first aspect when executing the computer program.

In a fifth aspect, embodiments of the present application provide a storage medium having stored thereon a computer program which, when executed by a processor, implements the eye image processing method as described in the first aspect above.

Compared with the related art, the eye image processing method, the eye image processing device, the eye movement tracking system and the electronic device provided by the embodiment of the application acquire eye image information; performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a pupil rough positioning image; performing edge detection processing on the first preprocessing image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the pupil rough positioning image; according to at least the target pupil positioning result, an eye movement characteristic extraction result aiming at the eye image information is generated, the problems of low accuracy and efficiency of eye image processing are solved, and efficient and accurate eye image processing is realized.

The details of one or more embodiments of the application are set forth in the accompanying drawings and the description below to provide a more thorough understanding of the other features, objects, and advantages of the application.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:

FIG. 1 is an application environment diagram of an eye image processing method according to an embodiment of the present application;

FIG. 2 is a flow chart of an eye image processing method according to an embodiment of the present application;

FIG. 3 is a flow chart of another method of eye image processing according to an embodiment of the present application;

FIG. 4A is a schematic illustration of an eyelid positioning sub-image according to an embodiment of the present application;

FIG. 4B is a schematic diagram of a target eyelid positioning result according to an embodiment of the present application;

FIG. 5 is a flow chart of an eyelid positioning algorithm according to an embodiment of the present application;

FIG. 6A is a schematic illustration of a third set of contour points according to an embodiment of the present application;

fig. 6B is a schematic diagram of a target pupil positioning result according to an embodiment of the present application;

Fig. 7 is a flowchart of a pupil positioning algorithm according to an embodiment of the present application;

FIG. 8 is a flowchart of yet another method of eye image processing according to an embodiment of the present application;

FIG. 9 is a schematic diagram of a target spot positioning result according to an embodiment of the present application;

FIG. 10 is a flowchart of a corneal reflection spot positioning algorithm according to an embodiment of the present application;

FIG. 11 is a flowchart of a method of eye image processing in accordance with a preferred embodiment of the present application;

fig. 12 is a block diagram of an eye image processing apparatus according to an embodiment of the present application;

fig. 13 is a block diagram of the interior of a computer device according to an embodiment of the present application.

Detailed Description

In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described and illustrated below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden on the person of ordinary skill in the art based on the embodiments provided herein, are intended to be within the scope of the present application. Moreover, it should be appreciated that while such a development effort might be complex and lengthy, it would nevertheless be a routine undertaking of design, fabrication, or manufacture for those of ordinary skill having the benefit of this disclosure, and thus should not be construed as having the benefit of this disclosure.

Reference in the specification to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. It is to be expressly and implicitly understood by those of ordinary skill in the art that the embodiments described herein can be combined with other embodiments without conflict.

Unless defined otherwise, technical or scientific terms used herein should be given the ordinary meaning as understood by one of ordinary skill in the art to which this application belongs. Reference to "a," "an," "the," and similar terms herein do not denote a limitation of quantity, but rather denote the singular or plural. The terms "comprising," "including," "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, article, or apparatus that comprises a list of steps or modules (elements) is not limited to only those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "connected," "coupled," and the like in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The term "plurality" as used herein means greater than or equal to two. "and/or" describes an association relationship of an association object, meaning that there may be three relationships, e.g., "a and/or B" may mean: a exists alone, A and B exist together, and B exists alone. The terms "first," "second," "third," and the like, as used herein, are merely distinguishing between similar objects and not representing a particular ordering of objects.

The eye image processing method provided by the application can be applied to an application environment shown in fig. 1. Wherein the image acquisition device 12 communicates with the server device 14 via a network. The server device 14 obtains eye image information through the image acquisition device 12, performs inverse binarization processing on the eye image information to obtain a first preprocessed image, performs contour extraction processing on the first preprocessed image to obtain a coarse positioning image of a pupil, performs edge detection processing on the first preprocessed image to obtain an edge image, and obtains a target pupil positioning result according to the edge image and the coarse positioning image of the pupil so as to generate an eye movement feature extraction result for the eye image information at least according to the target pupil positioning result. The image capturing device 12 may be, but not limited to, a binocular camera, a dome camera or other devices for capturing images, and the server device 14 may be implemented as a stand-alone server or a server cluster of multiple servers.

The present embodiment provides an eye image processing method, and fig. 2 is a flowchart of an eye image processing method according to an embodiment of the present application, as shown in fig. 2, where the flowchart includes the following steps:

Step S220, acquiring eye image information.

Wherein, the eye image information refers to near-eye images to be processed such as pupil positioning; preferably, the eye image information may employ an infrared near-eye image. The eye image information may be obtained by extracting continuous frame images from the video stream acquired by the image acquisition device by the control device such as the server device or the processing chip.

Step S240, performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a pupil rough positioning image.

Specifically, in the above step S240, the extracted pupil area image may be subjected to binarization processing to obtain a preprocessed pupil area image, so as to extract the pupil contour, and prepare for the subsequent image processing steps such as pupil center positioning. In addition, in order to improve accuracy and efficiency of eye image processing, the control device may also perform scaling, subimage matting, filtering, and other processing on the eye image information, so as to finally obtain the first preprocessed image. And then, the control device extracts the contour of the connected region in the first preprocessing image to obtain a contour point set aiming at the pupil, and further obtains the pupil rough positioning image.

Step S260, performing edge detection processing on the first preprocessed image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the pupil rough positioning image.

Specifically, the control device may detect the pupil outline in the first preprocessed image by using an edge extraction algorithm; for example, the gray value of the pixel point on the first preprocessed image is set to 0 or 255, all pixels with gray values not smaller than the selected threshold value are regarded as target pupil areas, the gray value of the pixels is represented by 255, otherwise, the gray value of the pixels is 0, and the pixels represent the background or other object areas, so that the edge image is obtained by matting. And then carrying out AND operation on the edge image and the pupil rough positioning image by a control device to obtain a superposed point set, carrying out ellipse fitting and other processing based on the superposed point set, and finally obtaining the target pupil positioning result.

In some embodiments, the edge detection processing of the eye image information to obtain an edge image further includes the following steps: and performing image segmentation processing on the eye image information by using an OTSU algorithm to obtain a segmentation threshold value, and performing edge detection on the eye image according to the segmentation threshold value to obtain the edge image.

Step S280, generating an eye movement feature extraction result aiming at the eye image information at least according to the target pupil positioning result.

Wherein the eye movement characteristic extraction result is a data analysis result reflecting eye movement behaviors; and determining an eye movement characteristic extraction result comprising data such as a movement track, an orientation and the like of the eye movement behavior based on the target pupil positioning result, so that man-machine interaction based on eye movement control can be realized.

Through the steps S220 to S280, the first preprocessed image is obtained by performing inverse binarization on the eye image information, the pupil rough positioning image is obtained based on the first preprocessed image, and the target pupil positioning result is obtained according to the pupil rough positioning image and the edge image obtained by edge detection, so that the eye image processing method for performing binarization processing on the image to obtain the pupil area and positioning the pupil by combining the edge information is realized, the algorithm is simple and the calculation efficiency is high, and meanwhile, the pupil detection accuracy is improved by performing pupil positioning twice, so that the problems of the eye image processing accuracy and low efficiency are solved, and the efficient and accurate eye image processing method is realized.

In some embodiments, the performing inverse binarization on the eye image information to obtain the first preprocessed image further includes the following steps:

And S241, calculating an integral graph according to the eye image information, calculating Haar features according to the integral graph, acquiring a Haar response maximum value and a Haar response radius according to the Haar features, and acquiring a first region-of-interest image according to the Haar response maximum value and the Haar response radius.

The control device calculates an integral graph of the eye image information, and searches a position with the maximum Haar response value through the integral graph; and if the Haar response value is maximum, the image information at the position is most similar to the preset Haar template. And then the control terminal takes the position with the maximum Haar response value as the center and takes the Haar response radius of 2 times as the side length, and the pupil initial positioning sub-image, namely the first region of interest image roiImage, is extracted from the eye image information.

In step S242, inverse binarization processing is performed on the first region of interest image to obtain the first preprocessed image.

Specifically, inverse binarization processing is performed on the first region of interest image, that is, the roiImage image, so that a low-brightness region of the roiImage is displayed as white, a high-brightness region is displayed as black, morphological filtering processing is performed, isolated noise points are eliminated, and finally the first preprocessed image is obtained.

Through the steps S241 to S242, the region of interest image is obtained through Haar response, and the obtained region of interest image is subjected to inverse binarization processing to obtain the first preprocessing image, so that preprocessing before the eye image processing is realized, and the accuracy of the eye image processing can be improved.

In some embodiments, there is provided an eye image processing method, and fig. 3 is a flowchart of another eye image processing method according to an embodiment of the present application, as shown in fig. 3, where the flowchart includes step S220, step S260, and step S280 shown in fig. 2, and further includes the following steps:

step S320, performing inverse binarization processing on the eye image information to obtain a first preprocessed image, performing contour extraction processing on the first preprocessed image to obtain a first optimal contour, and determining an eyelid positioning sub-image of the eye image information based on the first optimal contour; and calculating to obtain a pixel mean image according to the eyelid positioning sub-image, and calculating gradient information of the pixel mean image.

After the first preprocessed image is obtained in the step S320 or the steps S241 to S242, the control device may perform contour extraction processing on the connected region in the first preprocessed image, and screen to obtain the first optimal contour based on prior information that the pupil shape is approximately a circle and the pupil area ratio in the first preprocessed image is the largest; the control device can roughly calculate and obtain pupil center position and pupil radius information based on the first optimal contour. The control device obtains the x coordinate of the pupil center position by default according to the x coordinate of the pupil center position of the upper eyelid and the lower eyelid, thereby obtaining the eyelid positioning sub-image by utilizing minimum circumscribed rectangle matting in the eye image information so as to realize positioning in the vertical direction; specifically, the region marked by the black line frame in fig. 4A is the scratched eyelid positioning sub-image.

After the eyelid positioning sub-image is obtained, the control device may further perform filtering processing on the eyelid positioning sub-image to improve accuracy of image processing, and then calculate a gradient information gradient of the pixel mean image in the y direction according to a pixel mean value avgImage of each row in the eyelid positioning sub-image after statistical processing.

And S340, clustering the pixel mean image by using a clustering algorithm to obtain a clustering result, and acquiring a skin color brightness threshold according to the clustering result.

The clustering algorithm can be a K-means clustering algorithm, and the method comprises the steps of randomly selecting K pixel points as initial clustering centers, then calculating the distance between each pixel point and each initial clustering center, and distributing each pixel point to the initial clustering center closest to the pixel point; the initial cluster center and the pixel points distributed to the initial cluster center represent a cluster, each sample is distributed, and the cluster center of the cluster is recalculated according to the existing pixel points in the cluster; this process will repeat until no (or a minimum number of) pixels are reassigned to a different cluster, or no (or a minimum number of) cluster centers change again, ultimately generating a classification result for all pixels of the pixel mean image. The clustering algorithm may also be algorithms such as DBSCAN (Density-Based Spatial Clustering of Applications with Noise, density-based clustering method with noise), and the like, and will not be described herein.

The pixel mean image can be divided into different skin color areas, such as a periocular area, a pupil area, an iris area and the like, according to the clustering result of the pixel points by the clustering algorithm. The control means may determine pixel values of positions of the upper eyelid and the lower eyelid based on all of the divided skin tone regions, and use the pixel values as the skin tone brightness threshold value, so as to determine the eyelid positions in the eye image information in a subsequent step.

And step S360, comparing the pixel value obtained by traversing the pixel mean image with the skin color brightness threshold value respectively, and determining a target eyelid positioning result according to the comparison result and the gradient information.

In the pixel mean image, the control device may search from bottom to top for a row where the pixel value is smaller than the skin brightness threshold value for the first time, as the coordinate y in the preliminary vertical direction of the lower eyelid _init The method comprises the steps of carrying out a first treatment on the surface of the The control device uses the coordinate y on the pixel mean image _init In the region that is the center, a gradient maximum point is searched based on the gradient information obtained by the calculation, and the y coordinate of the point is taken as the final position of the lower eyelid. Then the control device can locate the upper eyelid, namely searching from top to bottom in the pixel mean image to find the row information that the first satisfied pixel value is smaller than the skin color brightness threshold value, then searching the row with the largest gradient downwards from the row, and taking the y value of the row as the final location position of the upper eyelid. For example, the detected upper eyelid position and lower eyelid position are marked with white lines in fig. 4B, respectively.

Step S380, obtaining the pupil rough positioning image based on the eyelid positioning sub-image under the condition that the eyelid distance in the target eyelid positioning result is larger than a preset eyelid distance threshold value; and obtaining a blink judgment result under the condition that the eyelid interval is smaller than or equal to the preset eyelid interval threshold value.

Wherein the eyelid distance between the upper eyelid position and the lower eyelid position may be calculated by the control means based on the target eyelid positioning result. If the eyelid spacing is greater than the preset eyelid spacing threshold, indicating that the user is not blinking, a subsequent pupil positioning step may be performed based on the eyelid positioning sub-images. If the eyelid spacing is smaller than or equal to the preset eyelid spacing threshold, the eyelid spacing is too small, namely, the user is in a blinking state, and the result is inaccurate if pupil detection is performed at the moment, so that a blinking judgment result can be directly obtained and the eye image processing flow is ended, and the accuracy and efficiency of eye image processing are improved.

Through the steps S320 to S380, an eyelid positioning procedure is added before pupil positioning, so that a method of judging whether the eye is in a blinking state or not by utilizing the eyelid distance and then realizing pupil positioning based on a judgment result is realized, the robustness of pupil detection is enhanced, and the accuracy and efficiency of eye image processing are effectively improved. In addition, at the subsequent cornea reflection spot positioning node, eyelid position information in this flow can also be used as a supplementary condition for filtering highlight information formed in the fundus region by spot secondary reflection.

It should be noted that, in this embodiment, an eyelid positioning algorithm is further provided, and fig. 5 is a flowchart of an eyelid positioning algorithm according to an embodiment of the present application, as shown in fig. 5, where the flowchart includes the following steps:

step S501, acquiring an infrared eye moving image as eye image information; in order to reduce the calculation amount, the eye image information is firstly subjected to 4 times downsampling processing, so that a reduced image smallImage is obtained.

Step S502, calculating an integral graph on the reduced image smallImage, searching a position with the maximum Haar response, taking the position with the maximum Haar response value as a center, taking the Haar response radius of 2 times as a side length, and extracting a pupil primary positioning sub-image, namely a first region-of-interest image roiImag, from the reduced image smallImage.

Step S503, performing inverse binarization processing on the roiImage image to obtain a binary image, displaying a low-brightness area of the image as white, displaying a high-brightness area as black, performing morphological filtering processing, and eliminating isolated noise points; and extracting the contour of the connected region in the processed binary image, and screening to obtain a first optimal contour based on prior information that the pupil shape is approximate to a circle and the pupil region in the roiImage has the largest proportion.

Step S504, based on the first optimal contour, roughly obtaining a pupil center position ptCenter and a pupil radius fRadius; eyelid positioning sub-images pupi_strip of the region shown as a black mark box in fig. 4A are scratched on the eye image information based on the pupil center position ptCenter.

In step S505, filtering the eyelid positioning sub-image pupil_strip, counting the pixel average value avgImage of each row, and calculating the gradient information gradient of the pixel average value image in the y direction.

Step S506, performing k-means clustering calculation on the pixel mean image avgImage to obtain a skin color brightness threshold value skinVal.

In step S507, in the pixel mean image avgImage, the line where the condition of the pixel value < skinVal is satisfied for the first time is searched from bottom to top as the coordinate in the preliminary lower eyelid vertical direction.

Step S508, searching a gradient maximum point based on gradient information gradient in a radius delta area with the vertical direction of the image as the center, and taking the y coordinate of the point as the final position of the lower eyelid.

Step S509, then, positioning the upper eyelid, and searching from top to bottom in the pixel mean image avgmage image: the row information meeting the pixel value < skinVal for the first time is searched, then the row with the largest gradient is searched downwards from the row, and the y value of the row is used as the final position of the upper eyelid.

Step S510, eyelid interval calculation is carried out, and whether the eyelid interval is too small or not is judged; if not, outputting a target eyelid positioning result; if yes, the eye blink is judged and the eye image processing flow is ended.

In some embodiments, the obtaining the pupil rough positioning image based on the eyelid positioning sub-image further includes the following steps:

step S381, performing inverse binarization processing on the eyelid positioning sub-image to obtain a second preprocessed image, and performing contour extraction processing on the second preprocessed image to obtain a second optimal contour.

Since the upper eyelid naturally forms a shadow on the eyeball, the control device may perform contrast enhancement processing on the eyelid positioning sub-image to enhance the edge information in order to increase the pupil detection rate. The control device performs inverse binarization operation on the eyelid positioning sub-image subjected to contrast processing, removes noise points through morphological filtering, and further obtains the second preprocessing image. The control device searches the outlines of all connected domains in the second preprocessing image, and because the pupil can be approximately considered to be positioned in the central area of the image in the second preprocessing image, traversing all the outlines by taking the pupil as a priori condition, and searching the outline with the largest area and the ratio of the x coordinate to the y coordinate of the central point of the area close to 1 as the second optimal outline pupil_contour; it can be understood that if the control device fails to find the second optimal contour meeting the condition, it is considered that no pupil exists in the current second preprocessed image, and the eye image processing flow can be directly ended, so as to improve the eye image processing efficiency.

Step S382, pupil center position and pupil radius information are obtained based on the first optimal contour, and screening processing is performed on the second optimal contour according to the pupil center position and the pupil radius information to obtain a first contour point set.

Specifically, based on the pupil center position ptCenter and the pupil radius information fpadius obtained in the eyelid positioning stage, the contour points too close to the upper eyelid area and the lower eyelid area in the second optimal contour are filtered to obtain a first screened contour point set clean_pupin_contour, so that interference of factors such as eyelashes is eliminated, and the accuracy of pupil positioning is improved.

Step S383, performing iterative fitting on the first contour point set to obtain a fitted circle result under the condition that the number of the first contour point set is larger than the preset contour point number; and obtaining a pupil contour point result according to the first contour point set and the fitted circle result, and obtaining the pupil rough positioning image according to the pupil contour point result.

The preset profile points may be preset by a user, for example, may be set to 5. When the control device detects that the number of the first contour point sets is larger than the preset contour point number, the detection is effective at the moment, a fitting circle result with the minimum loss can be obtained based on least square iterative fitting, the first contour point sets are matched with the fitting circle result, the matched contour points are used as pupil contour point results, and finally the pupil rough positioning image is obtained based on the pupil contour point results.

In step S384, the blink determination result is obtained when the number of the contour point sets is less than or equal to the preset contour point number.

Through the steps S381 to S384, the method of screening by using eyelid information to obtain the first contour point set and judging whether to blink based on the first contour point set is realized in the pupil rough positioning stage, so that the error of the detection result caused by pupil detection during blinking is avoided, the robustness of pupil detection is enhanced, and the accuracy and efficiency of eye image processing are further improved.

In some embodiments, the pupil contour point result obtained according to the first contour point set and the fitted circle result further includes the following steps: performing proportional amplification and proportional reduction on the fitted circle result respectively to obtain a circular template result; and obtaining the pupil contour point result according to the first contour point set and the circular ring template result.

Specifically, the circle radius of the fitting circle result is respectively enlarged and reduced according to a preset proportion; the preset ratio may be preset by a user, for example, set to 1.01 and 0.99, that is, the control device amplifies the fitted circle to 1.01 times of the radius of the circle according to the ratio 1 of 1.01, and reduces the fitted circle to 0.99 times of the radius of the circle according to the ratio 0.99, so as to form a circle based on the amplified and reduced result, and the result of the circular template is obtained. And then the control device screens out the contour points concentrated in the circular ring template from the first contour point set to serve as pupil contour points, so that a filtering mode of automatically eliminating noise points while avoiding contour point missed selection caused by that the first contour point set cannot be completely matched with a fitting circle is realized.

Through the embodiment, the ring template result is obtained by carrying out proportional amplification and proportional reduction on the fitting circle, and the first contour point set is further screened based on the ring template result to finally obtain the pupil contour point result, so that the isolated point (noise point) filtering method in the pupil rough positioning process is realized, and the pupil ellipse fitting accuracy is improved.

In some embodiments, the obtaining the target pupil positioning result according to the edge image and the pupil rough positioning image further includes the following steps:

step S261, performing scaling up and performing a and operation on the edge image and the pupil rough positioning image to obtain a first overlapping point set, and performing a fitting process on the first overlapping point set to obtain a first pupil fitting result.

It should be noted that, in order to improve accuracy of image processing, the control device may perform equal-proportion dilation processing on the edge image and the pupil rough positioning image, perform morphological dilation processing on an elliptical region in the pupil rough positioning image, and then perform and operation on the edge image and the pupil rough positioning image obtained after processing by the control device to obtain a first overlapping point set, and perform ellipse fitting based on the first overlapping point set to obtain the first pupil fitting result.

Step S262, performing AND operation on the edge image and the first pupil fitting result to obtain a second overlapping point set, performing fitting treatment on the second overlapping point set to obtain a second pupil fitting result, and performing highlight filtering treatment on the second pupil fitting result to obtain a second contour point set.

And the control device performs AND operation again on the edge image and the first pupil fitting result to obtain a second overlapping point set so as to realize secondary refinement, and performs ellipse fitting to obtain a secondary refined ellipse. The control device obtains a pixel point set on the first preprocessed image based on the secondary refined ellipse, then filters a high gray value, namely a highlight point, based on brightness information of the first preprocessed image, and then takes the filtered and filtered residual contour points as the second contour point set.

Step S263, sequentially calculating gradient direction results of the traversed second contour point set, screening the second contour point set based on the gradient direction results to obtain a third contour point set, and generating the target pupil positioning result according to the third contour point set.

The control device traverses the second contour point set, searches the position of the gradient maximum value in the local area in the direction of the two points connecting with the current contour point at the elliptical center of the second contour point set, and finely adjusts and updates the current contour point based on the searched position so as to ensure the accuracy of pupil positioning. Meanwhile, the control device sequentially calculates the gradient direction of each traversed profile point, the connecting line direction between the center of the ellipse and the current profile point is base_angle, the coincidence degree between a linear equation expressed by the base_angle and the gradient direction is compared, and if the linear equation is approximately coincident with the gradient direction, the corresponding current profile point is saved; if a large difference exists between the linear equation and the gradient direction, deleting the current contour point, further obtaining a third contour point set after the traversal is finished, and finally generating the target pupil positioning result according to fitting of the third contour point set. Specifically, the white point set in fig. 6A is based on the third contour point set, and a pupil region of an approximate ellipse is constituted by a plurality of white point sets; the white line marked in fig. 6B is the final fitting result of the target pupil positioning of the approximate ellipse.

Through the steps S261 to S263, the second contour point set is obtained through twice and operations of the edge image and the pupil rough positioning image, so that the accuracy of fitting the pupil ellipse through the contour point set is effectively improved, meanwhile, the gradient direction result of each point is obtained through traversing the second contour point set, and the second contour point set is screened based on the gradient direction result, so that the accuracy of pupil positioning is further improved, and the accuracy of eye image processing is further improved.

It should be noted that, in this embodiment, a pupil positioning algorithm is further provided, and fig. 7 is a flowchart of a pupil positioning algorithm according to an embodiment of the present application, as shown in fig. 7, where the flowchart includes the following steps:

step S701, acquiring an infrared eye moving image as eye image information; in the preprocessing process, the low-brightness region information is obtained based on steps S501 to S504 in the eyelid positioning algorithm, so as to obtain a pupil positioning region-of-interest image pupilImage, namely the eyelid positioning sub-image.

In step S702, since the upper eyelid naturally forms a shadow on the eyeball, in order to increase the pupil detection rate, a contrast-increasing process is performed on the pupilImage image to enhance the edge information.

In step S703, the constraint conditions of the upper and lower boundary of the contour point are set based on the initial positioning result, and the preprocessing process is ended.

Step S704, starting coarse positioning of pupils; and performing inverse binarization and morphological filtering on the pupilImage image to remove noise so as to obtain a second preprocessed image.

Step S705, finding the outline of all connected domains in the second preprocessing image to obtain outline conductors. Because the pupilImage image is obtained based on Haar response, the pupil can be approximately considered to be positioned in the central area of the image, all contours are traversed by taking the pupil as a priori condition, and the contour with the largest area and the ratio of the x coordinate to the y coordinate of the central point of the area close to 1 is searched as a second optimal contour pupil_contour.

Step S706, if a second optimal contour meeting the condition is found, contour points which are too close to the upper eyelid area and the lower eyelid area in the second optimal contour pupil_contour are filtered based on the pupil rough positioning center and the radius information obtained in the eyelid positioning stage, namely contour points are filtered based on upper and lower boundary constraints, so that interference of eyelashes and the like is eliminated, and a screened first contour point set clean_pupil_contour is obtained; otherwise, the current image is considered to have no pupil, and the pupil positioning algorithm is ended.

Step S707, judging whether the number of clean_public_conf points is greater than 5; if not, the rough positioning identification is described, the pupil positioning algorithm is ended, and a blink judgment result of blink judgment is output; if yes, based on a least square method, iteratively fitting a circle with the minimum fit loss.

In step S708, the circle radius is enlarged and reduced to obtain a circular template mask, and the points with pupil_contour points falling in the mask circle are selected as pupil contour points filteredPoints.

Step S709, fitting based on the pupil contour points filteredPoints to obtain an ellipse to locate the pupil, thereby obtaining a pupil coarse location image pupil_elipse and ending the pupil coarse location procedure.

Step S710, starting pupil accurate positioning; and obtaining a segmentation threshold th for the pubilimage image by using an OTSU algorithm, and carrying out Canny edge detection based on the segmentation threshold th to obtain an edge image edgeImage.

Step S711, performing scaling up on the edge image and the pupil coarse positioning image pupil_elipse to obtain a scaled-up edge image and a scaled-up pupil coarse positioning image respectively; performing morphological expansion processing on the ellipse described by the scale_pupil_elipse_image, performing AND operation on the graph and the scale_edge image to obtain a first overlapped point set boundaryPoints, and performing ellipse fitting based on the first overlapped point set to obtain a first pupil fitting result pupil_elipse 1.

Step S712, performing and operation on the edge image edgeImage of the original image size and the first pupil fitting result pupil_ellipse1, obtaining a contour edge point set again, namely a second overlapping point set boundaryPoints, and performing ellipse fitting to obtain a secondary refined ellipse pupil_ellipse2.

In step S713, a set of pixel points boundaryPoints on the pupilImage image is obtained based on the secondarily refined ellipse pupil_elipse 2, and then the highlighted points are filtered based on the highlight information, and then the filtered remaining points are taken as ellipse contour points, i.e., a second contour point set filtered_boundarypoints.

Step S714, traversing the contour point, searching the position of the gradient maximum value in the local area in the connecting line direction of the ellipse center and the current contour point pi, and updating the contour point pi based on the position fine adjustment of the gradient maximum value.

Step S715, sequentially calculating the gradient direction angle of each point, enabling the connecting line direction of the ellipse center and the current contour point pi to be base_angle, and if the straight line equations represented by the angle and the base_angle are approximately coincident, storing the current contour point; if the two linear equations have larger difference, deleting the current contour point; ending the pupil accurate positioning process; and obtaining and outputting contour point information and corresponding ellipse information attached to the pupil edge.

In some embodiments, there is provided an eye image processing method, and fig. 8 is a flowchart of still another eye image processing method according to an embodiment of the present application, as shown in fig. 8, and the flowchart includes steps S220 to S260 shown in fig. 2, and further includes the following steps:

Step S820, obtaining topology vector information of a light-compensating light source, where the light-compensating light source is used for compensating light in the process of collecting the eye image information.

It can be appreciated that in the eye tracking scheme based on pupil-cornea reflection method, an infrared LED lamp is generally used as a light supplementing light source, and a reference light spot is generated on the surface of the cornea of the eyeball which approximates to a sphere, and the change of the eye gaze movement of the human eye is represented as the change of the pupil position in an infrared image. Before cornea reflection light spots are detected, a user can input pre-deployed LED hardware structure design information to the control device; the control device constructs and generates a topological vector set of each light supplementing light source piece based on the design information of the LED hardware structure, namely the topological vector information.

Step S840, extracting a second region of interest image according to the pupil rough positioning image, performing contour extraction processing on the second region of interest image to obtain a contour sequence, and obtaining a bright spot contour sequence based on the contour sequence.

Specifically, the control device sequentially extracts a second region-of-interest image of each light spot from the pupil rough positioning image; it can be understood that the control device can also perform size amplification and other processing on the second region of interest image, so as to improve the accuracy of light spot positioning. The control device also carries out binarization processing on the second region of interest image to obtain a highlight information connected domain, and carries out contour extraction on the connected domain to obtain a contour sequence; the control device can also process the outline with the oversized area by using a morphological method, and divide the outline into a plurality of small-area areas. Then, the control device sequentially calculates the minimum circumscribed rectangle of each outline, and filters the outline which deviates from the center of the image obviously by combining the eyelid information; updating the contour set, and calculating to obtain a bright spot contour center list, namely the bright spot contour sequence.

Step S860, a target light spot positioning result is obtained according to the topology vector information and the bright spot contour sequence, and the eye movement feature extraction result is generated according to the target pupil positioning result and the target light spot positioning result.

The control device may calculate, based on the topology vector information, a sequence with the highest matching degree with the topology vector in the bright spot profile sequence as a cornea reflection light spot sequence, and obtain the target light spot positioning result based on the cornea reflection light spot sequence, and finally generate the eye movement feature extraction result.

The cornea reflection facula is used as reference information of pupil movement, and the positioning accuracy of the cornea reflection facula directly influences the accuracy of gaze point calculation. In the related art, the center of the light spot is usually calculated by a centroid method, a fitting method, a hough transformation method, a radial symmetry method and other common algorithms. The latter two kinds of calculation amount is large, and are not suitable for being used on low-power consumption equipment. The centroid method is simple and easy to realize, has high speed, but has poor anti-interference capability and low precision. Fitting relies on the extraction of edge contour points, which fails if the light spot is too small, resulting in contour points less than the number required for an ellipse/circle general equation fit. In addition, the LED infrared light source designed by the hardware structure of the eye tracking system is provided with mechanism information, the method is not combined with the structure design information, the hardware Id serial numbers corresponding to the light spots are matched, and the mapping relation from the image coordinates to the corresponding 3D coordinates in the world coordinate system is lacked, so that the calculation of the follow-up eye tracking result is influenced.

In the embodiment of the application, through the steps S820 to S860, the bright spot contour sequence is obtained based on the pupil rough positioning image by obtaining the topology vector information of the light supplementing light source, and then the target light spot positioning result is obtained according to the topology vector information and the bright spot contour sequence, so that the light spot positioning method combining with the hardware LED lamp distribution design is provided, the mapping from the image coordinates to the world coordinate system is realized, the calculation is simple, and the accuracy and the efficiency of the eye image processing are further improved.

In some embodiments, the obtaining the target spot positioning result according to the topology vector information and the bright spot contour sequence further includes the following steps:

step S861, sequentially taking the traversed bright spot contour points in the bright spot contour sequence as reference points, and calculating to obtain vector information between other bright spot contour points in the bright spot contour sequence and the reference points.

Step S862, a template vector is obtained according to the topology vector information and the vector information, a light source light spot sequence is obtained according to the combination of the topology vector information and the bright spot contour sequence, and a cornea reflection light spot sequence is obtained according to the light source light spot sequence and the template vector matching.

Specifically, after the vector information is obtained, searching a topology vector which is approximately collinear with the vector information from the topology vector information as a template vector; because the embodiment of the application refers to the hardware information of the light-compensating light source, each light spot in each group of sequences has a corresponding hardware ID, the above-mentioned bright spot profile sequences can be combined according to the hardware ID in the same way, for example, each light spot corresponding to the hardware ID 1 is combined into one group of light spot sequences, and each light spot corresponding to the hardware ID 2 is combined into another group of light spot sequences, so as to obtain the light source light spot sequences.

The control device calculates the sum of distances from each group of light spot sequences in the light source light spot sequences to the center of the pupil, takes the target light spot sequence with the smallest sum of distances and highest matching degree with the template vector as the cornea reflection light spot sequence, and calculates the center of each light spot in the cornea reflection light spot sequence as the light spot coarse positioning result.

Step S863, traversing the cornea reflection light spot sequence, sequentially obtaining third interested area images corresponding to each cornea reflection light spot in the cornea reflection light spot sequence, and carrying out contour extraction processing on each third interested area image to obtain a target light spot.

Specifically, the control device sequentially extracts the third region of interest image of each light spot from the eye image information, performs size amplification and binarization processing, and further extracts the outline and searches for the connected domain with the largest area. Traversing all the light spots, if the number of the connected domain contour points with the largest area corresponding to the current light spot is less than or equal to the preset number, discarding the current bright spot information, and re-extracting the third interested area images corresponding to other light spots until all the light spots are traversed; if the number of the connected domain contour points with the largest area is larger than the preset number, the corresponding current light spots are used as target light spots, ellipse fitting is conducted on the target light spots, and the fitted ellipse geometric center is calculated and stored.

Step S864, obtaining the target light spot positioning result based on the target light spot.

After traversing all the light spots through the step S863, counting the number of target light spots; if the target light spot number is less than the preset threshold value, the requirement of the subsequent cornea center calculation is not met, so that the processing of the current eye image information by the algorithm can be directly stopped; if the number of the target light spots is greater than or equal to a preset threshold value, cornea reflection light spot sequence information is stored in a (hardware ID, ellipse geometric center) data pair mode, and the cornea reflection light spot sequence information is used as a target light spot positioning result to perform subsequent eye movement calculation analysis. Specifically, four white dots marked around the pupil in fig. 9 are detected LED spot information.

Through the steps S861 to S864, the template vector is obtained through the topological vector information and the vector information in the bright spot contour sequence, so that a template matching algorithm based on any two led relative position vectors is designed, interference of other bright noise points in the eye image information on the extraction of the bright spots is weakened, and the accuracy of the extraction of the bright spots is improved.

It should be noted that, the present embodiment further provides a corneal reflection light spot positioning algorithm, and fig. 10 is a flowchart of a corneal reflection light spot positioning algorithm according to an embodiment of the present application, as shown in fig. 10, and the flowchart includes the following steps:

step S1001, obtaining input information of a cornea reflection spot positioning algorithm, including: infrared near-eye image, pupil coarse positioning image pupil_elipse extracted by pupil positioning, eyelid information eyeies and LED hardware structure design information.

Step S1002, starting rough positioning of light spots; the pupil coarse positioning image pupil_ellipse obtained based on the pupil positioning algorithm sets a region of interest image roiImage extracted from the cornea reflection spots.

In step S1003, binarization processing is performed on the roiImage image to obtain a highlight information connected domain.

Step S1004, extracting the contour of the connected domain to obtain a contour sequence contours.

Step S1005, traversing contours, processing the outline with the oversized area by using a morphological method, and cutting the outline into a plurality of small-area areas so as to realize the adhesion outline area segmentation processing; updating the contour set contours.

Step S1006, traversing the concours, sequentially calculating the minimum circumscribed rectangle of each contour, and filtering the contour which is obviously deviated from the center of the image by combining eyelid information to realize the filtering of the highlight region; updating the contour set contours, and calculating to obtain a bright spot contour center list fasula_center_list.

Step S1007, judging whether the number of the candidate combinations of the bright spots in the bright spot contour center list fasula_center_list is less than or equal to 0; if yes, ending the cornea reflection facula positioning process; if not, continuing to execute the subsequent steps.

Step S1008, constructing a topology vector set template among the LED light sources based on the hardware structure information; sequentially taking the center of each bright spot contour as a reference base, calculating the vector v of the center of the rest bright spot contour relative to the base, and searching from templates to find a template vector which is approximately collinear; and carrying out the same combination and combination on the bright spots in the bright spot contour center list according to the hardware id indicated by the hardware structure information.

Step S1009, judging whether the number of the bright spot combinations is more than 0; if yes, screening to obtain an optimal light spot combination based on the sum of the distances from the pupil center and the matching degree of the pupil center and the hardware template; if not, ending the cornea reflection facula positioning process.

And step S1010, calculating the center of the light spot, and ending the rough positioning process of the light spot.

Step S1011, starting fine positioning of the light spots; and reading the ith light spot information, and performing scale-time size amplification on the light spot image to obtain an amplified image scale_roi.

Step S1012, performing binarization processing on the scale_roi image, extracting the contour, and searching for the connected domain with the largest area, wherein the contour of the connected domain with the largest area is the optimal contour.

Step S1013, judging whether the number of the points of the optimal contour is more than 5; if not, the current spot information is discarded, and the step S1011 is re-executed; if yes, executing the subsequent steps.

Step S1014, elliptical fitting is carried out on the current light spot information, and the geometrical center of the light spot is calculated and recorded; and finishing the fine positioning process of the light spots.

Step S1015, judging whether the number of light spots is less than 2; if yes, directly ending the cornea reflection facula positioning process; if not, the cornea reflection light spot sequence information is stored in the form of (ID, center) data pairs and is output to a subsequent analysis module.

An embodiment of the present application will be described in detail with reference to a practical application scenario, and fig. 11 is a flowchart of an eye image processing method according to a preferred embodiment of the present application, as shown in fig. 11, where the flowchart includes the following steps:

step S1101, an infrared eye moving image is acquired as eye image information.

Step S1102, starting a pupil positioning process; a region of interest ROI image in the ocular image information is extracted based on the Haar response.

Step S1103, eyelid positioning is performed based on the ROI image, and whether the eyelid spacing is too small is judged based on the eyelid positioning result; if yes, step S1107 is executed.

In step S1104, if the determination result in step S1103 is no, the pupil area is roughly located, the pupil area is extracted from the eye image information, and inverse binarization processing is performed to generate a pupil roughly located image by roughly locating the pupil outline.

Step S1105, judging whether the rough positioning is successful; if not, step S1107 is performed.

Step S1106, if the determination result in the step S1105 is yes, performing Canny edge detection based on the adaptive threshold, and iteratively correcting the pupil outline based on the pupil rough positioning image and the edge image to obtain a target pupil positioning result; and ending the pupil positioning process.

Step S1107, judging that blinks; ending the eye image processing flow.

Step S1108, starting cornea reflection spot positioning; and (5) extracting the ROI image by matting, and carrying out binarization and highlight region contour extraction.

Step S1109, screening LED light spots based on topology prior information.

Step S1110, performing LED contour refinement based on the LED light spots to obtain a target light spot positioning result.

Through the steps S1101 to S1110, the feature extraction method with strong robustness is provided, the detection rate on the infrared near-eye image in the eye movement tracking project reaches more than 99%, and the detection precision reaches the sub-pixel level; in addition, the time cost can be less than 30ms when the eye image processing method is tested on a PC, and the real-time requirement is met, so that the accuracy and the efficiency of the eye image processing method are ensured.

It should be noted that the steps illustrated in the above-described flow or flow diagrams of the figures may be performed in a computer system, such as a set of computer-executable instructions, and that, although a logical order is illustrated in the flow diagrams, in some cases, the steps illustrated or described may be performed in an order other than that illustrated herein.

The embodiment also provides an eye image processing device, which is used for implementing the above embodiment and the preferred embodiment, and is not described in detail. As used below, the terms "module," "unit," "sub-unit," and the like may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

Fig. 12 is a block diagram of an eye image processing apparatus according to an embodiment of the present application, as shown in fig. 12, the apparatus includes: an acquisition module 122, a coarse positioning module 124, a target module 126, and a generation module 128; the acquiring module 122 is configured to acquire eye image information; the coarse positioning module 124 is configured to perform inverse binarization processing on the eye image information to obtain a first preprocessed image, and perform contour extraction processing on the first preprocessed image to obtain a pupil coarse positioning image; the target module 126 is configured to perform edge detection processing on the first preprocessed image to obtain an edge image, and obtain a target pupil positioning result according to the edge image and the pupil rough positioning image; the generating module 128 is configured to generate an eye movement feature extraction result for the eye image information at least according to the target pupil positioning result.

The above-described respective modules may be functional modules or program modules, and may be implemented by software or hardware. For modules implemented in hardware, the various modules described above may be located in the same processor; or the above modules may be located in different processors in any combination.

The embodiment also provides an eye movement tracking system, which comprises an image acquisition device and a control device; the image acquisition equipment is used for acquiring eye image information; the control device is configured to perform the steps in any one of the method embodiments described above with respect to the eye image information, so as to obtain an eye movement feature extraction result, and generate an eye movement tracking result based on the eye movement feature extraction result. The control device includes, but is not limited to, various server devices, processing chips, a single-chip microcomputer, a personal computer or other devices for controlling the rope skipping counting process to generate the target rope skipping detection result. In particular, the eye tracking system may be applied in an augmented Reality (Augmented Reality, abbreviated AR) or Virtual Reality (VR) device; taking the application of the eye tracking system to AR glasses as an example, the image acquisition equipment of the AR glasses acquires eye image information, the pupil detection unit and the LED reflection facula detection unit of the control device in the AR glasses acquire a target pupil positioning result and a target facula positioning result respectively through the embodiment of the method, a cornea center calculation unit in the control device calculates a cornea center position according to the target facula positioning result, a pupil center calculation unit in the control device calculates a pupil center position according to the target pupil positioning result, and a fixation point calculation unit of the control device calculates a fixation point according to the cornea center position and the fixation point calculation unit; the AR glasses ultimately implement interactions based on gaze point calculations. Through the embodiment, the pupil positioning and cornea reflection light spot detection method under the end-to-end eye movement tracking system is provided, and reliable characteristic information is provided for an eye movement tracking scheme based on a pupil-cornea reflection method.

In some of these embodiments, a computer device is provided, which may be a server, and fig. 13 is a structural diagram of an interior of the computer device according to an embodiment of the present application, as shown in fig. 13. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store eye movement feature extraction results. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement the above-described eye image processing method.

It will be appreciated by those skilled in the art that the structure shown in fig. 13 is merely a block diagram of a portion of the structure associated with the present application and is not limiting of the computer device to which the present application applies, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

The present embodiment also provides an electronic device comprising a memory having stored therein a computer program and a processor arranged to run the computer program to perform the steps of any of the method embodiments described above.

Optionally, the electronic apparatus may further include a transmission device and an input/output device, where the transmission device is connected to the processor, and the input/output device is connected to the processor.

Alternatively, in the present embodiment, the above-described processor may be configured to execute the following steps by a computer program:

s1, acquiring eye image information.

S2, performing inverse binarization processing on the eye image information to obtain a first preprocessed image, and performing contour extraction processing on the first preprocessed image to obtain a pupil rough positioning image.

And S3, performing edge detection processing on the first preprocessed image to obtain an edge image, and obtaining a target pupil positioning result according to the edge image and the pupil rough positioning image.

And S4, generating an eye movement characteristic extraction result aiming at the eye image information at least according to the target pupil positioning result.

It should be noted that, specific examples in this embodiment may refer to examples described in the foregoing embodiments and alternative implementations, and this embodiment is not repeated herein.

In addition, in combination with the eye image processing method in the above embodiment, the embodiment of the application may be implemented by providing a storage medium. The storage medium has a computer program stored thereon; the computer program, when executed by a processor, implements any of the eye image processing methods of the above embodiments.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the various embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), memory bus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.

It should be understood by those skilled in the art that the technical features of the above-described embodiments may be combined in any manner, and for brevity, all of the possible combinations of the technical features of the above-described embodiments are not described, however, they should be considered as being within the scope of the description provided herein, as long as there is no contradiction between the combinations of the technical features.

The above examples merely represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the invention. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application is to be determined by the claims appended hereto.

Claims

1. A method of eye image processing, the method comprising:

acquiring eye image information;

2. The method according to claim 1, wherein performing inverse binarization processing on the eye image information to obtain a first preprocessed image comprises:

and performing inverse binarization processing on the first region-of-interest image to obtain the first preprocessing image.

3. The method according to claim 1, wherein the performing contour extraction processing on the first preprocessed image to obtain the pupil rough positioning image comprises:

4. The eye image processing method in accordance with claim 3, wherein the obtaining the pupil rough positioning image based on the eyelid positioning sub-image comprises:

5. The method according to claim 4, wherein obtaining a pupil contour point result from the first contour point set and the fitted circle result comprises:

6. The method according to claim 1, wherein obtaining a target pupil positioning result from the edge image and the pupil rough positioning image comprises:

7. The eye image processing method of claim 1, wherein generating an eye movement feature extraction result for the eye image information based at least on the target pupil positioning result comprises:

8. The eye image processing method of claim 7, wherein obtaining the target spot positioning result according to the topology vector information and the bright spot contour sequence comprises:

obtaining a template vector according to the topology vector information and the vector information, and obtaining a light source light spot sequence according to the combination of the topology vector information and the bright spot contour sequence;

obtaining a cornea reflection light spot sequence according to the light source light spot sequence and the template vector matching, and obtaining a light spot coarse positioning result according to the cornea reflection light spot sequence;

traversing the cornea reflection light spot sequence, sequentially obtaining third interested area images corresponding to each cornea reflection light spot in the cornea reflection light spot sequence, and carrying out contour extraction processing on each third interested area image to obtain a target light spot;

And obtaining the target light spot positioning result based on the target light spot and the light spot coarse positioning result.

9. The eye image processing method according to any one of claims 1 to 8, wherein performing edge detection processing on the eye image information to obtain an edge image includes:

and performing image segmentation processing on the eye image information by using an OTSU algorithm to obtain a segmentation threshold, and performing edge detection on the eye image according to the segmentation threshold to obtain the edge image.

10. An eye image processing apparatus, the apparatus comprising: the device comprises an acquisition module, a coarse positioning module, a target module and a generation module;

the acquisition module is used for acquiring eye image information;

11. An eye tracking system, characterized in that the eye tracking system comprises an image acquisition device and a control means;

the image acquisition equipment is used for acquiring eye image information;

the control means for performing the eye image processing method according to any one of claims 1 to 9 with respect to the eye image information to obtain the eye movement feature extraction result, and generating an eye movement tracking result based on the eye movement feature extraction result.

12. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program, the processor being arranged to run the computer program to perform the ocular image processing method of any one of claims 1 to 9.

13. A storage medium having a computer program stored therein, wherein the computer program is arranged to perform the ocular image processing method of any one of claims 1 to 9 when run.