WO2019010959A1 - 用于确定视线的方法、设备和计算机可读存储介质 - Google Patents

用于确定视线的方法、设备和计算机可读存储介质 Download PDF

Info

Publication number
WO2019010959A1
WO2019010959A1 PCT/CN2018/074537 CN2018074537W WO2019010959A1 WO 2019010959 A1 WO2019010959 A1 WO 2019010959A1 CN 2018074537 W CN2018074537 W CN 2018074537W WO 2019010959 A1 WO2019010959 A1 WO 2019010959A1
Authority
WO
WIPO (PCT)
Prior art keywords
line
sight
feature vector
eye image
axis
Prior art date
Application number
PCT/CN2018/074537
Other languages
English (en)
French (fr)
Inventor
孙建康
张�浩
陈丽莉
楚明磊
孙剑
郭子强
闫桂新
王亚坤
Original Assignee
京东方科技集团股份有限公司
北京京东方光电科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司, 北京京东方光电科技有限公司 filed Critical 京东方科技集团股份有限公司
Priority to US16/305,882 priority Critical patent/US11294455B2/en
Publication of WO2019010959A1 publication Critical patent/WO2019010959A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/11—Complex mathematical operations for solving equations, e.g. nonlinear equations, general mathematical optimization problems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/012—Head tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G06T7/75—Determining position or orientation of objects or cameras using feature-based methods involving models
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/19—Sensors therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10048—Infrared image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30041—Eye; Retina; Ophthalmic
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person

Definitions

  • the present disclosure relates to the field of human-computer interaction, and more particularly to a method, apparatus, and computer readable storage medium for determining a line of sight.
  • Sight estimation techniques are techniques for determining the line of sight of a human or animal.
  • a conventional line-of-sight estimation technique uses a connected device to determine the line of sight, for example by using a specially designed contact lens, so that the direction of the line of sight can be determined by the contact lens as the eye moves.
  • another type of line-of-sight estimation technique uses a camera to capture an eye image of a subject, perform eye feature extraction, measure eye movement, and further estimate a line of sight direction or a position of an eye gaze point.
  • a method for determining a line of sight includes determining at least one line of sight feature vector from the eye image; determining a position of the line of sight drop based on the line of sight estimation model and the at least one line of sight feature vector.
  • the at least one line-of-sight feature vector includes at least one of: a first line of sight feature vector indicating a center of the first reference spot formed in the eye image by the first reference light source to the pupil center, indicating the pupil center And a second line of sight feature vector to a center of the second reference spot formed by the second reference source in the eye image and a third line of sight feature vector indicating the center of the second reference spot to the center of the first reference spot.
  • the step of determining at least one line of sight feature vector from the eye image includes elliptical fitting the pupil portion in the eye image to determine at least one of: a center of the fitted ellipse as a pupil center; The long axis of the fitted ellipse; the minor axis of the fitted ellipse; and the angle of rotation between the major axis and the horizontal direction.
  • the eye image is obtained with an annular reference light source as illumination.
  • the formula for the line of sight estimation model is:
  • x fix and y fix are the line-of-sight points corresponding to the eye image, respectively, on the X-axis of the observation object coordinate system.
  • x AI and y AI are components of the first line-of-sight feature vector in the X-axis and Y-axis directions of the eye image coordinate system, respectively
  • x IB and y IB are the second line-of-sight features, respectively.
  • the components of the vector in the X-axis and Y-axis directions of the eye image coordinate system, and x BA and y BA are components of the third line-of-sight feature vector in the X-axis and Y-axis directions of the eye image coordinate system, respectively.
  • the model parameters of the line of sight estimation model are determined by least squares method, and the number of calibration points used is at least 10.
  • the step of determining the position of the line of sight drop based on the line of sight estimation model and the at least one line of sight feature vector includes: a first line of sight feature vector, a second line of sight feature vector, and a third line of sight feature of the currently captured eye image
  • the vector is substituted into the line-of-sight estimation model in which the model parameters have been determined, and the coordinates of the corresponding line-of-sight points on the X-axis and the Y-axis of the observation object coordinate system are obtained.
  • the method further includes performing head motion compensation on the determined line of sight location to determine a compensated line of sight location.
  • the step of performing head motion compensation on the determined line of sight drop position to determine the compensated line of sight drop position includes: determining a head motion compensation feature vector according to the eye image; according to the head motion compensation feature vector Determining a head motion compensation value; and adjusting the determined line of sight drop position based on the head motion compensation value to determine a compensated line of sight drop position.
  • the head motion compensation feature vector includes at least one component value: a first component value indicating a head-to-back motion; a second component value indicating a head horizontal motion; and a third component indicating a head rotational motion value.
  • the first component value is the Euclidean distance of the center of two reference spots
  • the second component value is the ratio of the long and short axes of the elliptically fitted pupil
  • the third component value is an elliptical fit The angle of rotation between the long axis of the pupil and the horizontal direction.
  • the step of determining the head motion compensation value according to the head motion compensation feature vector includes: inputting the head motion compensation feature vector into the trained head vector compensation model based on the support vector regression machine to determine horizontal and vertical directions The corresponding head movement compensation value on it.
  • an apparatus for determining a line of sight includes: a line-of-sight feature vector determining unit configured to determine at least one line-of-sight feature vector according to the eye image; and a line-of-sight point position determining unit configured to determine a line of sight based on the line-of-sight estimating model and the at least one line-of-sight feature vector The location of the point.
  • the at least one line-of-sight feature vector includes at least one of: a first line of sight feature vector indicating a first reference spot center formed in the eye image by the first reference light source to a pupil center; indicating the pupil center a second line of sight feature vector to a center of the second reference spot formed by the second reference source in the eye image; and a third line of sight feature vector indicating the center of the second reference spot to the center of the first reference spot .
  • the line-of-sight feature vector determining unit is further configured to perform an ellipse fitting on the pupil portion in the eye image to determine at least one of: a center of the fitted ellipse as a pupil center; the fitted The major axis of the ellipse; the minor axis of the fitted ellipse; and the angle of rotation between the major axis and the horizontal direction.
  • the formula for the line of sight estimation model is:
  • x fix and y fix are the line-of-sight points corresponding to the eye image, respectively, on the X-axis of the observation object coordinate system.
  • x AI and y AI are components of the first line-of-sight feature vector in the X-axis and Y-axis directions of the eye image coordinate system, respectively
  • x IB and y IB are the second line-of-sight features, respectively.
  • the components of the vector in the X-axis and Y-axis directions of the eye image coordinate system, and x BA and y BA are components of the third line-of-sight feature vector in the X-axis and Y-axis directions of the eye image coordinate system, respectively.
  • the line-of-sight location determining unit is further configured to: substitute the first line of sight feature vector, the second line of sight feature vector, and the third line of sight feature vector of the currently captured eye image into the line of sight of the determined model parameter
  • the coordinates of the corresponding line of sight drop point on the X-axis and Y-axis of the observation object coordinate system are obtained.
  • the apparatus further includes a head motion compensation unit for performing head motion compensation on the determined line of sight drop position to determine a compensated line of sight drop position.
  • the head motion compensation unit is further configured to: determine a head motion compensation feature vector according to the eye image; determine a head motion compensation value according to the head motion compensation feature vector; and based on the head motion compensation value The determined line of sight drop position is adjusted to determine the compensated line of sight drop position.
  • the head motion compensation feature vector includes at least one component value: a first component value indicating a head-to-back motion; a second component value indicating a head horizontal motion; and a third component indicating a head rotational motion value.
  • the first component value is the Euclidean distance of the center of two reference spots
  • the second component value is the ratio of the long and short axes of the elliptically fitted pupil
  • the third component value is an elliptical fit The angle of rotation between the long axis of the pupil and the horizontal direction.
  • the head motion compensation unit is further configured to: input the head motion compensation feature vector into the trained head vector compensation model based on the support vector regression machine to determine a corresponding head motion compensation value in the horizontal and vertical directions.
  • an apparatus for determining a line of sight includes: a processor; a memory storing instructions that, when executed by the processor, cause the processor to: determine at least one line of sight feature vector based on an eye image; and based on a line of sight estimation model and said At least one line of sight feature vector to determine the location of the line of sight drop point.
  • a computer readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to the first aspect of the present disclosure.
  • FIG. 1 is a schematic diagram showing an example application scenario of a technical solution for determining a line of sight according to an embodiment of the present disclosure.
  • FIG. 2 is a diagram showing an example of determining respective line-of-sight feature vectors from an eye image according to an embodiment of the present disclosure.
  • FIG. 3 is a diagram showing an example of determining a head motion compensation vector from an eye image according to an embodiment of the present disclosure.
  • FIG. 4 is a diagram showing an example of determining a head motion compensation vector from an eye image according to another embodiment of the present disclosure.
  • FIG. 5 is a diagram showing an example of determining a head motion compensation vector from an eye image according to still another embodiment of the present disclosure.
  • FIG. 6 is a flow chart illustrating an example method for determining a line of sight, in accordance with an embodiment of the present disclosure.
  • FIG. 7 is a functional block diagram showing an example device for performing the method of FIG. 6 in accordance with an embodiment of the present disclosure.
  • FIG. 8 is a hardware layout diagram illustrating an example device for determining a line of sight, in accordance with an embodiment of the present disclosure.
  • the terms “include” and “including” and their derivatives are meant to be inclusive and not limiting; the term “or” is inclusive, meaning and/or.
  • the orientation terms used such as “upper”, “lower”, “left”, “right”, etc., are used to indicate relative positional relationships to assist those skilled in the art in understanding the present disclosure. Embodiments, and thus those skilled in the art should understand that “upper” / “lower” in one direction may become “lower” / “upper” in the opposite direction, and may become other in the other direction. Positional relationships, such as “left” / "right”, etc.
  • the present disclosure is described in detail by taking the present disclosure as an example of a human-computer interaction scenario.
  • the present disclosure is not limited thereto, and the present disclosure can also be applied to other fields such as augmented reality, virtual reality, user experience, psychological research, handicapped assistance, driver assistance, and the like.
  • specific embodiments are described below by taking a human user as an example, the present disclosure is not limited thereto.
  • the solution according to an embodiment of the present disclosure may also be applied to other animals or non-living bodies having similar eye features.
  • the traditional line-of-sight estimation techniques are mainly divided into a line-of-sight estimation method based on a two-dimensional mapping model and a line-of-sight estimation method based on a three-dimensional eyeball model.
  • the line-of-sight estimation method based on the two-dimensional mapping model is simple and rapid in terms of line-of-sight parameter extraction and line-of-sight feature recognition, and can meet the practical requirements.
  • the mapping model has low precision and poor stability, and requires the user's head to be stationary during use. Meet the comfort requirements.
  • the line-of-sight estimation method based on the three-dimensional eyeball model can detect the position of the user's head space and can adapt to the natural movement of the user's head, but its hardware configuration is complicated (at least two cameras and dual light sources are required), and the hardware cost of the device is high and the algorithm is implemented. Complex problems, and the need to obtain independent information of the user's eye parameters, accurate indirect estimation of eyeball independent parameters is difficult to achieve without the aid of other instruments.
  • FIG. 1 is a schematic diagram showing an example application scenario 10 of a technical solution for determining a line of sight, in accordance with an embodiment of the present disclosure.
  • the application scenario 10 may include a user 100, a target screen 120, a first reference light source 110A, and a second reference light source 110B (may not be collectively referred to as reference light source 110 hereinafter, unless otherwise specified), and image sensor 120. .
  • the principle for determining the gaze point 135 (ie, point O) of the user 100 on the target screen 130 is as follows.
  • the reference light is emitted to the user 100 by the first reference light source 110A and the second reference light source 110B, and then the reference light reflection image including the eye image of the user 100 is captured by the image sensor 120.
  • the image sensor 120 By performing the aforementioned image capturing process on the user's eyes a plurality of times while the user views a plurality of calibration points (sometimes referred to as reference points) on the target screen 130, an eye image corresponding to each of the calibration points can be obtained.
  • the model parameters of the line-of-sight estimation model can be determined, thereby implementing line-of-sight determination calibration. Then, the corresponding gaze point on the target screen 130 can be determined based on the user's eye image captured in real time.
  • reference light sources 110A and 110B are illustrated in FIG. 1, the present disclosure is not limited thereto. In fact, in other embodiments, a single reference source 110 or three or more reference sources 110 may also be used. According to the embodiments described in detail below, those skilled in the art can easily derive a scheme having other numbers of reference sources from the scheme of two reference sources.
  • reference light source is an annular reference light source in FIG. 1, the present disclosure is not limited thereto. In fact, in other embodiments, reference sources having other shapes, such as triangles, squares, rectangles, ellipses, hyperbolic shapes, or any other regular or irregular shape, may also be used.
  • both reference light sources 110 employ infrared light, such as infrared light having a wavelength of 850 nm, although the disclosure is not limited thereto.
  • infrared light such as infrared light having a wavelength of 850 nm
  • light waves of other wavelengths may also be employed.
  • near-infrared light in the visible range can be employed.
  • the human eye is not significantly affected by viewing due to the proximity to infrared light.
  • light waves of any other wavelength can also be used.
  • the two reference light sources 110 are respectively placed on the left and right sides of the image sensor 120 in FIG. 1, the present disclosure is not limited thereto. In fact, the two reference light sources 110 can be placed at any relative position of the image sensor 110 as long as the image sensor 110 can acquire the reference light of the two reference light sources 110 reflected by the eyes of the user 100.
  • the image sensor 120 (for example, a high definition camera) is located at a lower portion of the target screen 130 in FIG. 1, the present disclosure is not limited thereto. In fact, image sensor 120 can be located at any suitable location on target screen 130, such as left, right, upper, and the like.
  • FIG. 2 is a diagram for determining respective line-of-sight feature vectors (eg, vectors from eye images 20 in accordance with an embodiment of the present disclosure. with An example schematic of ).
  • line-of-sight feature vectors eg, vectors from eye images 20 in accordance with an embodiment of the present disclosure. with An example schematic of .
  • externally visible portions included in the eye of the user 100 can be seen in the eye image 20, including but not limited to: the pupil 200, the iris 210, and the sclera 220.
  • the iris 210 is a dark portion of the eye with an opening therebetween, i.e., the pupil 200, for light to enter the interior of the eye and be perceived and imaged by the photoreceptor cells on the retina.
  • the iris 210 is responsible for adjusting the size of the pupil 200 in accordance with the intensity of ambient light to enable the eye to adapt to different environments. For example, in a strong light environment, the iris 210 relaxes, causing the pupil 200 to contract, reducing the amount of light entering; instead, in the low light environment, the iris 210 contracts, causing the pupil 200 to be enlarged, increasing the amount of light entering.
  • the sclera 220 is a white portion of the eye, which is mainly a relatively rigid outer shell composed of elastic fibers or the like, and is responsible for protecting the eyeball. Furthermore, the cornea is actually covered above the pupil 200, the iris 210 and the sclera 220, and since it is transparent, it is not directly observable in the eye image. However, in the embodiment of the present disclosure, since the reference light emitted by the two reference light sources 110 is reflected when reaching the cornea, a reflective reference spot is formed (for example, the first reference spot 230 and the second reference shown in FIG. 2). Spot 240), thus indicating the presence of the cornea.
  • the light emitted by the reference source 110 will actually reflect on both the front and back surfaces of the cornea, and thus virtually each reference source 110 will form two spots on the cornea.
  • the brightness of the spot formed on the rear surface is significantly lower than that of the spot formed on the front surface, it is necessary to use a very high sensitivity, high resolution image sensor to observe, and thus is ignored in the embodiment shown in FIG.
  • embodiments of the present disclosure are equally applicable to similar operations on spots formed on the back surface, and those skilled in the art can readily derive similar schemes for back surface spots in accordance with the embodiments described herein, thus The description and description are concise and are omitted here.
  • the light emitted by the first reference light source 110A and the second reference light source 110B may form a first reference spot 230 and a second reference spot 240 on the cornea, respectively.
  • the centers of the two reference spots are a first reference spot center 235 (point A shown in Fig. 2) and a second reference spot center 245 (point B shown in Fig. 2).
  • the determination of the spot center may be performed by elliptical or circular fitting of the spot area detected in the eye image 20. Thereby, the coordinate positions A(x A , y A ) and B(x B , y B ) of the points A and B in the eye image coordinate system can be determined separately.
  • the determination of the spot binarization threshold can be performed by the histogram bimodal method, and the input pre-processed eye image is binarized to obtain a spot binarized image. Then, the spot binarized image can be etched and expanded, and then the median filter is used for secondary denoising to obtain a spot region image. Next, the extracted component region extraction may be performed on the extracted spot region image, and the centroids of the extracted two spot connecting components are calculated to obtain the first reflected spot center A (x A , y A ) and the second reflection respectively. Spot center B (x B , y B ). In other embodiments, other methods may be used to determine the reflected spot information.
  • the position of the pupil 200 in the eye image 20 can be determined while or before or after determining the center of the spot.
  • the position of the pupil 200 in the eye image 20 can be determined using light pupil or dark pupil techniques.
  • the pupil means that when the reference light source (eg, reference light source 110) is on the same optical axis as the image sensor (eg, image sensor 120), the light is reflected back at the fundus and passes through the pupil (eg, pupil 200) back to the image sensor, thereby The pupil is rendered in a bright state in the eye image captured by the image sensor.
  • the dark spot means that when the reference light source is not on the same optical axis as the image sensor 120, since the light does not reach the image sensor through the pupil after the fundus is reflected, the pupil is in the eye image captured by the image sensor. Dark state. The position, range, and the like of the pupil 200 in the eye image 20 can be determined whether or not the pupil or the dark is used.
  • a dark squeegee technique is employed.
  • the present disclosure is not limited thereto, but a diaphragm technique may also be employed.
  • the center 205 (or point I) of the pupil 200 can be determined in addition to the centers A and B of the two reference spots as previously described.
  • the pupil binarization threshold can be determined by the image segmentation and/or the histogram bimodal method, and the preprocessed eye image is subjected to the image processing. Binarization processing to obtain a pupil binarized image. Then, the pupil binarized image can be subjected to corrosion and expansion processing, and then the median filter is used for secondary denoising to obtain an image of the pupil region.
  • the desired pupil information may include at least one of the following: the center of the fitted ellipse (ie, the pupil center) I(x I , y I ), the long axis length r 1 of the fitted ellipse, the fit The short axis length r 2 of the ellipse, and the rotation angle ⁇ of the long axis and the horizontal direction (for example, as shown in FIG. 5) and the like.
  • other methods may be used to determine the pupil information.
  • the present disclosure is not limited thereto.
  • the ellipse major axis r 1 and the minor axis r 2 may also be a vertical axis and a horizontal axis, respectively.
  • the rotation angle between the major axis of the ellipse and the horizontal axis is not shown, as shown in FIG. 5, when the head is deflected to one side, the eye image 50 and The long axis r 1 ' of the bore 500 will appear at an angle to the horizontal axis, ie, the angle of rotation ⁇ .
  • the third feature vector can be determined.
  • all three feature vectors are still used to illustrate the following operations.
  • these three feature vectors can be expressed as two feature vectors in fact without affecting the implementation of the technical solution.
  • the specific parameters of the line-of-sight estimation model can be determined by the calibration points.
  • the established line of sight estimation model is as follows:
  • (x fix , y fix ) is the position coordinate of the preset calibration point on the target screen 130
  • (x AI , y AI ) is the first line of sight feature vector obtained as described above when the user views the corresponding calibration point.
  • (x IB , y IB ) is the second line of sight feature vector obtained as described above when the user views the corresponding calibration point
  • a 0 to a 9 and b 0 to b 9 are the line-of-sight estimation model parameters to be solved.
  • the corresponding data of the plurality of calibration points can be fitted by the least squares method to determine a 0 to a 9 and b 0 to b 9 to complete the calibration (also referred to as calibration).
  • the above-described line-of-sight estimation model can be calibrated or calibrated using 12 calibration points.
  • the present disclosure is not limited thereto.
  • other suitable numbers of calibration points may also be used for calibration.
  • 10 or more calibration points may be employed.
  • the stage of use can be entered.
  • the image sensor 120 may acquire the eye image 20 of the user 100, and determine a corresponding line of sight feature vector according to the acquired, for example, the aforementioned pupil center position, each spot center position, and the like (eg, the first and second sums).
  • the human user 100 Since the human user 100 is observing, for example, the target screen 130, its head usually involuntarily shifts, rotates, etc., and is unlikely to be completely still, so the actual gaze point of the user 100 and the gaze determined by the above-described line of sight determination scheme. There is an error between the points.
  • the coordinates O(x fix , y fix ) obtained by the above-described scheme can be compensated as in the embodiment shown in FIGS. 3 to 5.
  • the compensation scheme for different types of head movement will be described in detail with reference to FIGS. 3 to 5.
  • FIG. 3 is a diagram showing an example of determining a head motion compensation vector from an eye image in which a head of the user 100 moves back and forth in a direction perpendicular to the target screen 130, according to an embodiment of the present disclosure.
  • 4 is a diagram showing an example of determining a head motion compensation vector from an eye image in which a head of the user 100 moves horizontally left and right in a plane parallel to the target screen 130, according to another embodiment of the present disclosure.
  • FIG. 5 is a diagram showing an example of determining a head motion compensation vector from an eye image in which a head of the user 100 is rotated in a plane parallel to the target screen 130, according to still another embodiment of the present disclosure.
  • FIGS. 1 is a diagram showing an example of determining a head motion compensation vector from an eye image in which a head of the user 100 moves back and forth in a direction perpendicular to the target screen 130, according to an embodiment of the present disclosure.
  • 4 is a diagram showing an example of determining a head motion
  • 3 to 5 respectively show different types of head movements, in practice, it may be any one or a combination of any of the three types of head movements. Therefore, in order to comprehensively consider these various conditions, the head motion compensation model based on the support vector regression machine described below can be used.
  • the corresponding head motion compensation vector can be determined according to the eye image (for example, the eye images 30, 40, and/or 50), and then the head motion compensation vector is input to the aforementioned head motion compensation model based on the support vector regression machine. In order to determine the head movement compensation value. Then, the aforementioned coordinates O(x fix , y fix ) are adjusted accordingly according to the head motion compensation value.
  • the head of the user 100 moves back and forth in a direction perpendicular to the target screen 130, resulting in the first reference spot center 335 (ie, point A) and the second in the eye image 30.
  • the Euclidean distance between the reference spot centers 345 changes. More specifically, in the embodiment shown in FIG. 3, it can be seen that since the head of the user 100 moves in a direction away from the target screen 130, its third line of sight feature vector The length is shortened. Note that the dotted circle is the position of the two reference spots before the movement, and the distance between the centers is greater than the distance after the movement.
  • the first component value of the head motion of the head motion compensation vector can be determined as the Euclidean distance between the centers of the two reference spots. More specifically, the first component value L may be determined as the Euclidean distance between the first reference spot center 335 (ie, point A) and the second reference spot center 345 (ie, point B):
  • the second component value ⁇ indicating the horizontal movement of the head movement compensation vector can be determined as the long axis r 1 of the elliptical pupil capable of characterizing the left and right movement of the head in the horizontal direction parallel to the plane of the target screen 130.
  • the third component value ⁇ can be determined as the rotation angle ⁇ between the long axis r 1 of the ellipse and the horizontal direction capable of characterizing the rotational movement of the head in a plane parallel to the target screen 130.
  • the above respective component values may be obtained in whole or in part according to the pupil information determined in conjunction with FIG. Therefore, in practical applications, in the processing for head motion compensation, it is not necessary to calculate these component values again.
  • the head motion compensation vector C gaze (L, ⁇ , ⁇ )
  • it can be input into the trained head motion compensation model based on the support vector regression machine to obtain the head motion in the horizontal and vertical directions.
  • the compensation value (x horizontal , y vertical ) based on the calculated head motion compensation value (x horizontal , y vertical ), the position of the aforementioned determined line of sight drop point can be adjusted. More specifically, the calculated head motion compensation value (x horizontal , y vertical ) may be summed with the previously determined preliminary line of sight drop point O(x fix , y fix ), thereby obtaining the user 100 on the target screen 130.
  • the head motion compensation model based on the support vector regression machine algorithm, the error caused by the user's head motion can be easily compensated, and the anti-head motion interference capability of the line-of-sight estimation system and method is enhanced, and the user's head is allowed to be used when the system is used.
  • Natural motion reduces the restriction on the user's head when using the line-of-sight estimation system, which improves the comfort and naturalness of the line-of-sight estimation system.
  • FIG. 6 is a flow chart showing an example method 600 for determining a line of sight in accordance with an embodiment of the present disclosure.
  • method 600 can include steps S610 and S620.
  • steps S610 and S620 some of the steps of method 600 may be performed separately or in combination, and may be performed in parallel or sequentially, and are not limited to the specific order of operations illustrated in FIG.
  • method 600 can be performed by device 700 shown in FIG. 7 or device 800 shown in FIG.
  • FIG. 7 is a block diagram showing an example device 700 for determining a line of sight, in accordance with an embodiment of the disclosure.
  • the device 700 may include a line-of-sight feature vector determining unit 710 and a line-of-sight point position determining unit 720.
  • the line-of-sight feature vector determining unit 710 may be configured to determine at least one line-of-sight feature vector from the eye image.
  • the line-of-sight feature vector determining unit 710 may be a central processing unit (CPU) of the device 700, a digital signal processor (DSP), a microprocessor, a microcontroller, etc., which may be associated with an image sensor of the device 700 (eg, an infrared camera, a visible light camera, a camera, etc.) and/or a communication portion (for example, an Ethernet card, a WiFi chip, an RF chip, etc.), according to an eye image captured by an image sensor or an eye image received from a remote device through a communication portion To determine at least one line of sight feature vector.
  • CPU central processing unit
  • DSP digital signal processor
  • microprocessor e.g., a microcontroller, etc.
  • a communication portion for example, an Ethernet card, a WiFi chip, an RF chip, etc.
  • the line of sight drop position determining unit 720 may be configured to determine a position of the line of sight drop based on the line of sight estimation model and the at least one line of sight feature vector.
  • the line of sight placement location determining unit 720 can also be a central processing unit (CPU), digital signal processor (DSP), microprocessor, microcontroller, etc. of the device 700, which can be based on pre-trained and/or real-time training
  • the line-of-sight estimation model and the at least one line-of-sight feature vector determined by the line-of-sight feature vector determining unit 710 determine the position of the line-of-sight point.
  • device 700 may also include other units not shown in FIG. 7, such as a head motion compensation unit or the like.
  • the head motion compensation unit can be configured to perform head motion compensation on the determined line of sight drop position to determine a compensated line of sight drop position.
  • the head motion compensation unit may be further configured to determine a head motion compensation feature vector according to the eye image; determine a head motion compensation value according to the head motion compensation feature vector; and adjust the determined value based on the head motion compensation value The line of sight is positioned to determine the position of the compensated line of sight.
  • the head motion compensation unit may be further configured to: input a head motion compensation feature vector into the trained support vector regression machine based head motion compensation model to determine a corresponding head motion compensation value in the horizontal and vertical directions. .
  • device 700 may also include other functional units not shown in FIG. 7, such as: bus, memory, power supply, antenna, communication portion, storage portion. However, they do not affect the understanding of the principles of the present application, and thus their detailed descriptions are omitted herein.
  • a method 600 and an apparatus 700 for determining a line of sight performed on a device 700 in accordance with an embodiment of the present disclosure will be described in detail below with reference to FIGS. 6 and 7.
  • the method 600 begins in step S610, in which at least one line of sight feature vector may be determined by the line of sight feature vector determining unit 710 of the device 700 based on the eye image.
  • step S620 the position of the line of sight drop point may be determined by the line of sight drop position determining unit 720 of the device 700 based on the line of sight estimation model and the at least one line of sight feature vector.
  • the at least one line-of-sight feature vector comprises at least one of: a first line of sight feature vector indicating a center of the first reference spot formed in the eye image by the first reference source to the pupil center, wherein x AI and y AI is a component of the first line of sight feature vector in the X-axis and Y-axis directions of the eye image coordinate system, respectively; a second line of sight indicating the center of the pupil to the center of the second reference spot formed by the second reference light source in the eye image a feature vector, wherein x IB and y IB are components of the second line of sight feature vector in the X-axis and Y-axis directions of the eye image coordinate system, respectively; and a third line of sight indicating the center of the second reference spot to the center of the first reference spot
  • the feature vector, wherein x BA and y BA are components of the third line of sight feature vector in the X-axis and Y-axis directions of the eye image coordinate system, respectively.
  • the step of determining the at least one line of sight feature vector from the eye image comprises elliptical fitting the pupil portion in the eye image to determine at least one of: a center of the fitted ellipse as a pupil Center; the long axis of the fitted ellipse; the short axis of the fitted ellipse; and the angle of rotation between the long axis and the horizontal direction.
  • the ocular image is obtained with an annular reference source as illumination.
  • the formula for the line of sight estimation model is:
  • the model parameters of the line of sight estimation model are determined by least squares and the number of calibration points used is at least 10.
  • the step of determining a position of the line of sight drop based on the line of sight estimation model and the at least one line of sight feature vector comprises: a first line of sight feature vector, a second line of sight feature vector, and a third of the currently captured eye image
  • the line-of-sight feature vector is substituted into the line-of-sight estimation model in which the model parameters have been determined, and the coordinates of the corresponding line-of-sight points on the X-axis and the Y-axis of the observation object coordinate system are obtained.
  • the method further comprises performing head motion compensation on the determined line of sight location to determine a compensated line of sight location.
  • the step of performing head motion compensation on the determined line of sight location to determine the compensated line of sight location comprises: determining a head motion compensation feature vector from the eye image; Determining a head motion compensation value; and adjusting the determined line of sight drop position based on the head motion compensation value to determine a compensated line of sight drop position.
  • the head motion compensation feature vector includes at least one of: a first component value indicating a head-to-back motion; a second component value indicating a head horizontal motion; and a first component indicating a head rotational motion Three component values.
  • the first component value is the Euclidean distance of the center of the two reference spots
  • the second component value is the ratio of the long and short axes of the elliptically fitted pupil
  • the third component value is the elliptically fitted pupil The angle of rotation between the long axis and the horizontal direction.
  • the step of determining the head motion compensation value based on the head motion compensation feature vector comprises: inputting a head motion compensation feature vector into the trained head vector compensation model based on the support vector regression machine to determine horizontal and vertical directions The corresponding head movement compensation value on it.
  • FIG. 8 is a block diagram showing an example hardware arrangement 800 of the apparatus 700 of FIG. 7 in accordance with an embodiment of the present disclosure.
  • Hardware arrangement 800 includes a processor 806 (eg, a digital signal processor (DSP)).
  • DSP digital signal processor
  • Processor 806 can be a single processing unit or a plurality of processing units for performing different acts of the flows described herein.
  • the arrangement 800 can also include an input unit 802 for receiving signals from other entities, and an output unit 804 for providing signals to other entities.
  • Input unit 802 and output unit 804 can be arranged as a single entity or as separate entities.
  • arrangement 800 can include at least one readable storage medium 808 in the form of a non-volatile or volatile memory, such as an electrically erasable programmable read only memory (EEPROM), flash memory, and/or a hard drive.
  • the readable storage medium 808 includes a computer program 810 that includes code/computer readable instructions that, when executed by the processor 806 in the arrangement 800, cause the hardware arrangement 800 and/or the device 700 including the hardware arrangement 800.
  • Computer program 810 can be configured as computer program code having a computer program module 810A-810B architecture, for example.
  • the code in the computer program of arrangement 800 includes a module 810A for determining at least one line of sight feature vector from the eye image.
  • the code in the computer program further includes a module 810B for determining a position of the line of sight drop based on the line of sight estimation model and the at least one line of sight feature vector.
  • the computer program module can substantially perform various actions in the flows illustrated in Figures 1-6 to simulate device 700.
  • different computer program modules when different computer program modules are executed in processor 806, they may correspond to the different units described above in device 700.
  • code means in the embodiment disclosed above in connection with FIG. 8 is implemented as a computer program module that, when executed in processor 806, causes hardware arrangement 800 to perform the actions described above in connection with FIGS. 1-6, however, in alternative implementations In an example, at least one of the code means can be implemented at least partially as a hardware circuit.
  • the processor may be a single CPU (Central Processing Unit), but may also include two or more processing units.
  • a processor can include a general purpose microprocessor, an instruction set processor, and/or a related chipset and/or a special purpose microprocessor (eg, an application specific integrated circuit (ASIC)).
  • the processor may also include an onboard memory for caching purposes.
  • the computer program can be carried by a computer program product connected to the processor.
  • the computer program product can comprise a computer readable medium having stored thereon a computer program.
  • the computer program product can be flash memory, random access memory (RAM), read only memory (ROM), EEPROM, and the computer program modules described above can be distributed to different computers in the form of memory within the UE in alternative embodiments. In the program product.
  • functions described herein as being implemented by pure hardware, software, and/or firmware may also be implemented by dedicated hardware, a combination of general-purpose hardware and software, and the like.
  • functions described as being implemented by dedicated hardware eg, Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), etc.
  • general purpose hardware eg, central processing unit (CPU), digital signal processing (DSP) is implemented in a way that is combined with software and vice versa.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Ophthalmology & Optometry (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Optimization (AREA)
  • Computational Mathematics (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Medical Informatics (AREA)
  • Computing Systems (AREA)
  • Operations Research (AREA)
  • Geometry (AREA)
  • Algebra (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

本公开提出了用于确定视线的方法、设备和计算机可读存储介质。该方法包括：根据眼部图像来确定至少一个视线特征向量；基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置。所述至少一个视线特征向量包括以下至少一项：指示由第一参考光源在所述眼部图像中形成的第一参考光斑中心到瞳孔中心的第一视线特征向量；指示所述瞳孔中心到由第二参考光源在所述眼部图像中形成的第二参考光斑中心的第二视线特征向量；以及指示所述第二参考光斑中心到所述第一参考光斑中心的第三视线特征向量。

Description

用于确定视线的方法、设备和计算机可读存储介质
本申请要求于2017年7月10日提交的、申请号为201710558905.4的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本公开涉及人机交互领域，更具体地涉及用于确定视线的方法、设备和计算机可读存储介质。
背景技术
视线估计技术(有时也称为眼球跟踪技术)是一种用于确定人类或动物视线落点的技术。传统的一种视线估计技术使用接的触式的设备来确定视线，例如通过使用特殊设计的隐形眼镜，从而可以通过隐形眼镜随着眼球的动作来判断视线方向。此外，另一类视线估计技术是利用相机来捕获对象的眼部图像，进行眼球特征提取，测量眼球运动情况，进一步估计视线方向或眼睛注视点位置。
发明内容
根据本公开的第一方面，提供了一种用于确定视线的方法。该方法包括：根据眼部图像来确定至少一个视线特征向量；基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置。
例如，所述至少一个视线特征向量包括以下至少一项：指示由第一参考光源在所述眼部图像中形成的第一参考光斑中心到瞳孔中心的第一视线特征向量，指示所述瞳孔中心到由第二参考光源在所述眼部图像中形成的第二参考光斑中心的第二视线特征向量以及指示所述第二参考光斑中心到所述第一参考光斑中心的第三视线特征向量。
例如，根据眼部图像来确定至少一个视线特征向量的步骤包括对所述眼部图像中的瞳孔部分进行椭圆拟合，以确定以下至少一项：所拟合的椭圆的中心，作为瞳孔中心；所拟合的椭圆的长轴；所拟合的椭圆的短轴；以及所述长轴与水平方向之间的旋转角。例如，所述眼部图像是以环形参考光源作为照明来获得的。例如，所述视线估计模型的公式为：
Figure PCTCN2018074537-appb-000001
Figure PCTCN2018074537-appb-000002
其中，a i和b j为模型参数，i＝0…9且j＝0…9，x fix和y fix分别是与所述眼部图像相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标，x AI和y AI分别是所述第一视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，x IB和y IB分别是所述第二视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，以及x BA和y BA分别是所述第三视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量。例如，所述视线估计模型的模型参数是通过最小二乘法来确定的，以及所使用的标定点数目至少为10个。例如，基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置的步骤包括：将当前捕捉到的眼部图像的第一视线特征向量、第二视线特征向量和第三视线特征向量代入已确定过模型参数的所述视线估计模型中，得到相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标。例如，所述方法还包括：对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置。例如，对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置的步骤包括：根据所述眼部图像来确定头动补偿特征向量；根据所述头动补偿特征向量来确定头动补偿值；以及基于所述头动补偿值来调整所确定的视线落点位置，以确定经补偿的视线落点位置。例如，所述头动补偿特征向量包括以下至少一项分量值：指示头部前后运动的第一分量值；指示头部水平运动的第二分量值；以及指示头部旋转运动的第三分量值。例如，所述第一分量值是两个参考光斑中心的欧式距离，所述第二分量值是经椭圆拟合的瞳孔的长短轴之比，以及所述第三分量值是经椭圆拟合的瞳孔的长轴与水平方向之间的旋转角。例如，根据所述头动补偿特征向量来确定头动补偿值的步骤包括：向经训练的基于支持向量回归机的头动补偿模型中输入所述头动补偿特征向量，以确定水平和垂直方向上的相应头动补偿值。
根据本公开的第二方面，提供了一种用于确定视线的设备。该设备包括：视线特征向量确定单元，用于根据眼部图像来确定至少一个视线特征向量；以及视线落点位置确定单元，用于基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置。
例如，所述至少一个视线特征向量包括以下至少一项：指示由第一参考光源在所述眼部图像中形成的第一参考光斑中心到瞳孔中心的第一视线特征向量；指示所述瞳孔中心到由第二参考光源在所述眼部图像中形成的第二参考光斑中心的第二视线特征向量；以及指示所述第二参考光斑中心到所述第一参考光斑中心的第三视线特征向量。
例如，所述视线特征向量确定单元还用于对所述眼部图像中的瞳孔部分进行椭圆拟合，以确定以下至少一项：所拟合的椭圆的中心，作为瞳孔中心；所拟合的椭圆的长轴；所拟合的椭圆的短轴；以及所述长轴与水平方向之间的旋转角。例如，所述视线估计模型的公式为：
Figure PCTCN2018074537-appb-000003
Figure PCTCN2018074537-appb-000004
其中，a i和b j为模型参数，i＝0…9且j＝0…9，x fix和y fix分别是与所述眼部图像相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标，x AI和y AI分别是所述第一视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，x IB和y IB分别是所述第二视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，以及x BA和y BA分别是所述第三视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量。例如，所述视线落点位置确定单元还用于：将当前捕捉到的眼部图像的第一视线特征向量、第二视线特征向量和第三视线特征向量代入已确定过模型参数的所述视线估计模型中，得到相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标。例如，所述设备还包括：头动补偿单元，用于对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置。例如，所述头动补偿单元还用于：根据所述眼部图像来确定头动补偿特征向量；根据所述头动补偿特征向量来确定头动补偿值；以及基于所述头动补偿值来调整所确定的视线落点位置，以确定经补偿的视线落点位置。例如，所述头动补偿特征向量包括以下至少一项分量值：指示头部前后运动的第一分量值；指示头部水平运动的第二分量值；以及指示头部旋转运动的第三分量值。例如，所述第一分量值是两个参考光斑中心的欧式距离，所述第二分量值是经椭圆拟合的瞳孔的长短轴之比，以及所述第三分量值是经椭圆拟合的瞳孔的长轴与水平方向之间的旋转角。例如，所述头动补偿单元还用于：向经训练的基于支持向量回归机 的头动补偿模型中输入所述头动补偿特征向量，以确定水平和垂直方向上的相应头动补偿值。
根据本公开的第三方面，提供了一种用于确定视线的设备。该设备包括：处理器；存储器，存储有指令，所述指令在由所述处理器执行时使得所述处理器：根据眼部图像来确定至少一个视线特征向量；以及基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置。
根据本公开的第四方面，提供了一种存储有指令的计算机可读存储介质，所述指令在由处理器执行时使得所述处理器执行根据本公开第一方面所述的方法。
附图说明
通过下面结合附图说明本公开的优选实施例，将使本公开的上述及其它目的、特征和优点更加清楚，其中：
图1是示出了根据本公开实施例的用于确定视线的技术方案的示例应用场景的示意图。
图2是示出了根据本公开实施例的用于根据眼部图像来确定各个视线特征向量的示例示意图。
图3是示出了根据本公开实施例的用于根据眼部图像来确定头动补偿向量的示例示意图。
图4是示出了根据本公开另一实施例的用于根据眼部图像来确定头动补偿向量的示例示意图。
图5是示出了根据本公开又一实施例的用于根据眼部图像来确定头动补偿向量的示例示意图。
图6是示出了根据本公开实施例的用于确定视线的示例方法的流程图。
图7是示出了根据本公开实施例的用于执行图6所示方法的示例设备的功能框图。
图8是示出了根据本公开实施例的用于确定视线的示例设备的硬件布置图。
具体实施方式
下面参照附图对本公开的部分实施例进行详细说明，在描述过程中省略了对于本公开来说是不必要的细节和功能，以防止对本公开的理解造成混淆。在本说明书中，下述用于描述本公开原理的各种实施例只是说明，不应该以任何方式解释为限制公开的范围。 参照附图的下述描述用于帮助全面理解由权利要求及其等同物限定的本公开的示例性实施例。下述描述包括多种具体细节来帮助理解，但这些细节应认为仅仅是示例性的。因此，本领域普通技术人员应认识到，在不脱离本公开的范围和精神的情况下，可以对本文中描述的实施例进行多种改变和修改。此外，为了清楚和简洁起见，省略了公知功能和结构的描述。此外，贯穿附图，相同的附图标记用于相同或相似的功能和操作。此外，在附图中，各部分并不一定按比例来绘制。换言之，附图中的各部分的相对大小、长度等并不一定与实际比例相对应。
在本公开中，术语“包括”和“含有”及其派生词意为包括而非限制；术语“或”是包含性的，意为和/或。此外，在本公开的以下描述中，所使用的方位术语，例如“上”、“下”、“左”、“右”等均用于指示相对位置关系，以辅助本领域技术人员理解本公开实施例，且因此本领域技术人员应当理解：在一个方向上的“上”/“下”，在相反方向上可变为“下”/“上”，且在另一方向上，可能变为其他位置关系，例如“左”/“右”等。
以下，以本公开应用于人机交互场景为例，对本公开进行了详细描述。但本公开并不局限于此，本公开也可以应用于其它领域，例如增强现实、虚拟现实、用户体验、心理学研究、残疾人辅助、驾驶辅助等领域。此外，尽管下文中以人类用户为例来描述了具体实施例，但本公开不限于此。事实上，也可以针对其它动物或具有类似眼部特征的非生命体应用根据本公开实施例的方案。
传统的视线估计技术主要分为基于二维映射模型的视线估计方法和基于三维眼球模型的视线估计方法。基于二维映射模型的视线估计方法虽然在视线参数提取和视线特征识别方面简单、迅速，能够满足实用性要求，但是其映射模型的精度低、稳定性差，并且使用时需要用户头部静止，难以满足舒适性要求。而基于三维眼球模型的视线估计方法虽然能够检测用户头部空间位置，能够适应用户头部的自然运动，但是其硬件配置复杂(至少需要双相机和双光源)，存在设备硬件成本高和算法实现复杂的问题，并且需要获得用户眼球参数的独立信息，在不借助其他仪器的情况下，准确的间接估计眼球独立参数是很难实现的。
由于视线估计技术中存在的以上问题，使得基于非侵入式的视线估计系统难以得到广泛使用。因此，需要一种相对于现有的视线估计技术能够更易于实现、可满足实用要求，同时无需过高硬件配置和容许头部自然运动的视线估计系统和方法。
图1是示出了根据本公开实施例的用于确定视线的技术方案的示例应用场景10的示意图。如图1所示，应用场景10可以包括用户100、目标屏幕120、第一参考光源110A和第二参考光源110B(当不特别指明时，以下有时可统称为参考光源110)、以及图像传感器120。
大体上，用于确定用户100在目标屏幕130上的注视点135(即，点O)的原理如下。由第一参考光源110A和第二参考光源110B向用户100发射参考光，然后由图像传感器120来捕捉包括用户100的眼部图像在内的参考光反射图像。通过在用户观看目标屏幕130上的多个标定点(有时也称为参考点)时对用户眼部多次执行前述图像捕捉过程，可以获得与每个标定点相对应的眼部图像。进而基于与这些标定点相关的标定数据(包括例如下文中将提到的根据眼部图像所确定的各个视线特征向量等)，可以确定视线估计模型的模型参数，从而实现视线确定校准。然后，可根据实时捕捉到的用户眼部图像来确定其在目标屏幕130上的相应注视点。
尽管在图1中示出了使用两个参考光源110A和110B，然而本公开不限于此。事实上，在其他实施例中，也可以使用单独一个参考光源110或三个或三个以上参考光源110。根据下文中详细描述的实施例，本领域技术人员可以容易地根据两个参考光源的方案来推导出具有其他数目参考光源的方案。
此外，尽管在图1中参考光源是环形参考光源，然而本公开不限于此。事实上，在其他实施例中，也可以使用具有其它形状的参考光源，例如三角形、正方形、矩形、椭圆形、双曲型或任何其他规则或不规则的形状。
此外，为了避免影响用户100观看目标屏幕130，在图1所示实施例中，两个参考光源110都采用了红外光，例如波长为850nm的红外光，然而本公开不限于此。事实上，在其他实施例中，也可以采用其它波长的光波。例如，在一些实施例中，可以采用可见光范围内的近红外光。尽管其属于可见光范围，但是由于接近红外光，人眼在观看时并不会受到明显影响。此外，也可以使用任何其他波长的光波。
此外，尽管在图1中两个参考光源110被分别放置在图像传感器120的左右两侧，然而本公开不限于此。事实上，这两个参考光源110可以放置在图像传感器110的任何相对位置处，只要图像传感器110可以获取到由用户100的眼部所反射的这两个参考光源110的参考光即可。此外，尽管在图1中图像传感器120(例如，高清摄像头)位于目标屏幕130的下部，然而本公开不限于此。事实上，图像传感器120可以位于目标屏幕130的任何恰当位置处，例如，左侧、右侧、上部等等。
接下来，将参考图2并结合图1来详细描述根据本公开实施例的用于确定视线的示例方案。
图2是示出了根据本公开实施例的用于根据眼部图像20来确定各个视线特征向量(例如，向量
Figure PCTCN2018074537-appb-000005
和
Figure PCTCN2018074537-appb-000006
)的示例示意图。如图2所示，在眼部图像20中可以看到用户100眼睛中包括的外部可见的各个部分，包括(但不限于)：瞳孔200、虹膜210和巩膜220。
虹膜210是眼睛中的深色部分，其中间具有开口，即瞳孔200，以供光线进入眼睛内部并在视网膜上被感光细胞所感知并成像。虹膜210负责根据周边环境光的强度来调节瞳孔200的大小，以使得眼睛能够适应不同的环境。例如，在强光环境下，虹膜210舒张，使得瞳孔200收缩，减少进光量；相反在弱光环境下，虹膜210收缩，使得瞳孔200放大，增加进光量。
巩膜220是眼睛中的白色部分，其主要是由弹性纤维等构成的相对坚硬的外壳，负责保护眼球。此外，在瞳孔200、虹膜210和巩膜220的上方实际上还覆盖有角膜，由于其是透明的，因此在眼部图像中并不能直接观察到。然而，在本公开实施例中，由于两个参考光源110所发射的参考光在到达角膜时会发生反射，从而形成反射参考光斑(例如，图2所示的第一参考光斑230和第二参考光斑240)，因此可以表明角膜的存在。
此外，参考光源110发射的光线实际上在角膜的前表面和后表面上都会发生反射，且因此实际上每个参考光源110在角膜上会形成两个光斑。然而由于后表面上形成的光斑亮度显著低于前表面上形成的光斑，需要使用非常高感光度、高解析度的图像传感器才能观察到，因此在图1所示实施例中予以忽略。需要注意的是：本公开实施例同样也适用于对后表面上形成的光斑进行类似操作，本领域技术人员可以根据本文所描述的实施例来容易地导出针对后表面光斑的类似方案，因此为了描述和说明的简洁，此处予以省略。
如图2所示，第一参考光源110A和第二参考光源110B发出的光在角膜上可分别形成第一参考光斑230和第二参考光斑240。这两个参考光斑的中心分别为第一参考光斑中心235(图2所示点A)和第二参考光斑中心245(图2所示点B)。在本实施例中，对光斑中心的确定可以是通过对眼部图像20中检测到的光斑区域进行椭圆或圆形拟合来进行的。从而，可以分别确定A点和B点在眼部图像坐标系中的坐标位置A(x A，y A)和B(x B，y B)。
更具体地，可以通过直方图双峰法进行光斑二值化阈值的确定，对输入预处理后的 眼部图像进行二值化处理，以获得光斑二值化图像。然后可以对光斑二值化图像进行腐蚀和膨胀处理，再利用中值滤波进行二次去噪，得到光斑区域图像。接下来，可以对提取到的光斑区域图像进行连通组元提取，并计算提取到的两个光斑连通组元的质心，分别得到第一反射光斑中心A(x A，y A)和第二反射光斑中心B(x B，y B)。在其他实施例中，也可以采用其它方式来确定上述反射光斑信息。
在确定光斑中心的同时或之前或之后，可以确定瞳孔200在眼部图像20中的位置。例如，可以采用光瞳(light pupil)或暗瞳(dark pupil)技术来确定眼部图像20中的瞳孔200的位置。光瞳是指当参考光源(例如，参考光源110)与图像传感器(例如图像传感器120)在同一光轴上时，由于光线在眼底反射并穿过瞳孔(例如瞳孔200)回到图像传感器，从而使得瞳孔在图像传感器所捕捉到的眼部图像中呈现出明亮的状态。类似地，暗瞳是指当参考光源与图像传感器120不在同一光轴上时，由于光线在眼底反射之后不会通过瞳孔到达图像传感器，从而使得瞳孔在图像传感器所捕捉到的眼部图像中处于黑暗的状态。不论是使用光瞳还是暗瞳，都可以确定瞳孔200在眼部图像20中的位置、范围等。在图1所示实施例中，由于参考光源110和图像传感器120不在同一光轴上，因此采用的是暗瞳技术。然而，本公开不限于此，而是也可以采用光瞳技术。
当获得了眼部图像20时，除了如前所述确定两个参考光斑的中心A点和B点之外，还可以确定瞳孔200的中心205(或点I)。例如，当获得了具有如前所述的暗瞳的眼部图像20时，可以通过图像分割和/或直方图双峰法进行瞳孔二值化阈值的确定，对预处理后的眼部图像进行二值化处理，从而获得瞳孔二值化图像。然后，可以对瞳孔二值化图像进行腐蚀和膨胀处理，再利用中值滤波进行二次去噪，得到瞳孔区域图像。接下来，可以进行边缘检测，得到边缘点，并利用得到的边缘点采用最小二乘法进行瞳孔的椭圆拟合，得到所需瞳孔信息。例如，所需瞳孔信息可以包括以下至少一项：所拟合的椭圆的中心(即，瞳孔中心)I(x I，y I)、所拟合的椭圆的长轴长度r 1、所拟合的椭圆的短轴长度r 2、以及长轴与水平方向的旋转角θ(例如，如图5所示)等。在其他实施例中，也可以采用其它方式来确定上述瞳孔信息。
在图2所示实施例中，尽管椭圆长轴r 1和短轴r 2分别是水平轴和垂直轴，然而本公开不限于此。事实上，由于人类个体的差异，瞳孔虽然大体是圆形，但并不都是横向宽而纵向窄。事实上，椭圆长轴r 1和短轴r 2也可以分别是垂直轴和水平轴。此外，在图2所示实施例中，尽管并未示出椭圆长轴与水平轴之间存在旋转角，但是如图5所示，当头部向一侧偏斜时，眼部图像50以及瞳孔500的长轴r 1′会与水平轴出现夹角，即旋转 角θ。
需要注意的是：为了初步确定视线落点135，并不一定需要上述全部瞳孔信息。例如，在图2所示实施例中，只需要瞳孔中心的坐标信息即可。接下来，将详细描述如何根据第一参考光斑中心235(点A)、第二参考光斑中心245(点B)和瞳孔中心205(点I)的坐标信息来确定各个视线特征向量。
在图2所示实施例中，可以将第一视线特征向量
Figure PCTCN2018074537-appb-000007
确定为第一参考光斑中心A(x A，y A)到瞳孔中心I(x I，y I)的向量，即
Figure PCTCN2018074537-appb-000008
其中x AI＝x I-x A，y AI＝y I-y A。此外，可以将第二视线特征向量
Figure PCTCN2018074537-appb-000009
确定为瞳孔中心I(x I，y I)到第二参考光斑中心B(x B，y B)的向量，即
Figure PCTCN2018074537-appb-000010
其中x IB＝x B-x I，y IB＝y B-y I。此外，可以将第三视线特征向量
Figure PCTCN2018074537-appb-000011
确定为第二参考光斑中心B(x B，y B)到第一参考光斑中心A(x A，y A)的特征向量，即
Figure PCTCN2018074537-appb-000012
其中x BA＝x A-x B，y BA＝y A-y B。由于这三个特征向量满足公式(1)：
Figure PCTCN2018074537-appb-000013
因此，只要知道其中任意两个特征向量，就可以确定第三个特征向量。为了以下描述的方便，依然用全部三个特征向量来说明以下操作。然而本领域技术人员应当理解：事实上可以将这三个特征向量表达为两个特征向量，而不影响技术方案的实现。
之后，如前面结合图1所描述的，可以通过标定点来确定视线估计模型的具体参数。在本实施例中，所建立的视线估计模型如下：
Figure PCTCN2018074537-appb-000014
Figure PCTCN2018074537-appb-000015
其中，(x fix，y fix)为预设标定点在目标屏幕130上的位置坐标，(x AI，y AI)为用户观看相应标定点时如上所述获得的第一视线特征向量
Figure PCTCN2018074537-appb-000016
(x IB，y IB)为用户观看相应标定点时如上所述获得的第二视线特征向量
Figure PCTCN2018074537-appb-000017
为用户观看相应标定点时如上所述获得的第三视线特征向量
Figure PCTCN2018074537-appb-000018
以及a 0～a 9和b 0～b 9是待求解的视线估计模型参数。针对该视线估计模型，可通过最小二乘法对多个标定点的对应数据进行拟合，从而确定a 0～a 9和b 0～b 9，完成标定(也称为校准)。
在图2所示实施例中，可以采用12个标定点对上述视线估计模型进行标定或校准。然而本公开不限于此。在其他实施例中，也可以采用其它恰当数目的标定点来校准。例如，在一些实施例中，可以采用10个或10个以上的标定点。
接下来，当确定了视线估计模型的各个参数(即，视线确定设备或系统经过标定或校准之后)之后，可以进入使用阶段。在该阶段中，图像传感器120可以获取用户100的眼部图像20，并根据获取到的例如前述瞳孔中心位置、各光斑中心位置等来确定相应的视线特征向量(例如，第一、第二和第三视线特征向量)，将这些视线特征向量的值代入已确定过模型参数的上述视线估计模型的公式(2)和(3)中，分别得到相对应的视线落点135在观察对象坐标系(例如，目标屏幕130的坐标系)的X轴和Y轴上的坐标O(x fix，y fix)。
从而，根据如图2所示的实施例，可以提供一种易于实现、可满足实用要求的视线估计方案。其降低了视线估计方案的硬件配置要求，在单相机和双红外光源的硬件条件下便能实现对用户视线的估计。此外，根据本公开实施例的方案中所使用的视线估计模块中构建有上述视线估计模型，其可以提高视线确定方案的精度。
由于人类用户100在观察例如目标屏幕130时，其头部通常会不由自主地发生平移、转动等情况，不太可能完全静止不动，因此用户100实际的注视点与上述视线确定方案所确定的注视点之间存在误差。为了补偿由于头动所造成的该误差，还可以如图3～图5所示实施例一样，对前述方案所得到的坐标O(x fix，y fix)进行补偿。以下，将结合图3～图5来详细描述针对不同类型头动的补偿方案。
图3是示出了根据本公开实施例的用于根据眼部图像来确定头动补偿向量的示例示意图，其中，用户100的头部在垂直于目标屏幕130的方向上前后移动。图4是示出了根据本公开另一实施例的用于根据眼部图像来确定头动补偿向量的示例示意图，其中，用户100的头部在平行于目标屏幕130的平面中水平左右移动。图5是示出了根据本公开又一实施例的用于根据眼部图像来确定头动补偿向量的示例示意图，其中，用户100的头部在平行于目标屏幕130的平面内旋转。然而，需要注意的是，尽管上面图3～图5分别示出了不同类型的头部运动，然而实际情况中可能是这三种类型头部运动中任意一种或任意多种的组合。因此，为了综合考虑这几种状况，可以使用下文中描述的基于支持向量回归机的头动补偿模型。
首先，可以根据眼部图像(例如，眼部图像30、40和/或50)来确定相应的头动补偿向量，然后将该头动补偿向量输入到前述基于支持向量回归机的头动补偿模型中，以确定头动补偿值。然后根据头动补偿值来相应调整前述坐标O(x fix，y fix)。
具体地，如图3所示，由于用户100的头部在垂直于目标屏幕130的方向上前后移动，从而导致在眼部图像30中第一参考光斑中心335(即，点A)和第二参考光斑中心 345(即，点B)之间的欧式距离发生变化。更具体地，在图3所示实施例中，可以看到由于用户100的头部在远离目标屏幕130的方向上移动，因此其第三视线特征向量
Figure PCTCN2018074537-appb-000019
的长度缩短。注意到：虚线圆形为移动前的两个参考光斑的位置，而其中心之间的距离大于移动后的距离。因此，可以将头动补偿向量的指示头部前后运动的第一分量值确定为两个参考光斑中心之间的欧式距离。更具体地，可以将第一分量值L确定为第一参考光斑中心335(即，点A)和第二参考光斑中心345(即，点B)之间的欧式距离：
Figure PCTCN2018074537-appb-000020
此外，如图4所示，由于用户100的头部在平行于目标屏幕130的平面内沿水平方向左右移动，从而导致在眼部图像40中所拟合的椭圆瞳孔的长短轴长度r 1和r 2发生改变。更具体地，在图4所示实施例中，可以看到由于用户100的头部在平行于目标屏幕130的平面内远离目标屏幕130，所拟合的椭圆瞳孔的长短轴发生了变化，使得原来的长轴r 1变为实际上的短轴r 1′。换言之，尽管长短轴的长度可能都缩短，但由于头部在长轴的方向上移动，因此长轴缩短的更为明显，从而使得长短轴之比发生变化。因此，可以将头动补偿向量的指示头部水平运动的第二分量值μ确定为能够表征头部在平行于目标屏幕130的平面内的水平方向的左右运动的椭圆瞳孔的长轴r 1和短轴r 2的比值μ＝r 1/r 2。
此外，如图5所示，由于用户100的头部在平行于目标屏幕130的平面内旋转，从而导致在眼部图像50中所拟合的椭圆瞳孔长轴r 1与水平轴之间发生旋转，形成旋转角。因此，可以将第三分量值θ确定为能够表征头部在与目标屏幕130平行的平面中旋转运动的椭圆的长轴r 1与水平方向之间的旋转角θ。
上述各个分量值可以是全部或部分根据结合图2中所确定的瞳孔信息来得到的。因此，在实际应用中，在针对头动补偿的处理中，可以无需再次计算这些分量值。
在确定了头动补偿向量C gaze＝(L，μ，θ)之后，可以将其输入到训练好的基于支持向量回归机的头动补偿模型中，以得到水平和竖直方向上的头动补偿值(x horizontal，y vertical)。然后，根据计算出的头动补偿值(x horizontal，y vertical)，可以调整前述确定的视线落点的位置。更具体地，可以将计算出的头动补偿值(x horizontal，y vertical)与前述确定的初步视线落点O(x fix，y fix)进行求和运算，从而得到用户100在目标屏幕130上的最终视线落点F(X fix，Y fix)，其中X fix＝x fix+x horizontal且Y fix＝y fix+y horizontal。
因此，通过使用基于支持向量回归机算法的头动补偿模型，可以容易地补偿用户头部运动带来的误差，增强了视线估计系统和方法的抗头动干扰能力，容许系统使用时用 户头部自然运动，降低了以往视线估计系统使用时要求用户头部静止的限制，提高了视线估计系统使用时的舒适性和自然性。
图6是示出了根据本公开实施例的用于确定视线的示例方法600的流程图。如图6所示，方法600可以包括步骤S610和S620。根据本公开，方法600的一些步骤可以单独执行或组合执行，以及可以并行执行或顺序执行，并不局限于图6所示的具体操作顺序。在一些实施例中，方法600可以由图7所示的设备700或图8所示的设备800执行。
图7是示出了根据本公开实施例的用于确定视线的示例设备700的框图。如图7所示，设备700可以包括：视线特征向量确定单元710和视线落点位置确定单元720。
视线特征向量确定单元710可以用于根据眼部图像来确定至少一个视线特征向量。视线特征向量确定单元710可以是设备700的中央处理单元(CPU)、数字信号处理器(DSP)、微处理器、微控制器等等，其可以与设备700的图像传感器(例如，红外相机、可见光相机、摄像头等)和/或通信部分(例如，以太网卡、WiFi芯片、RF芯片等)相配合，根据通过图像传感器捕捉到的眼部图像或者通过通信部分从远程设备接收到的眼部图像来确定至少一个视线特征向量。
视线落点位置确定单元720可以用于基于视线估计模型和至少一个视线特征向量来确定视线落点的位置。视线落点位置确定单元720也可以是设备700的中央处理单元(CPU)、数字信号处理器(DSP)、微处理器、微控制器等等，其可以基于预先训练的和/或实时训练的视线估计模型以及由视线特征向量确定单元710所确定的至少一个视线特征向量来确定视线落点的位置。
此外，设备700还可以包括图7中未示出的其他单元，例如头动补偿单元等。在一些实施例中，头动补偿单元可以用于对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置。在一些实施例中，头动补偿单元还可以用于根据眼部图像来确定头动补偿特征向量；根据头动补偿特征向量来确定头动补偿值；以及基于头动补偿值来调整所确定的视线落点位置，以确定经补偿的视线落点位置。在一些实施例中，头动补偿单元还可以用于：向经训练的基于支持向量回归机的头动补偿模型中输入头动补偿特征向量，以确定水平和垂直方向上的相应头动补偿值。
此外，设备700还可以包括图7中未示出的其他功能单元，例如：总线、存储器、电源、天线、通信部分、存储部分。然而，它们并不影响对本申请的原理的理解，且因此此处省略对它们的详细描述。
以下将结合图6和图7，对根据本公开实施例的在设备700上执行的用于确定视线 的方法600和设备700进行详细的描述。
方法600开始于步骤S610，在步骤S610中，可以由设备700的视线特征向量确定单元710根据眼部图像来确定至少一个视线特征向量。
在步骤S620中，可以由设备700的视线落点位置确定单元720基于视线估计模型和至少一个视线特征向量来确定视线落点的位置。
在一些实施例中，至少一个视线特征向量包括以下至少一项：指示由第一参考光源在眼部图像中形成的第一参考光斑中心到瞳孔中心的第一视线特征向量，其中，x AI和y AI分别是第一视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量；指示瞳孔中心到由第二参考光源在眼部图像中形成的第二参考光斑中心的第二视线特征向量，其中，x IB和y IB分别是第二视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量；以及指示第二参考光斑中心到第一参考光斑中心的第三视线特征向量，其中，x BA和y BA分别是第三视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量。
在一些实施例中，根据眼部图像来确定至少一个视线特征向量的步骤包括对眼部图像中的瞳孔部分进行椭圆拟合，以确定以下至少一项：所拟合的椭圆的中心，作为瞳孔中心；所拟合的椭圆的长轴；所拟合的椭圆的短轴；以及长轴与水平方向之间的旋转角。在一些实施例中，眼部图像是以环形参考光源作为照明来获得的。在一些实施例中，视线估计模型的公式为：
Figure PCTCN2018074537-appb-000021
Figure PCTCN2018074537-appb-000022
其中，a i和b j为模型参数，i＝0…9且j＝0…9，以及x fix和y fix分别是与眼部图像相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标。在一些实施例中，视线估计模型的模型参数是通过最小二乘法来确定的，以及所使用的标定点数目至少为10个。在一些实施例中，基于视线估计模型和至少一个视线特征向量来确定视线落点的位置的步骤包括：将当前捕捉到的眼部图像的第一视线特征向量、第二视线特征向量和第三视线特征向量代入已确定过模型参数的视线估计模型中，得到相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标。在一些实施例中，方法还包括：对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置。在一些实施例中，对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置的步骤包括：根据眼部图像来确定头动补 偿特征向量；根据头动补偿特征向量来确定头动补偿值；以及基于头动补偿值来调整所确定的视线落点位置，以确定经补偿的视线落点位置。在一些实施例中，头动补偿特征向量包括以下至少一项分量值：指示头部前后运动的第一分量值；指示头部水平运动的第二分量值；以及指示头部旋转运动的第三分量值。在一些实施例中，第一分量值是两个参考光斑中心的欧式距离，第二分量值是经椭圆拟合的瞳孔的长短轴之比，以及第三分量值是经椭圆拟合的瞳孔的长轴与水平方向之间的旋转角。在一些实施例中，根据头动补偿特征向量来确定头动补偿值的步骤包括：向经训练的基于支持向量回归机的头动补偿模型中输入头动补偿特征向量，以确定水平和垂直方向上的相应头动补偿值。
图8是示出了根据本公开实施例的图7所示设备700的示例硬件布置800的框图。硬件布置800包括处理器806(例如，数字信号处理器(DSP))。处理器806可以是用于执行本文描述的流程的不同动作的单一处理单元或者是多个处理单元。布置800还可以包括用于从其他实体接收信号的输入单元802、以及用于向其他实体提供信号的输出单元804。输入单元802和输出单元804可以被布置为单一实体或者是分离的实体。
此外，布置800可以包括具有非易失性或易失性存储器形式的至少一个可读存储介质808，例如是电可擦除可编程只读存储器(EEPROM)、闪存、和/或硬盘驱动器。可读存储介质808包括计算机程序810，该计算机程序810包括代码/计算机可读指令，其在由布置800中的处理器806执行时使得硬件布置800和/或包括硬件布置800在内的设备700可以执行例如上面结合图1～6所描述的流程及其任何变形。
计算机程序810可被配置为具有例如计算机程序模块810A～810B架构的计算机程序代码。因此，在例如设备700中使用硬件布置800时的示例实施例中，布置800的计算机程序中的代码包括：模块810A，用于根据眼部图像来确定至少一个视线特征向量。计算机程序中的代码还包括：模块810B，用于基于视线估计模型和至少一个视线特征向量来确定视线落点的位置。
计算机程序模块实质上可以执行图1～6中所示出的流程中的各个动作，以模拟设备700。换言之，当在处理器806中执行不同计算机程序模块时，它们可以对应于设备700中的上述不同单元。
尽管上面结合图8所公开的实施例中的代码手段被实现为计算机程序模块，其在处理器806中执行时使得硬件布置800执行上面结合图1～6所描述的动作，然而在备选实施例中，该代码手段中的至少一项可以至少被部分地实现为硬件电路。
处理器可以是单个CPU(中央处理单元)，但也可以包括两个或更多个处理单元。例如，处理器可以包括通用微处理器、指令集处理器和/或相关芯片组和/或专用微处理器(例如，专用集成电路(ASIC))。处理器还可以包括用于缓存用途的板载存储器。计算机程序可以由连接到处理器的计算机程序产品来承载。计算机程序产品可以包括其上存储有计算机程序的计算机可读介质。例如，计算机程序产品可以是闪存、随机存取存储器(RAM)、只读存储器(ROM)、EEPROM，且上述计算机程序模块在备选实施例中可以用UE内的存储器的形式被分布到不同计算机程序产品中。
至此已经结合优选实施例对本公开进行了描述。应该理解，本领域技术人员在不脱离本公开的精神和范围的情况下，可以进行各种其它的改变、替换和添加。因此，本公开的范围不局限于上述特定实施例，而应由所附权利要求所限定。
此外，在本文中被描述为通过纯硬件、纯软件和/或固件来实现的功能，也可以通过专用硬件、通用硬件与软件的结合等方式来实现。例如，被描述为通过专用硬件(例如，现场可编程门阵列(FPGA)、专用集成电路(ASIC)等)来实现的功能，可以由通用硬件(例如，中央处理单元(CPU)、数字信号处理器(DSP))与软件的结合的方式来实现，反之亦然。

Claims (22)

  1. 一种用于确定视线的方法，包括：
    根据眼部图像来确定至少一个视线特征向量；
    基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置。
  2. 根据权利要求1所述的方法，其中，所述至少一个视线特征向量包括以下至少一项：
    指示由第一参考光源在所述眼部图像中形成的第一参考光斑中心到瞳孔中心的第一视线特征向量；
    指示所述瞳孔中心到由第二参考光源在所述眼部图像中形成的第二参考光斑中心的第二视线特征向量；以及
    指示所述第二参考光斑中心到所述第一参考光斑中心的第三视线特征向量。
  3. 根据权利要求1所述的方法，其中，根据眼部图像来确定至少一个视线特征向量的步骤包括对所述眼部图像中的瞳孔部分进行椭圆拟合，以确定以下至少一项：
    所拟合的椭圆的中心，作为瞳孔中心；
    所拟合的椭圆的长轴；
    所拟合的椭圆的短轴；以及
    所述长轴与水平方向之间的旋转角。
  4. 根据权利要求1所述的方法，其中，所述眼部图像是以环形参考光源作为照明来获得的。
  5. 根据权利要求2所述的方法，其中，所述视线估计模型的公式为：
    Figure PCTCN2018074537-appb-100001
    Figure PCTCN2018074537-appb-100002
    其中，a i和b j为模型参数，i＝0…9且j＝0…9，以及x fix和y fix分别是与所述眼部图像相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标；x AI和y AI分别是所述第一视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，x IB和y IB分别是所述第二视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，x BA和y BA分别是所述第三视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量。
  6. 根据权利要求5所述的方法，其中，所述视线估计模型的模型参数是通过最小二乘法来确定的，以及所使用的标定点数目至少为10个。
  7. 根据权利要求5所述的方法，其中，基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置的步骤包括：
    将当前捕捉到的眼部图像的第一视线特征向量、第二视线特征向量和第三视线特征向量代入已确定过模型参数的所述视线估计模型中，得到相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标。
  8. 根据权利要求1所述的方法，还包括：
    对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置。
  9. 根据权利要求8所述的方法，其中，对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置的步骤包括：
    根据所述眼部图像来确定头动补偿特征向量；
    根据所述头动补偿特征向量来确定头动补偿值；以及
    基于所述头动补偿值来调整所确定的视线落点位置，以确定经补偿的视线落点位置。
  10. 根据权利要求9所述的方法，其中，所述头动补偿特征向量包括以下至少一项分量值：
    指示头部前后运动的第一分量值；
    指示头部水平运动的第二分量值；以及
    指示头部旋转运动的第三分量值。
  11. 根据权利要求10所述的方法，其中，所述第一分量值是两个参考光斑中心的欧式距离，所述第二分量值是经椭圆拟合的瞳孔的长短轴之比，以及所述第三分量值是经椭圆拟合的瞳孔的长轴与水平方向之间的旋转角。
  12. 根据权利要求10所述的方法，其中，根据所述头动补偿特征向量来确定头动补偿值的步骤包括：
    向经训练的基于支持向量回归机的头动补偿模型中输入所述头动补偿特征向量，以确定水平和垂直方向上的相应头动补偿值。
  13. 一种用于确定视线的设备，包括：
    视线特征向量确定单元，用于根据眼部图像来确定至少一个视线特征向量；以及
    视线落点位置确定单元，用于基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置。
  14. 根据权利要求13所述的设备，其中，所述至少一个视线特征向量包括以下至少一项：
    指示由第一参考光源在所述眼部图像中形成的第一参考光斑中心到瞳孔中心的第一视线特征向量；
    指示所述瞳孔中心到由第二参考光源在所述眼部图像中形成的第二参考光斑中心的第二视线特征向量；以及
    指示所述第二参考光斑中心到所述第一参考光斑中心的第三视线特征向量。
  15. 根据权利要求13所述的设备，其中，所述视线特征向量确定单元还用于对所述眼部图像中的瞳孔部分进行椭圆拟合，以确定以下至少一项：所拟合的椭圆的中心，作为瞳孔中心；所拟合的椭圆的长轴；所拟合的椭圆的短轴；以及所述长轴与水平方向之间的旋转角。
  16. 根据权利要求13所述的设备，其中，所述视线估计模型的公式为：
    Figure PCTCN2018074537-appb-100003
    Figure PCTCN2018074537-appb-100004
    其中，a i和b j为模型参数，i＝0…9且j＝0…9，以及x fix和y fix分别是与所述眼部图像相对应的视线落点在观察对象坐标系的X轴和Y轴上的坐标；x AI和y AI分别是所述第一视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，x IB和y IB分别是所述第二视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量，x BA和y BA分别是所述第三视线特征向量在眼部图像坐标系的X轴和Y轴方向的分量。
  17. 根据权利要求13所述的设备，还包括：头动补偿单元，用于对所确定的视线落点位置进行头动补偿以确定经补偿的视线落点位置。
  18. 根据权利要求17所述的设备，其中，所述头动补偿单元还用于：根据所述眼部图像来确定头动补偿特征向量；根据所述头动补偿特征向量来确定头动补偿值；以及基于所述头动补偿值来调整所确定的视线落点位置，以确定经补偿的视线落点位置。
  19. 根据权利要求18所述的设备，其中，所述头动补偿特征向量包括以下至少一项分量值：指示头部前后运动的第一分量值；指示头部水平运动的第二分量值；以及指示 头部旋转运动的第三分量值。
  20. 根据权利要求19所述的设备，其中，所述第一分量值是两个参考光斑中心的欧式距离，所述第二分量值是经椭圆拟合的瞳孔的长短轴之比，以及所述第三分量值是经椭圆拟合的瞳孔的长轴与水平方向之间的旋转角。
  21. 一种用于确定视线的设备，包括：
    处理器；
    存储器，存储有指令，所述指令在由所述处理器执行时使得所述处理器：
    根据眼部图像来确定至少一个视线特征向量；以及
    基于视线估计模型和所述至少一个视线特征向量来确定视线落点的位置。
  22. 一种存储有指令的计算机可读存储介质，所述指令在由处理器执行时使得所述处理器执行根据权利要求1～12中任一项所述的方法。
PCT/CN2018/074537 2017-07-10 2018-01-30 用于确定视线的方法、设备和计算机可读存储介质 WO2019010959A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/305,882 US11294455B2 (en) 2017-07-10 2018-01-30 Method and device for determining gaze placement, computer readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710558905.4A CN107357429B (zh) 2017-07-10 2017-07-10 用于确定视线的方法、设备和计算机可读存储介质
CN201710558905.4 2017-07-10

Publications (1)

Publication Number Publication Date
WO2019010959A1 true WO2019010959A1 (zh) 2019-01-17

Family

ID=60293404

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/074537 WO2019010959A1 (zh) 2017-07-10 2018-01-30 用于确定视线的方法、设备和计算机可读存储介质

Country Status (3)

Country Link
US (1) US11294455B2 (zh)
CN (1) CN107357429B (zh)
WO (1) WO2019010959A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11112602B2 (en) * 2018-11-14 2021-09-07 Beijing 7Invensun Technology Co., Ltd. Method, apparatus and system for determining line of sight, and wearable eye movement device

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107357429B (zh) * 2017-07-10 2020-04-07 京东方科技集团股份有限公司 用于确定视线的方法、设备和计算机可读存储介质
CN108334810B (zh) 2017-12-25 2020-12-11 北京七鑫易维信息技术有限公司 视线追踪设备中确定参数的方法和装置
CN108510542B (zh) * 2018-02-12 2020-09-11 北京七鑫易维信息技术有限公司 匹配光源与光斑的方法和装置
CN108478184A (zh) * 2018-04-26 2018-09-04 京东方科技集团股份有限公司 基于vr的视力测量方法及装置、vr设备
JP7180209B2 (ja) * 2018-08-30 2022-11-30 日本電信電話株式会社 眼情報推定装置、眼情報推定方法、プログラム
CN109645956B (zh) * 2018-12-25 2021-08-06 重庆远视科技有限公司 眼睛屈光度测量装置
US11156829B2 (en) * 2019-07-29 2021-10-26 Facebook Technologies, Llc Pupil expander cailibration
CN111208905A (zh) * 2020-01-08 2020-05-29 北京未动科技有限公司 一种多模组视线追踪方法、系统和视线追踪设备
CN113448428B (zh) * 2020-03-24 2023-04-25 中移(成都)信息通信科技有限公司 一种视线焦点的预测方法、装置、设备及计算机存储介质
US20230333642A1 (en) * 2022-03-28 2023-10-19 Apple Inc. Calibrating a Gaze Tracker
CN117666706A (zh) * 2022-08-22 2024-03-08 北京七鑫易维信息技术有限公司 一种电子设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN201477518U (zh) * 2009-08-31 2010-05-19 北京科技大学 一种基于瞳孔角膜反射方法的视线追踪装置
CN101866215A (zh) * 2010-04-20 2010-10-20 复旦大学 在视频监控中采用视线跟踪的人机交互装置和方法
CN102125422A (zh) * 2010-01-12 2011-07-20 北京科技大学 视线追踪系统中基于瞳孔-角膜反射的视线估计方法
CN103761519A (zh) * 2013-12-20 2014-04-30 哈尔滨工业大学深圳研究生院 一种基于自适应校准的非接触式视线追踪方法
CN104951084A (zh) * 2015-07-30 2015-09-30 京东方科技集团股份有限公司 视线追踪方法及装置
CN107357429A (zh) * 2017-07-10 2017-11-17 京东方科技集团股份有限公司 用于确定视线的方法、设备和计算机可读存储介质

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5231674A (en) * 1989-06-09 1993-07-27 Lc Technologies, Inc. Eye tracking method and apparatus
US7306337B2 (en) * 2003-03-06 2007-12-11 Rensselaer Polytechnic Institute Calibration-free gaze tracking under natural head movement
CA2545202C (en) * 2003-11-14 2014-01-14 Queen's University At Kingston Method and apparatus for calibration-free eye tracking
US7963652B2 (en) * 2003-11-14 2011-06-21 Queen's University At Kingston Method and apparatus for calibration-free eye tracking
WO2006108017A2 (en) * 2005-04-04 2006-10-12 Lc Technologies, Inc. Explicit raytracing for gimbal-based gazepoint trackers
US8077914B1 (en) * 2006-08-07 2011-12-13 Arkady Kaplan Optical tracking apparatus using six degrees of freedom
US8457352B2 (en) * 2007-05-23 2013-06-04 The University Of British Columbia Methods and apparatus for estimating point-of-gaze in three dimensions
EP2238889B1 (en) * 2009-04-01 2011-10-12 Tobii Technology AB Adaptive camera and illuminator eyetracker
WO2012020760A1 (ja) * 2010-08-09 2012-02-16 国立大学法人静岡大学 注視点検出方法及び注視点検出装置
US9329683B2 (en) * 2010-12-08 2016-05-03 National University Corporation Shizuoka University Method for detecting point of gaze and device for detecting point of gaze
US8879801B2 (en) * 2011-10-03 2014-11-04 Qualcomm Incorporated Image-based head position tracking method and system
US8971570B1 (en) * 2011-11-04 2015-03-03 Google Inc. Dual LED usage for glint detection
US8824779B1 (en) * 2011-12-20 2014-09-02 Christopher Charles Smyth Apparatus and method for determining eye gaze from stereo-optic views
US8913789B1 (en) * 2012-01-06 2014-12-16 Google Inc. Input methods and systems for eye positioning using plural glints
JP5689850B2 (ja) * 2012-05-25 2015-03-25 株式会社ソニー・コンピュータエンタテインメント 映像解析装置、映像解析方法、および注視点表示システム
CN102930252B (zh) * 2012-10-26 2016-05-11 广东百泰科技有限公司 一种基于神经网络头部运动补偿的视线跟踪方法
CN103809737A (zh) * 2012-11-13 2014-05-21 华为技术有限公司 一种人机交互方法及装置
US11389059B2 (en) * 2013-01-25 2022-07-19 Wesley W. O. Krueger Ocular-performance-based head impact measurement using a faceguard
US9370302B2 (en) * 2014-07-08 2016-06-21 Wesley W. O. Krueger System and method for the measurement of vestibulo-ocular reflex to improve human performance in an occupational environment
US9244529B2 (en) * 2013-01-27 2016-01-26 Dmitri Model Point-of-gaze estimation robust to head rotations and/or device rotations
EP2975997B1 (en) * 2013-03-18 2023-07-12 Mirametrix Inc. System and method for on-axis eye gaze tracking
KR101539923B1 (ko) * 2013-12-27 2015-07-29 가톨릭대학교 산학협력단 이중 기계학습 구조를 이용한 생체신호 기반의 안구이동추적 시스템 및 이를 이용한 안구이동추적 방법
US9684827B2 (en) * 2014-03-26 2017-06-20 Microsoft Technology Licensing, Llc Eye gaze tracking based upon adaptive homography mapping
JP6583734B2 (ja) * 2014-08-22 2019-10-02 国立大学法人静岡大学 角膜反射位置推定システム、角膜反射位置推定方法、角膜反射位置推定プログラム、瞳孔検出システム、瞳孔検出方法、瞳孔検出プログラム、視線検出システム、視線検出方法、視線検出プログラム、顔姿勢検出システム、顔姿勢検出方法、および顔姿勢検出プログラム
CN105138965B (zh) * 2015-07-31 2018-06-19 东南大学 一种近眼式视线跟踪方法及其系统
CN109690553A (zh) * 2016-06-29 2019-04-26 醒眸行有限公司 执行眼睛注视跟踪的系统和方法
WO2018030515A1 (ja) * 2016-08-12 2018-02-15 国立大学法人静岡大学 視線検出装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN201477518U (zh) * 2009-08-31 2010-05-19 北京科技大学 一种基于瞳孔角膜反射方法的视线追踪装置
CN102125422A (zh) * 2010-01-12 2011-07-20 北京科技大学 视线追踪系统中基于瞳孔-角膜反射的视线估计方法
CN101866215A (zh) * 2010-04-20 2010-10-20 复旦大学 在视频监控中采用视线跟踪的人机交互装置和方法
CN103761519A (zh) * 2013-12-20 2014-04-30 哈尔滨工业大学深圳研究生院 一种基于自适应校准的非接触式视线追踪方法
CN104951084A (zh) * 2015-07-30 2015-09-30 京东方科技集团股份有限公司 视线追踪方法及装置
CN107357429A (zh) * 2017-07-10 2017-11-17 京东方科技集团股份有限公司 用于确定视线的方法、设备和计算机可读存储介质

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11112602B2 (en) * 2018-11-14 2021-09-07 Beijing 7Invensun Technology Co., Ltd. Method, apparatus and system for determining line of sight, and wearable eye movement device

Also Published As

Publication number Publication date
US20210223859A1 (en) 2021-07-22
CN107357429A (zh) 2017-11-17
US11294455B2 (en) 2022-04-05
CN107357429B (zh) 2020-04-07

Similar Documents

Publication Publication Date Title
US11294455B2 (en) Method and device for determining gaze placement, computer readable storage medium
US10878237B2 (en) Systems and methods for performing eye gaze tracking
US9398848B2 (en) Eye gaze tracking
WO2017211066A1 (zh) 基于虹膜与瞳孔的用于头戴式设备的视线估计方法
JP6800091B2 (ja) 視線計測装置及びプログラム
CN104809424B (zh) 一种基于虹膜特征实现视线追踪的方法
US6659611B2 (en) System and method for eye gaze tracking using corneal image mapping
KR100949743B1 (ko) 고글 형태를 갖는 착용형 시선 추적 장치 및 방법
US20150029322A1 (en) Method and computations for calculating an optical axis vector of an imaged eye
US20220100268A1 (en) Eye tracking device and a method thereof
WO2015051606A1 (zh) 定位方法及定位系统
JP7030317B2 (ja) 瞳孔検出装置及び瞳孔検出方法
JP6631951B2 (ja) 視線検出装置及び視線検出方法
JP2016173313A (ja) 視線方向推定システム、視線方向推定方法及び視線方向推定プログラム
WO2015051605A1 (zh) 图像采集定位方法及图像采集定位装置
US20220207919A1 (en) Methods, devices and systems for determining eye parameters
US10866635B2 (en) Systems and methods for capturing training data for a gaze estimation model
WO2020141537A1 (en) System and method for eye tracking
JP3711053B2 (ja) 視線測定装置及びその方法と、視線測定プログラム及びそのプログラムを記録した記録媒体
US11751764B2 (en) Measuring a posterior corneal surface of an eye
JP2016095584A (ja) 瞳孔検出装置、瞳孔検出方法及び瞳孔検出プログラム
JP2015123262A (ja) 角膜表面反射画像を利用した視線計測方法及びその装置
CN113616153A (zh) 一种眼角膜厚度的测量方法、装置、介质及电子设备
Meriläinen Adapting a gaze tracking system to mobile environments
CN112528713A (zh) 一种注视点估计方法、系统、处理器及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18831192

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: 1205A 19.05.2020

122 Ep: pct application non-entry in european phase

Ref document number: 18831192

Country of ref document: EP

Kind code of ref document: A1