WO2020042541A1 - 眼球追踪交互方法和装置 - Google Patents

眼球追踪交互方法和装置 Download PDF

Info

Publication number
WO2020042541A1
WO2020042541A1 PCT/CN2019/073763 CN2019073763W WO2020042541A1 WO 2020042541 A1 WO2020042541 A1 WO 2020042541A1 CN 2019073763 W CN2019073763 W CN 2019073763W WO 2020042541 A1 WO2020042541 A1 WO 2020042541A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
image
eye
eyeball
human eye
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/073763
Other languages
English (en)
French (fr)
Inventor
蒋壮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Water World Co Ltd
Original Assignee
Shenzhen Water World Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Water World Co Ltd filed Critical Shenzhen Water World Co Ltd
Publication of WO2020042541A1 publication Critical patent/WO2020042541A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/19—Sensors therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/193—Preprocessing; Feature extraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • G06V40/197—Matching; Classification

Definitions

  • the present application relates to the field of human-computer interaction technology, and in particular, to an eye tracking interaction method and device.
  • the eye movement control method is a non-contact human-computer interaction method.
  • the position of the eye's fixation point is calculated by tracking the position of the eyeball.
  • Eye movement control is a great help for users who ca n’t use both hands.
  • gaming computers with eye tracking capabilities make players more immersive in the game scene.
  • eye tracking technology requires special equipment, such as an eye tracker.
  • users need to control the device according to the eye movement methods defined in the instruction manual. Users cannot control the device according to their eye movement habits, and the user experience is not high.
  • the trend of human-computer interaction is human-centered, more friendly and convenient, so eye tracking is also moving towards controlling the device according to the user's eye movement habits.
  • the purpose of this application is to provide an eye-tracking interaction method and device, which aims to solve the problem that in the prior art, eye movement control requires special equipment and cannot achieve accurate gaze positioning according to the user's eye movement habits.
  • the present application proposes an eye-tracking interaction method, which includes: obtaining a user image of a user looking at a specified viewing area through a camera; searching for a human eye image and an eyeball image from the user image, and obtaining human eye position data and eyeball position data; Calculate feature data of the human eye position data and the eyeball position data; and calculate, based on preset calibration data and the feature data, the feature points corresponding to the feature data that the user looks at in the designated viewing area Coordinates; wherein the preset calibration data is calibration data of a plurality of positioning points in a designated viewing area.
  • the present application also proposes an eye-tracking interaction device, including: an image acquisition module for acquiring a user image of a user looking at a specified viewing area through a camera; an image analysis module for finding a human eye image from the user image and Eyeball image to obtain human eye position data and eyeball position data; a data calculation module for calculating feature data based on the human eye position data and the eyeball position data; a line of sight positioning module for obtaining preset calibration data and all The feature data is used to calculate the coordinates of the feature point that the user looks at corresponding to the feature data in the designated viewing area; wherein the preset calibration data is calibration data of a plurality of positioning points in the designated viewing area.
  • the eyeball tracking interaction method and device of the present application collect user images through a common camera, find human eyes and eyeballs from user images, calculate human eye positions and feature data of eyeball positions, and according to the feature data and preset calibration data The calculation is performed to obtain the coordinates of the user's line of sight corresponding to the feature data in the designated viewing area, thereby achieving the line of sight positioning.
  • the feature data and calibration data of this application are collected according to the user's eye movement habits, and the human-computer interaction mode is friendly, easy to implement, and requires no additional equipment, and the cost is low.
  • FIG. 1 is a schematic flowchart of an eye tracking interaction method according to an embodiment of the present application.
  • Figure 2a of this application is a schematic diagram of each positioning point
  • Figure 2b is a schematic diagram of the division of the left region and the right region
  • Figure 2c is a schematic diagram of the division of the upper region and the lower region of the present application
  • FIG. 3 is a schematic block diagram of a structure of an eye-tracking interactive device according to an embodiment of the present application.
  • FIG. 4 is a schematic block diagram of a structure of a data calculation module in FIG. 3;
  • FIG. 5 is a schematic block diagram of a structure of an eye-tracking interactive device according to another embodiment of the present application.
  • FIG. 6 is a schematic block diagram of the structure of the line-of-sight positioning module in FIG. 5;
  • FIG. 7 is a schematic block diagram of a structure of a position preliminary judgment unit in FIG. 6;
  • FIG. 7 is a schematic block diagram of a structure of a position preliminary judgment unit in FIG. 6;
  • FIG. 8 is a schematic structural diagram of a computer device according to an embodiment of the present application.
  • an embodiment of the present application provides an eye tracking interaction method, including:
  • the designated viewing area includes a terminal device interface for user-machine interaction, such as a smartphone display, a flat panel display, a smart TV display, a personal computer display, a laptop display, etc .; the camera includes The front camera and external camera of the terminal device, such as the front camera of a mobile phone.
  • the user looks at the characteristic points of the mobile phone display at an appropriate distance from the mobile phone display according to his habit, and collects human eyes through the front camera of the mobile phone.
  • An image of the feature point Specifically, a camera can be used to continuously collect images in real time, and distinguish the state of the human eye through a pre-trained classifier.
  • the preset states include gaze, eye movement, single eye blink, double eye blink, multiple blinks, etc. When any of the above states, user images are collected in real time. This application does not specifically limit the types of classifiers involved.
  • step S2 of this embodiment in order to improve the search efficiency and accuracy of the human eye image and eyeball image search, first search the human face image from the user image; then search the human eye image from the human face image, and according to The human eye image acquires human eye position data; finally, an eyeball image is searched from the human eye image, and eyeball position data is obtained according to the eyeball image.
  • Face image From the image. If no face image is found in the image, return to the step of obtaining the user image and adjust the relative position of the user and the specified viewing area until the face image can be found in the user image obtained by the camera .
  • face rules such as the distribution of eyes, nose, mouth, etc.
  • features that are invariant to the face such as skin color, edges, textures
  • face detection on the input image describe the facial features of the face with a standard face template.
  • the face detection When performing face detection, first calculate the correlation value between the input image and the standard face template, and then The obtained correlation value is compared with a preset threshold value to determine whether a face exists in the input image; the face area is regarded as a type of pattern, and a large amount of face data is used as a sample training to learn potential rules
  • a classifier is constructed to detect faces by discriminating all possible region pattern attributes in the image.
  • the found face image is marked with a rectangular frame.
  • Human eye search methods include template-based methods, statistics-based methods, and knowledge-based methods.
  • the method based on template matching includes a gray projection template and a geometric feature template.
  • the gray projection method refers to the horizontal and vertical projection of a gray image of a human face, and respectively counts the gray value and / or in two directions. The value of the gray function, find specific change points, and then combine the positions of change points in different directions according to prior knowledge to obtain the position of the human eye; the geometric feature template is implemented using the individual features and distribution features of the eyes as the basis Human eye detection.
  • Statistics-based methods generally train and learn a large number of target samples and non-target samples to obtain a set of model parameters, and then build a classifier or filter to detect the target based on the model.
  • the knowledge-based method is to determine the application environment of the image, summarize the knowledge (such as contour information, color information, position information) that can be used for human eye detection under specific conditions, and summarize them into rules that guide human eye detection.
  • This embodiment uses a rectangular frame to frame the left-eye image and the right-eye image, respectively, to obtain the following human eye position data, including:
  • r 1 the distance from the upper left vertex of the rectangular frame of the left-eye image to the left-most face image
  • t 1 the distance from the upper left vertex of the rectangular frame of the left eye image to the uppermost edge of the face image
  • w 1 the width of the rectangular frame of the left-eye image
  • h 1 the height of the rectangular frame of the left-eye image
  • r 2 the distance from the upper-left vertex of the rectangular frame of the right-eye image to the left-most face image
  • t 2 the distance from the top left vertex of the rectangular frame of the right eye image to the uppermost edge of the face image
  • w 2 the width of the rectangular frame of the right-eye image
  • h 2 the height of the rectangular frame of the right-eye image.
  • Finding an eyeball image from a human eye image includes finding a left eyeball image from a left eye image, and finding a right eyeball image from a right eye image. If no eyeball image is found, return to the step of obtaining a user image, and reacquire the user image until the eyeball image can be found in this step.
  • Eyeball search methods include neural network method, extreme point position discrimination method of edge point integral projection curve, template matching method, multi-resolution mosaic map method, geometric and symmetry detection method, and Hough transform-based method.
  • the left eyeball image and the right eyeball image are found, and the left eyeball image and the right eyeball image are respectively framed by rectangular frames, and the following eyeball position data is obtained, including:
  • r 3 the distance between the top left vertex of the rectangular frame of the left eyeball image and the leftmost face of the face image
  • t 3 the distance from the top left vertex of the rectangular frame of the left eyeball image to the uppermost edge of the face image
  • w 3 the width of the rectangular frame of the left eyeball image
  • h 3 the height of the rectangular frame of the left eyeball image
  • r 4 the distance between the top left vertex of the rectangular frame of the right eyeball image and the leftmost face of the face image
  • t 4 the distance from the top left vertex of the rectangular frame of the right eyeball image to the uppermost edge of the face image
  • w 4 the width of the rectangular frame of the right eyeball image
  • h 4 the height of the rectangular frame of the right eyeball image.
  • eyeball position data can also be obtained from a face image.
  • the feature data is calculated according to the human eye position data and the eyeball position data obtained in step S2, and is compared with the calibration data of the positioning points in the designated viewing area collected in advance and calculated to obtain the user's line of sight. The coordinates of the feature point in the specified viewing area.
  • step S3 of calculating feature data based on the human eye position data and the eyeball position data includes:
  • step S31 the specific process of calculating the distance feature data in step S31 is as follows:
  • the distance d x between the center of the left eye and the center of the right eye is calculated by formula (3), and d x is the distance feature data.
  • the characteristic data (d x, m x, n x) by at step S31, the user looks to the feature points is obtained.
  • step S1 of obtaining a user image of the user looking at the designated viewing area through the camera the method further includes: S01, retrieving a memory to determine whether the preset calibration data exists in the memory; S02, if not, then Storing the preset calibration data.
  • FIG. 2a it is a schematic diagram of the positioning points of the designated viewing area, including 9 positioning points of upper left, upper middle, upper right, left middle, middle middle, right middle, lower left, middle lower, and lower right; see FIG. 2b.
  • the designated viewing area surrounded by top left, middle left, bottom left, bottom middle, middle middle, and top middle is the left area
  • the designated viewing area surrounded by top right, middle right, bottom right, middle bottom, middle middle, and top middle is the right area
  • the designated viewing areas surrounded by top left, middle left, center middle, right middle, top right, and top middle are top areas
  • step S02 the user looks at a positioning point of the mobile phone display at an appropriate distance from the display screen of the mobile phone according to his habit, and collects an image of the human eye looking at the positioning point through the front camera of the mobile phone.
  • the gaze time can be set in advance to remind the user to keep looking at the anchor point.
  • the camera obtains the shooting instruction and collects the image; it can also use the camera to continuously collect the image in real time and distinguish it by the trained classifier.
  • the state of the human eye If it is determined that the human eye is in a fixation state, any frame image in the fixation state is acquired.
  • the user first looks at the upper left positioning point, and the camera collects the image of the human eye gazing at the upper left positioning point, searches for the human eye image and the eyeball image from the image, obtains the human eye position data and the eyeball position data, calculates the calibration data, and records the Correspondence between the calibration data and the upper left anchor point; the user starts to look at the upper left anchor point, and the rest of the steps are the same as the upper left anchor point; until the upper left, middle upper, right upper, left middle, middle middle, right middle, left lower, middle lower, and right The corresponding data of the calibration data and positioning points of the next 9 positioning points are collected.
  • the method for obtaining the eye position data and the eye position data of the fixation point of the human eye from the image in this step S02 is the same as that of step S2, and details are not described herein.
  • the calibration data in this step S02 includes distance calibration data, horizontal calibration data, and vertical calibration data.
  • the calculation method of the distance calibration data is the same as the calculation method of the distance feature data in step S31.
  • the calculation method of the horizontal calibration data is the same as the calculation method of the horizontal feature data in step S31.
  • the calculation method of the vertical calibration data is the same as that of step S31.
  • the calculation method of the data is the same, and will not be repeated here.
  • the difference between the calibration data and the feature data in this embodiment is that the calibration data corresponds to the anchor points, and the feature data corresponds to the feature points pointed by the user's line of sight. Both are obtained according to the user's eye movement habits and use the same calculation method. The calculation is helpful to improve the accuracy of the calculation of the feature point coordinates.
  • d 11 , d 12 , d 13 , d 21 , d 22 , d 23 , d 31 , d 32 and d 33 are distance calibration data of each positioning point
  • m 11 , m 12 , m 13 , m 21 , m 22 , M 23 , m 31 , m 32 and m 33 are the lateral calibration data of each positioning point
  • n 11 , n 12 , n 13 , n 21 , n 22 , n 23 , n 31 , n 32 and n 33 are each positioning Point vertical calibration data.
  • the preset calibration data includes distance calibration data, horizontal calibration data, and vertical calibration data; and based on the preset calibration data and the feature data, calculating the user's orientation corresponding to the feature data
  • Step S4 of the coordinates of the feature point in the designated viewing area includes: S41. Determine whether the distance feature data is within a calibration range of the distance calibration data; S42. If yes, perform a preliminary judgment on the position of the feature point To obtain a position interval where the feature point is located in the designated viewing area; S43. Calculate the coordinates of the feature point according to a preset calculation formula corresponding to the position interval.
  • step S41 the extracted nine anchor point distance calibration data d 11, d 12, d 13 , d 21, d 22, d 23, d 31, d 32 and d 33 are the maximum value and the minimum Value to obtain the calibration range of the distance calibration data, and determine whether the distance characteristic data d x is between the maximum value and the minimum value of the distance calibration data.
  • Step S42 the feature if the distance between the data d x is not above the maximum value and the minimum value, adjusting the user specified distance viewing area, wherein the data until the distance d x falls between the maximum and minimum values.
  • the distance feature data d x is between the above-mentioned maximum value and minimum value, a preliminary judgment is made on the position of the feature point to determine in which interval the feature point is located in the designated viewing area, such as in the left area or the right area, in the upper area or The lower area. Characterized by the required distance range data from the calibration data during calibration d x falls, so the test specified user views the same distance from the measurement area or when very close, to improve the accuracy of gaze tracking.
  • step S43 the coordinates (x i , y i ) of the feature point that the user is looking at within a designated viewing area are calculated according to preset calculation formulas corresponding to different position intervals, so as to implement line-of-sight tracking.
  • the abscissa x i of the feature point is calculated according to formula (12).
  • R x is the total width pixel value of the specified viewing area / 2
  • m x is the lateral characteristic data of the feature point
  • m min is the minimum lateral calibration data of the position interval where the feature point is
  • m max is the feature point Maximum lateral calibration data for the location interval.
  • the ordinate y i of the feature point is calculated according to formula (13).
  • n x is the longitudinal characteristic data of the feature point
  • n min is the minimum longitudinal calibration data of the position interval where the feature point is
  • n max is the feature point Maximum longitudinal calibration data for the location interval.
  • n min is the minimum value among n 11 , n 12 and n 13
  • n max is the maximum value among n 21 , n 22 and n 21
  • Q y R y
  • m min is the minimum value of n 21 , n 22, and n 21
  • the step of performing a preliminary judgment on the position of the feature point to obtain a position interval where the feature point is located in the designated viewing area includes: S431, performing the process by comparing the lateral feature data with the lateral calibration data. Compare the sizes to obtain the lateral position interval where the feature points are located in the designated viewing area; and obtain the feature points located in the longitudinal direction of the designated viewing area by comparing the longitudinal feature data with the longitudinal calibration data. Location interval.
  • min (m 11 , m 21 , m 31 ) refers to the minimum value of m 11 , m 21 , m 31
  • max (m 13 , m 23 , m 33 ) refers to the maximum value of m 13 , m 23 , m 33 .
  • min (n 11 , n 12 , n 13 ) refers to the minimum value of n 11 , n 12 , n 13
  • max (n 31 , n 32 , n 33 ) refers to the maximum value of n 31 , n 32 , n 33 .
  • an eye-tracking interaction device including:
  • An image acquisition module 10 is configured to acquire a user image of a user looking at a specified viewing area through a camera; an image analysis module 20 is configured to find a human eye image and an eyeball image from the user image, and obtain human eye position data and eyeball position data
  • a data calculation module 30 configured to calculate feature data based on the human eye position data and the eyeball position data; a line of sight positioning module 40 configured to calculate a correspondence of the feature data according to preset calibration data and the feature data The coordinate of the feature point that the user is looking at is in the designated viewing area; wherein the preset calibration data is calibration data of a plurality of positioning points in the designated viewing area.
  • the designated viewing area includes a terminal device interface for user-machine interaction, such as a smartphone display, a flat panel display, a smart TV display, a personal computer display, a laptop display, etc .;
  • the camera includes The front camera and external camera of the terminal device, such as the front camera of a mobile phone.
  • the image acquisition module 10 of this embodiment taking the eye movement control of the mobile phone display as an example, the user looks at the characteristic points of the mobile phone display at an appropriate distance from the mobile phone display according to his habits, and collects through the mobile phone's front camera. The human eye looks at the image of the feature point.
  • the real-time image acquisition unit can continuously acquire images in real time with a camera, and then distinguish the state of the human eye through a pre-trained classifier in the classification unit.
  • the preset states include gaze, eye movement, single-eye blink, double-eye blink, When blinking multiple times, etc., when it is determined that the human eye is in any of the above states, the user image is collected in real time by the image acquisition unit.
  • the human face image is first searched from the user image through the human face search unit;
  • the human eye image is searched in the face image, and the human eye position data is obtained according to the human eye image; finally, the eyeball image is searched from the human eye image through the eyeball search unit, and the eyeball position data is obtained according to the eyeball image.
  • Face image From the image. If no face image is found in the image, return to the step of obtaining the user image and adjust the relative position of the user and the specified viewing area until the face image can be found in the user image obtained by the camera .
  • face rules such as the distribution of eyes, nose, mouth, etc.
  • features that are invariant to the face such as skin color, edges, textures
  • face detection on the input image describe the facial features of the face with a standard face template.
  • the face detection When performing face detection, first calculate the correlation value between the input image and the standard face template, and then The obtained correlation value is compared with a preset threshold value to determine whether a face exists in the input image; the face area is regarded as a type of pattern, and a large amount of face data is used as a sample training to learn potential rules
  • a classifier is constructed to detect faces by discriminating all possible region pattern attributes in the image.
  • the found face image is marked with a rectangular frame.
  • Human eye search methods include template-based methods, statistics-based methods, and knowledge-based methods.
  • the method based on template matching includes a gray projection template and a geometric feature template.
  • the gray projection method refers to the horizontal and vertical projection of a gray image of a human face, and respectively counts the gray value and / or in two directions. The value of the gray function, find specific change points, and then combine the positions of change points in different directions according to prior knowledge to obtain the position of the human eye; the geometric feature template is implemented using the individual features and distribution features of the eyes as the basis Human eye detection.
  • Statistics-based methods generally train and learn a large number of target samples and non-target samples to obtain a set of model parameters, and then build a classifier or filter to detect the target based on the model.
  • the knowledge-based method is to determine the application environment of the image, summarize the knowledge (such as contour information, color information, position information) that can be used for human eye detection under specific conditions, and summarize them into rules that guide human eye detection.
  • the left-eye image and the right-eye image are framed by rectangular frames, and the following eye position data is obtained, including:
  • r 1 the distance from the upper left vertex of the rectangular frame of the left-eye image to the left-most face image
  • t 1 the distance from the upper left vertex of the rectangular frame of the left eye image to the uppermost edge of the face image
  • w 1 the width of the rectangular frame of the left-eye image
  • h 1 the height of the rectangular frame of the left-eye image
  • r 2 the distance from the upper-left vertex of the rectangular frame of the right-eye image to the left-most face image
  • t 2 the distance from the top left vertex of the rectangular frame of the right eye image to the uppermost edge of the face image
  • w 2 the width of the rectangular frame of the right-eye image
  • h 2 the height of the rectangular frame of the right-eye image.
  • Finding an eyeball image from a human eye image includes finding a left eyeball image from a left eye image, and finding a right eyeball image from a right eye image. If no eyeball image is found, return to the step of obtaining a user image, and reacquire the user image until the eyeball image can be found in this step.
  • Eyeball search methods include neural network method, extreme point position discrimination method of edge point integral projection curve, template matching method, multi-resolution mosaic map method, geometric and symmetry detection method, and Hough transform-based method. This embodiment uses a rectangular frame to frame the left eyeball image and the right eyeball image, respectively, and obtains the following eyeball position data, including:
  • r 3 the distance between the top left vertex of the rectangular frame of the left eyeball image and the leftmost face of the face image
  • t 3 the distance from the top left vertex of the rectangular frame of the left eyeball image to the uppermost edge of the face image
  • w 3 the width of the rectangular frame of the left eyeball image
  • h 3 the height of the rectangular frame of the left eyeball image
  • r 4 the distance between the top left vertex of the rectangular frame of the right eyeball image and the leftmost face of the face image
  • t 4 the distance from the top left vertex of the rectangular frame of the right eyeball image to the uppermost edge of the face image
  • w 4 the width of the rectangular frame of the right eyeball image
  • h 4 the height of the rectangular frame of the right eyeball image.
  • eyeball position data can also be obtained from a face image.
  • the feature data is calculated according to the human eye position data and the eye position data obtained by the image analysis module 20, and is compared with the calibration data of the positioning points in the specified viewing area collected in advance and Calculate to obtain the coordinates of the feature point that the user looks at in the specified viewing area.
  • the data calculation module 30 includes: a first data obtaining unit 301, configured to calculate distance feature data when a user looks at the feature point according to the position data of the human eye; and second data acquisition A unit 302 is configured to calculate, according to the position data of the human eye and the position data of the eyeball, the lateral feature data of the eyeball position and the longitudinal feature data of the eyeball position when the user looks at the feature point.
  • the specific process of calculating the distance feature data by the first data acquisition unit 301 is as follows: calculating the coordinates (x 1 , y 1 ) of the center position of the left eye by using the first calculation subunit according to formula (14),
  • the distance d x between the center of the left eye and the center of the right eye is calculated by the second calculation subunit according to formula (16), where d x is distance feature data.
  • the specific process of calculating the horizontal feature data and the vertical feature data by the second data obtaining unit 302 is as follows: calculate the coordinates (x 3 , y 3 ) of the center position of the left eyeball by using formula (17),
  • the eyeball tracking interaction device further includes:
  • a judgment module 01 is configured to retrieve a memory to determine whether the preset calibration data exists in the memory; a calibration module 02 is configured to store the preset if the preset calibration data does not exist in the memory Calibration data.
  • FIG. 4 it is a schematic diagram of an anchor point for a designated viewing area, including nine anchor points of upper left, upper middle, upper right, left middle, middle middle, right middle, lower left, middle lower, and lower right.
  • the designated viewing area surrounded by left middle, bottom left, bottom middle, middle middle, and top middle is the left area
  • the designated viewing area surrounded by top right, middle right, bottom right, middle bottom, middle, and middle top is the right area, top left, left
  • the designated viewing areas surrounded by middle, middle, right, middle right, top and middle are the top areas
  • the designated viewing areas surrounded by bottom left, left middle, middle, right middle, bottom right, and bottom middle are the bottom areas.
  • the gaze time can be set in advance to remind the user to keep looking at the anchor point.
  • the camera obtains the shooting instruction and collects the image; it can also use the camera to continuously collect the image in real time and distinguish it by the trained classifier.
  • the state of the human eye If it is determined that the human eye is in a fixation state, any frame image in the fixation state is acquired. Further, the human eye image and the eyeball image are searched from the obtained image to obtain human eye position data and eyeball position data; a series of calibration data is calculated according to the human eye position data and the eyeball position data, and the calibration data and The correspondence between the anchor points.
  • the user first looks at the upper left positioning point, and the camera collects the image of the human eye gazing at the upper left positioning point, searches for the human eye image and the eyeball image from the image, obtains the human eye position data and the eyeball position data, calculates the calibration data, and records the Correspondence between the calibration data and the upper left anchor point; the user starts to look at the upper left anchor point, and the rest of the steps are the same as the upper left anchor point; until the upper left, middle upper, right upper, left middle, middle middle, right middle, left lower, middle lower, and right The corresponding data of the calibration data and positioning points of the next 9 positioning points are collected.
  • the method for obtaining the eye position data and the eye position data of the fixation point of the human eye from the image in the calibration module 02 is the same as that of the image analysis module 20, and details are not described herein.
  • the preset calibration data in the calibration module 02 includes distance calibration data, horizontal calibration data, and vertical calibration data.
  • the calculation method of the distance calibration data is the same as the calculation method of the first data acquisition unit 301
  • the calculation method of the horizontal calibration data and the vertical calibration data is the same as the calculation method of the second data acquisition unit 302, and details are not described herein.
  • the difference between the calibration data and the feature data in this embodiment is that the calibration data corresponds to the anchor points, and the feature data corresponds to the feature points pointed by the user's line of sight.
  • the calibration module 02 obtains upper left (d 11 , m 11 , n 11 ), upper middle (d 12 , m 12 , n 12 ), upper right (d 13 , m 13 , n 13 ), left middle (d 21 , m 21 , n 21 ), middle (d 22 , m 22 , n 22 ), right middle (d 23 , m 23 , n 23 ), bottom left (d 31 , m 31 , n 31 ), bottom middle (d 32 , m 32 , n 32 ) and the calibration data of the 9 anchor points at the bottom right (d 33 , m 33 , n 33 ).
  • d 11 , d 12 , d 13 , d 21 , d 22 , d 23 , d 31 , d 32 and d 33 are distance calibration data of each positioning point
  • m 11 , m 12 , m 13 , m 21 , m 22 , M 23 , m 31 , m 32 and m 33 are the lateral calibration data of each positioning point
  • n 11 , n 12 , n 13 , n 21 , n 22 , n 23 , n 31 , n 32 and n 33 are each positioning Point vertical calibration data.
  • the preset calibration data includes distance calibration data, horizontal calibration data, and vertical calibration data;
  • the line-of-sight positioning module 40 includes:
  • the distance judging unit 401 is configured to judge whether the distance feature data is within a calibration range of the distance calibration data; the position preliminary judgment unit 402 is configured to, if the distance feature data is within a calibration range of the distance calibration data, Performing a preliminary judgment on the position of the feature point to obtain a position interval where the feature point is located in the designated viewing area; a coordinate calculation unit 403 is configured to calculate the feature point according to a preset calculation formula corresponding to the position interval; coordinate of.
  • the distance judgment unit 401 extracts the largest of the distance calibration data d 11 , d 12 , d 13 , d 21 , d 22 , d 23 , d 31 , d 32 and d 33 of the nine positioning points. Value and minimum value to obtain the calibration range of the distance calibration data, and determine whether the distance characteristic data d x is between the maximum value and the minimum value of the distance calibration data. Initial impression of the position unit 402, if the distance between the feature data d x is not above the maximum value and the minimum value, re-entering the image acquisition module 10 acquires user image.
  • the distance feature data d x is between the above-mentioned maximum value and minimum value, a preliminary judgment is made on the position of the feature point to determine in which interval the feature point is located in the designated viewing area, such as in the left area or the right area, in the upper area or The lower area. Characterized by the required distance range data from the calibration data during calibration d x falls, so the test specified user views the same distance from the measurement area or when very close, to improve the accuracy of gaze tracking.
  • the coordinates (x i , y i ) of the feature point that the user is looking at within a specified viewing area are calculated according to preset calculation formulas corresponding to different position intervals, so as to implement line-of-sight tracking.
  • the abscissa x i of the feature point is calculated according to formula (12).
  • R x is the total width pixel value of the specified viewing area / 2
  • m x is the lateral characteristic data of the feature point
  • m min is the minimum lateral calibration data of the position interval where the feature point is
  • m max is the feature point Maximum lateral calibration data for the location interval.
  • the ordinate y i of the feature point is calculated according to formula (13).
  • n x is the longitudinal characteristic data of the feature point
  • n min is the minimum longitudinal calibration data of the position interval where the feature point is
  • n max is the feature point Maximum longitudinal calibration data for the location interval.
  • n min is the minimum value among n 11 , n 12 and n 13
  • n max is the maximum value among n 21 , n 22 and n 21
  • Q y R y
  • m min is the minimum value of n 21 , n 22, and n 21
  • the position preliminary judgment unit 402 includes a first preliminary judgment sub-unit 4021 configured to obtain the feature point at the position by comparing the lateral feature data with the lateral calibration data.
  • a second preliminary sub-unit 4022 is configured to obtain a vertical position interval of the feature point in the specified viewing area by comparing the vertical feature data with the vertical calibration data.
  • min (m 11 , m 21 , m 31 ) refers to the minimum value of m 11 , m 21 , m 31
  • max (m 13 , m 23 , m 33 ) refers to the maximum value of m 13 , m 23 , m 33 .
  • min (n 11 , n 12 , n 13 ) refers to the minimum value of n 11 , n 12 , n 13
  • max (n 31 , n 32 , n 33 ) refers to the maximum value of n 31 , n 32 , n 33 .
  • the present application also proposes a computer device 03, which includes a processor 04, a memory 01, and a computer program 02 stored on the memory 01 and executable on the processor 04.
  • the processor 04 is implemented when the computer program 02 is executed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本申请揭示了一种眼球追踪交互方法和装置，包括获取用户图像；从用户图像中获取人眼位置数据和眼球位置数据；根据人眼位置数据和所述眼球位置数据计算特征数据；根据预设的校准数据以及特征数据，计算用户看向的特征点的坐标。本申请的眼球追踪交互方法和装置可以根据用户的眼动习惯来控制设备。

Description

眼球追踪交互方法和装置 技术领域
本申请涉及人机交互技术领域，具体涉及一种眼球追踪交互方法和装置。
背景技术
眼动控制方法是一种非接触的人机互动方式，通过追踪眼球位置来计算眼睛的注视点的位置。眼动控制对于无法双手操作的用户起到重大帮助。随着智能终端的发展，具有眼球追踪功能的游戏电脑使玩家在游戏场景中更为身临其境。现有技术中，眼球追踪技术需要用到专用设备，如眼动仪。在这些专用设备使用过程中，用户需要根据说明书限定的眼动方式才可控制设备，用户不可根据自己的眼动习惯来控制设备，用户体验不高。人机交互方式的趋势是以人为中心、更为友好和便捷，因此眼动追踪也朝着根据用户眼动习惯来控制设备的方向发展。但现有技术中，不使用专用设备来进行眼球追踪，视线的定位准确度较低，经常会出现用户实际所看向的区域与通过图像分析计算得到的计算数据不符的现象，从而影响人机交互的进行，用户体验不高。
技术问题
本申请的目的在于提供一种眼球追踪交互方法和装置，旨在解决现有技术中眼动控制需要用专用设备且不能根据用户眼动习惯来实现准确的视线定位的问题。
技术解决方案
本申请提出一种眼球追踪交互方法，包括：通过摄像头获取用户看向指定观看区域的用户图像；从所述用户图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据；根据所述人眼位置数据和所述眼球位置数据计算特征数据；根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标；其中，所述预设的校准数据为指定观看区域内多个定位点的校准数据。
本申请还提出了一种眼球追踪交互装置，包括：图像获取模块，用于通过摄像头获取用户看向指定观看区域的用户图像；图像分析模块，用于从所述用户图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据；数据计算模块，用于根据所述人眼位置数据和所述眼球位置数据计算特征数据；视线定位模块，用于根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标；其中，所述预设的 校准数据为指定观看区域内多个定位点的校准数据。
有益效果
本申请的眼球追踪交互方法和装置，通过普通摄像头采集用户图像，从用户图像中查找人眼和眼球，对人眼位置和眼球位置的特征数据进行计算，并根据特征数据和预设的校准数据进行计算，获得特征数据对应的用户视线在指定观看区域的坐标，实现视线定位。本申请的特征数据和校准数据均根据用户的眼动习惯来采集，人机交互方式友好，易于实现，且无需额外设备，成本较低。
附图说明
图1是本申请一实施例的眼动追踪交互方法的流程示意图；
本申请图2a为各定位点的示意图，图2b为左边区域和右边区域的划分示意图，图2c为上边区域和下边区域的划分示意图本申请；
图3是本申请一实施例的眼球追踪交互装置的结构示意框图；
图4是图3中数据计算模块的结构示意框图；
图5是本申请又一实施例的眼球追踪交互装置的结构示意框图；
图6是图5中视线定位模块的结构示意框图；
图7是图6中位置初判单元的结构示意框图；
图8是本申请一实施例的计算机设备的结构示意图。
本发明的最佳实施方式
参照图1，本申请实施例提供了一种眼球追踪交互方法，包括：
S1、通过摄像头获取用户看向指定观看区域的用户图像；
S2、从所述用户图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据；
S3、根据所述人眼位置数据和所述眼球位置数据计算特征数据；
S4、根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标；其中，所述预设的校准数据为指定观看区域内多个定位点的校准数据。
本实施例中，指定观看区域包括于用户进行人机交互的终端设备界面，例如可以是智能手机显示屏、平板显示屏、智能电视显示屏、个人电脑显示屏、笔记本电脑显示屏等；摄像头包括终端设备自带的前置摄像头、外接摄像头，如手机前置摄像头等。本实施例步骤S1中，以眼动控制手机显示屏为例，用户根据自己的习惯在距离手机显示屏合适的距离处，眼睛看向手机显示屏的特征点，通过手机前置摄像头采集人眼看向该特征点的图像。具体 地，可以用摄像头持续实时采集图像，通过预先训练好的分类器区分人眼的状态，上述预设状态包括注视、眼神移动、单眼眨眼、双眼眨眼、多次眨眼等，当判断人眼处于任一上述状态时，则实时采集用户图像。本申请对涉及的分类器类型不做具体限定。
本实施例步骤S2中，为了提高人眼图像和眼球图像查找的查找效率和准确度，先从所述用户图像中查找人脸图像；再从所述人脸图像中查找人眼图像，以及根据所述人眼图像获取人眼位置数据；最后从所述人眼图像中查找眼球图像，以及根据所述眼球图像获取眼球位置数据。
先从图像中查找人脸图像，如果在图像中没有查找到人脸图像，则返回获取用户图像步骤，调整用户和指定观看区域的相对位置，直至摄像头获取的用户图像中能查找到人脸图像。人脸图像的查找方法较多，比如：利用人脸规则(如眼睛、鼻子、嘴巴等的分布)对输入图像进行人脸检测；通过寻找人脸面部不变的特征(如肤色、边缘、纹理)来对输入图像进行人脸检测；将人脸的面部特征用一个标准的人脸模板来描述，进行人脸检测时，先计算输入图像与标准人脸模板之间的相关值，然后再将求得的相关值与事先设定的阂值进行比较，以判别输入图像中是否存在人脸；将人脸区域看作一类模式，使用大量的人脸数据作样本训练，来学习潜在的规则并构造分类器，通过判别图像中所有可能区域模式属性来实现人脸的检测。本实施例将查找到的人脸图像用矩形框标出。
从人脸图像中查找人眼图像，如果没有查找到人眼图像，则返回获取用户图像步骤，重新获取用户图像，直至本步骤能查找到人眼图像。人眼查找的方法包括基于模板匹配的方法、基于统计的方法和基于知识的方法。其中基于模板匹配的方法包括灰度投影模板和几何特征模板：灰度投影法是指对人脸灰度图像进行水平和垂直方向的投影，分别统计出两个方向上的灰度值和/或灰度函数值，找出特定变化点，然后根据先验知识将不同方向上的变化点位置相结合，即得到人眼的位置；几何特征模板是利用眼睛的个体特征以及分布特征作为依据来实施人眼检测。基于统计的方法一般是通过对大量目标样本和非目标样本进行训练学习得到一组模型参数，然后基于模型构建分类器或者滤波器来检测目标。基于知识的方法是确定图像的应用环境，总结特定条件下可用于人眼检测的知识(如轮廓信息、色彩信息、位置信息)等，把它们归纳成指导人眼检测的规则。本实施例用矩形框分别框出左眼图像和右眼图像，获得下述人眼位置数据，包括：
r 1：左眼图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 1：左眼图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 1：左眼图像的矩形框的宽度；h 1：左眼图像的矩形框的高度；
r 2：右眼图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 2：右眼图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 2：右眼图像的矩形框的宽度；h 2：右眼图像的矩形框的高度。
从人眼图像中查找眼球图像包括从左眼图像中查找到左眼球图像，从右眼图像中查找右眼球图像。如果没有查找到眼球图像，则返回获取用户图像步骤，重新获取用户图像，直至本步骤中能查找到眼球图像。眼球查找的方法包括神经网络法、边缘点积分投影曲线的极值位置判别法、模板匹配法、多分辨率的马赛克图法、几何及对称性检测法、基于霍夫变换法等。本实施例中查找到左眼球图像和右眼球图像，用矩形框分别框出左眼球图像和右眼球图像，获得得到下述眼球位置数据，包括：
r 3：左眼球图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 3：左眼球图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 3：左眼球图像的矩形框的宽度；h 3：左眼球图像的矩形框的高度；
r 4：右眼球图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 4：右眼球图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 4：右眼球图像的矩形框的宽度；h 4：右眼球图像的矩形框的高度。
本实施例中给出了从人眼图像中获取眼球位置数据的具体参数。基于本申请的发明理念，也可以从人脸图像中获取眼球位置数据。
本步骤S3-S4中，根据步骤S2获得的人眼位置数据和眼球位置数据来计算特征数据，与预先收集的指定观看区域内定位点的校准数据进行对照并计算，得到用户视线所看向的特征点在指定观看区域的坐标。
进一步地，所述根据所述人眼位置数据和所述眼球位置数据计算特征数据的步骤S3，包括：
S31、根据所述人眼位置数据，计算用户看向所述特征点时的距离特征数据；以及根据所述人眼位置数据和所述眼球位置数据，计算用户看向所述特征点时的眼球位置横向特征数据与眼球位置纵向特征数据。
本实施例中，步骤S31计算距离特征数据的具体过程如下：
通过公式(1)计算左眼中心位置坐标(x 1,y 1)，
Pot(x 1,y 1)＝Pot(r 1+w 1/2,t 1+h 1/2)    (1)
通过公式(2)计算右眼中心位置坐标(x 2,y 2)，
Pot(x 2,y 2)＝Pot(r 2+w 2/2,t 2+h 2/2)    (2)
通过公式(3)计算左眼中心与右眼中心的距离d x，d x即为距离特征数据。
Figure PCTCN2019073763-appb-000001
计算横向特征数据和纵向特征数据的具体过程如下：
通过公式(4)计算左眼球中心位置坐标(x 3,y 3)，
Pot(x 3,y 3)＝Pot(r 3+w 3/2,t 3+h 3/2)   (4)
通过公式(5)计算右眼球中心位置坐标(x 4,y 4)，
Pot(x 4,y 4)＝Pot(r 4+w 4/2,t 4+h 4/2)    (5)
通过公式(6)计算左眼球中心与左眼图像的最左边之间的第一横向距离d 1：d 1＝x 3–r 1      (6)
通过公式(7)计算左眼球中心与左眼图像的最上边之间的第一纵向距离d 3：d 3＝y 3–t 1      (7)
通过公式(8)计算右眼球中心与右眼图像的最右边之间的第二横向距离d 2：d 2＝r 2+w 2–x 4     (8)
通过公式(9)计算右眼球中心与右眼图像的最下边之间的第二纵向距离d 4：d 4＝t 2+h 2–y 4       (9)
通过公式(10)计算横向特征数据m x：m x＝d 1/d 2  (10)
通过公式(11)计算纵向特征数据n x：n x＝d 3/d 4  (11)
通过步骤S31，获得用户看向特征点时的特征数据(d x,m x,n x)。
进一步地，所述通过摄像头获取用户看向指定观看区域的用户图像的步骤S1前还包括：S01、检索存储器，判断所述存储器中是否有所述预设的校准数据；S02、若否，则存储所述预设的校准数据。
用户在开始眼动控制手机显示屏前，需要先判断是否已经进行过校准，如果存储器中查找不到校准数据，则先进行眼动控制校准。具体地，参照图2a，为指定观看区域的定位点的示意图，包括左上、中上、右上、左中、中中、右中、左下、中下和右下的9个定位点；参照图2b，其中左上、左中、左下、中下、中中和中上包围的指定观看区域为左边区域，右上、右中、右下、中下、中中和中上包围的指定观看区域为右边区域；参照图2c，左上、左中、中中、右中、右上和中上包围的指定观看区域为上边区域，左下、左中、中中、右中、右下和中下包围的指定观看区域为下边区域。
步骤S02中，用户根据自己的习惯在距离手机显示屏合适的距离处，眼睛 注视手机显示屏的一个定位点，通过手机前置摄像头采集人眼注视该定位点的图像。比如，可以预先设置注视时间，提醒用户持续注视该定位点，在达到预设的注视时间时长时，摄像头获得拍摄指令，采集图像；也可以用摄像头持续实时采集图像，通过训练好的分类器区分人眼的状态，如果判断人眼处于注视状态，则获取注视状态中的任一帧图像。进一步从获取的图像中查找人眼图像和眼球图像，获取到人眼位置数据和眼球位置数据；根据所述人眼位置数据和所述眼球位置数据计算一系列校准数据，依次记录所述校准数据与所述定位点的对应关系。具体地，用户首先看向左上定位点，摄像头采集人眼注视左上定位点的图像，从该图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据，计算校准数据，记录该校准数据与左上定位点的对应关系；用户再开始看向中上定位点，其余步骤同左上定位点；直至左上、中上、右上、左中、中中、右中、左下、中下和右下的9个定位点的校准数据和定位点的对应关系均采集完毕。本步骤S02中的从图像中获取人眼注视定位点的人眼位置数据和眼球位置数据的方法与步骤S2相同，在此不做赘述。本步骤S02中的校准数据包括距离校准数据、横向校准数据和纵向校准数据。其中，距离校准数据的计算方法和步骤S31的距离特征数据的计算方法相同，横向校准数据的计算方法和步骤S31的横向特征数据的计算方法相同，纵向校准数据的计算方法和步骤S31的纵向特征数据的计算方法相同，在此均不做赘述。本实施例的校准数据和特征数据的不同之处在于校准数据对应于定位点，而特征数据对应于用户视线指向的特征点，两者均根据用户的眼动习惯获取，且采用相同的计算方法进行计算，有利于提高特征点坐标计算的准确度。
本步骤S02获得左上(d 11,m 11,n 11)、中上(d 12,m 12,n 12)、右上(d 13,m 13,n 13)、左中(d 21,m 21,n 21)、中中(d 22,m 22,n 22)、右中(d 23,m 23,n 23)、左下(d 31,m 31,n 31)、中下(d 32,m 32,n 32)和右下(d 33,m 33,n 33)的9个定位点的校准数据。其中d 11、d 12、d 13、d 21、d 22、d 23、d 31、d 32和d 33是各个定位点的距离校准数据，m 11、m 12、m 13、m 21、m 22、m 23、m 31、m 32和m 33是各个定位点的横向校准数据，n 11、n 12、n 13、n 21、n 22、n 23、n 31、n 32和n 33是各个定位点的纵向校准数据。
进一步地，所述预设的校准数据包括距离校准数据、横向校准数据和纵向校准数据；所述根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标的步骤S4，包括：S41、判断所述距离特征数据是否在所述距离校准数据的校准范围内；S42、若是，则对所述特征点进行位置初判，得到所述特征点位于所述指定 观看区域的位置区间；S43、根据所述位置区间所对应的预设计算公式计算所述特征点的坐标。
本实施例中，步骤S41，提取出9个定位点的距离校准数据d 11、d 12、d 13、d 21、d 22、d 23、d 31、d 32和d 33中的最大值和最小值，得到距离校准数据的校准范围，判断距离特征数据d x是否在上述的距离校准数据的最大值和最小值之间。步骤S42中，如果距离特征数据d x不在上述的最大值和最小值之间，则调整用户与指定观看区域的距离，直至距离特征数据d x落入上述的最大值和最小值之间。如果距离特征数据d x在上述的最大值和最小值之间，则对特征点位置进行初判，判断特征点在指定观看区域位于哪个位置区间，比如位于左边区域还是右边区域，位于上边区域还是下边区域。通过要求距离特征数据d x落入校准时的距离校准数据范围，使得测试时用户与指定观看区域的距离与校准时相同或非常接近，以提高视线追踪的准确度。
步骤S43中，根据不同位置区间所对应的预设计算公式计算用户所看向的特征点在指定观看区域内的坐标(x i,y i)，从而实现视线追踪。
具体地，根据公式(12)计算特征点的横坐标x i。
x i＝Q x+R x*((m x–m min)/(m max-m min))    (12)
其中Q x为常数，R x为指定观看区域的总宽度像素值/2，m x为特征点的横向特征数据，m min为特征点所处位置区间的最小横向校准数据，m max为特征点所处位置区间的最大横向校准数据。
如果特征点位于左边区域，则Q x＝0，m min为m 11、m 21和m 31中的最小值，m max为m 12、m 22和m 32中的最大值；如果特征点位于右边区域，则Q x＝R x，m min为m 12、m 22和m 32中的最小值，m max为m 13、m 23和m 33中的最大值；如果特征点位于左边区域和右边区域的交界处，则x i＝R x。
根据公式(13)计算特征点的纵坐标y i。
y i＝Q y+R y*((n x–n min)/(n max-n min))      (13)
其中Q y为常数，R y为指定观看区域的总高度像素值/2，n x为特征点的纵向特征数据，n min为特征点所处位置区间的最小纵向校准数据，n max为特征点所处位置区间的最大纵向校准数据。
如果特征点位于上边区域，则Q y＝0，n min为n 11、n 12和n 13中的最小值，n max为n 21、n 22和n 21中的最大值；如果特征点位于下边区域，则Q y＝R y，m min为n 21、n 22和n 21中的最小值，n max为n 31、n 32和n 33中的最大值；如果特征点位于上边区域和下边区域的交界处，则y i＝R y。
进一步地，所述对所述特征点进行位置初判，得到所述特征点位于所述指定观看区域的位置区间的步骤，包括：S431、通过将所述横向特征数据与所述横向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的横向位置区间；以及通过将所述纵向特征数据与所述纵向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的纵向位置区间。
步骤S431中横向位置区间的判断，若min(m 11,m 21,m 31)＜m x＜m 22，则特征点位于左边区域；若m 22＜m x＜max(m 13,m 23,m 33)，则特征点位于右边区域；若m x＝m 22，则特征点位于左边区域和右边区域的交界；若m x＜min(m 11,m 21,m 31)或m x＞max(m 13,m 23,m 33)，则特征点不在指定观看区域上，需要重新进入步骤S1获取用户图像。其中min(m 11,m 21,m 31)指m 11,m 21,m 31中的最小值，max(m 13,m 23,m 33)指m 13,m 23,m 33中的最大值。纵向位置区间的判断，若min(n 11,n 12,n 13)＜n x＜n 22，则特征点位于上边区域；若n 22＜n x＜max(n 31,n 32,n 33)，则特征点位于下边区域；若n x＝n 22，则特征点位于上边区域和下边区域的交界；若n x＜min(n 11,n 12,n 13)或n x＞max(n 31,n 32,n 33)，则特征点不在指定观看区域上，需要重新进入步骤S1获取用户图像。其中min(n 11,n 12,n 13)指n 11,n 12,n 13中的最小值，max(n 31,n 32,n 33)指n 31,n 32,n 33中的最大值。
参照图3，本申请还提出了一种眼球追踪交互装置，包括：
图像获取模块10，用于通过摄像头获取用户看向指定观看区域的用户图像；图像分析模块20，用于从所述用户图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据；数据计算模块30，用于根据所述人眼位置数据和所述眼球位置数据计算特征数据；视线定位模块40，用于根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标；其中，所述预设的校准数据为指定观看区域内多个定位点的校准数据。
本实施例中，指定观看区域包括于用户进行人机交互的终端设备界面，例如可以是智能手机显示屏、平板显示屏、智能电视显示屏、个人电脑显示屏、笔记本电脑显示屏等；摄像头包括终端设备自带的前置摄像头、外接摄像头，如手机前置摄像头等。本实施例图像获取模块10中，以眼动控制手机显示屏为例，用户根据自己的习惯在距离手机显示屏合适的距离处，眼睛看向手机显示屏的特征点，通过手机前置摄像头采集人眼看向该特征点的图像。具体地，可以通过实时图像获取单元用摄像头持续实时采集图像，再通 过分类单元中的预先训练好的分类器区分人眼的状态，上述预设状态包括注视、眼神移动、单眼眨眼、双眼眨眼、多次眨眼等，当判断人眼处于任一上述状态时，则通过图像获取单元实时采集用户图像。
本实施例图像分析模块20中，为了提高人眼图像和眼球图像查找的查找效率和准确度，先通过人脸查找单元从所述用户图像中查找人脸图像；再通过人眼查找单元从所述人脸图像中查找人眼图像，以及根据所述人眼图像获取人眼位置数据；最后通过眼球查找单元从所述人眼图像中查找眼球图像，以及根据所述眼球图像获取眼球位置数据。
先从图像中查找人脸图像，如果在图像中没有查找到人脸图像，则返回获取用户图像步骤，调整用户和指定观看区域的相对位置，直至摄像头获取的用户图像中能查找到人脸图像。人脸图像的查找方法较多，比如：利用人脸规则(如眼睛、鼻子、嘴巴等的分布)对输入图像进行人脸检测；通过寻找人脸面部不变的特征(如肤色、边缘、纹理)来对输入图像进行人脸检测；将人脸的面部特征用一个标准的人脸模板来描述，进行人脸检测时，先计算输入图像与标准人脸模板之间的相关值，然后再将求得的相关值与事先设定的阂值进行比较，以判别输入图像中是否存在人脸；将人脸区域看作一类模式，使用大量的人脸数据作样本训练，来学习潜在的规则并构造分类器，通过判别图像中所有可能区域模式属性来实现人脸的检测。本实施例将查找到的人脸图像用矩形框标出。
从人脸图像中查找人眼图像，如果没有查找到人眼图像，则返回获取用户图像步骤，重新获取用户图像，直至本步骤能查找到人眼图像。人眼查找的方法包括基于模板匹配的方法、基于统计的方法和基于知识的方法。其中基于模板匹配的方法包括灰度投影模板和几何特征模板：灰度投影法是指对人脸灰度图像进行水平和垂直方向的投影，分别统计出两个方向上的灰度值和/或灰度函数值，找出特定变化点，然后根据先验知识将不同方向上的变化点位置相结合，即得到人眼的位置；几何特征模板是利用眼睛的个体特征以及分布特征作为依据来实施人眼检测。基于统计的方法一般是通过对大量目标样本和非目标样本进行训练学习得到一组模型参数，然后基于模型构建分类器或者滤波器来检测目标。基于知识的方法是确定图像的应用环境，总结特定条件下可用于人眼检测的知识(如轮廓信息、色彩信息、位置信息)等，把它们归纳成指导人眼检测的规则。本实施例用矩形框分别框出左眼图像和右眼图像，并获得下述人眼位置数据，包括：
r 1：左眼图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 1：左眼图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 1：左眼图像的矩形框的宽度；h 1：左眼图像的矩形框的高度；
r 2：右眼图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 2：右眼图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 2：右眼图像的矩形框的宽度；h 2：右眼图像的矩形框的高度。
从人眼图像中查找眼球图像包括从左眼图像中查找到左眼球图像，从右眼图像中查找右眼球图像。如果没有查找到眼球图像，则返回获取用户图像步骤，重新获取用户图像，直至本步骤中能查找到眼球图像。眼球查找的方法包括神经网络法、边缘点积分投影曲线的极值位置判别法、模板匹配法、多分辨率的马赛克图法、几何及对称性检测法、基于霍夫变换法等。本实施例用矩形框分别框出左眼球图像和右眼球图像，获得得到下述眼球位置数据，包括：
r 3：左眼球图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 3：左眼球图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 3：左眼球图像的矩形框的宽度；h 3：左眼球图像的矩形框的高度；
r 4：右眼球图像的矩形框的左上顶点距离人脸图像的最左边的距离；
t 4：右眼球图像的矩形框的左上顶点距离人脸图像的最上边的距离；
w 4：右眼球图像的矩形框的宽度；h 4：右眼球图像的矩形框的高度。
本实施例中给出了从人眼图像中获取眼球位置数据的具体参数。基于本申请的发明理念，也可以从人脸图像中获取眼球位置数据。
本实施例数据计算模块30和视线定位模块40中，根据图像分析模块20获得的人眼位置数据和眼球位置数据来计算特征数据，与预先收集的指定观看区域内定位点的校准数据进行对照并计算，得到用户视线所看向的特征点在指定观看区域的坐标。
进一步地，参照图4，所述数据计算模块30包括：第一数据获取单元301，用于根据所述人眼位置数据，计算用户看向所述特征点时的距离特征数据；第二数据获取单元302，用于用于根据所述人眼位置数据和所述眼球位置数据，计算用户看向所述特征点时的眼球位置横向特征数据与眼球位置纵向特征数据。
本实施例中，第一数据获取单元301计算距离特征数据的具体过程如下：通过第一计算子单元根据公式(14)计算左眼中心位置坐标(x 1,y 1)，
Pot(x 1,y 1)＝Pot(r 1+w 1/2,t 1+h 1/2)    (14)
通过第一计算子单元根据公式(15)计算右眼中心位置坐标(x 2,y 2)，
Pot(x 2,y 2)＝Pot(r 2+w 2/2,t 2+h 2/2)    (15)
通过第二计算子单元根据公式(16)计算左眼中心与右眼中心的距离d x，d x即为距离特征数据。
Figure PCTCN2019073763-appb-000002
本实施例中，第二数据获取单元302计算横向特征数据和纵向特征数据的具体过程如下：通过公式(17)计算左眼球中心位置坐标(x 3,y 3)，
Pot(x 3,y 3)＝Pot(r 3+w 3/2,t 3+h 3/2)   (17)
通过公式(18)计算右眼球中心位置坐标(x 4,y 4)，
Pot(x 4,y 4)＝Pot(r 4+w 4/2,t 4+h 4/2)   (18)
通过公式(19)计算左眼球中心与左眼图像的最左边之间的第一横向距离d 1：d 1＝x 3–r 1       (19)
通过公式(20)计算左眼球中心与左眼图像的最上边之间的第一纵向距离d 3：d 3＝y 3–t 1     (20)
通过公式(21)计算右眼球中心与右眼图像的最右边之间的第二横向距离d 2：d 2＝r 2+w 2–x 4       (21)
通过公式(22)计算右眼球中心与右眼图像的最下边之间的第二纵向距离d 4：d 4＝t 2+h 2–y 4     (22)
通过公式(23)计算横向特征数据m x：m x＝d 1/d 2  (23)
通过公式(24)计算纵向特征数据n x：n x＝d 3/d 4  (24)
通过第一数据获取单元301和第二数据获取单元302，获得用户看向特征点时的特征数据(d x,m x,n x)。
进一步地，参照图5，所述眼球追踪交互装置还包括：
判断模块01，用于检索存储器，判断所述存储器中是否有所述预设的校准数据；校准模块02，用于若所述存储器中没有所述预设的校准数据，则存储所述预设的校准数据。
本实施例中，用户在开始眼动控制手机显示屏之前，需要先判断是否已经进行过校准，如果存储器中查找不到校准数据，则先进行眼动控制校准。具体地，参照图4，为指定观看区域的定位点的示意图，包括左上、中上、右上、左中、中中、右中、左下、中下和右下的9个定位点，其中左上、左中、左下、中下、中中和中上包围的指定观看区域为左边区域，右上、右中、右下、中下、中中和中上包围的指定观看区域为右边区域，左上、左中、中中、右中、右上 和中上包围的指定观看区域为上边区域，左下、左中、中中、右中、右下和中下包围的指定观看区域为下边区域。校准模块02中，用户根据自己的习惯在距离手机显示屏合适的距离处，眼睛注视手机显示屏的一个定位点，通过手机前置摄像头采集人眼注视该定位点的图像。比如，可以预先设置注视时间，提醒用户持续注视该定位点，在达到预设的注视时间时长时，摄像头获得拍摄指令，采集图像；也可以用摄像头持续实时采集图像，通过训练好的分类器区分人眼的状态，如果判断人眼处于注视状态，则获取注视状态中的任一帧图像。进一步从获取的图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据；根据所述人眼位置数据和所述眼球位置数据计算一系列校准数据，依次记录所述校准数据与所述定位点的对应关系。具体地，用户首先看向左上定位点，摄像头采集人眼注视左上定位点的图像，从该图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据，计算校准数据，记录该校准数据与左上定位点的对应关系；用户再开始看向中上定位点，其余步骤同左上定位点；直至左上、中上、右上、左中、中中、右中、左下、中下和右下的9个定位点的校准数据和定位点的对应关系均采集完毕。校准模块02中的从图像中获取人眼注视定位点的人眼位置数据和眼球位置数据的方法与图像分析模块20相同，在此不做赘述。校准模块02中的预设的校准数据包括距离校准数据、横向校准数据和纵向校准数据。其中，距离校准数据的计算方法和第一数据获取单元301的计算方法相同，横向校准数据和纵向校准数据的计算方法与第二数据获取单元302的计算方法相同，在此均不做赘述。本实施例的校准数据和特征数据的不同之处在于校准数据对应于定位点，而特征数据对应于用户视线指向的特征点，两者均根据用户的眼动习惯获取，且采用相同的计算方法进行计算，有利于提高特征点坐标计算的准确度。校准模块02获得左上(d 11,m 11,n 11)、中上(d 12,m 12,n 12)、右上(d 13,m 13,n 13)、左中(d 21,m 21,n 21)、中中(d 22,m 22,n 22)、右中(d 23,m 23,n 23)、左下(d 31,m 31,n 31)、中下(d 32,m 32,n 32)和右下(d 33,m 33,n 33)的9个定位点的校准数据。其中d 11、d 12、d 13、d 21、d 22、d 23、d 31、d 32和d 33是各个定位点的距离校准数据，m 11、m 12、m 13、m 21、m 22、m 23、m 31、m 32和m 33是各个定位点的横向校准数据，n 11、n 12、n 13、n 21、n 22、n 23、n 31、n 32和n 33是各个定位点的纵向校准数据。
进一步地，参照图6，所述预设的校准数据包括距离校准数据、横向校准数据和纵向校准数据；所述视线定位模块40包括：
距离判断单元401，用于判断所述距离特征数据是否在所述距离校准数 据的校准范围内；位置初判单元402，用于若所述距离特征数据在所述距离校准数据的校准范围内，则对所述特征点进行位置初判，得到所述特征点位于所述指定观看区域的位置区间；坐标计算单元403，用于根据所述位置区间所对应的预设计算公式计算所述特征点的坐标。
本实施例中，距离判断单元401中，提取出9个定位点的距离校准数据d 11、d 12、d 13、d 21、d 22、d 23、d 31、d 32和d 33中的最大值和最小值，得到距离校准数据的校准范围，判断距离特征数据d x是否在上述的距离校准数据的最大值和最小值之间。位置初判单元402中，如果距离特征数据d x不在上述的最大值和最小值之间，则重新进入图像获取模块10获取用户图像。如果距离特征数据d x在上述的最大值和最小值之间，则对特征点位置进行初判，判断特征点在指定观看区域位于哪个位置区间，比如位于左边区域还是右边区域，位于上边区域还是下边区域。通过要求距离特征数据d x落入校准时的距离校准数据范围，使得测试时用户与指定观看区域的距离与校准时相同或非常接近，以提高视线追踪的准确度。坐标计算单元403中，根据不同位置区间所对应的预设计算公式计算用户所看向的特征点在指定观看区域内的坐标(x i,y i)，从而实现视线追踪。
具体地，根据公式(12)计算特征点的横坐标x i。
x i＝Q x+R x*((m x–m min)/(m max-m min))    (12)
其中Q x为常数，R x为指定观看区域的总宽度像素值/2，m x为特征点的横向特征数据，m min为特征点所处位置区间的最小横向校准数据，m max为特征点所处位置区间的最大横向校准数据。
若特征点位于左边区域，则Q x＝0，m min为m 11、m 21和m 31中的最小值，m max为m 12、m 22和m 32中的最大值；若特征点位于右边区域，则Q x＝R x，m min为m 12、m 22和m 32中的最小值，m max为m 13、m 23和m 33中的最大值；若特征点位于左边区域和右边区域的交界处，则x i＝R x。
根据公式(13)计算特征点的纵坐标y i。
y i＝Q y+R y*((n x–n min)/(n max-n min))     (13)
其中Q y为常数，R y为指定观看区域的总高度像素值/2，n x为特征点的纵向特征数据，n min为特征点所处位置区间的最小纵向校准数据，n max为特征点所处位置区间的最大纵向校准数据。
若特征点位于上边区域，则Q y＝0，n min为n 11、n 12和n 13中的最小值，n max为n 21、n 22和n 21中的最大值；若特征点位于下边区域，则Q y＝R y，m min 为n 21、n 22和n 21中的最小值，n max为n 31、n 32和n 33中的最大值；若特征点位于上边区域和下边区域的交界处，则y i＝R y。
进一步地，参照图7，所述位置初判单元402包括，第一初判子单元4021，用于通过将所述横向特征数据与所述横向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的横向位置区间；第二初判子单元4022，用于通过将所述纵向特征数据与所述纵向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的纵向位置区间。
第一初判子单元4021中，若min(m 11,m 21,m 31)＜m x＜m 22，则特征点位于左边区域；若m 22＜m x＜max(m 13,m 23,m 33)，则特征点位于右边区域；若m x＝m 22，则特征点位于左边区域和右边区域的交界；若m x＜min(m 11,m 21,m 31)或m x＞max(m 13,m 23,m 33)，则特征点不在指定观看区域上，需要重新进入图像获取模块10获取用户图像。其中min(m 11,m 21,m 31)指m 11,m 21,m 31中的最小值，max(m 13,m 23,m 33)指m 13,m 23,m 33中的最大值。第二初判子单元4022中，若min(n 11,n 12,n 13)＜n x＜n 22，则特征点位于上边区域；若n 22＜n x＜max(n 31,n 32,n 33)，则特征点位于下边区域；若n x＝n 22，则特征点位于上边区域和下边区域的交界；若n x＜min(n 11,n 12,n 13)或n x＞max(n 31,n 32,n 33)，则特征点不在指定观看区域上，需要重新进入图像获取模块10获取用户图像。其中min(n 11,n 12,n 13)指n 11,n 12,n 13中的最小值，max(n 31,n 32,n 33)指n 31,n 32,n 33中的最大值。
本申请还提出一种计算机设备03，其包括处理器04、存储器01及存储于所述存储器01上并可在所述处理器04上运行的计算机程序02，处理器04执行计算机程序02时实现上述的眼动控制校准数据获取方法。

Claims (17)

  1. 一种眼球追踪交互方法，其特征在于，包括：
    通过摄像头获取用户看向指定观看区域的用户图像；
    从所述用户图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据；
    根据所述人眼位置数据和所述眼球位置数据计算特征数据；
    根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标；其中，所述预设的校准数据为指定观看区域内多个定位点的校准数据。
  2. 如权利要求1所述的眼球追踪交互方法，其特征在于，所述根据所述人眼位置数据和所述眼球位置数据计算特征数据的步骤，包括：
    根据所述人眼位置数据，计算用户看向所述特征点时的距离特征数据；以及根据所述人眼位置数据和所述眼球位置数据，计算用户看向所述特征点时的眼球位置横向特征数据与眼球位置纵向特征数据。
  3. 如权利要求2所述的眼球追踪交互方法，其特征在于，所述根据所述人眼位置数据，计算用户看向所述特征点时的距离特征数据的步骤，包括：
    根据所述人眼位置数据所包括的左眼位置数据，计算左眼中心位置坐标；以及根据所述人眼位置数据所包括的右眼中心位置数据，计算右眼中心位置坐标；根据所述左眼中心位置坐标以及所述右眼中心位置坐标，计算左眼中心与右眼中心的距离，获得所述距离特征数据。
  4. 如权利要求1所述的眼球追踪交互方法，其特征在于，所述通过摄像头获取用户看向指定观看区域的用户图像的步骤前还包括：
    检索存储器，判断所述存储器中是否有所述预设的校准数据；
    若否，则存储所述预设的校准数据。
  5. 如权利要求4所述的眼球追踪交互方法，其特征在于，所述预设的校准数据包括距离校准数据、横向校准数据和纵向校准数据；所述根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标的步骤，包括：
    判断所述距离特征数据是否在所述距离校准数据的校准范围内；
    若是，则对所述特征点进行位置初判，得到所述特征点位于所述指定观看区域的位置区间；
    根据所述位置区间所对应的预设计算公式计算所述特征点的坐标。
  6. 如权利要求5所述的眼球追踪交互方法，其特征在于，所述对所述特征点进行位置初判，得到所述特征点位于所述指定观看区域的位置区间的步骤，包括：
    通过将所述横向特征数据与所述横向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的横向位置区间；以及通过将所述纵向特征数据与所述纵向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的纵向位置区间。
  7. 如权利要求1所述的眼球追踪交互方法，其特征在于，所述从所述用户图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据的步骤，包括：
    从所述用户图像中查找人脸图像；
    从所述人脸图像中查找人眼图像，以及根据所述人眼图像获取人眼位置数据；从所述人眼图像中查找眼球图像，以及根据所述眼球图像获取眼球位置数据。
  8. 如权利要求1所述的眼球追踪交互方法，其特征在于，所述通过摄像头获取用户看向指定观看区域的用户图像的步骤，包括：
    获取摄像头实时采集的图像；
    分别通过预训练的分类器对实时采集的所述图像内所包含的人眼的状态进行分类；
    当所述人眼处于预设状态时，则从实时采集的所述图像中获取所述用户图像。
  9. 一种眼球追踪交互装置，其特征在于，包括：
    图像获取模块，用于通过摄像头获取用户看向指定观看区域的用户图像；
    图像分析模块，用于从所述用户图像中查找人眼图像和眼球图像，获取人眼位置数据和眼球位置数据；
    数据计算模块，用于根据所述人眼位置数据和所述眼球位置数据计算特征数据；
    视线定位模块，用于根据预设的校准数据以及所述特征数据，计算所述特征数据对应的所述用户看向的特征点在所述指定观看区域的坐标；其中，所述预设的校准数据为指定观看区域内多个定位点的校准数据。
  10. 如权利要求9所述的眼球追踪交互装置，其特征在于，所述数据计算模块包括：
    第一数据获取单元，用于根据所述人眼位置数据，计算用户看向所述特征点时的距离特征数据；
    第二数据获取单元，用于根据所述人眼位置数据和所述眼球位置数据，计算用户看向所述特征点时的眼球位置横向特征数据与眼球位置纵向特征数据。
  11. 如权利要求10所述的眼球追踪交互装置，其特征在于，所述第一数据获取单元，包括：
    第一计算子单元，用于根据所述人眼位置数据所包括的左眼位置数据，计算左眼中心位置坐标；以及根据所述人眼位置数据所包括的右眼中心位置数据，计算右眼中心位置坐标；
    第二计算子单元，用于根据所述左眼中心位置坐标以及所述右眼中心位置坐标，计算左眼中心与右眼中心的距离，获得所述距离特征数据。
  12. 如权利要求9所述的眼球追踪交互装置，其特征在于，所述图像获取模块前还包括：
    判断模块，用于检索存储器，判断所述存储器中是否有所述预设的校准数据；校准模块，用于若所述存储器中没有所述预设的校准数据，则存储所述预设的校准数据。
  13. 如权利要求12所述的眼球追踪交互装置，其特征在于，所述预设的校准数据包括距离校准数据、横向校准数据和纵向校准数据；所述视线定位模块包括：
    距离判断单元，用于判断所述距离特征数据是否在所述距离校准数据的校准范围内；
    位置初判单元，用于若所述距离特征数据在所述距离校准数据的校准范围内，则对所述特征点进行位置初判，得到所述特征点位于所述指定观看区域的位置区间；
    坐标计算单元，用于根据所述位置区间所对应的预设计算公式计算所述特征点的坐标。
  14. 如权利要求13所述的眼球追踪交互装置，其特征在于，所述位置初判单元包括，
    第一初判子单元，用于通过将所述横向特征数据与所述横向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的横向位置区间；
    第二初判子单元，用于通过将所述纵向特征数据与所述纵向校准数据进行大小比较，得到所述特征点位于所述指定观看区域的纵向位置区间。
  15. 如权利要求9所述的眼球追踪交互装置，其特征在于，所述图像分析模块，包括：
    人脸查找单元，用于从所述用户图像中查找人脸图像；
    人眼查找单元，用于从所述人脸图像中查找人眼图像，以及根据所述人眼图像获取人眼位置数据；
    眼球查找单元，用于从所述人眼图像中查找眼球图像，以及根据所述眼球图像获取眼球位置数据。
  16. 如权利要求9所述的眼球追踪交互装置，其特征在于，所述图像获取模块，包括：
    实时图像获取单元，用于获取摄像头实时采集的图像；
    分类单元，用于分别通过预训练的分类器对实时采集的所述图像内所包含的人眼的状态进行分类；
    图像获取单元，用于当所述人眼处于预设状态时，则从实时采集的所述图像中获取所述用户图像。
  17. 一种计算机设备，其特征在于，其包括处理器、存储器及存储于所述存储器上并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如权利要求1～8任一项所述的眼球追踪交互方法。
PCT/CN2019/073763 2018-08-31 2019-01-29 眼球追踪交互方法和装置 Ceased WO2020042541A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811011423.8A CN109375765B (zh) 2018-08-31 2018-08-31 眼球追踪交互方法和装置
CN201811011423.8 2018-08-31

Publications (1)

Publication Number Publication Date
WO2020042541A1 true WO2020042541A1 (zh) 2020-03-05

Family

ID=65404744

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/073763 Ceased WO2020042541A1 (zh) 2018-08-31 2019-01-29 眼球追踪交互方法和装置

Country Status (2)

Country Link
CN (1) CN109375765B (zh)
WO (1) WO2020042541A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113255476A (zh) * 2021-05-08 2021-08-13 西北大学 一种基于眼动追踪的目标跟踪方法、系统及存储介质

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109343700B (zh) * 2018-08-31 2020-10-27 深圳市沃特沃德股份有限公司 眼动控制校准数据获取方法和装置
CN109976535B (zh) * 2019-05-05 2022-12-02 北京七鑫易维信息技术有限公司 一种校准方法、装置、设备及存储介质
CN110338750B (zh) * 2019-07-08 2022-04-05 北京七鑫易维信息技术有限公司 一种眼球追踪设备
CN110516553A (zh) * 2019-07-31 2019-11-29 北京航空航天大学 工作状态的监测方法和装置
CN110780742B (zh) * 2019-10-31 2021-11-02 Oppo广东移动通信有限公司 眼球追踪处理方法及相关装置
CN111399659B (zh) * 2020-04-24 2022-03-08 Oppo广东移动通信有限公司 界面显示方法及相关装置
CN113867541A (zh) * 2021-10-19 2021-12-31 歌尔光学科技有限公司 显示画面调节控制方法、装置、设备及存储介质
CN114529972B (zh) * 2022-02-22 2023-04-07 山西医科大学第一医院 一种肌萎缩侧索硬化患者自主呼叫处理方法及系统

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106020461A (zh) * 2016-05-13 2016-10-12 陈盛胜 基于眼球追踪技术的视频交互方法
CN106897426A (zh) * 2017-02-27 2017-06-27 上海禹放信息科技有限公司 基于眼球追踪技术的特定数据生成系统及方法
CN108427503A (zh) * 2018-03-26 2018-08-21 京东方科技集团股份有限公司 人眼追踪方法及人眼追踪装置

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102830793B (zh) * 2011-06-16 2017-04-05 北京三星通信技术研究有限公司 视线跟踪方法和设备
CN102662476B (zh) * 2012-04-20 2015-01-21 天津大学 一种视线估计方法
CN106056092B (zh) * 2016-06-08 2019-08-20 华南理工大学 基于虹膜与瞳孔的用于头戴式设备的视线估计方法
US9996744B2 (en) * 2016-06-29 2018-06-12 International Business Machines Corporation System, method, and recording medium for tracking gaze using only a monocular camera from a moving screen
CN107831900B (zh) * 2017-11-22 2019-12-10 中国地质大学（武汉） 一种眼控鼠标的人机交互方法及系统

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106020461A (zh) * 2016-05-13 2016-10-12 陈盛胜 基于眼球追踪技术的视频交互方法
CN106897426A (zh) * 2017-02-27 2017-06-27 上海禹放信息科技有限公司 基于眼球追踪技术的特定数据生成系统及方法
CN108427503A (zh) * 2018-03-26 2018-08-21 京东方科技集团股份有限公司 人眼追踪方法及人眼追踪装置

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113255476A (zh) * 2021-05-08 2021-08-13 西北大学 一种基于眼动追踪的目标跟踪方法、系统及存储介质
CN113255476B (zh) * 2021-05-08 2023-05-19 西北大学 一种基于眼动追踪的目标跟踪方法、系统及存储介质

Also Published As

Publication number Publication date
CN109375765A (zh) 2019-02-22
CN109375765B (zh) 2020-10-09

Similar Documents

Publication Publication Date Title
CN109343700B (zh) 眼动控制校准数据获取方法和装置
CN109375765B (zh) 眼球追踪交互方法和装置
Li et al. Learning to predict gaze in egocentric video
CN105184246B (zh) 活体检测方法和活体检测系统
US10671156B2 (en) Electronic apparatus operated by head movement and operation method thereof
US9750420B1 (en) Facial feature selection for heart rate detection
US9075453B2 (en) Human eye controlled computer mouse interface
US10957069B2 (en) Head pose estimation from local eye region
KR101288447B1 (ko) 시선 추적 장치와 이를 이용하는 디스플레이 장치 및 그 방법
CN114391117A (zh) 眼睛跟踪延迟增强
WO2023071884A1 (zh) 注视检测方法、电子设备的控制方法及相关设备
CN108985210A (zh) 一种基于人眼几何特征的视线追踪方法及系统
CN110032271A (zh) 对比度调节装置及其方法、虚拟现实设备及存储介质
CN105912126B (zh) 一种手势运动映射到界面的增益自适应调整方法
CN106529409A (zh) 一种基于头部姿态的眼睛注视视角测定方法
CN104143086A (zh) 人像比对在移动终端操作系统上的应用技术
US12189844B2 (en) Eye-gaze tracking apparatus and a method of eye-gaze tracking
WO2023071882A1 (zh) 人眼注视检测方法、控制方法及相关设备
WO2020063000A1 (zh) 神经网络训练、视线检测方法和装置及电子设备
CN107729871A (zh) 基于红外光的人眼活动轨迹追踪方法及装置
CN112183200A (zh) 一种基于视频图像的眼动追踪方法和系统
WO2024113275A1 (zh) 凝视点获取方法和装置、电子设备和存储介质
CN110794963A (zh) 基于深度相机的眼控辅助输入方法
KR102657095B1 (ko) 탈모 상태 정보 제공 방법 및 장치
CN114895790B (zh) 一种人机交互方法、装置、电子设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19854781

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19854781

Country of ref document: EP

Kind code of ref document: A1