CN110341617B - Eyeball tracking method, device, vehicle and storage medium - Google Patents

Eyeball tracking method, device, vehicle and storage medium Download PDF

Info

Publication number
CN110341617B
CN110341617B CN201910611395.1A CN201910611395A CN110341617B CN 110341617 B CN110341617 B CN 110341617B CN 201910611395 A CN201910611395 A CN 201910611395A CN 110341617 B CN110341617 B CN 110341617B
Authority
CN
China
Prior art keywords
eye
vehicle
information
parameter information
eyeball tracking
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910611395.1A
Other languages
Chinese (zh)
Other versions
CN110341617A (en
Inventor
姚涛
杨鑫
赵志忠
黄通兵
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing 7Invensun Technology Co Ltd
Original Assignee
Beijing 7Invensun Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing 7Invensun Technology Co Ltd filed Critical Beijing 7Invensun Technology Co Ltd
Priority to CN201910611395.1A priority Critical patent/CN110341617B/en
Publication of CN110341617A publication Critical patent/CN110341617A/en
Application granted granted Critical
Publication of CN110341617B publication Critical patent/CN110341617B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R16/00—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for
    • B60R16/02—Electric or fluid circuits specially adapted for vehicles and not otherwise provided for; Arrangement of elements of electric or fluid circuits specially adapted for vehicles and not otherwise provided for electric constitutive elements
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K35/00—Instruments specially adapted for vehicles; Arrangement of instruments in or on vehicles
    • B60K35/10—Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/18—Eye characteristics, e.g. of the iris
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K2360/00—Indexing scheme associated with groups B60K35/00 or B60K37/00 relating to details of instruments or dashboards
    • B60K2360/149—Instrument input by detecting viewing direction not otherwise provided for

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Mechanical Engineering (AREA)
  • Combustion & Propulsion (AREA)
  • Chemical & Material Sciences (AREA)
  • Transportation (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Ophthalmology & Optometry (AREA)
  • Optics & Photonics (AREA)
  • Health & Medical Sciences (AREA)
  • Eye Examination Apparatus (AREA)

Abstract

The embodiment of the invention discloses an eyeball tracking method, an eyeball tracking device, a vehicle and a storage medium, wherein the method is applied to eyeball tracking equipment, and the eyeball tracking equipment is configured on the vehicle and comprises the following steps: acquiring an eye image of a user; determining eye parameter information of the user according to the eye image and the target calibration coefficient; and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters. The embodiment of the invention realizes the determination of the gazing information by arranging the portable eyeball tracking equipment on the vehicle, has small occupied space, higher convenience, expandability and robustness compared with the prior art, has no limitation on the vehicle, greatly improves the utilization rate of the eyeball tracking equipment in the vehicle and further improves the safety of the vehicle.

Description

Eyeball tracking method, device, vehicle and storage medium
Technical Field
The embodiment of the invention relates to the technical field of vehicles, in particular to an eyeball tracking method, an eyeball tracking device, a vehicle and a storage medium.
Background
With the development of automatic driving technology, the change of vehicle interior design is continuously promoted, and various devices of different types are presented in the vehicle to support rich man-machine interaction, wherein eye movement interaction is an important interaction mode.
In the prior art, the way to implement eye movement interaction in a vehicle is generally to configure an expensive eye movement module device or a special eye movement identification system in the vehicle. However, the existing eyeball tracking method in the vehicle usually needs to be used together with equipment in the vehicle, and different vehicles need to be provided with special equipment or systems for realizing eye movement interaction, so that the cost is high, the use limitation is strong, and the requirements of people cannot be met.
Disclosure of Invention
The embodiment of the invention provides an eyeball tracking method, an eyeball tracking device, a vehicle and a storage medium, and can solve the technical problems of high cost and strong limitation in the prior art.
In a first aspect, an embodiment of the present invention provides an eyeball tracking method applied to an eyeball tracking device, where the eyeball tracking device is configured on a vehicle, and the method includes:
acquiring an eye image of a user;
determining eye parameter information of the user according to the eye image and the target calibration coefficient;
and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters.
Further, before acquiring the eye image of the user, the method further includes:
and calibrating the initialized calibration coefficient to obtain the target calibration coefficient.
Further, calibrating the initialized calibration coefficient to obtain the target calibration coefficient includes:
projecting calibration light through a projection module according to preset target position information to form a calibration mark;
and calibrating the initialized calibration coefficient according to the eye image when the user gazes at the calibration identifier and the target position information to obtain the target calibration coefficient.
Further, acquiring an eye image of the user includes:
acquiring a user image through an image acquisition module;
and extracting an eye image in the user image through an eye detection algorithm.
Further, determining the eye parameter information of the user according to the eye image and the target calibration coefficient includes:
extracting eye feature information in the eye image;
and determining the eye parameter information according to the eye feature information and the target calibration coefficient, wherein the eye parameter information comprises eyeball center positions and sight line direction vectors.
Further, the target calibration coefficient includes at least one of a corneal curvature radius, a corneal aqueous humor refractive index, an angle between an optical axis of the eye and a visual axis, and a distance from a corneal curvature center to a pupil center position.
Further, the vehicle configuration parameters include a location parameter of at least one vehicle device located within the vehicle cabin.
In a second aspect, an embodiment of the present invention further provides an eyeball tracking method applied to a vehicle, where an eyeball tracking device is configured in the vehicle, including:
receiving eye parameter information sent by the eyeball tracking equipment
Determining the gaze information of the user inside the vehicle according to the eye parameter information, which is determined by the eye tracking method according to the first aspect.
Further, determining the gazing information of the user in the vehicle according to the eye parameter information includes:
acquiring first position information of the eyeball tracking device and second position information of the vehicle;
determining a vehicle coordinate conversion matrix according to the first position information and the second position information;
obtaining processed vehicle eye parameter information according to the vehicle coordinate transformation matrix and the eye parameter information;
determining gaze information of a user inside the vehicle based on the vehicle eye parameter information.
Further, after determining the gazing information of the user in the vehicle according to the eye parameter information, the method further includes:
and executing preset operation according to the watching information.
In a third aspect, an embodiment of the present invention further provides an eyeball tracking apparatus, which is disposed on an eyeball tracking device disposed on a vehicle, and includes:
the image acquisition module is used for acquiring an eye image of a user;
the information determining module is used for determining the eye parameter information of the user according to the eye image and the target calibration coefficient;
and the information sending module is used for sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters.
Further, the eyeball tracking device further comprises:
and the calibration module is used for calibrating the initialized calibration coefficient before acquiring the eye image of the user to obtain the target calibration coefficient.
Further, the calibration module is specifically configured to:
projecting calibration light through a projection module according to preset target position information to form a calibration mark;
and calibrating the initialized calibration coefficient according to the eye image when the user gazes at the calibration identifier and the target position information to obtain the target calibration coefficient.
Further, the image acquisition module is specifically configured to:
acquiring a user image through an image acquisition module;
and extracting an eye image in the user image through an eye detection algorithm.
Further, the information determination module includes:
a feature extraction unit, configured to extract eye feature information in the eye image;
and the eye parameter information unit is used for determining the eye parameter information according to the eye characteristic information and the target calibration coefficient, wherein the eye parameter information comprises an eyeball center position and a sight line direction vector.
Further, the target calibration coefficient includes at least one of a corneal curvature radius, a corneal aqueous humor refractive index, an angle between an optical axis of the eye and a visual axis, and a distance from a corneal curvature center to a pupil center position.
In a fourth aspect, an embodiment of the present invention further provides an eyeball tracking apparatus, which is applied to a vehicle, where an eyeball tracking device is configured, and includes:
an information receiving module for receiving the eye parameter information sent by the eyeball tracking equipment
A gaze information module configured to determine gaze information of a user inside the vehicle according to the eye parameter information, which is determined by the eye tracking method according to the first aspect.
Further, the gaze information module is specifically configured to:
acquiring first position information of the eyeball tracking device and second position information of the vehicle;
determining a vehicle coordinate conversion matrix according to the first position information and the second position information;
obtaining processed vehicle eye parameter information according to the vehicle coordinate transformation matrix and the eye parameter information;
determining gaze information of a user inside the vehicle based on the vehicle eye parameter information.
Further, the eyeball tracking device further comprises;
and the operation module is used for executing preset operation according to the watching information after the watching information of the user in the vehicle is determined according to the eye parameter information.
In a fifth aspect, an embodiment of the present invention further provides a vehicle, where an eyeball tracking apparatus is configured, where the vehicle includes:
one or more first processors;
a first storage device for storing one or more first programs;
when the one or more first programs are executed by the one or more first processors, causing the one or more first processors to implement the eye tracking method according to the second aspect;
wherein the eye tracking device comprises:
one or more second processors;
a projection module for projecting light;
the communication module is used for sending the eye parameter information to the vehicle;
the image acquisition module is used for acquiring a user image;
the light source module is used for providing a visible light source for the projection equipment;
second storage means for storing one or more second programs;
when the one or more second programs are executed by the one or more second processors, the one or more second processors are caused to implement the eye tracking method according to the first aspect.
In a sixth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, which when executed by a processor implements the eye tracking method according to the first aspect and/or the eye tracking method according to the second aspect.
According to the embodiment of the invention, an eye image of a user is acquired through an eyeball tracking device arranged on a vehicle, and the eye parameter information of the user is determined according to the eye image and a target calibration coefficient; and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters. The embodiment of the invention realizes the determination of the gazing information by arranging the portable eyeball tracking equipment on the vehicle, has small occupied space, higher convenience, expandability and robustness compared with the prior art, has no limitation on the vehicle, greatly improves the utilization rate of the eyeball tracking equipment in the vehicle and further improves the safety of the vehicle.
Drawings
Fig. 1 is a flowchart illustrating an eye tracking method according to a first embodiment of the invention;
fig. 2 is a schematic view illustrating an eyeball tracking method according to a first embodiment of the invention;
FIG. 3 is an external view of an eye tracking apparatus according to a first embodiment of the present invention;
FIG. 4 is a flowchart illustrating an eye tracking method according to a second embodiment of the present invention;
FIG. 5 is a flowchart illustrating an eye tracking method according to a third embodiment of the present invention;
FIG. 6 is a diagram illustrating spatial transformation according to a third embodiment of the present invention;
fig. 7 is a schematic structural diagram of an eye tracking device according to a fourth embodiment of the present invention;
fig. 8 is a schematic structural diagram of an eye tracking device according to a fifth embodiment of the present invention;
fig. 9 is a schematic structural view of a vehicle in a sixth embodiment of the invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of an eyeball tracking method in an embodiment of the present invention, which may be implemented by an eyeball tracking apparatus installed on an eyeball tracking device, where the eyeball tracking apparatus may be implemented in software and/or hardware. The eyeball tracking device is arranged in the vehicle and can interact with the vehicle.
Fig. 2 is a schematic diagram of an eye tracking method according to an embodiment of the present invention, the eye tracking method in this embodiment can be applied to an eye tracking device 11 disposed on a vehicle, and a specific position of the eye tracking device 11 can be set according to actual conditions, for example, the eye tracking device 11 can be disposed in a reserved slot of a carriage, and a position of the eye tracking device 11 in the reserved slot can be adjusted up, down, left, and right within a preset range according to a user's requirement. It is understood that the position of the reserved slot is not limited in this embodiment, and may be, for example, in front of a gear controller or in front of a windshield. The eyeball tracking device 11 can be connected with the vehicle through a communication protocol in the communication module, so that information interaction is realized.
The eyeball tracking device 11 in the figure is described by taking an example of being arranged in front of the gear controller 16, the eyeball tracking device 11 in the figure acquires an eye image of an eyeball 14 of a user, obtains eye parameter information according to the eye image of the user and a pre-calibrated target calibration coefficient, and sends the eye parameter information to a vehicle, and the vehicle can determine the gaze information of the user in the vehicle according to the eye parameter information and vehicle configuration parameters, namely, a gaze point in the figure is located at a point a in the display screen 12. The image capture area of the eye tracking device 11 is shown as box 12, in which the steering wheel 15 of the vehicle is in front of the user's eyes 14.
Fig. 3 is an external schematic view of an eye tracking device according to a first embodiment of the present invention, where the eye tracking device according to the present embodiment may be a portable device, and may be integrated into an electronic device shaped like a cube, and in fig. 3, the projection module 120, the communication module 130, and the image capturing module 140 may be shown outside the eye tracking device 11, and other modules (such as a control module, a sound module, and a light source module) may be arranged inside the eye tracking device.
As shown in fig. 1, the method may specifically include:
and S110, acquiring an eye image of the user.
The eye image is an image including eyeballs of the user.
Specifically, the eyeball tracking device can acquire a user image through the image acquisition module and extract an eye image in the user image through an eye detection algorithm. The user image may be an image including a human face and an image not including a human face. And after the user image is acquired, searching the face in the user image, and returning to acquire the user image again if the face is not searched in the user image until the face can be searched in the acquired user image. In this embodiment, the method for searching a face is not limited, for example, a haar-like face detector algorithm is used to detect a face or a face feature (for example, skin color or nose) is used to detect a face of a user image, and whether the user image includes a face is determined according to a matching value between the face feature in the user image and a standard feature.
Further, the eye image in the user image is extracted by an eye detection algorithm, wherein the eye detection algorithm is not limited, for example, the eye detection algorithm may include a template matching based method or a statistical based method, etc. In this embodiment, a statistical-based method is taken as an example to describe, a set of model parameters is obtained by training and learning a large number of samples including eye images and samples not including eye images, and then a classifier or a filter is constructed based on the model to detect the eye images in the user image. After an eye image is detected in the user image, the eye image is extracted.
And S120, determining the eye parameter information of the user according to the eye image and the target calibration coefficient.
The target calibration coefficients are used for estimating the gazing information by measuring the eye movement condition, and the gazing information may include a sight line or a gazing point and the like. The specific method for determining the target calibration coefficient may be set according to actual conditions, and this embodiment is described by taking a pupil-cornea reflection method in an optical recording method as an example, and other methods that can implement gaze tracking are also applicable.
Specifically, determining the eye parameter information of the user according to the eye image and the target calibration coefficient may include: extracting eye feature information in the eye image; and determining eye parameter information according to the eye characteristic information and the target calibration coefficient, wherein the eye parameter information comprises eyeball center positions and sight line direction vectors. The eye feature information may include a pupil center position, a pupil shape, an iris position, an iris shape, an eyelid position, an eye corner position, a light spot (also referred to as purkinje spot) position, and the like.
In this step, the eye image may be processed to obtain a gradient value of the gray scale in a set direction, a position where a maximum gradient value of the gray scale is located may be determined as a pupil edge feature of the eye image, then the pupil edge feature may be fitted to determine a center position of the fitted image, and the center position may be determined as a pupil center position in the image. It is to be understood that the means for determining the pupil center position is not limited thereto. Further, the gray image after the eye image is processed may be obtained by a spot recognition algorithm that the region with the higher gray value is a purkinje spot, and the obtained position of the purkinje spot is a light spot position.
Further, the eye parameter information may include an eyeball center position and a gaze direction vector, the eye feature information may include a pupil center position and a light spot position, and determining the eye parameter information according to the eye feature information and the target calibration coefficient may include: according to the light spot position and the corneal curvature radius in the target calibration coefficient, a corneal curvature center is obtained, and the corneal curvature center is determined as the eyeball center position; and obtaining a sight line direction vector according to the pupil center position, the cornea curvature center and the included angle between the optical axis of the eye part and the visual axis in the target calibration coefficient. The optical axis is a connecting line from the pupil center to the retina center, the visual axis is a connecting line from the extraocular fixation point to the macula lutea through a node, and an included angle of 4-5 degrees is formed between the optical axis of the eyes and the visual axis. The target calibration coefficients in this embodiment may include the corneal radius of curvature, the refractive index of the aqueous humor of the cornea, the angle between the optical axis of the eye and the visual axis, the distance from the center of the cornea to the center of the pupil, and the like.
And S130, transmitting the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters.
Wherein the eye parameter information is based on a coordinate system of the eye tracking device. The vehicle configuration parameters comprise position parameters of at least one vehicle device located in the vehicle compartment, see fig. 2, which may comprise a steering wheel 15 or a windshield, a HUD (not shown in fig. 2), etc., the position parameters being coordinates of one vehicle device based on a vehicle coordinate system.
Specifically, after the eyeball tracking device acquires the eye parameter information, the eye parameter information may be sent to the vehicle, so that the vehicle converts the eye parameter information into the eye parameter information in the vehicle coordinate system according to the coordinate transformation matrix determined in advance, and determines the user's gazing information in the vehicle according to the eye parameter information and the vehicle configuration parameters.
According to the technical scheme provided by the embodiment, the eye image of the user is acquired through the eyeball tracking equipment arranged on the vehicle, and the eye parameter information of the user is determined according to the eye image and the target calibration coefficient; and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters. The present embodiment has realized gazing information's affirmation through set up a portable eyeball tracking equipment on the vehicle, compares in prior art, and occupation space is little, has bigger convenience, scalability and robustness to no restriction to the vehicle itself, greatly improved eyeball tracking equipment's rate of utilization in the vehicle, further improved the security of vehicle.
Example two
Fig. 4 is a flowchart of an eyeball tracking method in a second embodiment of the present invention, wherein the eyeball tracking method is further optimized in the present embodiment based on the above-mentioned embodiment. Correspondingly, the method of the embodiment specifically includes:
and S210, calibrating the initialized calibration coefficient to obtain a target calibration coefficient.
In this embodiment, before the eyeball tracking in the vehicle is performed, the calibration coefficient in the eyeball tracking device is calibrated in advance. A specific method for determining the target calibration coefficient is described by taking a pupil-cornea reflection method as an example in the optical recording method.
Specifically, calibrating the initialized calibration coefficient to obtain the target calibration coefficient may include: projecting calibration light through a projection module according to preset target position information to form a calibration mark; and calibrating the initialized calibration coefficient according to the eye image and the target position information when the user watches the calibration mark to obtain a target calibration coefficient.
The target position information is set as position information on a target plane, and the target plane may be a surface of any planar object, such as a horizontal desktop. If the spatial relationship between the calibration device for eye tracking and the target position is predetermined, the target position is not limited to a plane. The number of the calibration marks can be multiple, the calibration marks are not limited to one mark point, and the calibration marks can also be a calibration image and a calibration picture which play a role in marking. For example, the shape of the calibration mark in the present embodiment may be a circle, a square, a line, other irregular shapes, and the like, and the specific shape may be set according to actual situations.
The eyeball tracking device can extract eye feature information obtained from a user image when the user gazes at the calibration mark, obtain a sight error according to the eye feature information, the target position information and the initialized calibration coefficient, calibrate the target calibration coefficient according to the sight error, and calibrate and determine the target calibration coefficient according to the sight error. The calibration coefficient may be a parameter to be determined in an eye tracking model for performing gaze estimation, the initialized calibration coefficient is a calibration coefficient determined according to prior knowledge, and the calibration coefficient is actually different for different users, so that calibration is required.
Wherein, the sight line error can be defined as the angle difference between the eye visual axis direction vector and the estimated direction vector from the eye to the calibration mark. The eyeball tracking device can obtain the visual axis direction vector of the eye and the direction vector from the eye to the calibration mark based on the eye characteristic information, the target position and the initialized target calibration coefficient; based on the angle error formula, the eye visual axis direction vector and the direction vector from the eye to the calibration mark, the error of the eye visual axis direction can be obtained, and the error of the eye visual axis direction is determined as the sight error. The calibration unit may then determine a minimum error of the line-of-sight error based on a given error optimization algorithm, and determine a calibration coefficient corresponding to the minimum error as a calibrated target calibration coefficient.
And S220, acquiring an eye image of the user.
And S230, extracting eye feature information in the eye image.
And S240, determining the eye parameter information of the user according to the eye characteristic information and the target calibration coefficient.
Specifically, the eye parameter information may include an eyeball center position and a gaze direction vector, the eye feature information may include a pupil center position and a light spot position, and the determining of the eye parameter information according to the eye feature information and a target calibration coefficient in the target calibration coefficient may include: according to the light spot position and the corneal curvature radius in the target calibration coefficient, a corneal curvature center is obtained, and the corneal curvature center is determined as the eyeball center position; and obtaining a sight line direction vector according to the pupil center position, the cornea curvature center and the included angle between the optical axis of the eye part and the visual axis in the target calibration coefficient.
Wherein, the corneal curvature center can be represented by C, the pupil center position can be represented by P, and the eye visual axis direction vector can be represented by VSAnd (4) showing. The corneal center of curvature may be defined by C ═ f1(gi,LiR) determination of where giIndicating the spot position, LiCoordinates of a light source in the eye tracking apparatus are represented, and R represents a corneal radius of curvature in the target calibration coefficient. The pupil center position can be defined by P ═ f2(p, C, K, n), wherein p represents the pupil center position coordinates in the eye image, K represents the cornea center to pupil center position distance in the target calibration coefficient, and n represents the cornea aqueous humor refractive index in the target calibration coefficient. The direction vector of the optical axis of the eye can pass
Figure BDA0002122485850000131
Is determined in which
Figure BDA0002122485850000133
And spherical coordinate representation of unit vectors representing the direction of the optical axis of the eye. The visual line direction vector (also called eye visual axis direction vector) can pass through
Figure BDA0002122485850000132
And determining, wherein (alpha, beta) represents the included angle between the optical axis of the eye and the visual axis.
And S250, sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters.
Wherein the eye parameter information is based on a coordinate system of the eye tracking device. The vehicle configuration parameters comprise position parameters of at least one vehicle device located in the vehicle compartment, see fig. 2, which may comprise a steering wheel 15 or a windshield, a HUD (not shown in fig. 2), etc., the position parameters being coordinates of one vehicle device based on a vehicle coordinate system.
Specifically, after the eyeball tracking device acquires the eye parameter information, the eye parameter information may be sent to the vehicle, so that the vehicle converts the eye parameter information into the eye parameter information in the vehicle coordinate system according to the coordinate transformation matrix determined in advance, and determines the user's gazing information in the vehicle according to the eye parameter information and the vehicle configuration parameters.
According to the technical scheme provided by the embodiment, the eye image of the user is acquired through the eyeball tracking equipment arranged on the vehicle, and the eye parameter information of the user is determined according to the eye image and the calibrated target calibration coefficient; and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters. The portable eyeball tracking equipment is arranged on the vehicle, so that the fixation information is determined, compared with the prior art, the portable eyeball tracking equipment has the advantages of small occupied space, higher convenience, expandability and robustness, no limitation on the vehicle, great improvement on the utilization rate of the eyeball tracking equipment in the vehicle, and further improvement on the safety of the vehicle; compared with the prior art that recalibration is needed when eyeball tracking is carried out each time, in the embodiment, the target calibration coefficient of the eyeball tracking equipment is calibrated in advance, interaction can be directly carried out with a vehicle, and the eye movement interaction efficiency of a user is improved.
EXAMPLE III
Fig. 5 is a flowchart of an eyeball tracking method in a third embodiment of the invention. The present embodiment is applicable to the case of implementing eyeball tracking in a vehicle, and the method may be executed by an eyeball tracking device disposed in a vehicle, and the device may be implemented in a software and/or hardware manner. An eye tracking device is configured in the vehicle. The method specifically comprises the following steps:
and S310, receiving eye parameter information sent by the eyeball tracking equipment.
The eye parameter information may be determined by the eye tracking method in the first and second embodiments, and the eye parameter information is expressed based on the coordinate system of the eye tracking device. In this embodiment, the communication module disposed in the eyeball tracking apparatus in the vehicle may receive the eye parameter information sent by the eyeball tracking device.
And S320, determining the watching information of the user in the vehicle according to the eye parameter information.
Specifically, determining the gaze information of the user in the vehicle according to the eye parameter information sent by the eyeball tracking device may include: acquiring first position information of eyeball tracking equipment and second position information of a vehicle; determining a vehicle coordinate conversion matrix according to the first position information and the second position information; obtaining processed vehicle eye parameter information according to the vehicle coordinate transformation matrix and the eye parameter information; gaze information of a user inside a vehicle is determined based on the vehicle eye parameter information.
The second position information of the vehicle may be an origin of a coordinate system of the vehicle, and the setting of the origin is not limited in this embodiment, for example, the origin may be set at the center of the windshield, and the like. The first location information of the eye tracking device may be determined after the eye tracking device is connected to the vehicle. The first position information of the eye tracking apparatus and the second position information of the vehicle may be represented by a unified virtual coordinate system, and a vehicle coordinate transformation matrix may be determined based on the first position information and the second position information. In this embodiment, the predetermined vehicle coordinate transformation matrix may be stored in the vehicle configuration parameters and directly obtained. The vehicle coordinate transformation matrix may include a rotation matrix D and a translation matrix T, referring to fig. 6, fig. 6 is a schematic diagram of spatial transformation in the third embodiment of the present invention, where the coordinate system (x1, y1, z1) is a coordinate system of the eyeball tracking device, the coordinate system (x2, y2, z2) is a vehicle coordinate system, the received eye parameter information of the eyeball 14 may include an eyeball center position P and a gaze direction vector V, the received eye parameter information is represented based on the coordinate system of the eyeball tracking device, and the coordinate transformation matrix is [ D, T ] in the diagram. The vehicle eye parameter information based on the vehicle coordinate system can be obtained through the coordinate transformation matrix and the received eye parameter information, and particularly through Transform (p, v) ═ D | T ] (p, v) ═ p1, v1, wherein p1 is the eyeball center position based on the vehicle coordinate system, and v1 is the sight line direction vector based on the vehicle coordinate system.
After the vehicle eye parameter information based on the vehicle coordinate system is acquired, the intersection point of the sight line direction vector v1 in the vehicle eye parameter information in the compartment, namely the fixation point, can be obtained by solving. For example, assuming that the normal vector of the interaction plane whose sight line direction falls within the vehicle compartment is n1, based on the spatial geometric formula, the intersection point of the straight line with the starting point p1 and the vector direction v1 and the interaction plane can be obtained. If the intersection point is located on the display screen of the vehicle, the pixel coordinates of the intersection point on the display screen can be further obtained.
It can be understood that, if the position of the eyeball tracking device in the reserved slot of the vehicle can be adjusted vertically and horizontally within a preset range according to the requirements of a user, the eyeball tracking device can correspond to a plurality of adjusting positions at the moment, and the coordinate transformation matrix corresponding to each adjusting position is different, so that the corresponding vehicle coordinate transformation matrix is stored in each adjusting position, and the eyeball tracking can be performed more efficiently subsequently.
And S330, executing preset operation according to the gazing information.
The preset operation can be set according to actual conditions.
The eyeball tracking method in the embodiment has no requirement on the vehicle, and is suitable for any vehicle needing eyeball tracking, for example, whether the vehicle comprises a display screen or not is not limited. When the vehicle includes a display screen, referring to fig. 2, if the fixation point is determined to be point a in the display screen 12, the vehicle can mark and display point a for the user to see. When the vehicle includes a display screen, the preset operation may be an operation generally implemented by physical contact in the prior art, such as clicking or selecting, implemented on the display screen, and specifically, when the gaze information satisfies the operation condition, the preset operation corresponding to the operation condition may be executed. The operation conditions may correspond to the preset operations one to one, and the specific operation conditions may be set as needed, for example, the operation conditions may be that the gazing time is greater than a preset time threshold, and the corresponding preset operation is to click on a position corresponding to the current gazing point on the display screen.
When the vehicle does not comprise a display screen, the vehicle configuration parameters are known, and after the vehicle determines the gaze information of the user in the vehicle according to the eye parameter information sent by the eyeball tracking device, the vehicle device with the gaze point of the user specifically located in the compartment can be determined. If the vehicle device where the user's point of regard is located is the same as the preset vehicle device, the preset operation may be performed on the preset vehicle device. The preset operation can be set according to actual conditions, for example, when the preset vehicle device is a radio in a carriage, the preset operation can be set to be operations such as starting or closing; for another example, when the preset vehicle device is a front windshield, the preset operation may be set to an operation of starting a wiper or the like built in the front windshield.
In addition, after the vehicle determines the gaze information of the user in the vehicle according to the eye parameter information sent by the eyeball tracking device, if the sight line direction of the user is not in the preset direction, the user can be determined to be in an exhausted state or an unconcentrated state, and the preset operation can be fatigue prompt or alarm prompt through a sound device in the vehicle or starting an emergency braking measure of the vehicle, so as to ensure the driving safety of the user. The preset direction may be a front road direction or a left-right road direction.
According to the technical scheme provided by the embodiment, the vehicle receives the eye parameter information sent by the eyeball tracking equipment arranged in the vehicle, the watching information of the user in the vehicle is determined according to the eye parameter information, and the preset operation is executed according to the watching information. The eyeball tracking method in the embodiment has no limitation on the vehicle, so that the use rate of the eyeball tracking equipment in the vehicle is greatly improved, and the safety of the vehicle is further improved.
Example four
Fig. 7 is a schematic structural diagram of an eyeball tracking device in a fourth embodiment of the present invention, which is suitable for implementing eyeball tracking in a vehicle. The eyeball tracking device provided by the embodiment of the invention can execute the eyeball tracking method provided by the first embodiment and the second embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method. The eye tracking apparatus may be disposed in an eye tracking device, which may be disposed in a vehicle.
The device specifically comprises an image acquisition module 410, an information determination module 420 and an information sending module 430, wherein:
an image acquisition module 410, configured to acquire an eye image of a user;
an information determining module 420, configured to determine eye parameter information of the user according to the eye image and the target calibration coefficient;
the information sending module 430 is configured to send the eye parameter information to the vehicle, so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameter.
According to the embodiment of the invention, an eye image of a user is acquired through an eyeball tracking device arranged on a vehicle, and the eye parameter information of the user is determined according to the eye image and a target calibration coefficient; and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters. The embodiment of the invention realizes the determination of the gazing information by arranging the portable eyeball tracking equipment on the vehicle, has small occupied space, higher convenience, expandability and robustness compared with the prior art, has no limitation on the vehicle, greatly improves the utilization rate of the eyeball tracking equipment in the vehicle and further improves the safety of the vehicle.
Further, the eye tracking apparatus further includes:
and the calibration module is used for calibrating the initialized calibration coefficient before acquiring the eye image of the user to obtain a target calibration coefficient.
Further, the calibration module is specifically configured to:
projecting calibration light through a projection module according to preset target position information to form a calibration mark;
and calibrating the initialized calibration coefficient according to the eye image and the target position information when the user watches the calibration mark to obtain a target calibration coefficient.
Further, the image obtaining module 410 is specifically configured to:
acquiring a user image through an image acquisition module;
and extracting an eye image in the user image through an eye detection algorithm.
Further, the information determining module 420 includes:
a feature extraction unit configured to extract eye feature information in the eye image;
and the eye parameter information unit is used for determining eye parameter information according to the eye characteristic information and the target calibration coefficient, wherein the eye parameter information comprises eyeball center positions and sight line direction vectors.
Further, the target calibration factor includes at least one of a corneal radius of curvature, a corneal aqueous humor refractive index, an angle between an optical axis of the eye and a visual axis, and a distance from a center of corneal curvature to a pupil center position.
The eyeball tracking device provided by the embodiment of the invention can execute the eyeball tracking method provided by the first embodiment and the second embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.
EXAMPLE five
Fig. 8 is a schematic structural diagram of an eyeball tracking device in a fifth embodiment of the present invention, which is suitable for implementing eyeball tracking in a vehicle. The eyeball tracking device provided by the embodiment of the invention can execute the eyeball tracking method provided by the third embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. The eyeball tracking device is arranged in a vehicle, and eyeball tracking equipment is arranged in the vehicle.
The apparatus specifically includes an information receiving module 510 and a gaze information module 520, wherein:
an information receiving module 510, configured to receive eye parameter information sent by an eyeball tracking device;
a gazing information module 520, configured to determine gazing information of the user in the vehicle according to the eye parameter information sent by the eye tracking device, where the eye parameter information is determined by the eye tracking method in the first embodiment and the second embodiment.
In the embodiment of the invention, the vehicle receives the eye parameter information sent by the eyeball tracking equipment arranged in the vehicle, and determines the gazing information of the user in the vehicle according to the eye parameter information. The embodiment of the invention has no limitation on the vehicle, greatly improves the utilization rate of the eyeball tracking equipment in the vehicle, and further improves the safety of the vehicle.
Further, the gaze information module 510 is specifically configured to:
acquiring first position information of eyeball tracking equipment and second position information of a vehicle;
determining a vehicle coordinate transformation matrix according to the first position information and the vehicle configuration parameters;
obtaining processed vehicle eye parameter information according to the vehicle coordinate transformation matrix and the eye parameter information;
gaze information of a user inside a vehicle is determined based on the vehicle eye parameter information.
Further, the eyeball tracking device also comprises;
and the operation module is used for executing preset operation according to the watching information after the watching information of the user in the vehicle is determined according to the eye parameter information.
The eyeball tracking device provided by the embodiment of the invention can execute the eyeball tracking method provided by the third embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE six
Fig. 9 is a schematic structural diagram of a vehicle in a sixth embodiment of the invention, and fig. 9 shows a block diagram of an exemplary vehicle suitable for implementing an embodiment of the invention. The vehicle shown in fig. 9 is only an example, and should not bring any limitation to the function and the range of use of the embodiment of the present invention.
As shown in fig. 9, the vehicle is configured with an eye tracking device, which may include but is not limited to: one or more first processors 60, a first memory device 61, a first communication device 62, a first input device 63 and a first output device 64. A first processor 60 is shown as an example. The number of the first storage devices 61 in the vehicle may be one or more, and one first storage device 61 is exemplified in fig. 9. The first processor 60, the first storage device 61, the first communication device 62, the first input device 63 and the first output device 64 of the vehicle may be connected by a bus or other means, and fig. 9 illustrates the connection by a bus as an example.
The eye tracking device in fig. 9 may include, but is not limited to: one or more second processors 70, a second memory device 71, a second communication device 72, a second input device 73, and a second output device 74. A second processor 70 is shown as an example. The number of the second storage devices 71 in the vehicle may be one or more, and one second storage device 71 is exemplified in fig. 9. The second processor 70, the second storage device 71, the second communication device 72, the second input device 73, and the second output device 74 of the vehicle may be connected by a bus or other means, and fig. 9 illustrates the connection by a bus as an example. The eyeball tracking device in the figure can further comprise a projection module, a communication module, an image acquisition module and a light source module, wherein the projection module is used for projecting calibration light; the communication module is used for sending the eye parameter information to the vehicle; the image acquisition module is used for acquiring a user image; and the light source module is used for providing a visible light source for the projection equipment.
The vehicle may be connected via the first communication means 62 thereof to the second communication means 72 of the eye tracking device therein. The specific implementation composition of the first communication device 62 and the second communication device 72 can be set according to actual situations, and the specific connection mode is not limited in this embodiment. For example, the first communication device 62 and the second communication device 72 may be connected via bluetooth.
The first storage device 61 and the second storage device 71 are a computer-readable storage medium that can be used to store software programs, computer-executable programs, and modules, the first storage device 61 being used to store one or more first programs, and the second storage device 71 being used to store one or more second programs. The first storage device 61 and the second storage device 71 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to use of the device, and the like. Further, the first storage means 61 and the second storage means 71 may comprise high speed random access memory, and may further comprise non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the first and second storage devices 61, 71 may further include memories remotely located with respect to the first and second processors 60, 70, respectively, which may be connected to the device via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The first and second input means 63 and 73 may be used to receive input numeric or character information and generate key signal inputs related to user settings and function control, and may also be a camera for acquiring images. The first output device 64 and the second output device 74 may include audio devices such as speakers. It should be noted that the specific composition of the first input device 63, the second input device 73, the first output device 64 and the second output device 7 can be set according to actual situations.
The first processor 60 executes various functional applications and data processing by executing the first program stored in the first storage device 61, for example, implements an eye tracking method provided by a third embodiment of the present invention, and is applied to a vehicle in which an eye tracking apparatus is disposed, and the method includes:
receiving eye parameter information sent by eyeball tracking equipment;
and determining the gaze information of the user in the vehicle according to the eye parameter information, wherein the eye parameter information is determined by the eyeball tracking method of the first embodiment and the second embodiment of the invention.
The second processor 70 executes various functional applications and data processing by executing the second program stored in the second storage device 71, for example, implementing the eyeball tracking method provided in the first and second embodiments of the present invention, and is applied to an eyeball tracking device, which is disposed on a vehicle, and includes:
acquiring an eye image of a user;
determining eye parameter information of the user according to the eye image and the target calibration coefficient;
and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters.
EXAMPLE seven
A seventh embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the eyeball tracking method according to the first and second embodiments of the present invention and/or the eyeball tracking method according to the third embodiment of the present invention.
The eyeball tracking method provided by the first embodiment and the second embodiment of the invention is applied to eyeball tracking equipment, and the eyeball tracking equipment is configured on a vehicle and comprises the following steps:
acquiring an eye image of a user;
determining eye parameter information of the user according to the eye image and the target calibration coefficient;
and sending the eye parameter information to the vehicle so that the vehicle determines the gazing information of the user in the vehicle according to the eye parameter information and the vehicle configuration parameters.
The eyeball tracking method provided by the third embodiment of the invention is applied to a vehicle, and eyeball tracking equipment is configured in the vehicle, and the eyeball tracking method comprises the following steps:
receiving eye parameter information sent by eyeball tracking equipment;
and determining the gaze information of the user in the vehicle according to the eye parameter information, wherein the eye parameter information is determined by the eyeball tracking method of the first embodiment and the second embodiment of the invention.
Computer storage media for embodiments of the invention may employ any combination of one or more computer-readable media. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims (10)

1. An eyeball tracking method is applied to eyeball tracking equipment, the eyeball tracking equipment is arranged in a reserved slot of a vehicle, the position of the eyeball tracking equipment in the reserved slot can be adjusted up, down, left and right within a preset range, the eyeball tracking equipment comprises a projection module, and the method comprises the following steps:
projecting calibration light through the projection module according to preset target position information to form a calibration mark;
calibrating the initialized calibration coefficient according to the eye image when the user watches the calibration identifier and the target position information to obtain a target calibration coefficient;
acquiring an eye image of a user;
determining eye parameter information of the user according to the eye image and the target calibration coefficient;
the eye parameter information is sent to the vehicle, so that the vehicle converts the eye parameter information into eye parameter information under a vehicle coordinate system according to a coordinate transformation matrix determined in advance, and the user's watching information in the vehicle is determined according to the eye parameter information and vehicle configuration parameters; the eyeball tracking equipment corresponds to a plurality of adjusting positions when being adjusted up, down, left and right within a preset range, and each adjusting position is correspondingly stored with a different coordinate transformation matrix;
the eye parameter information comprises eyeball center positions and sight line direction vectors, and the eye parameter information of the user is determined according to the eye image and the target calibration coefficient, and comprises the following steps:
extracting eye characteristic information in the eye image, wherein the eye characteristic information comprises a pupil center position and a light spot position;
according to the light spot position and the corneal curvature radius in the target calibration coefficient, a corneal curvature center is obtained, and the corneal curvature center is determined as the eyeball center position;
and obtaining a sight line direction vector according to the pupil center position, the cornea curvature center and the included angle between the optical axis of the eye part and the visual axis in the target calibration coefficient.
2. The method of claim 1, wherein obtaining an image of the user's eyes comprises:
acquiring a user image through an image acquisition module;
and extracting an eye image in the user image through an eye detection algorithm.
3. The method of claim 1, wherein calibrating the initialized calibration coefficients according to the eye image when the user gazes at the calibration identifier and the target position information to obtain target calibration coefficients comprises:
extracting the eye feature information from the user image when the user watches the calibration mark;
obtaining a sight error according to the eye feature information, the target position information and the initialized calibration coefficient;
determining a target calibration coefficient according to the sight error calibration;
the method comprises the steps of obtaining eye visual axis direction vectors and eye to calibration identification direction vectors based on eye characteristic information, target position information and initialized calibration coefficients, obtaining eye visual axis direction vectors and eye to calibration identification direction vectors based on an angle error formula, the eye visual axis direction vectors and the eye to calibration identification direction vectors, obtaining eye visual axis direction errors, determining the eye visual axis direction errors as visual line errors, determining the minimum error of the visual line errors based on a given error optimization algorithm, and determining calibration coefficients corresponding to the minimum error as calibrated target calibration coefficients.
4. The method of claim 1, wherein the vehicle configuration parameters include a location parameter of at least one vehicle device located within a vehicle cabin.
5. An eyeball tracking method is applied to a vehicle, the vehicle comprises a reserved slot, eyeball tracking equipment is configured in the reserved slot, the eyeball tracking equipment comprises a projection module, and the position of the eyeball tracking equipment in the reserved slot can be adjusted up, down, left and right within a preset range, and the method comprises the following steps:
receiving eye parameter information sent by the eyeball tracking equipment;
the method comprises the steps of converting eye parameter information into eye parameter information under a vehicle coordinate system according to a coordinate transformation matrix determined in advance, determining the watching information of a user in a vehicle according to the eye parameter information and vehicle configuration parameters, wherein the eye parameter information is determined through the eyeball tracking method according to any one of claims 1 to 4, the eyeball tracking device corresponds to a plurality of adjusting positions when being adjusted up, down, left and right within a preset range, and different coordinate transformation matrices are stored in each adjusting position correspondingly.
6. The method of claim 5, wherein after determining user gaze information inside the vehicle from the ocular parameter information, further comprising:
and executing preset operation according to the watching information.
7. An eyeball tracking device, which is characterized in that the eyeball tracking device is arranged in a reserved slot of a vehicle, the position in the reserved slot can be adjusted up, down, left and right within a preset range, the eyeball tracking device comprises a projection module, and the device comprises:
the calibration module is used for projecting calibration light through the projection module according to preset target position information to form a calibration mark; calibrating the initialized calibration coefficient according to the eye image when the user watches the calibration identifier and the target position information to obtain a target calibration coefficient;
the image acquisition module is used for acquiring an eye image of a user;
the information determining module is used for determining the eye parameter information of the user according to the eye image and the target calibration coefficient;
the information sending module is used for sending the eye parameter information to the vehicle so that the vehicle converts the eye parameter information into eye parameter information under a vehicle coordinate system according to a coordinate transformation matrix determined in advance, and the information of the user gazing inside the vehicle is determined according to the eye parameter information and vehicle configuration parameters; the eyeball tracking equipment corresponds to a plurality of adjusting positions when being adjusted up, down, left and right within a preset range, and each adjusting position is correspondingly stored with a different coordinate transformation matrix;
the eye parameter information comprises eyeball center positions and sight line direction vectors, and the eye parameter information of the user is determined according to the eye image and the target calibration coefficient, and comprises the following steps:
extracting eye characteristic information in the eye image, wherein the eye characteristic information comprises a pupil center position and a light spot position;
according to the light spot position and the corneal curvature radius in the target calibration coefficient, a corneal curvature center is obtained, and the corneal curvature center is determined as the eyeball center position;
and obtaining a sight line direction vector according to the pupil center position, the cornea curvature center and the included angle between the optical axis of the eye part and the visual axis in the target calibration coefficient.
8. The utility model provides an eyeball tracer, its characterized in that sets up in the vehicle, include in the vehicle and reserve the slot, it has eyeball tracking equipment to dispose in the slot to reserve, eyeball tracking equipment contains projection module, eyeball tracking equipment is in position in the slot of reserving can be in the within range of predetermineeing about adjusting from top to bottom, the device includes:
the information receiving module is used for receiving the eye parameter information sent by the eyeball tracking equipment;
the eye tracking device comprises a fixation information module, a coordinate transformation matrix and an eyeball tracking module, wherein the fixation information module is used for converting eye parameter information into eye parameter information under a vehicle coordinate system according to the determined coordinate transformation matrix, determining the fixation information of a user in a vehicle according to the eye parameter information and vehicle configuration parameters, and the eye parameter information is determined by the eyeball tracking method according to any one of claims 1 to 4, wherein the eyeball tracking device corresponds to a plurality of adjusting positions when being adjusted up, down, left and right within a preset range, and different coordinate transformation matrices are stored in each adjusting position correspondingly.
9. The utility model provides a vehicle, its characterized in that contains in the vehicle and reserves the slot, it has eyeball tracking equipment to dispose in the slot of reserving, eyeball tracking equipment is in the position in reserving the slot can be adjusted about from top to bottom in the predetermined limit, the vehicle still includes:
one or more first processors;
a first storage device for storing one or more first programs;
when executed by the one or more first processors, cause the one or more first processors to implement the eye tracking method of any one of claims 5-6;
wherein the eye tracking device comprises:
one or more second processors;
a projection module for projecting the calibration light;
the communication module is used for sending the eye parameter information to the vehicle;
the image acquisition module is used for acquiring a user image;
the light source module is used for providing a visible light source for the projection module;
second storage means for storing one or more second programs;
when executed by the one or more second processors, cause the one or more second processors to implement the eye tracking method according to any one of claims 1-4.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the eye tracking method according to any one of claims 1-4 and/or the eye tracking method according to any one of claims 5-6.
CN201910611395.1A 2019-07-08 2019-07-08 Eyeball tracking method, device, vehicle and storage medium Active CN110341617B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910611395.1A CN110341617B (en) 2019-07-08 2019-07-08 Eyeball tracking method, device, vehicle and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910611395.1A CN110341617B (en) 2019-07-08 2019-07-08 Eyeball tracking method, device, vehicle and storage medium

Publications (2)

Publication Number Publication Date
CN110341617A CN110341617A (en) 2019-10-18
CN110341617B true CN110341617B (en) 2021-05-28

Family

ID=68178450

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910611395.1A Active CN110341617B (en) 2019-07-08 2019-07-08 Eyeball tracking method, device, vehicle and storage medium

Country Status (1)

Country Link
CN (1) CN110341617B (en)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111159678B (en) * 2019-12-26 2023-08-18 联想(北京)有限公司 Identity recognition method, device and storage medium
DE102020200221A1 (en) * 2020-01-09 2021-07-15 Volkswagen Aktiengesellschaft Method and device for estimating an eye position of a driver of a vehicle
CN113534945A (en) * 2020-04-17 2021-10-22 北京七鑫易维信息技术有限公司 Method, device and equipment for determining eyeball tracking calibration coefficient and storage medium
CN111857333B (en) * 2020-06-29 2022-09-13 维沃移动通信有限公司 Eye movement tracking method and device and electronic equipment
CN112232255B (en) * 2020-10-26 2022-11-01 北京鹰瞳科技发展股份有限公司 Identity authentication method and equipment based on multi-mode data
WO2023272453A1 (en) * 2021-06-28 2023-01-05 华为技术有限公司 Gaze calibration method and apparatus, device, computer-readable storage medium, system, and vehicle

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106203297A (en) * 2016-06-30 2016-12-07 北京七鑫易维信息技术有限公司 A kind of personal identification method and device
WO2018006019A1 (en) * 2016-07-01 2018-01-04 Itay Katz System and method for driver monitoring
CN109542217A (en) * 2018-10-12 2019-03-29 深圳市元征科技股份有限公司 A kind of eyeball tracking display methods and device
CN109558012A (en) * 2018-12-26 2019-04-02 北京七鑫易维信息技术有限公司 Eyeball tracking method and device
CN109933205A (en) * 2019-03-25 2019-06-25 一汽轿车股份有限公司 A kind of vehicle-mounted expression in the eyes interactive device

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106203297A (en) * 2016-06-30 2016-12-07 北京七鑫易维信息技术有限公司 A kind of personal identification method and device
WO2018006019A1 (en) * 2016-07-01 2018-01-04 Itay Katz System and method for driver monitoring
CN109542217A (en) * 2018-10-12 2019-03-29 深圳市元征科技股份有限公司 A kind of eyeball tracking display methods and device
CN109558012A (en) * 2018-12-26 2019-04-02 北京七鑫易维信息技术有限公司 Eyeball tracking method and device
CN109933205A (en) * 2019-03-25 2019-06-25 一汽轿车股份有限公司 A kind of vehicle-mounted expression in the eyes interactive device

Also Published As

Publication number Publication date
CN110341617A (en) 2019-10-18

Similar Documents

Publication Publication Date Title
CN110341617B (en) Eyeball tracking method, device, vehicle and storage medium
CN110167823B (en) System and method for driver monitoring
EP3571673B1 (en) Method for displaying virtual image, storage medium and electronic device therefor
CN107111753B (en) Gaze detection offset for gaze tracking models
US8958599B1 (en) Input method and system based on ambient glints
CN104573623B (en) Face detection device and method
EP4361771A1 (en) Gesture recognition method and apparatus, system, and vehicle
EP4383193A1 (en) Line-of-sight direction tracking method and apparatus
CN112287795B (en) Abnormal driving gesture detection method, device, equipment, vehicle and medium
WO2023272453A1 (en) Gaze calibration method and apparatus, device, computer-readable storage medium, system, and vehicle
CN109144250B (en) Position adjusting method, device, equipment and storage medium
WO2023231653A1 (en) Vehicle-mounted camera calibration method and apparatus, computer device, storage medium and product
CN112083795A (en) Object control method and device, storage medium and electronic equipment
KR102487926B1 (en) Electronic device and method for measuring heart rate
CN114255204A (en) Amblyopia training method, device, equipment and storage medium
CN110338750B (en) Eyeball tracking equipment
CN113160260A (en) Head-eye double-channel intelligent man-machine interaction system and operation method
CN114661152B (en) AR display control system and method for reducing visual fatigue
CN112651270A (en) Gaze information determination method and apparatus, terminal device and display object
CN112771480B (en) Interaction method and electronic equipment
CN114463832A (en) Traffic scene sight tracking method and system based on point cloud
KR20190040797A (en) Appartus and method for tracking eye of user
US11240482B2 (en) Information processing device, information processing method, and computer program
CN113641238A (en) Control method, control device, terminal equipment, controlled equipment and storage medium
WO2023220916A1 (en) Part positioning method and apparatus

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant