WO2016184107A1 - Wearable apparatus for sight line focus positioning and method for sight line focus positioning - Google Patents

Wearable apparatus for sight line focus positioning and method for sight line focus positioning Download PDF

Info

Publication number
WO2016184107A1
WO2016184107A1 PCT/CN2015/098877 CN2015098877W WO2016184107A1 WO 2016184107 A1 WO2016184107 A1 WO 2016184107A1 CN 2015098877 W CN2015098877 W CN 2015098877W WO 2016184107 A1 WO2016184107 A1 WO 2016184107A1
Authority
WO
WIPO (PCT)
Prior art keywords
infrared light
right eye
iris
sight
line
Prior art date
Application number
PCT/CN2015/098877
Other languages
French (fr)
Chinese (zh)
Inventor
宋展
聂颖
Original Assignee
中国科学院深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院 filed Critical 中国科学院深圳先进技术研究院
Publication of WO2016184107A1 publication Critical patent/WO2016184107A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer

Definitions

  • the present invention relates to the field of line-of-sight tracking technology, and more particularly to a wearable device and a line-of-sight focus positioning method for line-of-sight focus positioning.
  • the existing gaze tracking technology often collects facial and facial images through a camera, and optically calculates and tracks the direction of the line of sight.
  • gaze tracking devices can be divided into wearable and non-wearable.
  • the wearable device collects the eye image through the camera, first determines the direction of the line of sight in the camera coordinate system; and then determines the direction of the line of sight in the world coordinate system by adding peripherals and the like.
  • the eye is captured by a plurality of external cameras and the eye image is acquired, and the spatial coordinates of the tester's line of sight are directly obtained.
  • the biggest disadvantage of non-wearable devices is that they need to accurately illuminate the eyes with infrared point light sources, so they can't move the head position casually.
  • the camera has high precision requirements, and it is also highly susceptible to environmental factors and is difficult to promote.
  • the line-of-sight tracking technology can be divided into 2D line-of-sight tracking technology and 3D line-of-sight tracking technology.
  • the 2D tracking technique is represented by the pupil-corneal reflex method, which uses a vector composed of the pupil center and the corneal bright spot to characterize the line of sight.
  • There are many similar devices in the 2D tracking technology For example, there is an operating pointer indicating control device based on human eye image and gaze tracking in the prior art, which can be worn on the human head and can operate the operating pointer through the eye movement.
  • the human-computer interaction is realized by the operation control, and in the process of the interactive control, the human-computer interaction effect of the indicated position of the operation pointer and the real scene observed by the human eye line is achieved.
  • the device adopts an infrared ranging sensor to measure the position of the human eye, and also emits infrared light in real time to the position of the human eye through its infrared emitter, thereby serving as an auxiliary infrared light source, and using infrared light in the human eye.
  • Reflecting on the cornea to form a corneal reflection spot correspondingly using a miniature infrared camera to effectively capture the original infrared image of the position of the human eye for human eye recognition and line-of-sight tracking, avoiding interference caused by dark conditions such as visible light, and cornea Reflective spots can be used later
  • the position of the center of the corneal reflection spot is positioned, and then the position of the center of the pupil is combined to assist in capturing the line of sight of the human eye.
  • the device also divides the image of the human eye region, and the image region of the segmented human eye region image is relatively fixed regardless of whether the position of the human eye in the original image changes displacement due to head motion.
  • the 3D tracking technology uses the dual camera or multi-camera to obtain the coordinates of each reference point in the 3D space for line-of-sight tracking. Compared with the 2D tracking technology, the accuracy and stability are improved.
  • the main problem is that the available 3D features are relatively small, and only the approximate 3D motion information of the eyeball can be obtained through several feature points, and high-precision line-of-sight positioning cannot be achieved.
  • Embodiments of the present invention provide a wearable device for line-of-sight focus positioning, which is used to achieve high-precision line-of-sight focus positioning and ignore the influence of head motion.
  • the device includes:
  • a plurality of infrared light sources respectively mounted on the left and right eyeglass frames of the eyeglass holder; the plurality of infrared light sources are distributed at different positions on the left and right eyeglass frames for illuminating the iris from different angles;
  • a camera for capturing an iris image illuminated by an infrared light source is also mounted on the frame;
  • a signal processing unit is further mounted on the eyeglass holder, and the signal processing unit includes:
  • a synchronization control module configured to control the plurality of infrared light sources to be respectively turned on in a set order; and, when the infrared light source is turned on, the camera on the frame of the infrared light source that is synchronously triggered to turn on the iris image illuminated by the infrared light source;
  • the image processing module is configured to process the iris image to establish a 3D model of the left and right eye irises; and to locate the left and right eye line of sight according to the 3D model of the left and right eye irises.
  • the synchronization control module is specifically configured to control each of the left and right eyeglass frames to have an infrared light source turned on at the same time, and control the camera on the left and right eyeglass frames to simultaneously capture an iris image illuminated by the turned infrared light source.
  • the image processing module is specifically configured to simultaneously process an iris image captured by a camera on the left and right eyeglass frames.
  • the image processing module is specifically configured to:
  • the iris images illuminated by the different infrared light sources from different angles are processed to establish a 3D model of the left and right eye irises;
  • the normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
  • the focus position of the left and right eyesight lines in the world coordinate system is determined.
  • the camera is equipped with an infrared filter.
  • the signal processing unit further includes:
  • a communication module configured to send the positioned left and right eye sight position to the controlled device
  • the embodiment of the invention further provides a line-of-sight focus positioning method for realizing high-precision line-of-sight focus positioning, the method comprising:
  • the left and right eyesight focus is positioned.
  • the left and right eyes are each illuminated by an infrared light source, and an iris image of the left and right eyes illuminated by the infrared light source is obtained;
  • the obtained iris images of the left and right eyes illuminated by the infrared light source are processed.
  • the obtained iris image is processed to create a 3D model of the left and right eye irises, including:
  • the iris images of the left and right eyes are processed by different infrared light sources from different angles to establish a 3D model of the left and right eye irises;
  • the left and right eyesight focus is located, including:
  • the normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
  • the focus position of the left and right eyesight lines in the world coordinate system is determined.
  • determining the focus position of the left and right eye lines of sight in the world coordinate system according to the left and right eye line of sight directions including:
  • the focus position of the left and right eyesight lines in the world coordinate system is determined.
  • the method further includes:
  • the positioned left and right eye line focus positions are sent to the controlled device.
  • the wearable device and the line-of-sight focus positioning method for line-of-sight focus positioning use a plurality of infrared light sources to respectively illuminate the left and right eyes from different angles, and obtain a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles;
  • the obtained iris image is processed to establish a 3D model of the left and right eye irises; according to the 3D model of the left and right eye irises, the left and right eye line of sight is positioned; since the 3D model of the left and right eye irises is based on the iris image is composed of multiple infrared light sources respectively
  • the different angles are obtained by illuminating the left and right eyes.
  • the wearable device for line-of-sight focus positioning of the embodiment of the present invention is also designed in the shape of glasses to achieve a wearable operation, and the influence of head movement can be ignored.
  • FIG. 1 is a schematic diagram of a wearable device for line-of-sight focus positioning according to an embodiment of the present invention
  • FIG. 2 is a detailed working principle diagram of a wearable device for line-of-sight focus positioning according to an embodiment of the present invention
  • FIG. 3 is a schematic diagram of a line-of-sight focus positioning method according to an embodiment of the present invention.
  • an embodiment of the present invention provides a wearable device and a line-of-sight focus positioning method for line-of-sight focus positioning, which utilizes a three-dimensional reconstruction technique to reconstruct a three-dimensional shape of an iris in an eyeball, and analyzes the three-dimensional model.
  • the line of sight of the two eyeballs is obtained, thereby obtaining the intersection of the binocular line of sight direction, thereby achieving the purpose of tracking the line of sight focus; in the case of obtaining the iris image in the eyeball in real time and performing analysis processing, the embodiment of the present invention can track the line of sight focus in real time. .
  • FIG. 1 is a schematic diagram of a wearable device for line-of-sight focus positioning according to an embodiment of the present invention. As shown in FIG. 1, the device may include:
  • a plurality of infrared light sources 2 are respectively mounted on the left and right eyeglass frames of the eyeglass holder 1; the plurality of infrared light sources 2 are distributed at different positions on the left and right eyeglass frames for illuminating the iris from different angles;
  • the left and right eyeglass frames are respectively mounted with a camera 3 for capturing an iris image illuminated by the infrared light source 2;
  • a signal processing unit 4 is further mounted on the eyeglass holder 1 , and the signal processing unit 4 includes:
  • a synchronization control module configured to control the plurality of infrared light sources 2 to be respectively turned on in a set order; and, while the infrared light source 2 is turned on, the camera 3 on the frame of the infrared light source 2 that is synchronously triggered to turn on is photographed by the infrared light source 2 Underlying iris image;
  • the image processing module is configured to process the iris image to establish a 3D model of the left and right eye irises; and to locate the left and right eye line of sight according to the 3D model of the left and right eye irises.
  • the wearable device for gazing focus positioning uses a plurality of infrared light sources to respectively illuminate the left and right eyes from different angles, and obtains a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles; and processing the obtained iris images, A 3D model of the left and right eye irises is established; the left and right eye line of sight is positioned according to the 3D model of the left and right eye irises; the iris image based on the 3D model for establishing the left and right eye irises is obtained by illuminating the left and right eyes from different angles by a plurality of infrared light sources, respectively.
  • the device has a simple structure and low cost, and the infrared light source is safe to the human eye and does not affect the normal work of the person; the use in the form of glasses is naturally fixed with the head position, and is not affected by the head movement.
  • the dynamic 3D reconstruction of the iris can be realized based on the 3D optical principle, and the high-precision 3D shape of the iris can be obtained, thereby obtaining the line of sight information.
  • the left and right eye cameras can work at the same time to obtain the line of sight information of the left and right eyes, and then combine the calibration information of the two cameras to obtain the position of the line of sight focus; thus, the working speed is fast, real-time 3D reconstruction and line-of-sight focus positioning can be realized.
  • the synchronous control module can be specifically used to control the left and right glasses frames to have one infrared light source turned on at the same time, and control the iris image under the illumination of the open infrared light source while controlling the camera on the left and right eyeglass frames; the image processing module can be specifically used The iris image taken by the camera on the left and right eyeglass frames is simultaneously processed.
  • the image processing module may be specifically configured to: combine the iris direction image corresponding to the calibration direction parameter of the infrared light source, and based on the photometric 3D reconstruction principle, process the iris image illuminated by different infrared light sources from different angles to establish the left and right eye irises. 3D model; planar fitting of the 3D model of the left and right eye iris to obtain the 3D plane of the left and right eye iris; determining the normal direction of the 3D plane of the left and right eye iris as the line of sight of the left and right eyes respectively; combining the left and right glasses according to the direction of the left and right eyesight
  • the calibration parameters of the camera on the frame determine the focus position of the left and right eyesight lines in the world coordinate system.
  • the iris can be sequentially illuminated from different angles by using multiple infrared light sources.
  • the camera performs synchronous shooting, and based on the calibration information such as the direction of the light source, the photometric 3D reconstruction method can be used.
  • the high-precision 3D shape information of the iris, and then the direction information of the line of sight is analyzed, and the above-mentioned processing is performed on the left and right eyes simultaneously, and the spatial relative position information of the left and right cameras is obtained according to the calibration parameters of the left and right cameras, and the line of sight directions of the two eyes can be calculated.
  • the wearable device is designed into the shape of the glasses, so that the wearable operation can be realized, so that the influence of the movement of the head can be ignored.
  • the signal processing unit may further comprise a communication module for transmitting the positioned left and right eye line of sight focus positions to the controlled device.
  • the wearable device for line-of-sight focus positioning can implement a computer, a mobile phone, a tablet, etc. by using a built-in communication module, such as the wireless transmission unit, and transmitting the line-of-sight positioning information to the intelligent terminal.
  • the eye control operation function of the intelligent terminal may also include a power source for powering.
  • the wearable device for line-of-sight focus positioning of the embodiment of the present invention mainly comprises a special eyeglass bracket, a plurality of infrared light sources, such as an infrared LED module (eight in FIG. 1 ), and two cameras ( It can be equipped with an infrared filter), as well as a signal processing unit (which can include a synchronous control module, an image processing module, and a communication module and power supply).
  • a signal processing unit which can include a synchronous control module, an image processing module, and a communication module and power supply.
  • the system calibration part includes two parts, 1) the direction information of each LED light source (relative to the left/right camera coordinate system); 2) the calibration of the stereo vision system composed of the left and right cameras, mainly acquiring two The spatial positional relationship of the camera coordinate system;
  • 3D reconstruction part taking the left eye as an example, the main steps include:
  • the obtained 3D reconstruction model is also approximated to a plane, and the plane normal direction is the direction of the line of sight;
  • the focus position is sent to the operation terminal to realize the interactive operation.
  • Glasses bracket used to install camera, infrared LED light source, and signal processing unit and other hardware equipment;
  • Infrared LED light source In this example, at least four infrared LED light sources are respectively installed on the left and right eyeglass frames for illuminating the eyeball and the iris from different angles, and the infrared rays can penetrate the outer structure of the eyeball such as the cornea to obtain a clearer iris image.
  • the LED power is safe for the human eye, and the band can be selected from commonly used bands such as 850 nm and 940 nm;
  • camera left and right glasses frame respectively installed a camera, equipped with a filter corresponding to the LED band, used to isolate the interference of ambient light, to obtain a clear iris image;
  • Signal processing unit including the following main modules:
  • Synchronous control module used to control the switch of each LED, while the LED is illuminated, synchronously triggering the camera to take a photo
  • an image processing module for processing the acquired iris image, acquiring a 3D model of the iris through a 3D reconstruction algorithm, and performing calculation of a line of sight focus;
  • a communication module configured to transmit the calculated line-of-sight focus position information to the operated terminal.
  • FIG. 1 The detailed working principle of the wearable device for line-of-sight focus positioning according to the embodiment of the present invention is illustrated in FIG. 1 , as shown in FIG. 2 .
  • System parameter calibration including:
  • Synchronous control module controls the synchronous shooting of the camera and four LED lights. If the camera frequency is 60FPS, it can complete 15 cycles per second, and then get 15 times of 3D reconstruction model of iris to realize iris dynamics. 3D reconstruction process;
  • the photometric 3D reconstruction method can be used to obtain the iris 3D by combining the different light source direction information of each LED. model
  • the image processing module uses a high-speed computing module such as DSP or FPGA for real-time image processing and 3D reconstruction;
  • the plane fitting strategy is used to obtain the 3D plane of the iris (relative to the coordinate system of the camera), and the normal direction of the plane indicates the orientation of the pupil: Iris_L1, Iris_L2;
  • the 3D spatial plane position of the iris also changes, the line of sight direction and the line of sight focus also change, and the line of sight direction and focus information are transmitted to the controlled device in real time through the built-in communication module, thereby realizing dynamic Sight focus tracking and interworking functions.
  • the embodiment of the present invention adopts the principle of photometric 3D reconstruction, and forms a dynamic three-dimensional reconstruction system by using several infrared LED lights and a camera in the form of wearable glasses, thereby realizing dynamic 3D reconstruction of the iris, and further Get the normal direction of the plane where the pupil is located, that is, the direction of the line of sight.
  • the two cameras By calibrating the two cameras, the unified coordinate system of the two line-of-sight directions of the left and right eyes is realized, and the dynamic calculation of the line-of-sight focus is completed.
  • the LED light source can be selected by 4 or more, and the number of more light sources can improve the accuracy of the three-dimensional reconstruction; the position of the LED light source and the camera can be adjusted.
  • an embodiment of the present invention further provides a line-of-sight focus positioning method, as described in the following embodiments. Since the method for solving the problem is similar to the wearable device for line-of-sight focus positioning, the implementation of the method can be referred to the implementation of the wearable device for line-of-sight focus positioning, and the repeated description is not repeated.
  • FIG. 3 is a schematic diagram of a line-of-sight focus positioning method according to an embodiment of the present invention.
  • the line-of-sight focus positioning method in the embodiment of the present invention may include:
  • Step 301 Obtain a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles;
  • Step 302 Processing the obtained iris image to establish a 3D model of the left and right eye irises
  • Step 303 Position the left and right eyesight focus according to the 3D model of the left and right eye irises.
  • the left and right eyes can be illuminated by one infrared light source at the same time, and the iris images of the left and right eyes illuminated by the infrared light source are obtained; and the iris images obtained by the infrared light source are simultaneously processed.
  • the obtained iris image is processed to establish a 3D model of the left and right eye iris, which may include:
  • the iris images of the left and right eyes are processed by different infrared light sources from different angles to establish a 3D model of the left and right eye irises;
  • the left and right eyesight focus is located, including:
  • the normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
  • the focus position of the left and right eyesight lines in the world coordinate system is determined.
  • determining the focus position of the left and right eyesight lines in the world coordinate system according to the left and right eyesight directions may include:
  • the focus position of the left and right eyesight lines in the world coordinate system is determined.
  • the line-of-sight focus positioning method in the embodiment of the present invention may further include:
  • the positioned left and right eye line focus positions are sent to the controlled device.
  • the wearable device and the line-of-sight focus positioning method for line-of-sight focus positioning use a plurality of infrared light sources to respectively illuminate the left and right eyes from different angles, and obtain a plurality of infrared light sources respectively illuminating the left and right eyes from different angles.
  • the infrared light sources are respectively obtained by illuminating the left and right eyes from different angles.
  • the wearable device for line-of-sight focus positioning of the embodiment of the present invention is also designed in the shape of glasses to achieve a wearable operation, and the influence of head movement can be ignored.
  • embodiments of the present invention can be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.
  • computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • the computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device.
  • the apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.
  • These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device.
  • the instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Eye Examination Apparatus (AREA)
  • Studio Devices (AREA)

Abstract

A wearable apparatus for sight line focus positioning and a method for sight line focus positioning. The apparatus comprises: a glasses stand (1), wherein left and right eyeglass frames of the glasses stand (1) are respectively mounted with a plurality of infrared light sources (2), the plurality of infrared light sources (2) are distributed at different positions on the left and right eyeglass frames for illuminating irises from different angles, and the left and right eyeglass frames are further respectively mounted with a camera (3) used for shooting an iris image illuminated by the infrared light sources; the glasses stand (1) is further mounted with a signal processing unit (4), wherein the signal processing unit (4) comprises: a synchronization control module used for controlling turn-on of the plurality of infrared light sources respectively according to a set order, and while the infrared light sources are turned on, synchronously triggering the cameras (3) on the eyeglass frames where the infrared light sources are turned on to shoot the iris image illuminated by the infrared light sources, and an image processing module used for processing the iris image to establish a 3D model of the left-eye and right-eye irises; and a left-eye and right-eye sight line focus is positioned according to the 3D model of the left-eye and right-eye irises. Sight line focus positioning of high precision can be realized.

Description

用于视线焦点定位的可穿戴设备及视线焦点定位方法Wearable device for line-of-sight focus positioning and line-of-sight focus positioning method 技术领域Technical field
本发明涉及视线追踪技术领域，尤其涉及用于视线焦点定位的可穿戴设备及视线焦点定位方法。The present invention relates to the field of line-of-sight tracking technology, and more particularly to a wearable device and a line-of-sight focus positioning method for line-of-sight focus positioning.
背景技术Background technique
人类通过眼睛获取大量信息，同时使用眼睛来表达情绪与想法。人们可以通过追踪视线焦点的方式，获取眼睛传递出的丰富信息；进而进行有效的人机交互，以及传播学、心理学等领域的研究。视线追踪是利用各种检测手段，获取“注视点”的过程，并保证能在一段时间对其进行连续追踪的技术。该技术广泛应用于人机交互、虚拟现实等领域。Humans get a lot of information through their eyes while using their eyes to express emotions and thoughts. People can obtain rich information transmitted by the eyes by tracking the focus of the line of sight; and then carry out effective human-computer interaction, as well as research in the fields of communication and psychology. Sight tracking is a technique that uses various detection methods to acquire a "gaze point" and to ensure continuous tracking of it over a period of time. This technology is widely used in human-computer interaction, virtual reality and other fields.
现有的视线追踪技术往往通过摄像机采集人脸或脸部图像，利用光学方法计算获得并追踪视线方向。根据系统搭建的不同，视线追踪设备可分为穿戴式和非穿戴式。穿戴设备通过摄像头采集眼部图像，首先确定相机坐标系中的视线方向；再通过增加外设等方法以确定世界坐标系中的视线方向。非穿戴式设备，则是通过多个外部摄像机采集照射眼部并获取眼部图像，直接得到测试者视线的空间坐标。非穿戴设备最大的缺点是：需要红外点光源准确照射眼部，因此不能随便移动头部位置；同时对摄像头精度要求高，也极易受到环境因素影响，不易推广。The existing gaze tracking technology often collects facial and facial images through a camera, and optically calculates and tracks the direction of the line of sight. Depending on the system, gaze tracking devices can be divided into wearable and non-wearable. The wearable device collects the eye image through the camera, first determines the direction of the line of sight in the camera coordinate system; and then determines the direction of the line of sight in the world coordinate system by adding peripherals and the like. For non-wearing devices, the eye is captured by a plurality of external cameras and the eye image is acquired, and the spatial coordinates of the tester's line of sight are directly obtained. The biggest disadvantage of non-wearable devices is that they need to accurately illuminate the eyes with infrared point light sources, so they can't move the head position casually. At the same time, the camera has high precision requirements, and it is also highly susceptible to environmental factors and is difficult to promote.
根据光学方法的不同，视线追踪技术可分为2D视线追踪技术和3D视线追踪技术。2D追踪技术以瞳孔-角膜反射法为代表，利用瞳孔中心与角膜亮斑组成的向量表征视线方向。2D追踪技术已有不少类似设备，例如现有技术中已有一种基于人眼图像和视线追踪的操作指针指示控制设备，其能够佩戴于人体头部上且能够通过眼部动作对操作指针加以操作控制而实现人机交互，其在交互控制过程中，达到操作指针的指示位置与人眼视线所观察到的现实场景相叠加的人机交互效果。该设备采用了红外测距传感器对人眼所在位置进行测距，还通过其红外发射器实时地向人体眼睛所在位置发射出红外光，以此作为辅助的红外光源，并借助红外光在人眼角膜上反射而形成角膜反射光斑，相应地采用微型红外摄像头有效的拍摄人体眼睛所在位置的原始红外图像，用以进行人眼识别和视线追踪，避免了可见光环境阴暗等情况引起的干扰，并且角膜反射光斑可用于在后 续的视线追踪识别过程中对角膜反射光斑中心所在位置加以定位，进而结合瞳孔中心所在位置，辅助实现对人眼的视线方向加以捕捉。该设备还通过分割人眼区域图像的方式，不管原始图像中人眼位置是否因为头部运动而发生位移变化，分割出的人眼区域图像的图像区域范围是相对固定的。Depending on the optical method, the line-of-sight tracking technology can be divided into 2D line-of-sight tracking technology and 3D line-of-sight tracking technology. The 2D tracking technique is represented by the pupil-corneal reflex method, which uses a vector composed of the pupil center and the corneal bright spot to characterize the line of sight. There are many similar devices in the 2D tracking technology. For example, there is an operating pointer indicating control device based on human eye image and gaze tracking in the prior art, which can be worn on the human head and can operate the operating pointer through the eye movement. The human-computer interaction is realized by the operation control, and in the process of the interactive control, the human-computer interaction effect of the indicated position of the operation pointer and the real scene observed by the human eye line is achieved. The device adopts an infrared ranging sensor to measure the position of the human eye, and also emits infrared light in real time to the position of the human eye through its infrared emitter, thereby serving as an auxiliary infrared light source, and using infrared light in the human eye. Reflecting on the cornea to form a corneal reflection spot, correspondingly using a miniature infrared camera to effectively capture the original infrared image of the position of the human eye for human eye recognition and line-of-sight tracking, avoiding interference caused by dark conditions such as visible light, and cornea Reflective spots can be used later During the continuous line of sight tracking and identification process, the position of the center of the corneal reflection spot is positioned, and then the position of the center of the pupil is combined to assist in capturing the line of sight of the human eye. The device also divides the image of the human eye region, and the image region of the segmented human eye region image is relatively fixed regardless of whether the position of the human eye in the original image changes displacement due to head motion.
然而，此类2D视线追踪技术的缺点是利用的图像信息少，准确性和稳定性比较低。However, such 2D gaze tracking technology has the disadvantage of using less image information and lower accuracy and stability.
3D追踪技术是通过双摄像头或多摄像头获得3D空间中各个参考点的坐标进行视线追踪。相比2D追踪技术准确率和稳定性都有提升，其主要问题在于可用的3D特征比较少，只能通过若干特征点获取眼球的大致的3D运动信息，而无法实现高精度的视线定位。The 3D tracking technology uses the dual camera or multi-camera to obtain the coordinates of each reference point in the 3D space for line-of-sight tracking. Compared with the 2D tracking technology, the accuracy and stability are improved. The main problem is that the available 3D features are relatively small, and only the approximate 3D motion information of the eyeball can be obtained through several feature points, and high-precision line-of-sight positioning cannot be achieved.
发明内容Summary of the invention
本发明实施例提供一种用于视线焦点定位的可穿戴设备，用以实现高精度的视线焦点定位并忽略头部运动的影响，该设备包括：Embodiments of the present invention provide a wearable device for line-of-sight focus positioning, which is used to achieve high-precision line-of-sight focus positioning and ignore the influence of head motion. The device includes:
眼镜支架，所述眼镜支架的左右眼镜框上分别安装有多个红外光源；所述多个红外光源分布于所述左右眼镜框上的不同位置，用于从不同角度照射虹膜；所述左右眼镜框上还分别安装有用于拍摄红外光源照射下的虹膜图像的摄像头；a plurality of infrared light sources respectively mounted on the left and right eyeglass frames of the eyeglass holder; the plurality of infrared light sources are distributed at different positions on the left and right eyeglass frames for illuminating the iris from different angles; A camera for capturing an iris image illuminated by an infrared light source is also mounted on the frame;
所述眼镜支架上还安装有信号处理单元，所述信号处理单元包括：A signal processing unit is further mounted on the eyeglass holder, and the signal processing unit includes:
同步控制模块，用于控制所述多个红外光源按设定顺序分别开启；以及，在红外光源开启的同时，同步触发开启的红外光源所在眼镜框上的摄像头拍摄红外光源照射下的虹膜图像；a synchronization control module, configured to control the plurality of infrared light sources to be respectively turned on in a set order; and, when the infrared light source is turned on, the camera on the frame of the infrared light source that is synchronously triggered to turn on the iris image illuminated by the infrared light source;
图像处理模块，用于对虹膜图像进行处理，建立左右眼虹膜的3D模型；根据左右眼虹膜的3D模型，定位左右眼视线焦点。The image processing module is configured to process the iris image to establish a 3D model of the left and right eye irises; and to locate the left and right eye line of sight according to the 3D model of the left and right eye irises.
一个实施例中，所述同步控制模块具体用于控制所述左右眼镜框上同时各有一个红外光源开启，以及，控制所述左右眼镜框上的摄像头同时拍摄开启的红外光源照射下的虹膜图像；In one embodiment, the synchronization control module is specifically configured to control each of the left and right eyeglass frames to have an infrared light source turned on at the same time, and control the camera on the left and right eyeglass frames to simultaneously capture an iris image illuminated by the turned infrared light source. ;
所述图像处理模块具体用于同时处理所述左右眼镜框上的摄像头拍摄的虹膜图像。The image processing module is specifically configured to simultaneously process an iris image captured by a camera on the left and right eyeglass frames.
一个实施例中，所述图像处理模块具体用于：In an embodiment, the image processing module is specifically configured to:
结合虹膜图像对应红外光源的标定方向参数，基于光度学3D重建原理，对所述多个红外光源从不同角度照射的虹膜图像进行处理，建立左右眼虹膜的3D模型； Combining the iris image with the calibration direction parameter of the infrared light source, based on the photometric 3D reconstruction principle, the iris images illuminated by the different infrared light sources from different angles are processed to establish a 3D model of the left and right eye irises;
对左右眼虹膜的3D模型进行平面拟合，获得左右眼虹膜的3D平面；Plane fitting the 3D model of the left and right eye irises to obtain the 3D plane of the left and right eye irises;
将左右眼虹膜的3D平面的法向分别确定为左右眼视线方向；The normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
根据左右眼视线方向，结合所述左右眼镜框上的摄像头的标定参数，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, combined with the calibration parameters of the camera on the left and right eyeglass frames, the focus position of the left and right eyesight lines in the world coordinate system is determined.
一个实施例中，所述摄像头配备红外滤光片。In one embodiment, the camera is equipped with an infrared filter.
一个实施例中，所述信号处理单元还包括：In an embodiment, the signal processing unit further includes:
通信模块，用于将定位的左右眼视线焦点位置发送至被操控设备；a communication module, configured to send the positioned left and right eye sight position to the controlled device;
和/或，用于供电的电源。And / or power supply for power supply.
本发明实施例还提供一种视线焦点定位方法，用以实现高精度的视线焦点定位，该方法包括：The embodiment of the invention further provides a line-of-sight focus positioning method for realizing high-precision line-of-sight focus positioning, the method comprising:
获得多个红外光源分别从不同角度照射左右眼的虹膜图像；Obtaining a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles;
对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型；Processing the obtained iris image to establish a 3D model of the left and right eye irises;
根据左右眼虹膜的3D模型，定位左右眼视线焦点。According to the 3D model of the left and right eye irises, the left and right eyesight focus is positioned.
一个实施例中，左右眼同时各有一个红外光源照射，同时获得左右眼被红外光源照射的虹膜图像；In one embodiment, the left and right eyes are each illuminated by an infrared light source, and an iris image of the left and right eyes illuminated by the infrared light source is obtained;
同时处理获得的左右眼被红外光源照射的虹膜图像。At the same time, the obtained iris images of the left and right eyes illuminated by the infrared light source are processed.
一个实施例中，对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型，包括：In one embodiment, the obtained iris image is processed to create a 3D model of the left and right eye irises, including:
结合虹膜图像对应红外光源的标定方向参数，基于光度学3D重建原理，对多个红外光源分别从不同角度照射左右眼的虹膜图像进行处理，建立左右眼虹膜的3D模型；Combining the iris image with the calibration direction parameter of the infrared light source, based on the photometric 3D reconstruction principle, the iris images of the left and right eyes are processed by different infrared light sources from different angles to establish a 3D model of the left and right eye irises;
根据左右眼虹膜的3D模型，定位左右眼视线焦点，包括：According to the 3D model of the left and right eye iris, the left and right eyesight focus is located, including:
对左右眼虹膜的3D模型进行平面拟合，获得左右眼虹膜的3D平面；Plane fitting the 3D model of the left and right eye irises to obtain the 3D plane of the left and right eye irises;
将左右眼虹膜的3D平面的法向分别确定为左右眼视线方向；The normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
根据左右眼视线方向，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, the focus position of the left and right eyesight lines in the world coordinate system is determined.
一个实施例中，根据左右眼视线方向，确定世界坐标系下左右眼视线的焦点位置，包括：In one embodiment, determining the focus position of the left and right eye lines of sight in the world coordinate system according to the left and right eye line of sight directions, including:
根据左右眼视线方向，结合拍摄左右眼虹膜图像的摄像头的标定参数，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, combined with the calibration parameters of the camera that captures the left and right eye iris images, the focus position of the left and right eyesight lines in the world coordinate system is determined.
一个实施例中，该方法还包括：In one embodiment, the method further includes:
将定位的左右眼视线焦点位置发送至被操控设备。 The positioned left and right eye line focus positions are sent to the controlled device.
本发明实施例的用于视线焦点定位的可穿戴设备及视线焦点定位方法，利用多个红外光源分别从不同角度照射左右眼，获得多个红外光源分别从不同角度照射左右眼的虹膜图像；对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型；根据左右眼虹膜的3D模型，定位左右眼视线焦点；由于建立左右眼虹膜的3D模型所基于的虹膜图像是由多个红外光源分别从不同角度照射左右眼而获得的，因此相对于现有的3D追踪技术，可用的3D特征比较多，能够获取眼球的较为详细的3D运动信息，因而能够实现高精度的视线焦点定位。此外，本发明实施例的用于视线焦点定位的可穿戴设备还设计成眼镜的形状，从而实现穿戴式操作，可以忽略头部运动的影响。The wearable device and the line-of-sight focus positioning method for line-of-sight focus positioning according to the embodiment of the present invention use a plurality of infrared light sources to respectively illuminate the left and right eyes from different angles, and obtain a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles; The obtained iris image is processed to establish a 3D model of the left and right eye irises; according to the 3D model of the left and right eye irises, the left and right eye line of sight is positioned; since the 3D model of the left and right eye irises is based on the iris image is composed of multiple infrared light sources respectively The different angles are obtained by illuminating the left and right eyes. Therefore, compared with the existing 3D tracking technology, more 3D features are available, and more detailed 3D motion information of the eyeball can be acquired, thereby achieving high-precision line-of-sight focus positioning. In addition, the wearable device for line-of-sight focus positioning of the embodiment of the present invention is also designed in the shape of glasses to achieve a wearable operation, and the influence of head movement can be ignored.
附图说明DRAWINGS
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。在附图中：In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only It is a certain embodiment of the present invention, and other drawings can be obtained from those skilled in the art without any creative work. In the drawing:
图1为本发明实施例中用于视线焦点定位的可穿戴设备的示意图；1 is a schematic diagram of a wearable device for line-of-sight focus positioning according to an embodiment of the present invention;
图2为本发明实施例中用于视线焦点定位的可穿戴设备的详细工作原理图；2 is a detailed working principle diagram of a wearable device for line-of-sight focus positioning according to an embodiment of the present invention;
图3为本发明实施例中视线焦点定位方法的示意图。FIG. 3 is a schematic diagram of a line-of-sight focus positioning method according to an embodiment of the present invention.
具体实施方式detailed description
为使本发明实施例的目的、技术方案和优点更加清楚明白，下面结合附图对本发明实施例做进一步详细说明。在此，本发明的示意性实施例及其说明用于解释本发明，但并不作为对本发明的限定。The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The illustrative embodiments of the present invention and the description thereof are intended to explain the present invention, but are not intended to limit the invention.
为了实现高精度的视线焦点定位，本发明实施例提供一种用于视线焦点定位的可穿戴设备及视线焦点定位方法，利用三维重建技术重建出眼球内虹膜的三维形状，通过对三维模型的分析得出两个眼球的视线方向，进而得到双目视线方向的交点，从而达到追踪视线焦点的目的；在实时获得眼球内虹膜图像并作分析处理的情况下，本发明实施例可以实时追踪视线焦点。In order to achieve high-precision line-of-sight focus positioning, an embodiment of the present invention provides a wearable device and a line-of-sight focus positioning method for line-of-sight focus positioning, which utilizes a three-dimensional reconstruction technique to reconstruct a three-dimensional shape of an iris in an eyeball, and analyzes the three-dimensional model. The line of sight of the two eyeballs is obtained, thereby obtaining the intersection of the binocular line of sight direction, thereby achieving the purpose of tracking the line of sight focus; in the case of obtaining the iris image in the eyeball in real time and performing analysis processing, the embodiment of the present invention can track the line of sight focus in real time. .
下面先介绍本发明实施例的用于视线焦点定位的可穿戴设备。图1为本发明实施例的用于视线焦点定位的可穿戴设备的示意图。如图1所示，该设备可以包括： Hereinafter, a wearable device for line-of-sight focus positioning according to an embodiment of the present invention will be described. FIG. 1 is a schematic diagram of a wearable device for line-of-sight focus positioning according to an embodiment of the present invention. As shown in FIG. 1, the device may include:
眼镜支架1，所述眼镜支架1的左右眼镜框上分别安装有多个红外光源2；所述多个红外光源2分布于所述左右眼镜框上的不同位置，用于从不同角度照射虹膜；所述左右眼镜框上还分别安装有用于拍摄红外光源2照射下的虹膜图像的摄像头3；a plurality of infrared light sources 2 are respectively mounted on the left and right eyeglass frames of the eyeglass holder 1; the plurality of infrared light sources 2 are distributed at different positions on the left and right eyeglass frames for illuminating the iris from different angles; The left and right eyeglass frames are respectively mounted with a camera 3 for capturing an iris image illuminated by the infrared light source 2;
所述眼镜支架1上还安装有信号处理单元4，所述信号处理单元4包括：A signal processing unit 4 is further mounted on the eyeglass holder 1 , and the signal processing unit 4 includes:
同步控制模块，用于控制所述多个红外光源2按设定顺序分别开启；以及，在红外光源2开启的同时，同步触发开启的红外光源2所在眼镜框上的摄像头3拍摄红外光源2照射下的虹膜图像；a synchronization control module, configured to control the plurality of infrared light sources 2 to be respectively turned on in a set order; and, while the infrared light source 2 is turned on, the camera 3 on the frame of the infrared light source 2 that is synchronously triggered to turn on is photographed by the infrared light source 2 Underlying iris image;
图像处理模块，用于对虹膜图像进行处理，建立左右眼虹膜的3D模型；根据左右眼虹膜的3D模型，定位左右眼视线焦点。The image processing module is configured to process the iris image to establish a 3D model of the left and right eye irises; and to locate the left and right eye line of sight according to the 3D model of the left and right eye irises.
具体实施时，用于视线焦点定位的可穿戴设备利用多个红外光源分别从不同角度照射左右眼，获得多个红外光源分别从不同角度照射左右眼的虹膜图像；对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型；根据左右眼虹膜的3D模型，定位左右眼视线焦点；由于建立左右眼虹膜的3D模型所基于的虹膜图像是由多个红外光源分别从不同角度照射左右眼而获得的，因此相对于现有的3D追踪技术，可用的3D特征比较多，能够获取眼球的较为详细的3D运动信息，因而能够实现高精度的视线焦点定位。该设备结构简单，低成本，并且，红外光源对人眼安全，不影响人的正常工作；以眼镜的形式使用，与头部位置自然固定，不受头部运动影响。实施例中，可以基于3D光学原理，实现虹膜的动态的3D重建，能够得到虹膜的高精度的3D形状，进而得到视线信息。In a specific implementation, the wearable device for gazing focus positioning uses a plurality of infrared light sources to respectively illuminate the left and right eyes from different angles, and obtains a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles; and processing the obtained iris images, A 3D model of the left and right eye irises is established; the left and right eye line of sight is positioned according to the 3D model of the left and right eye irises; the iris image based on the 3D model for establishing the left and right eye irises is obtained by illuminating the left and right eyes from different angles by a plurality of infrared light sources, respectively. Therefore, compared with the existing 3D tracking technology, more 3D features are available, and more detailed 3D motion information of the eyeball can be acquired, thereby achieving high-precision line-of-sight focus positioning. The device has a simple structure and low cost, and the infrared light source is safe to the human eye and does not affect the normal work of the person; the use in the form of glasses is naturally fixed with the head position, and is not affected by the head movement. In the embodiment, the dynamic 3D reconstruction of the iris can be realized based on the 3D optical principle, and the high-precision 3D shape of the iris can be obtained, thereby obtaining the line of sight information.
具体实施时，左右眼的摄像头可以同时工作，得到左右眼的视线信息，进而结合2个摄像头的标定信息获取视线焦点的位置；这样工作起来速度快，可以实现实时的3D重建和视线焦点定位。实施时，同步控制模块具体可以用于控制左右眼镜框上同时各有一个红外光源开启，以及，控制左右眼镜框上的摄像头同时拍摄开启的红外光源照射下的虹膜图像；图像处理模块具体可以用于同时处理左右眼镜框上的摄像头拍摄的虹膜图像。In the specific implementation, the left and right eye cameras can work at the same time to obtain the line of sight information of the left and right eyes, and then combine the calibration information of the two cameras to obtain the position of the line of sight focus; thus, the working speed is fast, real-time 3D reconstruction and line-of-sight focus positioning can be realized. In the implementation, the synchronous control module can be specifically used to control the left and right glasses frames to have one infrared light source turned on at the same time, and control the iris image under the illumination of the open infrared light source while controlling the camera on the left and right eyeglass frames; the image processing module can be specifically used The iris image taken by the camera on the left and right eyeglass frames is simultaneously processed.
具体实施时，图像处理模块具体可以用于：结合虹膜图像对应红外光源的标定方向参数，基于光度学3D重建原理，对多个红外光源从不同角度照射的虹膜图像进行处理，建立左右眼虹膜的3D模型；对左右眼虹膜的3D模型进行平面拟合，获得左右眼虹膜的3D平面；将左右眼虹膜的3D平面的法向分别确定为左右眼视线方向；根据左右眼视线方向，结合左右眼镜框上的摄像头的标定参数，确定世界坐标系下左右眼视线的焦点位置。 In a specific implementation, the image processing module may be specifically configured to: combine the iris direction image corresponding to the calibration direction parameter of the infrared light source, and based on the photometric 3D reconstruction principle, process the iris image illuminated by different infrared light sources from different angles to establish the left and right eye irises. 3D model; planar fitting of the 3D model of the left and right eye iris to obtain the 3D plane of the left and right eye iris; determining the normal direction of the 3D plane of the left and right eye iris as the line of sight of the left and right eyes respectively; combining the left and right glasses according to the direction of the left and right eyesight The calibration parameters of the camera on the frame determine the focus position of the left and right eyesight lines in the world coordinate system.
由上述实施例可知，本发明实施例中，可以通过多个红外光源从不同角度对虹膜进行顺序照射，同时，摄像头进行同步拍摄，基于光源方向等标定信息，采用光度学3D重建方法，可以获取虹膜的高精度3D形状信息，进而分析出视线的方向信息，对左右眼同时进行上述处理，并根据左右摄像头的标定参数获得左右摄像头的空间相对位置信息，就可以计算出两个眼睛的视线方向，并统一到同一世界坐标系下，从而得到视线的焦点位置；将该可穿戴设备设计成眼镜形状，从而使可以实现穿戴式操作，这样就可以忽略头部的运动的影响。It can be seen from the above embodiments that in the embodiment of the present invention, the iris can be sequentially illuminated from different angles by using multiple infrared light sources. At the same time, the camera performs synchronous shooting, and based on the calibration information such as the direction of the light source, the photometric 3D reconstruction method can be used. The high-precision 3D shape information of the iris, and then the direction information of the line of sight is analyzed, and the above-mentioned processing is performed on the left and right eyes simultaneously, and the spatial relative position information of the left and right cameras is obtained according to the calibration parameters of the left and right cameras, and the line of sight directions of the two eyes can be calculated. And unified into the same world coordinate system to obtain the focus position of the line of sight; the wearable device is designed into the shape of the glasses, so that the wearable operation can be realized, so that the influence of the movement of the head can be ignored.
实施例中，信号处理单元还可以包括通信模块，用于将定位的左右眼视线焦点位置发送至被操控设备。例如，本发明实施例的用于视线焦点定位的可穿戴设备可以通过内置的通信模块，如该通信模块为无线发射单元，将视线定位信息传输到智能终端，就可以实现电脑、手机、平板等智能终端的眼控操作功能。实施例中，信号处理单元还可以包括用于供电的电源。In an embodiment, the signal processing unit may further comprise a communication module for transmitting the positioned left and right eye line of sight focus positions to the controlled device. For example, the wearable device for line-of-sight focus positioning according to the embodiment of the present invention can implement a computer, a mobile phone, a tablet, etc. by using a built-in communication module, such as the wireless transmission unit, and transmitting the line-of-sight positioning information to the intelligent terminal. The eye control operation function of the intelligent terminal. In an embodiment, the signal processing unit may also include a power source for powering.
下面再结合图1举例说明本发明实施例的用于视线焦点定位的可穿戴设备的基本原理。如图1所示，本发明实施例的用于视线焦点定位的可穿戴设备主要由一个特殊的眼镜支架，若干红外光源，例如红外LED模组(图1中为8个)，2个摄像头(可以配备红外滤光片)，以及信号处理单元(可以包括同步控制模块、图像处理模块、及通信模块和电源)。其基本原理如下：The basic principle of the wearable device for line-of-sight focus positioning according to the embodiment of the present invention will be exemplified below with reference to FIG. As shown in FIG. 1 , the wearable device for line-of-sight focus positioning of the embodiment of the present invention mainly comprises a special eyeglass bracket, a plurality of infrared light sources, such as an infrared LED module (eight in FIG. 1 ), and two cameras ( It can be equipped with an infrared filter), as well as a signal processing unit (which can include a synchronous control module, an image processing module, and a communication module and power supply). The basic principles are as follows:
1、系统标定部分，包括两部分，1)每个LED灯的光源方向信息(相对于其所在的左/右摄像头坐标系)；2)左右摄像头组成的立体视觉系统的标定，主要获取两个摄像头坐标系的空间位置关系；1. The system calibration part includes two parts, 1) the direction information of each LED light source (relative to the left/right camera coordinate system); 2) the calibration of the stereo vision system composed of the left and right cameras, mainly acquiring two The spatial positional relationship of the camera coordinate system;
2、三维重建部分，以左眼为例，主要步骤包括：2, 3D reconstruction part, taking the left eye as an example, the main steps include:
1)镜框上安装有4个红外LED及一个摄像头(可以配备红外滤光片，用于过滤掉红外LED之外的环境光线)，由同步控制模块控制四个LED等顺序亮灭，并由摄像头同步采集红外LED照射下的虹膜的图像；1) There are 4 infrared LEDs and one camera installed on the frame (can be equipped with infrared filters to filter out the ambient light outside the infrared LED), and the four LEDs controlled by the synchronous control module are sequentially turned on and off, and the camera is turned on. Simultaneously collecting images of the iris under the illumination of the infrared LED;
2)结合LED光源标定方向参数，基于光度学3D重建原理，得到虹膜的3D重建模型；2) Combining the calibration direction parameters of the LED light source, based on the principle of photometric 3D reconstruction, obtaining a 3D reconstruction model of the iris;
3)因为虹膜区域近似为一个平面，得到的3D重建模型也近似为一个平面，得到平面法向即表示视线方向；3) Since the iris region is approximately a plane, the obtained 3D reconstruction model is also approximated to a plane, and the plane normal direction is the direction of the line of sight;
4)对右眼同样进行上述操作，得到右眼的视线方向； 4) Perform the above operation on the right eye to obtain the line of sight direction of the right eye;
3、视线定位：根据左右眼的视线方向，结合2个摄像头的标定参数，就可以得到世界坐标系下视线的焦点位置；3, line of sight positioning: according to the line of sight of the left and right eyes, combined with the calibration parameters of the two cameras, you can get the focus position of the line of sight in the world coordinate system;
4、通过通信模块，将焦点位置发送到操作终端上实现互动操作。4. Through the communication module, the focus position is sent to the operation terminal to realize the interactive operation.
下面再结合图1举例说明本发明实施例的用于视线焦点定位的可穿戴设备的几个主要部分：Several main parts of the wearable device for line-of-sight focus positioning according to an embodiment of the present invention are exemplified below with reference to FIG. 1 :
1、眼镜支架：用于安置摄像头、红外LED光源，以及信号处理单元等硬件设备；1. Glasses bracket: used to install camera, infrared LED light source, and signal processing unit and other hardware equipment;
2、红外LED光源：本例中左右眼镜框上分别安装至少四个红外LED光源，用于从不同角度照射眼球及虹膜，红外线可以穿透角膜等眼球外部结构，可以获得更清晰的虹膜图像，LED功率为人眼安全功率，波段可选择850nm，940nm等常用波段；2. Infrared LED light source: In this example, at least four infrared LED light sources are respectively installed on the left and right eyeglass frames for illuminating the eyeball and the iris from different angles, and the infrared rays can penetrate the outer structure of the eyeball such as the cornea to obtain a clearer iris image. The LED power is safe for the human eye, and the band can be selected from commonly used bands such as 850 nm and 940 nm;
3、摄像头：左右眼镜框分别安装一个摄像头，装配有与LED波段对应的滤光片，用于隔离环境光的干扰，获取清晰的虹膜图像；3, camera: left and right glasses frame respectively installed a camera, equipped with a filter corresponding to the LED band, used to isolate the interference of ambient light, to obtain a clear iris image;
4、信号处理单元，包括以下主要模块：4. Signal processing unit, including the following main modules:
1)电源，用于系统供电，也可以采用外接电源式；1) Power supply for system power supply, or external power supply;
2)同步控制模块，用于控制每个LED的开关，LED发光的同时，同步触发摄像头拍摄照片；2) Synchronous control module, used to control the switch of each LED, while the LED is illuminated, synchronously triggering the camera to take a photo;
3)图像处理模块，用于处理所获取的虹膜图像，通过3D重建算法获取虹膜的3D模型，并进行视线焦点的计算；3) an image processing module for processing the acquired iris image, acquiring a 3D model of the iris through a 3D reconstruction algorithm, and performing calculation of a line of sight focus;
4)通信模块，用于将计算得到的视线焦点位置信息传输至所操作的终端。4) A communication module, configured to transmit the calculated line-of-sight focus position information to the operated terminal.
下面再结合图1举例说明本发明实施例的用于视线焦点定位的可穿戴设备的详细工作原理，如图2所示：The detailed working principle of the wearable device for line-of-sight focus positioning according to the embodiment of the present invention is illustrated in FIG. 1 , as shown in FIG. 2 .
1、系统参数标定，主要包括：1. System parameter calibration, including:
1)2个摄像头的内外部参数标定，用以获取每个摄像头的焦距、中心点等内部参数，以及两个摄像头间的旋转矩阵R和平移矩阵T，方法属于常用的立体视觉标定手段；1) Internal and external parameter calibration of two cameras to obtain the internal parameters such as the focal length and center point of each camera, and the rotation matrix R and translation matrix T between the two cameras. The method belongs to the commonly used stereo vision calibration method;
2)LED光源方向的标定：用于获取每个LED相对于其所在的摄像头的光源主方向向量L，如对于左摄像头，可以获得四个LED的光源方向Ll1，Ll2，Ll3，Ll4，对于右摄像头，可以获得四个LED的光源方向Lr1，Lr2，Lr3，Lr4；2) Calibration of LED light source direction: used to obtain the main direction vector L of each LED relative to the camera where it is located. For the left camera, the light source directions Ll1, Ll2, Ll3, Ll4 of four LEDs can be obtained. Camera, you can get the direction of the light source of four LEDs Lr1, Lr2, Lr3, Lr4;
2、虹膜图像拍摄： 2, iris image capture:
1)由同步控制模块控制摄像头和四个LED灯的同步拍摄，如摄像头频率为60FPS，则每秒钟可以完成15个周期的拍摄，进而得到15次虹膜的3D重建模型，从而实现虹膜的动态3D重建过程；1) Synchronous control module controls the synchronous shooting of the camera and four LED lights. If the camera frequency is 60FPS, it can complete 15 cycles per second, and then get 15 times of 3D reconstruction model of iris to realize iris dynamics. 3D reconstruction process;
2)左右眼镜框上的红外光源和摄像头同步进行工作，并将拍摄的图像传输到图像处理模块；2) The infrared light source on the left and right eyeglass frames and the camera work synchronously, and the captured image is transmitted to the image processing module;
3、虹膜的3D重建：3. 3D reconstruction of the iris:
由于LED是从不同方向照射虹膜的，因此每个拍摄周期可以拍摄到四张不同亮度分布的虹膜图像，结合每个LED的不同光源方向信息，采用光度学3D重建方法，就可以获得虹膜的3D模型；Since the LED illuminates the iris from different directions, four iris images of different brightness distributions can be captured in each shooting cycle, and the photometric 3D reconstruction method can be used to obtain the iris 3D by combining the different light source direction information of each LED. model;
对于左右两部分同时进行拍摄，同时进行处理，互不干涉；Shooting at the same time for the left and right parts, and processing at the same time, without mutual interference;
在图像处理模块采用DSP或者FPGA等高速计算模块进行实时的图像处理和3D重建；The image processing module uses a high-speed computing module such as DSP or FPGA for real-time image processing and 3D reconstruction;
4、对于获取的左右眼虹膜的3D模型，采用平面拟合策略，得到虹膜的3D平面(分别相对于所在的摄像头的坐标系)，该平面的法向即表示瞳孔的指向：Iris_L1，Iris_L2；4. For the acquired 3D model of the left and right eye iris, the plane fitting strategy is used to obtain the 3D plane of the iris (relative to the coordinate system of the camera), and the normal direction of the plane indicates the orientation of the pupil: Iris_L1, Iris_L2;
5、视线焦点的定位：5, the focus of the line of sight:
由于在标定阶段，可以得到两个摄像头的旋转和平移参数R和T，则很容易将两个视线方向统一到同一坐标系下，即{Iris_L1*R，Iris_L2}，求解两个向量的空间交点即为视线的焦点位置。Since the rotation and translation parameters R and T of the two cameras can be obtained during the calibration phase, it is easy to unify the two line-of-sight directions into the same coordinate system, ie {Iris_L1*R, Iris_L2}, to solve the spatial intersection of the two vectors. This is the focus position of the line of sight.
6、交互操作：6, interactive operation:
随着操作者眼睛的转动，虹膜的3D空间平面位置也发生变化，视线方向和视线焦点也发生变化，通过内置的通信模块将视线方向和焦点信息实时发送到被操控设备，即可实现动态的视线焦点跟踪和交互操作功能。As the operator's eyes rotate, the 3D spatial plane position of the iris also changes, the line of sight direction and the line of sight focus also change, and the line of sight direction and focus information are transmitted to the controlled device in real time through the built-in communication module, thereby realizing dynamic Sight focus tracking and interworking functions.
由上述实施例可知，本发明实施例采用光度学3D重建原理，以穿戴式眼镜的形式，用几个红外LED灯和摄像头组成了一个动态三维重建系统，实现了对虹膜的动态3D重建，进而得到瞳孔所在平面的法向即视线方向。通过对2个摄像头的标定，实现了左右眼两个视线方向的统一坐标系，完成视线焦点的动态计算。It can be seen from the above embodiments that the embodiment of the present invention adopts the principle of photometric 3D reconstruction, and forms a dynamic three-dimensional reconstruction system by using several infrared LED lights and a camera in the form of wearable glasses, thereby realizing dynamic 3D reconstruction of the iris, and further Get the normal direction of the plane where the pupil is located, that is, the direction of the line of sight. By calibrating the two cameras, the unified coordinate system of the two line-of-sight directions of the left and right eyes is realized, and the dynamic calculation of the line-of-sight focus is completed.
具体实施时，LED光源可以选择4个或者更多，更多的光源数目可以提升三维重建的精度；LED光源和相机的位置可以做出调整。 In specific implementation, the LED light source can be selected by 4 or more, and the number of more light sources can improve the accuracy of the three-dimensional reconstruction; the position of the LED light source and the camera can be adjusted.
基于同一发明构思，本发明实施例中还提供了一种视线焦点定位方法，如下面的实施例所述。由于该方法解决问题的原理与用于视线焦点定位的可穿戴设备相似，因此该方法的实施可以参见用于视线焦点定位的可穿戴设备的实施，重复之处不再赘述。Based on the same inventive concept, an embodiment of the present invention further provides a line-of-sight focus positioning method, as described in the following embodiments. Since the method for solving the problem is similar to the wearable device for line-of-sight focus positioning, the implementation of the method can be referred to the implementation of the wearable device for line-of-sight focus positioning, and the repeated description is not repeated.
图3为本发明实施例中视线焦点定位方法的示意图。如图3所示，本发明实施例中视线焦点定位方法可以包括：FIG. 3 is a schematic diagram of a line-of-sight focus positioning method according to an embodiment of the present invention. As shown in FIG. 3, the line-of-sight focus positioning method in the embodiment of the present invention may include:
步骤301、获得多个红外光源分别从不同角度照射左右眼的虹膜图像；Step 301: Obtain a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles;
步骤302、对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型；Step 302: Processing the obtained iris image to establish a 3D model of the left and right eye irises;
步骤303、根据左右眼虹膜的3D模型，定位左右眼视线焦点。Step 303: Position the left and right eyesight focus according to the 3D model of the left and right eye irises.
具体实施时，可以左右眼同时各有一个红外光源照射，同时获得左右眼被红外光源照射的虹膜图像；同时处理获得的左右眼被红外光源照射的虹膜图像。In the specific implementation, the left and right eyes can be illuminated by one infrared light source at the same time, and the iris images of the left and right eyes illuminated by the infrared light source are obtained; and the iris images obtained by the infrared light source are simultaneously processed.
具体实施时，对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型，可以包括：In a specific implementation, the obtained iris image is processed to establish a 3D model of the left and right eye iris, which may include:
结合虹膜图像对应红外光源的标定方向参数，基于光度学3D重建原理，对多个红外光源分别从不同角度照射左右眼的虹膜图像进行处理，建立左右眼虹膜的3D模型；Combining the iris image with the calibration direction parameter of the infrared light source, based on the photometric 3D reconstruction principle, the iris images of the left and right eyes are processed by different infrared light sources from different angles to establish a 3D model of the left and right eye irises;
根据左右眼虹膜的3D模型，定位左右眼视线焦点，包括：According to the 3D model of the left and right eye iris, the left and right eyesight focus is located, including:
对左右眼虹膜的3D模型进行平面拟合，获得左右眼虹膜的3D平面；Plane fitting the 3D model of the left and right eye irises to obtain the 3D plane of the left and right eye irises;
将左右眼虹膜的3D平面的法向分别确定为左右眼视线方向；The normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
根据左右眼视线方向，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, the focus position of the left and right eyesight lines in the world coordinate system is determined.
具体实施时，根据左右眼视线方向，确定世界坐标系下左右眼视线的焦点位置，可以包括：In a specific implementation, determining the focus position of the left and right eyesight lines in the world coordinate system according to the left and right eyesight directions may include:
根据左右眼视线方向，结合拍摄左右眼虹膜图像的摄像头的标定参数，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, combined with the calibration parameters of the camera that captures the left and right eye iris images, the focus position of the left and right eyesight lines in the world coordinate system is determined.
具体实施时，本发明实施例中视线焦点定位方法还可以包括：In a specific implementation, the line-of-sight focus positioning method in the embodiment of the present invention may further include:
将定位的左右眼视线焦点位置发送至被操控设备。The positioned left and right eye line focus positions are sent to the controlled device.
综上所述，本发明实施例的用于视线焦点定位的可穿戴设备及视线焦点定位方法，利用多个红外光源分别从不同角度照射左右眼，获得多个红外光源分别从不同角度照射左右眼的虹膜图像；对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型；根据左右眼虹膜的3D模型，定位左右眼视线焦点；由于建立左右眼虹膜的3D模型所基于的虹膜图像是由多个红外光源分别从不同角度照射左右眼而获得的，因此相对于现有的3D追踪技术，可用的3D特征比较多，能够获取眼球的较为详细的3D运动信息，因而能够 实现高精度的视线焦点定位。此外，本发明实施例的用于视线焦点定位的可穿戴设备还设计成眼镜的形状，从而实现穿戴式操作，可以忽略头部运动的影响。In summary, the wearable device and the line-of-sight focus positioning method for line-of-sight focus positioning according to the embodiment of the present invention use a plurality of infrared light sources to respectively illuminate the left and right eyes from different angles, and obtain a plurality of infrared light sources respectively illuminating the left and right eyes from different angles. Iris image; processing the obtained iris image to establish a 3D model of the left and right eye iris; positioning the left and right eye line of sight focus according to the 3D model of the left and right eye iris; the iris image based on the 3D model of the left and right eye iris is based on The infrared light sources are respectively obtained by illuminating the left and right eyes from different angles. Therefore, compared with the existing 3D tracking technology, more 3D features are available, and more detailed 3D motion information of the eyeball can be acquired, thereby enabling Achieve high-precision line-of-sight focus positioning. In addition, the wearable device for line-of-sight focus positioning of the embodiment of the present invention is also designed in the shape of glasses to achieve a wearable operation, and the influence of head movement can be ignored.
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。Those skilled in the art will appreciate that embodiments of the present invention can be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware. Moreover, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) including computer usable program code.
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (system), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or FIG. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine for the execution of instructions for execution by a processor of a computer or other programmable data processing device. Means for implementing the functions specified in one or more of the flow or in a block or blocks of the flow chart.
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。The computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture comprising the instruction device. The apparatus implements the functions specified in one or more blocks of a flow or a flow and/or block diagram of the flowchart.
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions can also be loaded onto a computer or other programmable data processing device such that a series of operational steps are performed on a computer or other programmable device to produce computer-implemented processing for execution on a computer or other programmable device. The instructions provide steps for implementing the functions specified in one or more of the flow or in a block or blocks of a flow diagram.
以上所述的具体实施例，对本发明的目的、技术方案和有益效果进行了进一步详细说明，所应理解的是，以上所述仅为本发明的具体实施例而已，并不用于限定本发明的保护范围，凡在本发明的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。 The above described specific embodiments of the present invention are further described in detail, and are intended to be illustrative of the embodiments of the present invention. All modifications, equivalent substitutions, improvements, etc., made within the spirit and scope of the invention are intended to be included within the scope of the invention.

Claims (10)

  1. 一种用于视线焦点定位的可穿戴设备，其特征在于，包括：A wearable device for line-of-sight focus positioning, comprising:
    眼镜支架，所述眼镜支架的左右眼镜框上分别安装有多个红外光源；所述多个红外光源分布于所述左右眼镜框上的不同位置，用于从不同角度照射虹膜；所述左右眼镜框上还分别安装有用于拍摄红外光源照射下的虹膜图像的摄像头；a plurality of infrared light sources respectively mounted on the left and right eyeglass frames of the eyeglass holder; the plurality of infrared light sources are distributed at different positions on the left and right eyeglass frames for illuminating the iris from different angles; A camera for capturing an iris image illuminated by an infrared light source is also mounted on the frame;
    所述眼镜支架上还安装有信号处理单元，所述信号处理单元包括：A signal processing unit is further mounted on the eyeglass holder, and the signal processing unit includes:
    同步控制模块，用于控制所述多个红外光源按设定顺序分别开启；以及，在红外光源开启的同时，同步触发开启的红外光源所在眼镜框上的摄像头拍摄红外光源照射下的虹膜图像；a synchronization control module, configured to control the plurality of infrared light sources to be respectively turned on in a set order; and, when the infrared light source is turned on, the camera on the frame of the infrared light source that is synchronously triggered to turn on the iris image illuminated by the infrared light source;
    图像处理模块，用于对虹膜图像进行处理，建立左右眼虹膜的3D模型；根据左右眼虹膜的3D模型，定位左右眼视线焦点。The image processing module is configured to process the iris image to establish a 3D model of the left and right eye irises; and to locate the left and right eye line of sight according to the 3D model of the left and right eye irises.
  2. 如权利要求1所述的设备，其特征在于，所述同步控制模块具体用于控制所述左右眼镜框上同时各有一个红外光源开启，以及，控制所述左右眼镜框上的摄像头同时拍摄开启的红外光源照射下的虹膜图像；The device according to claim 1, wherein the synchronization control module is specifically configured to control that each of the left and right eyeglass frames has an infrared light source turned on at the same time, and that the camera on the left and right eyeglass frames is simultaneously turned on. An iris image illuminated by an infrared source;
    所述图像处理模块具体用于同时处理所述左右眼镜框上的摄像头拍摄的虹膜图像。The image processing module is specifically configured to simultaneously process an iris image captured by a camera on the left and right eyeglass frames.
  3. 如权利要求1所述的设备，其特征在于，所述图像处理模块具体用于：The device according to claim 1, wherein the image processing module is specifically configured to:
    结合虹膜图像对应红外光源的标定方向参数，基于光度学3D重建原理，对所述多个红外光源从不同角度照射的虹膜图像进行处理，建立左右眼虹膜的3D模型；Combining the iris image with the calibration direction parameter of the infrared light source, based on the photometric 3D reconstruction principle, the iris images illuminated by the different infrared light sources from different angles are processed to establish a 3D model of the left and right eye irises;
    对左右眼虹膜的3D模型进行平面拟合，获得左右眼虹膜的3D平面；Plane fitting the 3D model of the left and right eye irises to obtain the 3D plane of the left and right eye irises;
    将左右眼虹膜的3D平面的法向分别确定为左右眼视线方向；The normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
    根据左右眼视线方向，结合所述左右眼镜框上的摄像头的标定参数，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, combined with the calibration parameters of the camera on the left and right eyeglass frames, the focus position of the left and right eyesight lines in the world coordinate system is determined.
  4. 如权利要求1所述的设备，其特征在于，所述摄像头配备红外滤光片。The device of claim 1 wherein said camera is provided with an infrared filter.
  5. 如权利要求1所述的设备，其特征在于，所述信号处理单元还包括：The device of claim 1, wherein the signal processing unit further comprises:
    通信模块，用于将定位的左右眼视线焦点位置发送至被操控设备；a communication module, configured to send the positioned left and right eye sight position to the controlled device;
    和/或，用于供电的电源。And / or power supply for power supply.
  6. 一种视线焦点定位方法，其特征在于，包括：A method for locating a focus of a line of sight, comprising:
    获得多个红外光源分别从不同角度照射左右眼的虹膜图像；Obtaining a plurality of infrared light sources respectively illuminating the iris images of the left and right eyes from different angles;
    对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型；Processing the obtained iris image to establish a 3D model of the left and right eye irises;
    根据左右眼虹膜的3D模型，定位左右眼视线焦点。 According to the 3D model of the left and right eye irises, the left and right eyesight focus is positioned.
  7. 如权利要求6所述的方法，其特征在于，左右眼同时各有一个红外光源照射，同时获得左右眼被红外光源照射的虹膜图像；The method according to claim 6, wherein each of the left and right eyes is simultaneously illuminated by an infrared light source, and an iris image of the left and right eyes illuminated by the infrared light source is obtained;
    同时处理获得的左右眼被红外光源照射的虹膜图像。At the same time, the obtained iris images of the left and right eyes illuminated by the infrared light source are processed.
  8. 如权利要求6所述的方法，其特征在于，对获得的虹膜图像进行处理，建立左右眼虹膜的3D模型，包括：The method according to claim 6, wherein the obtained iris image is processed to establish a 3D model of the left and right eye irises, including:
    结合虹膜图像对应红外光源的标定方向参数，基于光度学3D重建原理，对多个红外光源分别从不同角度照射左右眼的虹膜图像进行处理，建立左右眼虹膜的3D模型；Combining the iris image with the calibration direction parameter of the infrared light source, based on the photometric 3D reconstruction principle, the iris images of the left and right eyes are processed by different infrared light sources from different angles to establish a 3D model of the left and right eye irises;
    根据左右眼虹膜的3D模型，定位左右眼视线焦点，包括：According to the 3D model of the left and right eye iris, the left and right eyesight focus is located, including:
    对左右眼虹膜的3D模型进行平面拟合，获得左右眼虹膜的3D平面；Plane fitting the 3D model of the left and right eye irises to obtain the 3D plane of the left and right eye irises;
    将左右眼虹膜的3D平面的法向分别确定为左右眼视线方向；The normal directions of the 3D planes of the left and right eye irises are respectively determined as the line of sight directions of the left and right eyes;
    根据左右眼视线方向，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, the focus position of the left and right eyesight lines in the world coordinate system is determined.
  9. 如权利要求8所述的方法，其特征在于，根据左右眼视线方向，确定世界坐标系下左右眼视线的焦点位置，包括：The method according to claim 8, wherein determining the focus position of the left and right eyesight lines in the world coordinate system according to the direction of the left and right eyesight lines comprises:
    根据左右眼视线方向，结合拍摄左右眼虹膜图像的摄像头的标定参数，确定世界坐标系下左右眼视线的焦点位置。According to the direction of the left and right eyesight, combined with the calibration parameters of the camera that captures the left and right eye iris images, the focus position of the left and right eyesight lines in the world coordinate system is determined.
  10. 如权利要求6所述的方法，其特征在于，还包括：The method of claim 6 further comprising:
    将定位的左右眼视线焦点位置发送至被操控设备。 The positioned left and right eye line focus positions are sent to the controlled device.
PCT/CN2015/098877 2015-05-15 2015-12-25 Wearable apparatus for sight line focus positioning and method for sight line focus positioning WO2016184107A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510250492.4A CN104834381B (en) 2015-05-15 2015-05-15 Wearable device and sight line focus localization method for sight line focus location
CN2015102504924 2015-05-15

Publications (1)

Publication Number Publication Date
WO2016184107A1 true WO2016184107A1 (en) 2016-11-24

Family

ID=53812318

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2015/098877 WO2016184107A1 (en) 2015-05-15 2015-12-25 Wearable apparatus for sight line focus positioning and method for sight line focus positioning

Country Status (2)

Country Link
CN (1) CN104834381B (en)
WO (1) WO2016184107A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110347260A (en) * 2019-07-11 2019-10-18 歌尔科技有限公司 A kind of augmented reality device and its control method, computer readable storage medium
CN110826248A (en) * 2019-11-18 2020-02-21 杭州涂鸦信息技术有限公司 Method and system for simulating vignetting of infrared camera based on light source database
CN112578557A (en) * 2019-09-30 2021-03-30 三星显示有限公司 Electronic device

Families Citing this family (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104834381B (en) * 2015-05-15 2017-01-04 中国科学院深圳先进技术研究院 Wearable device and sight line focus localization method for sight line focus location
CN105128862A (en) * 2015-08-18 2015-12-09 上海擎感智能科技有限公司 Vehicle terminal eyeball identification control method and vehicle terminal eyeball identification control system
CN105141938B (en) * 2015-08-18 2017-12-01 深圳先进技术研究院 Sight positioner
CN105892634A (en) * 2015-11-18 2016-08-24 乐视致新电子科技（天津）有限公司 Anti-dizziness method and virtual reality display output device
KR101862499B1 (en) * 2015-11-27 2018-05-29 포브, 아이엔씨. Viewpoint detecting system, point of gaze determining method and point of gaze determining program
CN105487787A (en) * 2015-12-09 2016-04-13 东莞酷派软件技术有限公司 Terminal operation method and device based on iris recognition and terminal
US10114465B2 (en) * 2016-01-15 2018-10-30 Google Llc Virtual reality head-mounted devices having reduced numbers of cameras, and methods of operating the same
CN105931272B (en) * 2016-05-06 2019-04-05 上海乐相科技有限公司 A kind of Moving Objects method for tracing and system
CN106168855B (en) * 2016-07-01 2020-06-30 北京麦哲科技有限公司 Portable MR glasses, mobile phone and MR glasses system
CN106200965B (en) * 2016-07-08 2017-11-17 广东欧珀移动通信有限公司 A kind of screen display method, device and mobile terminal
CN106162144A (en) * 2016-07-21 2016-11-23 触景无限科技（北京）有限公司 A kind of visual pattern processing equipment, system and intelligent machine for overnight sight
CN106372569B (en) * 2016-08-18 2019-07-05 瑞声科技(新加坡)有限公司 Has the virtual reality device of iris capturing function
CN106339085B (en) 2016-08-22 2020-04-21 华为技术有限公司 Terminal with sight tracking function, method and device for determining user viewpoint
CN107784255B (en) * 2016-08-30 2024-05-14 北京亮亮视野科技有限公司 Head-wearing type user iris information acquisition equipment
CN106203410B (en) * 2016-09-21 2023-10-17 上海星寰投资有限公司 Identity verification method and system
SG11201907370XA (en) * 2017-02-12 2019-09-27 Lemnis Tech Pte Ltd Methods, devices and systems for focus adjustment of displays
CN109429060B (en) * 2017-07-07 2020-07-28 京东方科技集团股份有限公司 Pupil distance measuring method, wearable eye equipment and storage medium
CN107644443B (en) * 2017-09-01 2020-07-28 北京七鑫易维信息技术有限公司 Parameter setting method and device in sight tracking equipment
CN108334191B (en) * 2017-12-29 2021-03-23 北京七鑫易维信息技术有限公司 Method and device for determining fixation point based on eye movement analysis equipment
CN108470152A (en) * 2018-02-14 2018-08-31 天目爱视（北京）科技有限公司 Based on infrared 3D four-dimension iris data acquisition methods and system
CN108681403A (en) * 2018-05-18 2018-10-19 吉林大学 A kind of trolley control method using eye tracking
CN109044263A (en) * 2018-07-13 2018-12-21 北京七鑫易维信息技术有限公司 Detection method, device, equipment and the storage medium of eye sight line
JP6894880B2 (en) * 2018-11-07 2021-06-30 矢崎総業株式会社 Monitoring system
CN110674682B (en) * 2019-08-15 2021-12-14 武汉虹识技术有限公司 Binocular iris image acquisition device and method
CN113760083A (en) * 2020-06-01 2021-12-07 张也弛 Method and device for determining position of landing point of operator sight on screen of terminal equipment
CN111983803A (en) * 2020-08-19 2020-11-24 业成科技（成都）有限公司 Eyeball tracking module and electronic equipment
CN112541975B (en) * 2020-12-24 2024-03-12 华南理工大学 Three-dimensional head-based head-mounted product field of view calculation method
CN112926523B (en) * 2021-03-30 2022-07-26 青岛小鸟看看科技有限公司 Eyeball tracking method and system based on virtual reality
CN113138664A (en) * 2021-03-30 2021-07-20 青岛小鸟看看科技有限公司 Eyeball tracking system and method based on light field perception
CN114136209B (en) * 2021-11-24 2023-11-24 京东方科技集团股份有限公司 Eyeball position positioning circuit, eyeball position positioning method, substrate and virtual reality wearable device
CN115245630A (en) * 2022-07-15 2022-10-28 北京鹰瞳科技发展股份有限公司 Red light irradiation device and red light irradiation method for myopia physiotherapy

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120019645A1 (en) * 2010-07-23 2012-01-26 Maltz Gregory A Unitized, Vision-Controlled, Wireless Eyeglasses Transceiver
US20130285901A1 (en) * 2012-04-27 2013-10-31 Dongguk University Industry-Academic Cooperation Foundation System and method for tracking gaze at distance
CN103777351A (en) * 2012-10-26 2014-05-07 鸿富锦精密工业（深圳）有限公司 Multimedia glasses
US20150035745A1 (en) * 2013-07-30 2015-02-05 National Chiao Tung University Head-mount eye tracking system
CN104834381A (en) * 2015-05-15 2015-08-12 中国科学院深圳先进技术研究院 Wearable device for sight focus positioning and sight focus positioning method

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102830793B (en) * 2011-06-16 2017-04-05 北京三星通信技术研究有限公司 Sight tracing and equipment

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120019645A1 (en) * 2010-07-23 2012-01-26 Maltz Gregory A Unitized, Vision-Controlled, Wireless Eyeglasses Transceiver
US20130285901A1 (en) * 2012-04-27 2013-10-31 Dongguk University Industry-Academic Cooperation Foundation System and method for tracking gaze at distance
CN103777351A (en) * 2012-10-26 2014-05-07 鸿富锦精密工业（深圳）有限公司 Multimedia glasses
US20150035745A1 (en) * 2013-07-30 2015-02-05 National Chiao Tung University Head-mount eye tracking system
CN104834381A (en) * 2015-05-15 2015-08-12 中国科学院深圳先进技术研究院 Wearable device for sight focus positioning and sight focus positioning method

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110347260A (en) * 2019-07-11 2019-10-18 歌尔科技有限公司 A kind of augmented reality device and its control method, computer readable storage medium
CN112578557A (en) * 2019-09-30 2021-03-30 三星显示有限公司 Electronic device
CN110826248A (en) * 2019-11-18 2020-02-21 杭州涂鸦信息技术有限公司 Method and system for simulating vignetting of infrared camera based on light source database
CN110826248B (en) * 2019-11-18 2023-10-20 杭州涂鸦信息技术有限公司 Method and system for simulating infrared camera dark angle based on light source database

Also Published As

Publication number Publication date
CN104834381B (en) 2017-01-04
CN104834381A (en) 2015-08-12

Similar Documents

Publication Publication Date Title
WO2016184107A1 (en) Wearable apparatus for sight line focus positioning and method for sight line focus positioning
US11199706B2 (en) Head-mounted display for virtual and mixed reality with inside-out positional, user body and environment tracking
CN104090659B (en) Operating pointer based on eye image and Eye-controlling focus indicates control device
KR102456047B1 (en) Eyewear-mountable eye tracking device
US9916005B2 (en) Gaze tracking with projector
US9348141B2 (en) Low-latency fusing of virtual and real content
CN105574525B (en) A kind of complex scene multi-modal biological characteristic image acquiring method and its device
EP3371781B1 (en) Systems and methods for generating and using three-dimensional images
CN103293673B (en) Cap integrated with display, eye tracker and iris recognition instrument
WO2016122991A1 (en) Automatic generation of virtual materials from real-world materials
EP3092616A1 (en) Mapping glints to light sources
JPH0782539B2 (en) Pupil imager
US11712619B2 (en) Handle controller
EP3542308B1 (en) Method and device for eye metric acquisition
WO2018154272A1 (en) Systems and methods for obtaining information about the face and eyes of a subject
CN111524175A (en) Depth reconstruction and eye movement tracking method and system for asymmetric multiple cameras
WO2016101861A1 (en) Head-worn display device
RU2007117996A (en) METHOD FOR MEASURING FAST EYE MOVEMENT AND BINOCULAR VISION ANGLATION AND DEVICE FOR ITS IMPLEMENTATION
US20230104656A1 (en) Gaze tracking
Hornak et al. Point-of-Gaze Estimation on Embedded Platform
JP2022113973A (en) Display method, display device, and program
Combier et al. Towards an Augmented Reality Head Mounted Display System Providing Stereoscopic Wide Field of View for Indoor and Outdoor Environments with Interaction through the Gaze Direction
Garcia et al. A distributed real time eye-gaze tracking system
CN114578940A (en) Control method and device and electronic equipment
CN112965607A (en) Glasses mouse and control method thereof

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 15892481

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205N DATED 04.05.2018)

122 Ep: pct application non-entry in european phase

Ref document number: 15892481

Country of ref document: EP

Kind code of ref document: A1