WO2021249187A1 - 视线追踪方法、视线追踪装置、计算设备和介质 - Google Patents

视线追踪方法、视线追踪装置、计算设备和介质 Download PDF

Info

Publication number
WO2021249187A1
WO2021249187A1 PCT/CN2021/096007 CN2021096007W WO2021249187A1 WO 2021249187 A1 WO2021249187 A1 WO 2021249187A1 CN 2021096007 W CN2021096007 W CN 2021096007W WO 2021249187 A1 WO2021249187 A1 WO 2021249187A1
Authority
WO
WIPO (PCT)
Prior art keywords
coordinates
image
coordinate system
human eye
pupil
Prior art date
Application number
PCT/CN2021/096007
Other languages
English (en)
French (fr)
Inventor
薛亚冲
张�浩
陈丽莉
孙建康
李纲
吕耀宇
Original Assignee
京东方科技集团股份有限公司
北京京东方光电科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司, 北京京东方光电科技有限公司 filed Critical 京东方科技集团股份有限公司
Publication of WO2021249187A1 publication Critical patent/WO2021249187A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements

Definitions

  • the present disclosure relates to the technical field of gaze tracking, and in particular, to a gaze tracking method, a gaze tracking device, a computing device, and a medium.
  • a polynomial mapping model is commonly used in the gaze tracking system, and the model mainly uses a high-order polynomial to represent the mapping relationship between the pupil and the gaze point on the screen.
  • 9 calibration points are used for calibration to obtain the mapping relationship between the pupil and the screen.
  • this gaze tracking method has major drawbacks: first, the calibration process is more cumbersome, and the user needs to calibrate before each use.
  • the user needs to look at the calibration points appearing on the screen in turn, each time they need to look at 1-2s and the calibration time is longer, generally 15-25s; second, the calibration process is prone to errors, and the user requires accurate attention to the calibration points during the calibration process If there is an incorrect gaze at a calibration point, for example, the human eye does not look at the center of the calibration point well, it will cause an error in the mapping model, which will lead to a large error in the gaze point calculation and need to be recalibrated.
  • the present disclosure provides a gaze tracking method, a gaze tracking device, a computing device, and a computer-readable storage medium.
  • a line of sight tracking method including:
  • the multiple human eye images are trajectory images acquired when the user's eyes scan the screen along a predetermined trajectory;
  • the coordinates of the gaze point of the user on the screen in the image coordinate system are determined according to the spherical center coordinates and the multiple pupil coordinates in the world coordinate system.
  • the acquiring multiple human eye images and respectively determining multiple pupil coordinates in the world coordinate system in the multiple human eye images further includes:
  • Image processing is performed on the multiple human eye images respectively, and the pupil coordinates in the image coordinate system in each human eye image are determined;
  • the pupil coordinates in the image coordinate system in each human eye image are converted into the pupil coordinates in the world coordinate system.
  • the performing image processing on the multiple human eye images to determine pupil coordinates in the image coordinate system in each human eye image includes:
  • the outline of the pupil area of the binarized image is calculated separately, and the non-pupil outline is eliminated according to the size and shape of the outline, and the non-pupil outline is eliminated based on
  • the binarized image determines the pupil coordinates in the image coordinate system of the human eye image corresponding to the binarized image, wherein the origin of the image coordinate system is located at the upper left corner of the screen.
  • the separately preprocessing the multiple human eye images includes:
  • the performing binarization processing on multiple pre-processed human eye images respectively includes:
  • the gray value of the pupil in the obtained binarized image is set to zero, and an open operation is performed on the binarized image to remove the white holes in the pupil.
  • the gaze tracking method before acquiring multiple human eye images and respectively determining multiple pupil coordinates in the world coordinate system in the multiple human eye images, the gaze tracking method further includes:
  • the internal parameter calibration board and the external parameter calibration board are used to calibrate the internal parameter matrix and the external parameter matrix of the image collector respectively.
  • the calibration of the internal parameter matrix and the external parameter matrix of the image collector using an internal parameter calibration board and an external parameter calibration board respectively further includes:
  • the determining the coordinates of the gaze point of the user on the screen in the image coordinate system according to the spherical center coordinates and the multiple pupil coordinates in the world coordinate system further includes:
  • the coordinates of the gaze point of the user in the world coordinate system are converted into the coordinates of the gaze point of the user on the screen in the image coordinate system.
  • the trajectory image includes:
  • a line-of-sight tracking device including
  • the pupil positioning circuit is configured to acquire multiple human eye images and respectively determine multiple pupil coordinates in the world coordinate system in the multiple human eye images, and the multiple human eye images are the user's eyes scanning along a predetermined trajectory Trajectory image acquired on the screen;
  • the sphere center positioning circuit is configured to determine sphere center coordinates based on multiple pupil coordinates in the world coordinate system, where the sphere center coordinates are the coordinates of the sphere center of the sphere where the multiple pupil coordinates in the world coordinate system are located ;
  • the gaze point positioning circuit is configured to determine the coordinates of the user's gaze point on the screen in the image coordinate system according to the spherical center coordinates and multiple pupil coordinates in the world coordinate system.
  • the pupil positioning circuit includes an image collector and a light source, and the pupil positioning circuit is configured to:
  • Image processing is performed on the multiple human eye images respectively, and the pupil coordinates in the image coordinate system in each human eye image are determined;
  • the pupil coordinates in the image coordinate system in each human eye image are converted into the pupil coordinates in the world coordinate system.
  • the pupil positioning circuit further includes a calibration circuit for respectively calibrating the internal parameter matrix and the external parameter matrix of the image collector using an internal parameter calibration board and an external parameter calibration board.
  • a computer-readable storage medium having computer-executable instructions stored thereon, wherein, when the computer-executable instructions are executed by a processor, any one of the above-mentioned gaze tracking methods is executed .
  • a computing device including a processor and a memory storing computer-executable instructions, wherein the processor executes any of the above-mentioned computer-executable instructions when executing the computer-executable instructions. Sight tracking method.
  • Fig. 1 shows a flowchart of a line-of-sight tracking method according to an embodiment of the present disclosure
  • Fig. 2 shows a schematic structural diagram of a virtual reality device according to an embodiment of the present disclosure
  • Fig. 3 shows a schematic diagram of an internal reference calibration board according to an embodiment of the present disclosure
  • Fig. 4 shows a schematic diagram of an external reference calibration board according to an embodiment of the present disclosure
  • Fig. 5 shows a schematic diagram of calibration of the external parameter matrix of the image collector according to an embodiment of the present disclosure
  • Fig. 6 shows a schematic diagram of line-of-sight tracking according to an embodiment of the present disclosure
  • Fig. 7 shows a schematic diagram of coordinate system conversion according to an embodiment of the present disclosure
  • FIG. 8 shows a schematic structural diagram of a line-of-sight tracking device according to an embodiment of the present disclosure
  • Fig. 9 shows a schematic structural diagram of a computing device according to an embodiment of the present disclosure.
  • Fig. 1 shows a flow chart of a line-of-sight tracking method according to an embodiment of the present disclosure.
  • the gaze tracking method includes: S10, acquiring multiple human eye images and respectively determining multiple pupil coordinates in the world coordinate system in the multiple human eye images, and the multiple human eye images It is the trajectory image acquired when the user’s eyes scan the screen along a predetermined saccade; S12, judging whether the sphere center coordinates can be determined according to multiple pupil coordinates in the world coordinate system, and if the sphere center coordinates cannot be determined, reacquire Multiple human eye images, the spherical center coordinates are the coordinates of the spherical center of the sphere where the multiple pupil coordinates in the world coordinate system are located; S14, according to the spherical center coordinates and the multiple pupils in the world coordinate system The coordinates determine the coordinates of the user's gaze point on the screen in the image coordinate system.
  • the pupil coordinates in the world coordinate system of each human eye image are determined by using multiple human eye images with saccadic trajectories obtained when the human eye scans the screen, and then the pupil coordinates are determined according to the movement of the pupil on the surface of the eyeball. It is characterized by determining the spherical center coordinates of the spherical surface where the movement track is located by multiple pupil coordinates in the world coordinate system, and then determining where the user is based on the multiple pupil coordinates and spherical center coordinates in the world coordinate system.
  • the coordinates in the image coordinate system of the gaze point on the screen are described, so as to realize the gaze tracking of the human eye.
  • the pupil coordinates mentioned here are the pupil coordinates.
  • the human eye is used to scan the screen to obtain multiple pupil coordinates with saccade trajectories, which simplifies the step of calibrating sequentially using multiple (usually 9) fixed calibration points in the related technology, which is effective Avoiding the cumbersome process of calibration using polynomial mapping methods, speeding up the user's gaze tracking, improving the stability and calculation accuracy of gaze tracking, can effectively improve the user's experience, and has a wide range of application prospects.
  • the virtual reality device 100 includes a first lens 11 and a second lens 12. Considering that both eyes have the same line of sight, this embodiment uses monocular line-of-sight tracking for description.
  • the virtual reality device further includes an image collector 13 (for example, a camera) arranged directly below the first lens 11 and Surrounding light source 14.
  • the gaze tracking procedure is as follows.
  • step S10 Acquire multiple human eye images and respectively determine multiple pupil coordinates in the world coordinate system in the multiple human eye images, the multiple human eye images are acquired when the user's eyes scan the screen along a predetermined trajectory Trajectory image.
  • the step S10 may specifically include the following steps S100-S104.
  • S100 Control the image collector to collect multiple human eye images under the light provided by the light source.
  • the image collector is a camera, and the center axis of the camera points to the center position of the human eye area, so that the camera collects human eye images in an environment where the light source provides light.
  • the light source may be an infrared light source
  • the camera may be an infrared camera.
  • the camera may be a high-speed infrared camera, and multiple infrared light sources may be used in consideration of the power of the light source and uniform light supplementation.
  • the high-speed infrared camera used in this embodiment has a resolution of 640*480, a frame rate of 100fps, a field of view (FOV) of 60°, and the vertices of a regular hexagon around the first lens 11
  • Six infrared light sources are set in the position, and the wavelength of each infrared light source is 850nm.
  • the six infrared light sources can provide uniform ambient light to facilitate the high-speed infrared camera to collect human eye images, and are beneficial to segment the pupil from the iris area to obtain a clear pupil image.
  • the trajectory image is a trajectory image obtained when the user's eyes scan in a first direction and a second direction of the screen, respectively, where the first direction and the second direction are perpendicular.
  • the user may be prompted to scan the screen from left to right in the horizontal direction, and then the user may be prompted to scan the screen from top to bottom in the vertical direction to complete the process of scanning the screen. That is, in this process, the pupil of the user's eye moves from the left end of the screen to the right end, and then from the upper end to the lower end of the screen.
  • the high-speed infrared camera collects N frames of human eye images with pupil movement trajectories under the infrared light provided by the infrared light source.
  • the trajectory image is a trajectory image obtained when the user's eyes scan along the diagonal of the screen.
  • the user may be prompted to scan from one corner of the screen to another corner relative to the center of the screen to complete the process of scanning the screen, for example, from the upper left corner of the screen to the lower right corner (diagonal) of the screen. That is, the user's eyes scan the diagonal of the screen to obtain multiple human eye images with saccadic trajectories.
  • the image collector can collect multiple human eye images for gaze tracking while the user's eyes are scanning the screen.
  • the trajectory image is a trajectory image obtained when the user's eyes circle the screen for a glance.
  • the user may be prompted to start at a point on the periphery of the screen and scan around the screen for a week to complete the process of scanning the screen.
  • the image collector collects multiple human eye images for gaze tracking while the user's eyes are scanning the screen.
  • multiple images of the human eye can be collected during the saccade of the user's eyes, thereby effectively simplifying the calibration step of sequentially using 9 calibration points in the related technology.
  • S102 Perform image processing on the multiple human eye images respectively, and determine pupil coordinates in the image coordinate system in each human eye image.
  • obtaining pupil coordinates in the image coordinate system through multiple human eye images may specifically include:
  • preprocessing the multiple human eye images may include: converting the human eye images into grayscale images, and then filtering the grayscale images (for example, Gaussian filtering) to filter out the grayscale images Noise.
  • grayscale images for example, Gaussian filtering
  • the pre-processed multiple human eye images are binarized. Specifically, binarization is performed on each pixel in the filtered image to obtain a binarized image; the gray value of the pupil part in the obtained binarized image is set to 0, and the binarized image is opened Calculate to remove the white holes in the pupils.
  • the outline of the pupil area of the binarized image is calculated separately, and then the non-pupil outline is eliminated according to the size and shape of the outline, and the non-pupil is eliminated based on
  • the binarized image after the contour determines the pupil coordinates in the image coordinate system of the human eye image corresponding to the binarized image, wherein the origin of the image coordinate system is located at the upper left corner of the screen.
  • S104 Convert the pupil coordinates in the image coordinate system of each human eye image to the pupil coordinates in the world coordinate system according to the internal parameter matrix and the external parameter matrix pre-calibrated by the image collector.
  • the internal parameter matrix pre-calibrated by the high-speed infrared camera is used to convert the pupil coordinates in the image coordinate system to the pupil coordinates in the camera coordinate system, and then use the high-speed infrared
  • the camera's pre-calibrated external parameter matrix converts the pupil coordinates in the camera coordinate system to the pupil coordinates in the world coordinate system.
  • the pre-calibrated internal parameter matrix and external parameter matrix of the image collector can be calibrated before the high-speed infrared camera leaves the factory, or can be calibrated before use, which is not limited in this application.
  • the line-of-sight tracking method may further include: Step S01: Calibrate the internal parameter matrix and the external parameter matrix of the image collector using the internal parameter calibration board and the external parameter calibration board respectively.
  • the conversion relationship between the image coordinate system and the camera coordinate system is:
  • (u, v) are the coordinates in the image coordinate system
  • M is the camera internal parameter matrix
  • Fig. 3 shows a schematic diagram of an internal reference calibration board according to an embodiment of the present disclosure.
  • the internal parameter calibration board and the OpenCV open source camera calibration program are used to obtain the internal parameter matrix of the image collector, and the conversion between the image coordinate system and the camera coordinate system can be realized through the internal parameter matrix.
  • OpenCV is a cross-platform computer vision and machine learning software library based on the BSD license (open source) that can run on Linux, Windows, Android and Mac OS operating systems. It is lightweight and efficient—consisting of a series of C functions and a small number of C++ classes, it also provides interfaces to languages such as Python, Ruby, and MATLAB, and implements many common algorithms in image processing and computer vision.
  • obtaining the external parameter matrix of the image collector includes the first and second steps as described below.
  • the external parameter calibration board In the first step, according to the number of calibration points set on the external parameter calibration board, set the external parameter calibration board at different positions relative to the screen, and obtain the position image corresponding to each position, where the number of positions is, for example, It can correspond to the number of calibration points.
  • the center of the right screen of the VR device is set as the origin of the world coordinate system Ow
  • the center of the camera lens 30 is the origin of the camera coordinate system Oc.
  • the establishment of the two coordinate systems conforms to the right-hand rule.
  • five calibration points 21 are provided on the external reference calibration board 20.
  • Figure 5 shows a schematic diagram of obtaining the external parameter matrix of the high-speed infrared camera by using the external parameter calibration board.
  • the origin of the world coordinate system is represented by Ow
  • the three axes are Xw, Yw, Zw
  • the origin of the camera coordinate system is represented by Oc
  • the three axes are Xc, Yc, Zc
  • the lens of the high-speed infrared camera is set at the distance screen Is the position of d.
  • the coordinates of the corresponding points W1'-W10' of W1-W10 in the world coordinate system can be determined from the screen, respectively: W1'(s, s, d+d1) ), W2'(-s, s, d+d1), W3'(0, 0, d+d1), W4'(s, -s, d+d1), W5'(-s, -s, d +d1), W6'(s,s,d+d1+d2), W7'(-s,s,d+d1+d2), W8'(0,0,d+d1+d2), W9'( s, -
  • the second step is to obtain the external parameter matrix of the image collector according to the coordinates of the calibration points in the world coordinate system on the screen and the coordinates of the calibration points in the image coordinate system in the corresponding position images.
  • the conversion relationship between the world coordinate system and the camera coordinate system is:
  • the acquired coordinates of 10 points in the image coordinate system are brought into the above conversion relationship (2) to solve the 9 unknown parameters in the rotation matrix Rc matrix, so as to obtain the coordinates used to convert the camera coordinate system and the world.
  • the external parameter matrix of the coordinate system is
  • the number of calibration points on the external reference calibration board and the number of positions where the external reference calibration board is set in different positions are not limited in this application.
  • the number of locations is determined according to the number of calibration points on the external reference calibration board. Repeat it again.
  • the internal parameter matrix and the external parameter matrix are obtained by pre-calibrating the image collector once, so that the conversion between different coordinate systems can be realized through the internal parameter matrix and the external parameter matrix during the gaze tracking process.
  • the error-prone problems are calibrated through the polynomial mapping method, which effectively improves the stability and calculation accuracy of gaze tracking, and improves the user experience.
  • S12 Determine whether the spherical center coordinates can be determined according to the multiple pupil coordinates in the world coordinate system, if the spherical center coordinates can be determined, jump to S14, otherwise jump to S10, the spherical center coordinates Is the coordinates of the center of the sphere where the multiple pupil coordinates in the world coordinate system are located
  • the pupil rotates around the center of the eyeball on the surface of the eyeball, that is, the movement trajectory of the pupil is on the spherical surface, according to the pupil coordinates (x, y, z) in the world coordinate system ,
  • the following equation can be determined:
  • (x 0 , y 0 , z 0 ) are the coordinates of the center of the sphere in the world coordinate system, and R is the radius of the eyeball.
  • the sum of squared errors between the estimated value after fitting and the actual value is:
  • E(x 0 , y 0 , z 0 , R) is a function of x 0 , y 0 , z 0 , and R. Therefore, E(x 0 , y 0 , z 0 , R) is about x 0 , y 0 , Z 0 , the partial derivative of R is 0, namely:
  • S14 Determine the coordinates of the gaze point of the user on the screen according to the coordinates of the center of the sphere and the coordinates of the pupil.
  • the coordinates of the gaze point of the user on the screen in the image coordinate system are determined according to the spherical center coordinates and the multiple pupil coordinates in the world coordinate system.
  • Fig. 6 shows a schematic diagram of line-of-sight tracking according to an embodiment of the present disclosure.
  • the calculation is continued to determine the coordinates of the gaze point in the image coordinate system according to the spherical center coordinates in the world coordinate system and multiple pupil coordinates, that is, the pupil 52 is centered on the spherical center 51
  • the eyeball 50 moves, and the gaze point 41 is the point where the line of sight passes through the center of the sphere 51 and the pupil 52 and intersects on the screen 40, which specifically includes S140 and S142.
  • S140 Obtain a line-of-sight equation according to the multiple pupil coordinates in the world coordinate system and the spherical center coordinates, and obtain the coordinates of the user's gaze point in the world coordinate system according to the line-of-sight equation.
  • S142 Convert the coordinates of the gaze point of the user in the world coordinate system to the coordinates of the gaze point of the user on the screen in the image coordinate system.
  • the coordinates of the gaze point obtained by the above calculation are coordinates in the world coordinate system.
  • the coordinates of the gaze point need to be converted to coordinates in the image coordinate system.
  • the origin of the world coordinate system is at the center of the screen
  • the origin of the image coordinate system is at the upper left corner of the screen
  • the X and Y axes of the two coordinate systems are parallel to each other
  • the coordinates of the gaze point on the screen in the world coordinate system are (x k , y k ).
  • the shape of a single screen in a virtual reality device is generally square, that is, the horizontal resolution and the vertical resolution are equal, so the physical of the screen is set
  • the size is n*n
  • the resolution is m*m
  • the pixel pitch of the screen is: n/m.
  • (x t , y t ) are the coordinates of the gaze point in the image coordinate system.
  • multiple human eye images collected while the user’s eyes are scanning the screen are used to determine the pupil coordinates in each human eye image, and then multiple pupil coordinates are used to obtain the spherical center coordinates of the eyeball center, and then according to the multiple pupil coordinates and The coordinates of the center of the sphere determine the coordinates of the user's marked viewpoint on the screen, thereby realizing the tracking of the user's line of sight.
  • the polynomial mapping model that uses 9 calibration points in related technologies to achieve gaze tracking, it effectively simplifies the gaze tracking process, improves the stability and calculation accuracy of gaze tracking, and enhances user experience. It has a wide range of application prospects.
  • this application does not specifically limit the use of monocular line-of-sight tracking and binocular line-of-sight tracking, and the use of binocular line-of-sight tracking can further improve the accuracy of line-of-sight tracking.
  • Those skilled in the art should choose an appropriate method for gaze tracking according to actual application requirements, and take the ability to obtain pupil coordinates, determine the sphere center coordinates, and then determine the coordinates of the gaze point as the design criteria, which will not be repeated here.
  • an embodiment of the present application also provides a gaze tracking device. Since the gaze tracking device provided in this embodiment of the application corresponds to the gaze tracking method provided in the foregoing embodiments, the previous implementation manner is also applicable to the gaze tracking device provided in this embodiment, and will not be described in detail in this embodiment .
  • an embodiment of the present application further provides a gaze tracking device 800, which includes a pupil positioning circuit 801, a spherical center positioning circuit 802, and a gaze point positioning circuit 803.
  • the pupil positioning circuit 801 is configured to acquire multiple human eye images and respectively determine multiple pupil coordinates in the world coordinate system in the multiple human eye images, and the multiple human eye images are the user's eyes scanning along a predetermined trajectory Trajectory image acquired while on the screen.
  • the sphere center positioning circuit 802 is configured to determine sphere center coordinates according to multiple pupil coordinates in the world coordinate system, where the sphere center coordinates are the coordinates of the sphere center of the sphere where the multiple pupil coordinates in the world coordinate system are located.
  • the gaze point positioning circuit 803 is configured to determine the coordinates of the gaze point of the user on the screen in the image coordinate system according to the spherical center coordinates and multiple pupil coordinates in the world coordinate system.
  • the pupil positioning circuit 801 may include an image collector and a light source, and the pupil positioning circuit is configured to: control the image collector to collect multiple images of the human eye under the light provided by the light source; Image processing is performed to determine the pupil coordinates in the image coordinate system of each human eye image; the pupil coordinates in the image coordinate system of each human eye image are converted according to the internal parameter matrix and the external parameter matrix pre-calibrated by the image collector Is the pupil coordinates in the world coordinate system.
  • the pupil coordinates in the world coordinate system of each human eye image are determined by using multiple human eye images with saccadic trajectories obtained when the human eye scans the screen, and then the pupil coordinates are determined according to the movement of the pupil on the surface of the eyeball. It is characterized by determining the spherical center coordinates of the spherical surface where the movement track is located by multiple pupil coordinates in the world coordinate system, and then determining where the user is based on the multiple pupil coordinates and spherical center coordinates in the world coordinate system.
  • the coordinates in the image coordinate system of the gaze point on the screen are described, so as to realize the gaze tracking of the human eye.
  • the specific implementation of this embodiment is the same as the foregoing embodiment, and will not be repeated here.
  • the pupil positioning unit 801 may further include a calibration circuit 8011 for respectively calibrating the internal parameter matrix and the external parameter matrix of the image collector using an internal parameter calibration board and an external parameter calibration board.
  • the image collector is calibrated in advance to obtain the internal parameter matrix and the external parameter matrix, so that the conversion between different coordinate systems can be realized through the internal parameter matrix and the external parameter matrix during the gaze tracking process.
  • the error-prone problems are calibrated through the polynomial mapping method, which effectively improves the stability and calculation accuracy of gaze tracking, and improves the user experience.
  • the specific implementation of this embodiment is the same as the foregoing embodiment, and will not be repeated here.
  • pupil positioning circuit can be implemented as program modules, or implemented as various integrated circuits with data processing capabilities, such as processors, microcomputers, etc. Processors, programmable logic devices, etc.
  • Another embodiment of the present disclosure provides a computer-readable storage medium on which computer-executable instructions are stored.
  • the computer-executable instructions are executed by a processor, the following is achieved: S10: Obtain multiple images of human eyes and determine all images respectively.
  • S12 according to the world coordinate system Multiple pupil coordinates of, determine whether the spherical center coordinates can be determined, if the spherical center coordinates cannot be determined, reacquire multiple human eye images, the spherical center coordinates are the spherical surface where the multiple pupil coordinates in the world coordinate system are located The coordinates of the center of the sphere; S14, determine the coordinates of the user's gaze point on the screen in the image coordinate system according to the coordinates of the center of the sphere and multiple pupil coordinates in the world coordinate system.
  • the computer-readable storage medium may adopt any combination of one or more computer-readable media.
  • the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium.
  • Computer-readable storage medium refers to a medium and/or device that can store information persistently, and/or a tangible storage device. Therefore, computer-readable storage media refers to non-signal bearing media.
  • Computer-readable storage media include such as volatile and non-volatile, removable and non-removable media and/or suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements/circuits or other data) ) Hardware such as storage devices implemented by methods or technologies.
  • Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical storage devices, hard disks, cassette tapes, magnetic tapes, disk storage Apparatus or other magnetic storage devices, or other storage devices, tangible media, or articles suitable for storing desired information and that can be accessed by a computer.
  • the computer-readable storage medium may be any tangible medium that contains or stores executable instructions, and the executable instructions may be used by or in combination with an instruction execution system, apparatus, or device.
  • the computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and computer-readable program code (for example, computer-executable instructions) is carried therein.
  • This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing.
  • the computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium.
  • the computer-readable medium may send, propagate or transmit the program for use by or in combination with the instruction execution system, apparatus, or device .
  • the instructions contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.
  • the computer-executable instructions for executing the present disclosure can be written in one or more programming languages or a combination thereof.
  • the programming languages include object-oriented programming languages-such as Java, Smalltalk, C++, and also conventional Procedural programming language-such as "C" language or similar programming language.
  • the executable instructions may be executed entirely on the user's computer, partly on the user's computer, executed as an independent software package, partly on the user's computer and partly executed on a remote computer, or entirely executed on the remote computer or server.
  • the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to pass Internet connection).
  • LAN local area network
  • WAN wide area network
  • FIG. 9 a schematic structural diagram of a computing device provided by another embodiment of the present disclosure.
  • the computing device 900 shown in FIG. 9 is only an example, and should not bring any limitation to the function and scope of use of the embodiments of the present disclosure.
  • the computing device 900 is represented in the form of a general-purpose computing device.
  • the components of the computing device 900 may include, but are not limited to: one or more processors or processing units 916, a system memory 928, and a bus 918 connecting different system components (including the system memory 928 and the processing unit 916).
  • the bus 918 represents one or more of several types of bus structures, including a memory bus or a memory controller, a peripheral bus, or a local bus using any bus structure among multiple bus structures.
  • these architectures include but are not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC) bus, enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and peripheral component interconnection ( PCI) bus.
  • ISA industry standard architecture
  • MAC microchannel architecture
  • VESA Video Electronics Standards Association
  • PCI peripheral component interconnection
  • the system memory 928 may include computer-readable media in the form of volatile memory, such as random access memory (RAM) 930 and/or cache memory 932.
  • the computing device 900 may further include other removable/non-removable, volatile/nonvolatile computer storage media.
  • the storage system 934 may represent a non-removable, non-volatile magnetic medium (not shown in FIG. 9 and commonly referred to as a "hard drive").
  • a disk drive for reading and writing to a removable non-volatile disk such as a "floppy disk”
  • a removable non-volatile optical disk such as CD-ROM, DVD-ROM
  • each drive may be connected to the bus 918 through one or more data medium interfaces.
  • the system memory 928 may include at least one program product, the program product having a set (for example, at least one) program modules, and these program modules are configured to perform the functions of the various embodiments of the present disclosure.
  • a program/utility tool 940 having a set of (at least one) program module 942 may be stored in, for example, the system memory 928.
  • Such program module 942 includes but is not limited to an operating system, one or more application programs, other program modules, and programs Data, each of these examples or some combination may include the realization of the network environment.
  • the program module 942 generally executes the functions and/or methods in the embodiments described in the present disclosure.
  • the various circuits included in the gaze tracking device as described above can be implemented as program modules.
  • the computing device 900 may also communicate with one or more external devices 914 (such as a keyboard, pointing device, display 924, etc.), and may also communicate with one or more devices that enable a user to interact with the computing device 900, and/or communicate with Any device (such as a network card, modem, etc.) that enables the computing device 900 to communicate with one or more other computing devices. Such communication can be performed through an input/output (I/O) interface 922.
  • the computing device 900 may also communicate with one or more networks (for example, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through the network adapter 920. As shown in FIG.
  • the network adapter 920 communicates with other modules of the computing device 900 through the bus 918. It should be understood that although not shown in FIG. 9, other hardware and/or software modules can be used in conjunction with the computing device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, Tape drives and data backup storage systems, etc.
  • the processor unit 916 executes various functional applications and data processing by running programs stored in the system memory 928, for example, to implement a line-of-sight tracking method provided by an embodiment of the present disclosure.
  • the processor unit may be, for example, a central processing unit, a microprocessor, or one or more cores thereof, and so on.

Abstract

本申请描述了一种视线追踪方法、视线追踪装置、计算设备和介质，所述视线追踪方法包括：获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定轨迹扫视屏幕时获取的轨迹图像；根据所述世界坐标系下的多个瞳孔坐标，确定球心坐标，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标；根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。

Description

视线追踪方法、视线追踪装置、计算设备和介质
相关申请的交叉引用
本申请要求2020年6月9日提交的中国专利申请No.202010517378.4的权益，其全部公开内容通过引用合并于此。
技术领域
本公开涉及视线追踪技术领域，特别是涉及一种视线追踪方法、视线追踪装置、计算设备和介质。
背景技术
随着虚拟现实(Virtual Reality，VR)等人工智能技术的迅速发展，用户迫切需要一种方便、准确、鲁棒性好的交互系统，因此，非侵入式视线追踪技术成为了这一领域的研究热点。
目前，相关技术中，在视线追踪系统中普遍采用的是多项式映射模型，该模型主要采用高阶多项式表示瞳孔到屏幕注视点的映射关系。一般使用9个标定点进行标定，从而得出瞳孔到屏幕的映射关系，但是这种视线追踪方法存在较大的缺陷：第一，标定过程较为繁琐，每次使用前都需要用户进行标定，标定过程中需要用户依次注视屏幕上出现的标定点，每次需要注视1-2s并且标定时间较长，一般为15-25s；第二，标定过程容易出错，用户在标定过程中要求准确注视标定点，若有一个标定点出现错误注视，例如人眼没有很好的注视标定点中心，就会导致映射模型出错，从而导致注视点计算出现较大误差，需要进行重新标定。
发明内容
为了解决上述问题至少之一，本公开提供了一种视线追踪方法、视线追踪装置、计算设备和计算机可读存储介质。
根据本公开的第一方面，提供了一种视线追踪方法，包括：
获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定轨迹扫视屏幕时获取的轨迹图像；
根据所述世界坐标系下的多个瞳孔坐标，确定球心坐标，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标；
根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
可选地，所述获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标进一步包括：
控制图像采集器在光源提供的光下采集多张人眼图像；
分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标；
根据所述图像采集器预标定的内参矩阵和外参矩阵将所述各人眼图像中图像坐标系下的瞳孔坐标转换为世界坐标系下的瞳孔坐标。
可选地，所述分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标，包括：
分别对所述多张人眼图像进行预处理；
分别对预处理后的多张人眼图像进行二值化处理；
针对二值化处理后得到的每个二值化图像，分别计算所述二值化图像的瞳孔区域的轮廓，以及根据轮廓的大小和形状剔除其中的非瞳孔轮廓，并基于剔除非瞳孔轮廓后的所述二值化图像确定二值化图像对应的人眼图像的图像坐标系下的瞳孔坐标，其中，所述图像坐标系的原点位于屏幕左上角。
可选地，所述分别对所述多张人眼图像进行预处理，包括：
将所述多张人眼图像转化为多张灰度图像；
对所述多张灰度图像进行滤波，以滤除所述灰度图像中的噪声。
可选地，所述分别对预处理后的多张人眼图像进行二值化处理，包括：
对所述预处理后的图像中的各像素进行二值化处理，以获得二值化图像；
将获得的二值化图像中瞳孔部分灰度值设置为零，并对二值化图像采取开运算以去除瞳孔中的白色空洞。
可选地，在获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标之前，所述视线追踪方法还包括：
分别使用内参标定板和外参标定板标定所述图像采集器的内参矩 阵和外参矩阵。
可选地，所述分别使用内参标定板和外参标定板标定所述图像采集器的内参矩阵和外参矩阵进一步包括：
根据外参标定板上设置的标定点的数量，将外参标定板相对于屏幕分别设置在位置数量的不同位置上，并获取各位置对应的位置图像，其中所述位置数量与所述标定点的数量相对应；
根据所述屏幕上世界坐标系下的标定点的坐标、以及对应的各所述位置图像中图像坐标系下的标定点的坐标获取所述图像采集器的外参矩阵。
可选地，所述根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标进一步包括：
根据所述世界坐标系下的多个瞳孔坐标和所述球心坐标获取视线方程，并根据所述视线方程获取用户的注视点在世界坐标系下的坐标；
将所述用户的注视点在世界坐标系下的坐标转换为屏幕上用户的注视点在图像坐标系下的坐标。
可选地，所述轨迹图像包括：
所述用户的眼睛按照所述屏幕的对角线进行扫视时获取的轨迹图像；
或者
所述用户的眼睛分别按照所述屏幕的第一方向和第二方向进行扫视时获取的轨迹图像，其中第一方向和第二方向垂直；
或者
所述用户的眼睛环绕所述屏幕进行扫视时获取的轨迹图像。
根据本公开的第二方面，提供了一种视线追踪装置，包括
瞳孔定位电路，被配置成获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定轨迹扫视屏幕时获取的轨迹图像；
球心定位电路，被配置成根据所述世界坐标系下的多个瞳孔坐标，确定球心坐标，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标；
注视点定位电路，被配置成根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
可选地，所述瞳孔定位电路包括图像采集器和光源，并且所述瞳孔定位电路被配置成：
控制图像采集器在光源提供的光下采集多张人眼图像；
分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标；
根据所述图像采集器预标定的内参矩阵和外参矩阵将所述各人眼图像中图像坐标系下的瞳孔坐标转换为世界坐标系下的瞳孔坐标。
可选地，瞳孔定位电路还包括标定电路，用于分别使用内参标定板和外参标定板标定所述图像采集器的内参矩阵和外参矩阵。
根据本公开的第三方面，提供了一种计算机可读存储介质，其上存储有计算机可执行指令，其中，所述计算机可执行指令被处理器执行时执行如上所述的任一视线追踪方法。
根据本公开的第四方面，提供了一种计算设备，包括处理器和其存储有计算机可执行指令的存储器，其中，所述处理器执行所述计算机可执行指令时执行如上所述的任一视线追踪方法。
附图说明
为了更清楚地说明本公开实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1示出根据本公开的一个实施例所述的视线追踪方法的流程图；
图2示出根据本公开的一个实施例所述的虚拟现实设备的结构示意图；
图3示出根据本公开的一个实施例所述的内参标定板的示意图；
图4示出根据本公开的一个实施例所述的外参标定板的示意图；
图5示出根据本公开的一个实施例所述的图像采集器的外参矩阵的标定示意图；
图6示出根据本公开的一个实施例所述的视线追踪的示意图；
图7示出根据本公开的一个实施例所述的坐标系转换的示意图；
图8示出根据本公开的一个实施例所述的视线追踪装置的结构示意图；
图9示出根据本公开的一个实施例所述的一种计算设备的结构示意图。
具体实施方式
为了更清楚地说明本公开，下面结合实施例和附图对本公开做进一步的说明。附图中相似的部件以相同的附图标记进行表示。本领域技术人员应当理解，下面所具体描述的内容是说明性的而非限制性的，不应以此限制本公开的保护范围。
图1示出本公开的一个实施例所述的视线追踪方法的流程图。如图1所示，所述视线追踪方法包括：S10，获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定扫视轨迹扫视屏幕时获取的轨迹图像；S12，根据所述世界坐标系下的多个瞳孔坐标，判断是否能够确定球心坐标，若无法确定所述球心坐标则重新获取多张人眼图像，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标；S14，根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
在本实施例中，通过使用在人眼扫视屏幕时获取的具有扫视轨迹的多张人眼图像确定各人眼图像的世界坐标系下的瞳孔坐标，然后根据所述瞳孔在眼球球面上运动的特点，通过所述世界坐标系下的多个瞳孔坐标确定其运动轨迹所在的球面的球心坐标，再根据所述世界坐标系下的多个瞳孔坐标和球心坐标确定出所述用户在所述屏幕上的注视点的图像坐标系下的坐标，从而实现对人眼的视线追踪。这里所述的瞳孔坐标也即是瞳孔的坐标。
在本实施例的视线追踪过程中，利用人眼扫视屏幕获取具有扫视轨迹的多个瞳孔坐标，进而简化了相关技术中使用多个(通常为9个)固定的标定点依次标定的步骤，有效避免采用多项式映射方法进行标定的繁琐流程，加快对用户的视线追踪，提高视线追踪的稳定性和计算精度，能够有效提升用户的使用体验，具有广泛的应用前景。
在一个具体的实施例中，如图2所示，虚拟现实设备100包括第一镜头11和第二镜头12。考虑到两眼视线一致，本实施例采用单目视线追踪进行说明，所述虚拟现实设备还包括设置在第一镜头11正下方 的图像采集器13(例如，相机)和设置在第一镜头11周围的光源14。在这种情况下，视线追踪步骤如下所述。
在S10：获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定轨迹扫视屏幕时获取的轨迹图像。作为示例，所述步骤S10可以具体包括如下步骤S100-S104。
S100：控制图像采集器在所述光源提供的光下采集多张人眼图像。
在本实施例中，所述图像采集器为相机，所述相机的中心轴线指向人眼区域的中心位置，使得所述相机在光源提供光的环境下采集人眼图像。考虑到在夜晚或黑暗环境中采集人眼图像的场景，所述光源可以为红外光源，所述相机可以为红外相机。为快速准确采集人眼图像，所述相机可以为高速红外相机，同时考虑到光源的功率和均匀补光的问题可以使用多个红外光源。具体的，本实施例使用的高速红外相机的分辨率为640*480，帧速为100fps，视场角(FOV)为60°，并且在所述第一镜头11的周围按照正六边形的顶点位置设置六个红外光源，各红外光源的波长为850nm。所述六个红外光源能够提供均匀的环境光以便于高速红外相机采集人眼图像，并且有利于将瞳孔从虹膜区域分割以获取清晰的瞳孔图像。
在一个可选的实施例中，所述轨迹图像为：用户的眼睛分别按照所述屏幕的第一方向和第二方向进行扫视时获取的轨迹图像，其中第一方向和第二方向垂直。具体的，例如可以提示用户在水平方向从左到右扫视屏幕，然后提示用户在竖直方向从上到下扫视屏幕以完成扫视屏幕的过程。即在该过程中，用户的眼睛的瞳孔从屏幕左端移动到右端，再从屏幕的上端移动到下端，高速红外相机在红外光源提供的红外光下采集N帧具有瞳孔运动轨迹的人眼图像以用于视线追踪。本实施例中用户扫视屏幕预计耗时2s，相比于相关技术中通过9个标定点进行标定所需的15-25s的时间，大幅降低了标定时间，有效提高视线追踪的和标定的效率，提升了用户体验。
在一个可选的实施例中，所述轨迹图像为用户的眼睛按照所述屏幕的对角线进行扫视时获取的轨迹图像。例如可以提示用户从屏幕的一角开始扫视至相对于屏幕中心的另一角以完成扫视屏幕的过程，例如从屏幕的左上角开始扫视至屏幕的右下角(对角)。即，通过用户 的眼睛扫视屏幕的对角线获取具有扫视轨迹的多张人眼图像。图像采集器可以用户的眼睛扫视屏幕的过程中采集多张人眼图像用于视线追踪。
在一个可选的实施例中，所述轨迹图像为用户的眼睛环绕所述屏幕进行扫视时获取的轨迹图像。例如，可以提示用户从屏幕周边的一点开始，环绕屏幕扫视一周以完成扫视屏幕的过程。图像采集器在用户的眼睛扫视屏幕的过程中采集多张人眼图像用于视线追踪。
综上，本实施例中，可以在用户的眼睛的扫视过程中采集多张人眼图像，从而有效简化相关技术中依次使用9个标定点的标定步骤。
S102：分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标。
在本实施例中，通过多张人眼图像获取图像坐标系下的瞳孔坐标，具体可以包括：
首先，分别对所述多张人眼图像进行预处理。具体的，对所述多张人眼图像进行预处理可以包括：将人眼图像转化为灰度图像，然后对所述灰度图像进行滤波(例如，高斯滤波)，以滤除灰度图像中的噪声。
其次，分别对预处理后的多张人眼图像进行二值化处理。具体的，对滤波后的图像中的各像素进行二值化处理，以得到二值化图像；将获得的二值化图像中瞳孔部分灰度值设置为0，并对二值化图像采取开运算以去除瞳孔中的白色空洞。
最后，针对二值化处理后得到的每个二值化图像，分别计算所述二值化图像的瞳孔区域的轮廓，然后根据轮廓的大小和形状剔除其中的非瞳孔轮廓，并基于剔除非瞳孔轮廓后的所述二值化图像(例如，使用质心法)确定二值化图像对应的人眼图像的图像坐标系下的瞳孔坐标，其中，所述图像坐标系的原点位于屏幕左上角。
S104：根据所述图像采集器预标定的内参矩阵和外参矩阵将所述各人眼图像的图像坐标系下的瞳孔坐标转换为世界坐标系下的瞳孔坐标。
在本实施例中，根据各图像中图像坐标系下的瞳孔坐标，利用高速红外相机预先标定的内参矩阵将图像坐标系下的瞳孔坐标转换为相机坐标系下的瞳孔坐标，然后再利用高速红外相机预先标定的外参矩 阵将相机坐标系下的瞳孔坐标转换为世界坐标系下的瞳孔坐标。
在本实施例中，所述图像采集器预标定的内参矩阵和外参矩阵可以是高速红外相机出厂前标定的，也可以在使用前进行标定，本申请对此不做限定。
考虑到高速红外相机未在出厂前标定或者没有该高速红外相机的内参矩阵和外参矩阵数据，在一个可选的实施例中，在所述S 10之前，所述视线追踪方法还可以包括：步骤S01：分别使用内参标定板和外参标定板标定所述图像采集器的内参矩阵和外参矩阵。
首先，获取图像采集器的内参矩阵。
作为示例，具体地，图像坐标系和相机坐标系的转换关系为：
Figure PCTCN2021096007-appb-000001
其中，(u，v)为图像坐标系下的坐标，M为相机内参矩阵，
Figure PCTCN2021096007-appb-000002
为相机坐标系下的坐标。
图3示出根据本公开的一个实施例所述的内参标定板的示意图。在本实施例中，使用内参标定板和OpenCV开源的相机标定程序获取图像采集器的内参矩阵，通过该内参矩阵能够实现在图像坐标系和相机坐标系之间进行转换。OpenCV是一个基于BSD许可(开源)发行的跨平台计算机视觉和机器学习软件库，可以运行在Linux、Windows、Android和Mac OS操作系统上。它轻量级而且高效——由一系列C函数和少量C++类构成，同时提供了Python、Ruby、MATLAB等语言的接口，实现了图像处理和计算机视觉方面的很多通用算法。
其次，获取图像采集器的外参矩阵。
在一个可选的实施例中，获取图像采集器的外参矩阵包括如下所述的第一步和第二步。
第一步，根据外参标定板上设置的标定点的数量，分别将外参标定板相对于屏幕设置在位置数量的不同位置上，并获取各位置对应的位置图像，其中所述位置数量例如可以与所述标定点的数量相对应。
在本实施例中，如图2所示，设定VR设备的右屏幕中心为世界坐标系原点Ow，相机镜头30的中心为相机坐标系原点Oc，两坐标系的 建立都符合右手定则。如图4所示为外参标定板20上设置有5个标定点21。图5示出了利用外参标定板获取高速红外相机的外参矩阵的示意图。具体的，世界坐标系原点以Ow表示，三个轴分别为Xw、Yw、Zw，相机坐标系原点以Oc表示，三个轴分别为Xc、Yc、Zc；高速红外相机的镜头设置在距离屏幕为d的位置。
首先，如图5所示，将外参标定板平行于屏幕设置在距屏幕d+d1处，所述外参标定板的中心在世界坐标系的Zw轴上，然后使用高速红外相机拍照并获取标定点W1-W5的在相机坐标系下的5个点坐标。
其次，如图5所示，将外参标定板平行于屏幕设置在距屏幕d+d1+d2处，所述外参标定板的中心在世界坐标系的Zw轴上，然后使用高速红外相机拍照并获取标定点W6-W10的在相机坐标系下的5个点坐标。所述W1-W5与所述W6-W10是相对应的，是外参标定板上的相同点在不同位置处的表示。
在上述过程中，根据设置有5个标定点(W1-W5，W6-W10)的外参标定板，以及分别将外参标定板设置在不同的两个位置上采集位置图像，一共获得图像坐标系下的10个点坐标。同时，根据原点位于屏幕中心的世界坐标系，从屏幕上能够确定W1-W10在世界坐标系下的对应的点W1’-W10’的坐标，分别为：W1’(s，s，d+d1)，W2’(-s，s，d+d1)，W3’(0，0，d+d1)，W4’(s，-s，d+d1)，W5’(-s，-s，d+d1)，W6’(s，s，d+d1+d2)，W7’(-s，s，d+d1+d2)，W8’(0，0，d+d1+d2)，W9’(s，-s，d+d1+d2)和W10’(-s，-s，d+d1+d2)。
第二步，根据所述屏幕上世界坐标系下的标定点的坐标、以及对应的各所述位置图像中图像坐标系下的标定点的坐标获取所述图像采集器的外参矩阵。
作为示例，具体地，世界坐标系和相机坐标系的转换关系为：
Figure PCTCN2021096007-appb-000003
其中，
Figure PCTCN2021096007-appb-000004
为世界坐标系下的坐标，包括旋转矩阵Rc和平移矩阵Pc，其中，Rc为3*3的旋转矩阵，平移矩阵Pc为两坐标系原点的差值；
Figure PCTCN2021096007-appb-000005
为相机坐标系下的坐标。
在本实施例中，将获取的图像坐标系下的10个点的坐标带入上述转换关系式(2)求解旋转矩阵Rc矩阵中的9个未知参数，从而获取用于转换相机坐标系和世界坐标系的外参矩阵。
值得说明的是，本申请对外参标定板上标定点的数量、以及将外参标定板设置在不同位置的位置数量不作限定，根据外参标定板上标定点的数量确定位置数量，在此不再赘述。
在本实施例中，通过预先对图像采集器进行一次标定获取内参矩阵和外参矩阵，以便于在视线追踪过程中通过所述内参矩阵和外参矩阵实现不同坐标系之间的转换，相比于相关技术中通过多项式映射方式标定容易出错的问题，有效提高了视线追踪的稳定性和计算精度，并且提升了用户的使用体验。
S12：根据所述世界坐标系下的多个瞳孔坐标，判断是否能够确定球心坐标，若能够确定所述球心坐标则跳转至S 14，否则跳转至S 10，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标
在本实施例中，用户的眼睛在扫视屏幕时，瞳孔围绕眼球中心在眼球球面上进行转动，即瞳孔的运动轨迹位于球面上，则根据世界坐标系下的瞳孔坐标(x，y，z)，可以确定以下方程：
(x-x 0) 2+(y-y 0) 2+(z-z 0) 2＝R 2   (3)
其中，(x 0，y 0，z 0)为世界坐标系下的球心坐标，R为眼球半径。
具体的，对于球心坐标，拟合后的估计值与实际值的误差平方和为：
Figure PCTCN2021096007-appb-000006
其中，E(x 0，y 0，z 0，R)是x 0，y 0，z 0，R的函数，因此，E(x 0，y 0，z 0，R)关于x 0，y 0，z 0，R的偏导数为0，即：
Figure PCTCN2021096007-appb-000007
Figure PCTCN2021096007-appb-000008
Figure PCTCN2021096007-appb-000009
Figure PCTCN2021096007-appb-000010
Figure PCTCN2021096007-appb-000011
根据方程8可得：
Figure PCTCN2021096007-appb-000012
因此
Figure PCTCN2021096007-appb-000013
三个方程可以简化为：
Figure PCTCN2021096007-appb-000014
Figure PCTCN2021096007-appb-000015
Figure PCTCN2021096007-appb-000016
令：
Figure PCTCN2021096007-appb-000017
Figure PCTCN2021096007-appb-000018
Figure PCTCN2021096007-appb-000019
Figure PCTCN2021096007-appb-000020
Figure PCTCN2021096007-appb-000021
Figure PCTCN2021096007-appb-000022
并带入化简后的
Figure PCTCN2021096007-appb-000023
方程中，可得：
Figure PCTCN2021096007-appb-000024
Figure PCTCN2021096007-appb-000025
Figure PCTCN2021096007-appb-000026
Figure PCTCN2021096007-appb-000027
在方程(13)、(14)和(15)中减去方程(16)并化简成矩阵形式，可得：
Figure PCTCN2021096007-appb-000028
对方程(17)进行求解，若所述用户的眼睛扫视获得的世界坐标系下的多个瞳孔坐标能够解出矩阵方程(17)，则获得世界坐标系下的球心坐标(x 0，y 0，z 0)，否则跳转至S10重新采集用户的眼睛扫视屏幕的人眼图像。
S14：根据所述球心坐标和所述瞳孔坐标确定用户在屏幕上的注视点的坐标。根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
图6示出根据本公开的一个实施例所述的视线追踪的示意图。在本实施例中，如图6所示，根据世界坐标系下的球心坐标和多个瞳孔坐标继续计算以确定图像坐标系下的注视点的坐标，即瞳孔52以球心51为中心在眼球50上运动，注视点41为视线经过球心51和瞳孔52相交于屏幕40上的点，具体包括S140和S142。
S140：根据所述世界坐标系下的多个瞳孔坐标和所述球心坐标获取视线方程，并根据所述视线方程获取用户的注视点在世界坐标系下的坐标。
在本实施例中，根据世界坐标系下的球心坐标和瞳孔坐标确定以下视线方程：
Figure PCTCN2021096007-appb-000029
转化为一般方程为：
Figure PCTCN2021096007-appb-000030
在本实施例中，设定所述屏幕所在平面方程为：z＝0；将其代入模块方程(19)，能够确定世界坐标系下的注视点的坐标K(x k，y k，z k)：
Figure PCTCN2021096007-appb-000031
S142：将所述用户的注视点在世界坐标系下的坐标转换为屏幕上用户的注视点在图像坐标系下的坐标。
在本实施例中，上述计算得到的注视点的坐标为世界坐标系下的坐标，为了便于计算屏幕上的注视点的位置，需要将该注视点的坐标转换为图像坐标系下坐标。
作为示例，具体地，如图7所示，以屏幕为参照物，世界坐标系原点位于屏幕中心，图像坐标系原点位于屏幕左上角，并且两坐标系的X轴和Y轴相互平行，转换关系如下：
第一，注视点在屏幕上的世界坐标系的坐标为(x k，y k)。
第二，考虑到用户的单眼的横向视场角和纵向视场角大致相同，因此虚拟现实设备中的单个屏幕形状一般为正方形，即横向分辨率和纵向分辨率相等，因此设定屏幕的物理尺寸为n*n，分辨率为m*m，则屏幕的像素间距为：n/m。
则转换后的图像坐标为：
Figure PCTCN2021096007-appb-000032
则(x t，y t)即为图像坐标系下的注视点的坐标。
至此，利用在用户的眼睛扫视屏幕时而采集的多张人眼图像确定各人眼图像中的瞳孔坐标，再利用多个瞳孔坐标获取眼球中心的球心坐标，再根据所述多个瞳孔坐标和球心坐标确定用户在屏幕上的标注视点的坐标，从而实现对用户的视线的追踪。相比于相关技术中通过9个标定点采用多项式映射模型实现视线追踪，有效简化视线追踪流程、提高视线追踪的稳定性和计算精度，增强用户体验，具有广泛的应用前景。
值得说明的是，本申请对采用单目视线追踪和双目视线追踪不作具体限定，采用双目视线追踪能够进一步提高视线追踪的准确度。本领域技术人员应当根据实际应用需求选择适当的方式进行视线追踪，以能够获取瞳孔坐标、确定球心坐标、进而确定注视点的坐标为设计准则，在此不再赘述。
与上述实施例提供的视线追踪方法相对应，本申请的一个实施例还提供一种视线追踪装置。由于本申请实施例提供的视线追踪装置与上述几种实施例提供的视线追踪方法相对应，因此在前实施方式也适用于本实施例提供的视线追踪装置，在本实施例中不再详细描述。
如图8所示，本申请的一个实施例还提供一种视线追踪装置800，包括瞳孔定位电路801、球心定位电路802以及注视点定位电路803。
瞳孔定位电路801被配置成获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定轨迹扫视屏幕时获取的轨迹图像。球心定位电路802被配置成根据所述世界坐标系下的多个瞳孔坐标，确定球心坐标，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标。注视点定位电路803被配置成根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
所述述瞳孔定位电路801可以包括图像采集器和光源，并且所述瞳孔定位电路被配置成：控制图像采集器在光源提供的光下采集多张 人眼图像；分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标；根据所述图像采集器预标定的内参矩阵和外参矩阵将所述各人眼图像中图像坐标系下的瞳孔坐标转换为世界坐标系下的瞳孔坐标。
在本实施例中，通过使用在人眼扫视屏幕时获取的具有扫视轨迹的多张人眼图像确定各人眼图像的世界坐标系下的瞳孔坐标，然后根据所述瞳孔在眼球球面上运动的特点，通过所述世界坐标系下的多个瞳孔坐标确定其运动轨迹所在的球面的球心坐标，再根据所述世界坐标系下的多个瞳孔坐标和球心坐标确定出所述用户在所述屏幕上的注视点的图像坐标系下的坐标，从而实现对人眼的视线追踪。本实施例的具体实施方式同前述实施例，在此不再赘述。
瞳孔定位单元801还可以包括标定电路8011，用于分别使用内参标定板和外参标定板标定所述图像采集器的内参矩阵和外参矩阵。
在本实施例中，通过预先对图像采集器进行标定以获取内参矩阵和外参矩阵，以便于在视线追踪过程中通过所述内参矩阵和外参矩阵实现不同坐标系之间的转换，相比于相关技术中通过多项式映射方式标定容易出错的问题，有效提高了视线追踪的稳定性和计算精度，并且提升了用户的使用体验。本实施例的具体实施方式同前述实施例，在此不再赘述。
应当指出，上述所述的瞳孔定位电路、标定电路、球心定位电路、注视点定位电路等可以被实施为程序模块，或者实施为具有数据处理能力的各种的集成电路，例如处理器、微处理器、可编程逻辑器件等等。
本公开的另一个实施例提供了一种计算机可读存储介质，其上存储有计算机可执行指令，该计算机可执行指令被处理器执行时实现：S10，获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定扫视轨迹扫视屏幕时获取的轨迹图像；S12，根据所述世界坐标系下的多个瞳孔坐标，判断是否能够确定球心坐标，若无法确定所述球心坐标则重新获取多张人眼图像，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标；S14，根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
在实际应用中，所述计算机可读存储介质可以采用一个或多个计算机可读的介质的任意组合。计算机可读介质可以是计算机可读信号介质或者计算机可读存储介质。“计算机可读存储介质”是指能够持久存储信息的介质和/或设备，和/或有形的存储装置。因此，计算机可读存储介质是指非信号承载介质。计算机可读存储介质包括诸如易失性和非易失性、可移动和不可移动介质和/或以适用于存储信息(诸如计算机可读指令、数据结构、程序模块、逻辑元件/电路或其他数据)的方法或技术实现的存储设备之类的硬件。计算机可读存储介质的示例可以包括但不限于RAM、ROM、EEPROM、闪存或其它存储器技术、CD-ROM、数字通用盘(DVD)或其他光学存储装置、硬盘、盒式磁带、磁带，磁盘存储装置或其他磁存储设备，或其他存储设备、有形介质或适于存储期望信息并可以由计算机访问的制品。在本实施例中，计算机可读存储介质可以是任何包含或存储可执行指令的有形介质，该可执行指令可以被指令执行系统、装置或者器件使用或者与其结合使用。
计算机可读的信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的程序代码(例如，计算机可执行的指令)。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读的信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的程序。
计算机可读介质上包含的指令可以用任何适当的介质传输，包括但不限于无线、电线、光缆、RF等等，或者上述的任意合适的组合。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的计算机可执行指令，所述程序设计语言包括面向对象的程序设计语言-诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言-诸如“C”语言或类似的程序设计语言。可执行指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)-连接 到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
如图9所示，本公开的另一个实施例提供的一种计算设备的结构示意图。图9显示的计算设备900仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图9所示，计算设备900以通用计算设备的形式表现。计算设备900的组件可以包括但不限于：一个或者多个处理器或者处理单元916，系统存储器928，连接不同系统组件(包括系统存储器928和处理单元916)的总线918。
总线918表示几类总线结构中的一种或多种，包括存储器总线或者存储器控制器、外围总线、或者使用多种总线结构中的任意总线结构的局域总线。举例来说，这些体系结构包括但不限于工业标准体系结构(ISA)总线，微通道体系结构(MAC)总线，增强型ISA总线、视频电子标准协会(VESA)局域总线以及外围组件互连(PCI)总线。
系统存储器928可以包括易失性存储器形式的计算机可读介质，例如随机存取存储器(RAM)930和/或高速缓存存储器932。计算设备900可以进一步包括其它可移动/不可移动的、易失性/非易失性计算机存储介质。仅作为举例，存储系统934可以表示不可移动的、非易失性磁介质(图9未显示，通常称为“硬盘驱动器”)。尽管图9中未示出，可以提供用于对可移动非易失性磁盘(例如“软盘”)读写的磁盘驱动器，以及对可移动非易失性光盘(例如CD-ROM，DVD-ROM或者其它光介质)读写的光盘驱动器。在这些情况下，每个驱动器可以通过一个或者多个数据介质接口与总线918相连。系统存储器928可以包括至少一个程序产品，该程序产品具有一组(例如至少一个)程序模块，这些程序模块被配置以执行本公开各实施例的功能。
具有一组(至少一个)程序模块942的程序/实用工具940，可以存储在例如系统存储器928中，这样的程序模块942包括但不限于操作系统、一个或者多个应用程序、其它程序模块以及程序数据，这些示例中的每一个或某种组合中可能包括网络环境的实现。程序模块942通常执行本公开所描述的实施例中的功能和/或方法。如上面所述的视线追踪装置包括的各种电路可以被实施为程序模块。
计算设备900也可以与一个或多个外部设备914(例如键盘、指向 设备、显示器924等)通信，还可与一个或者多个使得用户能与该计算设备900交互的设备通信，和/或与使得该计算设备900能与一个或多个其它计算设备进行通信的任何设备(例如网卡，调制解调器等等)通信。这种通信可以通过输入/输出(I/O)接口922进行。并且，计算设备900还可以通过网络适配器920与一个或者多个网络(例如局域网(LAN)，广域网(WAN)和/或公共网络，例如因特网)通信。如图9所示，网络适配器920通过总线918与计算设备900的其它模块通信。应当明白，尽管图9中未示出，可以结合计算设备900一起使用其它硬件和/或软件模块，包括但不限于：微代码、设备驱动器、冗余处理单元、外部磁盘驱动阵列、RAID系统、磁带驱动器以及数据备份存储系统等。
处理器单元916通过运行存储在系统存储器928中的程序，从而执行各种功能应用以及数据处理，例如实现本公开实施例所提供的一种视线追踪方法。处理器单元例如可以是中央处理单元、微处理器、或者其一个或多个核心等等。
显然，本公开的上述实施例仅仅是为清楚地说明本公开所作的举例，而并非是对本公开的实施方式的限定，对于所属领域的普通技术人员来说，在上述说明的基础上还可以做出其它不同形式的变化或变动，这里无法对所有的实施方式予以穷举，凡是属于本公开的技术方案所引伸出的显而易见的变化或变动仍处于本公开的保护范围之列。

Claims (14)

  1. 一种视线追踪方法，包括：
    获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定轨迹扫视屏幕时获取的轨迹图像；
    根据所述世界坐标系下的多个瞳孔坐标，确定球心坐标，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标；
    根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
  2. 根据权利要求1所述的视线追踪方法，其中，所述获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标进一步包括：
    控制图像采集器在光源提供的光下采集多张人眼图像；
    分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标；
    根据所述图像采集器预标定的内参矩阵和外参矩阵将所述各人眼图像中图像坐标系下的瞳孔坐标转换为世界坐标系下的瞳孔坐标。
  3. 根据权利要求2所述的视线追踪方法，其中，所述分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标，包括：
    分别对所述多张人眼图像进行预处理；
    分别对预处理后的多张人眼图像进行二值化处理；
    针对二值化处理后得到的每个二值化图像，分别计算所述二值化图像的瞳孔区域的轮廓，以及根据轮廓的大小和形状剔除其中的非瞳孔轮廓，并基于剔除非瞳孔轮廓后的所述二值化图像确定二值化图像对应的人眼图像的图像坐标系下的瞳孔坐标，其中，所述图像坐标系的原点位于屏幕左上角。
  4. 根据权利要求3所述视线追踪方法，其中，分别对所述多张人眼图像进行预处理，包括：
    将所述多张人眼图像转化为多张灰度图像；
    对所述多张灰度图像进行滤波，以滤除所述灰度图像中的噪声。
  5. 根据权利要求3所述视线追踪方法，其中，分别对预处理后的多张人眼图像进行二值化处理，包括：
    对所述预处理后的图像中的各像素进行二值化处理，以获得二值化图像；
    将获得的二值化图像中瞳孔部分灰度值设置为零，并对二值化图像采取开运算以去除瞳孔中的白色空洞。
  6. 根据权利要求2所述的视线追踪方法，其中，在获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标之前，所述视线追踪方法还包括：
    分别使用内参标定板和外参标定板标定所述图像采集器的内参矩阵和外参矩阵。
  7. 根据权利要求6所述的视线追踪方法，其中，所述分别使用内参标定板和外参标定板标定所述图像采集器的内参矩阵和外参矩阵进一步包括：
    根据外参标定板上设置的标定点的数量，将外参标定板相对于屏幕分别设置在位置数量的不同位置上，并获取各位置对应的位置图像，其中所述位置数量与所述标定点的数量相对应；
    根据所述屏幕上世界坐标系下的标定点的坐标、以及对应的各所述位置图像中图像坐标系下的标定点的坐标获取所述图像采集器的外参矩阵。
  8. 根据权利要求1所述的视线追踪方法，其中，所述根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标进一步包括：
    根据所述世界坐标系下的多个瞳孔坐标和所述球心坐标获取视线方程，并根据所述视线方程获取用户的注视点在世界坐标系下的坐标；
    将所述用户的注视点在世界坐标系下的坐标转换为屏幕上用户的注视点在图像坐标系下的坐标。
  9. 根据权利要求1-8中任一项所述的视线追踪方法，其中，所述轨迹图像包括：
    所述用户的眼睛按照所述屏幕的对角线进行扫视时获取的轨迹图像；
    或者
    所述用户的眼睛分别按照所述屏幕的第一方向和第二方向进行扫视时获取的轨迹图像，其中第一方向和第二方向垂直；
    或者
    所述用户的眼睛环绕所述屏幕进行扫视时获取的轨迹图像。
  10. 一种视线追踪装置，其中，包括
    瞳孔定位电路，被配置成获取多张人眼图像并分别确定所述多张人眼图像中的世界坐标系下的多个瞳孔坐标，所述多张人眼图像为用户的眼睛沿预定轨迹扫视屏幕时获取的轨迹图像；
    球心定位电路，被配置成根据所述世界坐标系下的多个瞳孔坐标，确定球心坐标，所述球心坐标为所述世界坐标系下的多个瞳孔坐标所在球面的球心的坐标；
    注视点定位电路，被配置成根据所述球心坐标和所述世界坐标系下的多个瞳孔坐标确定屏幕上用户的注视点在图像坐标系下的坐标。
  11. 根据权利要求10所述的视线追踪装置，其中，所述瞳孔定位电路包括图像采集器和光源，并且所述瞳孔定位电路被配置成：
    控制图像采集器在光源提供的光下采集多张人眼图像；
    分别对所述多张人眼图像进行图像处理，确定各人眼图像中图像坐标系下的瞳孔坐标；
    根据所述图像采集器预标定的内参矩阵和外参矩阵将所述各人眼图像中图像坐标系下的瞳孔坐标转换为世界坐标系下的瞳孔坐标。
  12. 根据权利要求11所述的视线追踪装置，其中，瞳孔定位电路还包括标定电路，用于分别使用内参标定板和外参标定板标定所述图像采集器的内参矩阵和外参矩阵。
  13. 一种计算机可读存储介质，其上存储有计算机可执行指令，其中，所述计算机可执行指令被处理器执行时执行如权利要求1-9中任一项所述的视线追踪方法。
  14. 一种计算设备，包括处理器和其存储有计算机可执行指令的存储器，其中，所述处理器执行所述计算机可执行指令时执行如权利要求1-9中任一项所述的视线追踪方法。
PCT/CN2021/096007 2020-06-09 2021-05-26 视线追踪方法、视线追踪装置、计算设备和介质 WO2021249187A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010517378.4 2020-06-09
CN202010517378.4A CN111638799B (zh) 2020-06-09 2020-06-09 一种视线追踪方法、视线追踪装置、计算机设备和介质

Publications (1)

Publication Number Publication Date
WO2021249187A1 true WO2021249187A1 (zh) 2021-12-16

Family

ID=72329910

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/096007 WO2021249187A1 (zh) 2020-06-09 2021-05-26 视线追踪方法、视线追踪装置、计算设备和介质

Country Status (2)

Country Link
CN (1) CN111638799B (zh)
WO (1) WO2021249187A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114722570A (zh) * 2022-03-07 2022-07-08 北京航空航天大学 视线估计模型建立方法、装置、电子设备及存储介质

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111638799B (zh) * 2020-06-09 2023-10-27 京东方科技集团股份有限公司 一种视线追踪方法、视线追踪装置、计算机设备和介质
CN112308932B (zh) * 2020-11-04 2023-12-08 中国科学院上海微系统与信息技术研究所 一种注视检测方法、装置、设备及存储介质
CN113793389B (zh) * 2021-08-24 2024-01-26 国网甘肃省电力公司 一种增强现实系统虚实融合标定方法及装置
CN113723293B (zh) * 2021-08-30 2024-01-05 中国科学院上海微系统与信息技术研究所 一种视线方向的确定方法、装置、电子设备及存储介质
CN114035335B (zh) * 2021-11-29 2023-08-08 京东方科技集团股份有限公司 显示装置及其控制方法、显示系统
CN115797607B (zh) * 2023-02-13 2023-04-14 无锡文康科技有限公司 一种增强vr真实效果的图像优化处理方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102830793A (zh) * 2011-06-16 2012-12-19 北京三星通信技术研究有限公司 视线跟踪方法和设备
US8824779B1 (en) * 2011-12-20 2014-09-02 Christopher Charles Smyth Apparatus and method for determining eye gaze from stereo-optic views
CN108681699A (zh) * 2018-05-04 2018-10-19 上海像我信息科技有限公司 一种基于深度学习的视线估计方法及视线估计装置
CN110705504A (zh) * 2019-10-14 2020-01-17 京东方科技集团股份有限公司 视线定位方法、显示装置、电子设备以及存储介质
CN111638799A (zh) * 2020-06-09 2020-09-08 京东方科技集团股份有限公司 一种视线追踪方法、视线追踪装置、计算机设备和介质

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010259605A (ja) * 2009-05-01 2010-11-18 Nippon Hoso Kyokai <Nhk> 視線測定装置および視線測定プログラム
CN108427503B (zh) * 2018-03-26 2021-03-16 京东方科技集团股份有限公司 人眼追踪方法及人眼追踪装置
CN109947253B (zh) * 2019-03-25 2020-06-19 京东方科技集团股份有限公司 眼球追踪的模型建立方法、眼球追踪方法、设备、介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102830793A (zh) * 2011-06-16 2012-12-19 北京三星通信技术研究有限公司 视线跟踪方法和设备
US8824779B1 (en) * 2011-12-20 2014-09-02 Christopher Charles Smyth Apparatus and method for determining eye gaze from stereo-optic views
CN108681699A (zh) * 2018-05-04 2018-10-19 上海像我信息科技有限公司 一种基于深度学习的视线估计方法及视线估计装置
CN110705504A (zh) * 2019-10-14 2020-01-17 京东方科技集团股份有限公司 视线定位方法、显示装置、电子设备以及存储介质
CN111638799A (zh) * 2020-06-09 2020-09-08 京东方科技集团股份有限公司 一种视线追踪方法、视线追踪装置、计算机设备和介质

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114722570A (zh) * 2022-03-07 2022-07-08 北京航空航天大学 视线估计模型建立方法、装置、电子设备及存储介质
CN114722570B (zh) * 2022-03-07 2023-09-15 北京航空航天大学 视线估计模型建立方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN111638799B (zh) 2023-10-27
CN111638799A (zh) 2020-09-08

Similar Documents

Publication Publication Date Title
WO2021249187A1 (zh) 视线追踪方法、视线追踪装置、计算设备和介质
US11403757B2 (en) Sight line detection method and sight line detection device
JP6842520B2 (ja) 物体検出方法、装置、機器、記憶媒体及び車両
US11315281B2 (en) Pupil positioning method and apparatus, VR/AR apparatus and computer readable medium
TWI398796B (zh) Pupil tracking methods and systems, and correction methods and correction modules for pupil tracking
Itoh et al. Interaction-free calibration for optical see-through head-mounted displays based on 3d eye localization
WO2020125499A9 (zh) 一种操作提示方法及眼镜
CN108960045A (zh) 眼球追踪方法、电子装置及非暂态电脑可读取记录媒体
TWI680743B (zh) 眼球追蹤方法、電子裝置及非暫態電腦可讀取記錄媒體
WO2016195066A1 (ja) 眼球の運動を検出する方法、そのプログラム、そのプログラムの記憶媒体、及び、眼球の運動を検出する装置
Takemura et al. Estimation of a focused object using a corneal surface image for eye-based interaction
CN110555426A (zh) 视线检测方法、装置、设备及存储介质
US11011140B2 (en) Image rendering method and apparatus, and VR device
US20230080861A1 (en) Automatic Iris Capturing Method And Apparatus, Computer-Readable Storage Medium, And Computer Device
US10909363B2 (en) Image acquisition system for off-axis eye images
CN112017212B (zh) 人脸关键点跟踪模型的训练、跟踪方法及系统
CN115205286B (zh) 爬塔机器人机械臂螺栓识别与定位方法、存储介质、终端
CN116486250A (zh) 一种基于嵌入式的多路图像采集与处理方法及系统
CN114578952B (zh) 人机交互方法、系统、处理设备和计算机可读存储介质
Tordoff Active control of zoom for computer vision
CN116407080A (zh) 近视患者眼底结构的演变识别及3d可视化系统和方法
CN114792343B (zh) 图像获取设备的标定方法、获取图像数据的方法、装置
KR20090110348A (ko) 물체 형상 생성 방법, 물체 형상 생성 장치 및 프로그램
Cao et al. Gaze tracking on any surface with your phone
JP7177280B2 (ja) 画像認識装置、画像認識方法、及び、画像認識プログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21822516

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21822516

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 06.07.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 21822516

Country of ref document: EP

Kind code of ref document: A1