TWI758231B - Eye gesture tracking system - Google Patents

Abstract

An eye gesture tracking system including a machine with a display, the display including a plurality of tunable optical elements. The eye gesture tracking system can also include an eye gesture tracking device including circuitry configured to obtain an electrical signal that represents a measurement, by a photodetector, of an optical signal reflected from an eye. The circuitry can further be configured to determine a depth map of the eye based on phase differences between a reference signal and the electrical signal generated by the photodetector, and determine a gaze information that represents a gaze of the eye based on the depth map. Additionally, the eye gesture tracking system can include a signal processing unit in communication with the machine and the eye gesture tracking device, the signal processing unit can perform the operations including receiving, from the eye gesture tracking device, output data representing the gaze information and determining the gaze information representing the gaze of the eye in relation to the display of the machine.

Description

eye tracking system

The present invention is about eye tracking.

Light can be directed to an eye and reflected light can be observed. The reflected light can be processed to determine information relevant to the eye.

In some embodiments, an eye tracking method can be used to determine the gaze information of an eye. The aforementioned method includes demodulating the modulated optical signal reflected by the eye, and the demodulated optical signal can be processed to generate a depth map corresponding to the eye and further determine the line-of-sight information of the eye. The sight line information of the eye includes information corresponding to, for example, the pupil or iris of the eye, and can be used for various applications, such as judging user preference data, visually controlling human-machine interface devices, providing cross-platform peripheral control, and the like. In addition, by tracking the posture of the eye, the corresponding eye gaze information can be used to instantly refocus the adjustable optical element to change the light incident on the eye, creating a dizzy-free visual experience. Eye tracking methods can also be used on a variety of platforms to enhance the visual experience by dynamically refocusing optics, such as providing 3D gaze point images.

According to one embodiment of the innovative object of the present invention, the method includes: extracting an electrical signal generated by a photodetector from a modulated optical signal reflected from the eye, the electrical signal including a first phase, wherein the modulated optical signal is determined by Generated by one or more light sources biased by a modulation signal, the modulation signal includes a second phase; the eye is judged according to a phase difference between a first phase of an electrical signal generated by the photodetector and a third phase of a reference signal A depth map of the ; the aforementioned method further includes: judging the line of sight information corresponding to the sight line of the eye according to the depth map, and outputting output data corresponding to the line of sight information.

In other embodiments, corresponding systems, devices, and computer programs for performing the actions of the methods encoded on computer storage devices are included.

One or more of the following features may optionally be included in one embodiment of the present invention. For example, the foregoing methods may include providing one or more filters to remove non-target wavelength signals in the modulated optical signal reflected by the eye. Additionally, the aforementioned method can include providing one or more lenses to focus the modulated optical signal reflected by the eye on the photodetector. A depth map can contain one or more data sets of three-dimensional (3D) information. The sight line information can include one or more of an identification of a specific region of the eye, identification of a pupil of the eye, identification of an iris of the eye, and identification of a physiological structure of the eye. In certain aspects, providing output data for line-of-sight information includes providing output data corresponding to line-of-sight information as input to other devices, machines, or systems. The aforementioned method can include determining an eye pose based on the gaze information, and providing output data representing the eye pose. In this case, the eye posture can include movement of the eye, rotation of the eye, steady state of the eye, time corresponding to the steady state of the eye, closed state of the eye, time corresponding to the closed state of the eye, open state of the eye, One or more of the time corresponding to the open state of the eye, the blinking state of the eye, the time corresponding to the blinking state of the eye, and the frequency corresponding to the blinking state. Additionally, providing output data representing eye pose can include providing output data corresponding to gaze information as input to other devices, machines, or systems.

In certain aspects, the modulated optical signal reflected from the eye is generated by one or more light sources to which a modulated signal is applied, the modulated signal being synchronized with a reference signal. The aforementioned method can include generating a vertical iris vector in a plane tangent to the eye, and judging the sight line information representing the sight line of the eye according to the depth map and the iris vector. The aforementioned method can further include generating a pupil position in a plane tangent to the eye, and judging the sight line information representing the sight line of the eye according to the depth map and the pupil position.

In another embodiment according to the innovative object of the present invention, an eye tracking system includes: a machine having a display including a plurality of tunable optical elements. The system can also include: a device, the device including a circuit for capturing an electrical signal corresponding to a modulated light signal reflected by an eye measured by a photodetector, the electrical signal including a first phase, wherein The modulated optical signal is generated by one or more light sources biased by a modulation signal that includes a second phase. The aforementioned circuit can be further used to determine a depth map of the eye according to a phase difference between a third phase of a reference signal and a first phase of an electrical signal generated by the photodetector, and to determine a line of sight representing a line of sight of the eye according to the depth map News. In addition, the aforementioned system includes one or more processors capable of communicating with machines or devices, and the aforementioned one or more processors include one or more storage devices for storing instructions for operations; when executing the aforementioned instructions, The one or more processors are caused to perform operations including receiving output data representing line of sight information from the device and determining between the display of the machine and the line of sight information representing the line of sight of the eye.

In certain aspects, the aforementioned operations can further include determining a particular location on the display where the eye is focused; the aforementioned particular location is based on a relationship between the display and line-of-sight information representing the eye's line of sight, and providing an indication of the particular location on the display . The aforementioned operations can include determining a specific position on the display where the eye is focused; the aforementioned specific position is based on the relationship between the display and line-of-sight information representing the eye's line of sight, and providing a foveated image of the specific region on the display. The aforementioned plurality of tunable optical elements can include tunable lenses or tunable mirrors. In this case, enabling adjustment of a subset of the aforementioned plurality of tunable optical elements may depend on the relationship between the display and line-of-sight information representing the line of sight of the eye. Additionally, adjustment of a subset of the plurality of tunable elements can include dynamically refocusing light incident thereon.

The aforementioned system can include a wearable device coupled to the aforementioned machine, device, and one or more processors to form an integrated hardware package. Mechanical displays are opaque and visual images are displayed on the display through an array of one or more light sources. In some aspects, the aforementioned system can include a wearable device coupled to the mechanism and the device to form a one-piece hardware package, the display of the mechanism is opaque, and the visual image is displayed on an array of one or more light sources. on the display; one or more processors are located remotely and communicate with the integrated hardware package through a wireless or wired connection. In other aspects, the aforementioned system can include a wearable device coupled to the machine, the device, and one or more processors to form a one-piece hardware package; the machine's display is at least partially transparent to an image projected toward it, The image characteristics projected toward the display are adjusted by one or more adjustable optical elements on the display.

Additionally, the aforementioned system can include a wearable device coupled to the mechanism and the device to form a one-piece hardware package; the mechanism's display is at least partially transparent to the image projected toward it, through one or more Adjustable optics to adjust the characteristics of the image projected to the display; one or more mechanical processors are located remotely and communicate with an integral form of hardware package via a wireless or wired connection. The aforementioned system can also include a pluggable device coupled to the device and one or more processors to form an integral form of hardware package, mechanically located remotely, and connected to the integral form of hardware through a wireless or wired connection. The body package communicates; the mechanical display is opaque, and the visual image displayed on the display is realized through one or more light source arrays.

In some aspects, the aforementioned system can include a wearable device coupled to the device and one or more processors to form a hardware package in a unitary form; the mechanism is remotely located and integrated with the device through a wireless or wired connection The mechanical display is opaque, and the visual image displayed on the display is realized through an array of one or more light sources. In this case, the aforementioned operations can further include determining that the eye is focused on a specific position on the display; the aforementioned specific position is based on the relationship between the display and the line-of-sight information representing the line of sight of the eye, and providing an indication of the specific position on the display . In some aspects, the optical signal reflected by the eye is generated by one or more light sources to which a modulating signal is applied; the aforementioned modulating signal is synchronized with the reflected signal.

In another embodiment according to the innovative object of the present invention, the eye movement tracking device includes a plurality of adjustable optical elements to adjust the focal length. The wearable device can also include one or more processors, and the processor includes one or more storage devices to store the operation instructions; when the processor executes the aforementioned instructions, the one or more processors will be included to retrieve to a The photodetector measures an electrical signal corresponding to a modulated light signal reflected by the eye, and determines the operation of a depth map of the eye according to the phase difference between the electrical signal generated by the photodetector and a reference signal. The aforementioned operations can further include determining line of sight information representing a line of sight of an eye according to the depth map, the line of sight of the eye representing the relationship between the line of sight of the eye and a display of a remote device; and initiating adjustment of a subset of adjustable elements according to the line of sight information.

One or more of the following features may optionally be included in one embodiment of the present invention. The eye tracking method of the present invention can be used to provide cross-platform peripheral control. Cross-platform peripheral control can be used to exchange information between multiple devices. The aforementioned exchange information can include eye movement information, the position of an eye, and the like. Compared with the traditional eye tracking mechanism, the cross-platform peripheral control can be used to extend the operation area; thereby, due to the limited detection range of the traditional eye tracking mechanism to detect a single device, the eye tracking method of the present invention can provide Wider operating range than traditional eye tracking mechanisms. Furthermore, multiple users can simultaneously use the cross-platform peripheral to control multiple devices, which can effectively create user-to-user interaction.

Furthermore, the eye tracking method of the present invention can be used to provide a dizzy-free visual experience. In certain aspects, eye tracking information can be used in an optical system that refocuses tunable optics based on eye tracking information and known distance information. Adjustable optics adjust the angle of incident light to provide instant focus. The present invention can reduce the feeling of dizziness by making the depth perception between the user's eyes and the brain consistent with real-time focusing according to the eye tracking method. Furthermore, eye tracking information can be used to control a subset of adjustable optical elements to create a variable focus in which the focal lengths of various regions in the image presented to the viewer can be controlled to vary. The variable focus of the present invention provides a natural 3D effect through simple adjustable optics, which is different from the traditional artificial 3D effect provided through complex calculation algorithms.

The accompanying drawings and the description below set forth the details of one or more implementations of the invention. Other features and advantages of the present invention will become apparent from the description, drawings and claims.

Eye tracking methods can be used to determine gaze information about an eye tracking. The foregoing method includes illuminating the eye, and detecting the light signal reflected by the eye to track a gaze direction and focus of the eye. Determining the direction of the eye's gaze and the focus of the eye can be used to communicate with another device. For example, the gaze information of the eye can be used to provide one or more commands to another device. In some embodiments, line-of-sight information and/or other information such as gestures can be detected using the system of the present invention embedded in a mobile phone; the aforementioned mobile phone can act as a remote control for receiving user commands and connect to other devices, such as tablet computers, televisions, etc., to execute the aforementioned instructions.

In some implementations, the gaze information may include eye gestures. In some embodiments, the gaze information includes the pose of the eye. Eye gestures, such as eye movement, eye rotation, eye steady state, etc., can be used to indicate certain instructions to be provided to other devices. In some embodiments, the gaze information of the eye can be used to determine the position of the eye focus, eg, the eye is focused on a particular display. In this case, the position where the eye is focused on the display can be used to gather information of interest to the user. For example, if an advertisement is provided on the display, the user's eyes focus on the position of the advertisement displayed on the display to determine whether the user is interested in the advertisement. The location of the eye's line of sight and the length of time the eye maintains a particular line of sight can help determine the level of interest a user is interested in being provided in a particular display.

In some embodiments of the present invention, the eye tracking method can be integrated into a wearable device and/or a peripheral device. For example, a wearable device can provide eye illumination and detect light reflected from the eye. Wearable devices can include components such as an accelerometer, a gyroscope, or both to assist in tracking the eyes and the focus of the eyes on a particular display, thereby tracking the eyes more efficiently and accurately. In some embodiments, the wearable device may further comprise a plurality of tunable optical elements to adjust the light path. The aforementioned tunable optical elements include mirrors and/or lenses, and the mirrors and/or lenses are adjusted according to the movement or stillness of the eye being tracked. Tunable optics can be used to dynamically focus or defocus in real time to assist the eye to focus on a specific object or display. For example, tunable optics can solve the problem of inconsistent focus and vergence when viewing a virtual reality or augmented reality image. In some embodiments, the components of the wearable device can be implemented by an external distal device separate from the wearable device. Eye-tracking methods can provide as output, particularly eye gaze data, and use this output as commands to remote devices and/or tunable optics to assist in various visual experiences.

FIG. 1A is a schematic diagram of an eye movement tracking system 100 . The eye tracking system 100 can be used to process a user's eye information and generate a depth map corresponding to the eye. The eye tracking system 100 includes an eye tracking device 110 to track the movement of a user's eyes 120 , an image display 130 and a signal processing unit 140 to process eye data detected by the eye tracking system 100 . The eye tracking system 100 can optionally include a console 170 for the user to input according to practical applications. The user's eyes 120 may include one or both eyes for viewing the image display 130 .

The image display 130 can be one or more image displays on a computer, notebook computer, desktop computer, television, smartphone, tablet computer, or the like. The image display 130 includes a liquid crystal display, a light-emitting diode display, an organic light-emitting diode display, a head-mounted display, and the like. In some embodiments, the image display 130 can include adjustable optical elements, such as a mirror and/or an adjustable lens. In this case, the adjustable optics of the image display 130 can be used to instantly focus and defocus to assist the user's eyes 120 in viewing the image display 130 .

The eye tracking device 110 includes one or more signal processing units 140 in communication with the eye tracking device. The eye tracking device 110 can provide illumination for the user's eyes 120 and receive light signals reflected by the user's eyes 120 . The eye tracking device 110 can include a tunable light source that can illuminate the user's eye 120 at one or more specific wavelengths. The tunable light source may be able to comprise a single optical transmitter or multiple optical transmitters, which (etc.) may be modulated by a voltage source at a radio frequency or a microwave frequency to provide illumination. In some embodiments, the light emitter can be used to illuminate the entire user's eye 120 . In other embodiments, light emitters can be used to illuminate specific portions of a user's eye 120 . The aforementioned wavelength or wavelengths used in the eye tracking system 100 can be predetermined according to various criteria; eg, non-availability to the human eye, low solar spectral irradiance at sea level, eye safety, and the like.

In some embodiments, the eye-tracking device 110 includes one or more light detectors to receive the light signal reflected by the user's eye 120 ; the light signal reflected by the user's eye 120 is the adjustable light signal provided by the eye-tracking device 110 . In some embodiments, the eye tracking device 110 can detect the reflected tunable light signal with one or more light detectors. The aforementioned photodetector may be disclosed in US Patent Application No. 15/338,660 filed on October 31 and entitled "High-Speed Light Sensing Apparatus" and US Patent Application No. 15/228,282, filed in 2016 U.S. Patent Application No. 15/228,282 filed on August 4 with the patent name of "GERMANIUM-SILICON LIGHT SENSING APPARATUS" pending.

The signal processing unit 140 includes one or more signal processing units, which (etc.) are capable of communicating with the image display 130 and the eye tracking device 110 . The signal processing unit 140 can determine the sight line information 150 of the user's eyes 120 through the data corresponding to the eye tracking device 110 and the image display 130 . The eye tracking device 110 can be used to demodulate the reflected modulated optical signal. Additionally, the eye tracking device 110 can be used to create a depth map of the illuminated portion of the user's eye 120 . The aforementioned depth map corresponds to the reflected optical signal measured by the detector of the eye tracking device 110 . Specifically, the depth map may provide two-dimensional (2D) or three-dimensional (3D) information about the user's eye 120 . The signal processing unit 140 can process the depth map according to the data of the time difference ranging information of the reflected optical signal. In some embodiments, the depth map is generated from the phase difference between the reflected optical signal and the reference signal. For example, the eye tracking device 110 can provide a comparison between the reflected light signal and a reference signal, and use it to determine the depth map of the user's eye 120 . The depth map can further include a 3D model representing the user's eye 120 ; thereby, a 3D eye model can be generated and constructed, thereby allowing the signal processing unit 140 to determine the sight line information 150 of the user's eye 120 .

The signal processing unit 140 can be disposed close to the user's eyes 120 . For example, the signal processing unit 140 and the eye tracking device 110 may be disposed in a single wearable device adjacent to the user's eye 120 . The signal processing unit 140 and the eye tracking device 110 can also be disposed in a single peripheral device away from the user's eyes 120 . In other embodiments, the signal processing unit 140 can be provided separately from the eye tracking device 110 . For example, the signal processing unit 140 may be located in the image display 130 and communicate with the eye tracking device 110 located in a single wearable device or peripheral device.

The gaze information 150 can include, for example, information such as the user's gaze direction and focus. The signal processing unit 140 can determine the optical signal received by the eye tracking device 110 in the line of sight information 150 . The gaze information 150 can be used to analyze the behavior of the user's eyes. In addition, the sight line information 150 can be used to identify a focused position of the user's eyes 120 on the image display 130 . In this case, the line of sight information 150 can be used to determine which item the user's eyes 120 focus on on the image display 130 . Thereby, it is possible to judge what the user is interested in without actual driving of a special device. For example, the advertisement provider can use the user's eyes 120 to determine what the user is interested in without using a mouse, a trackpad, a touch screen, or the like. In other cases, the actual actuation of a special device may be used to perform some function of user or system interaction. Utilizing such a device can improve performance during complex interactions between the system and the user. For example, a pilot of a fighter jet can use the sight line information 150 of the eyes to determine/select a target of interest on the image display 130, and use the console 170 to perform tasks on the target, such as: target acquisition, target priority assignment, Weapon selection, etc.

In some embodiments, the line-of-sight information 150 can serve as instructions provided to other devices. In this case, the gaze information 150 may include eye tracking, such as: eye movement, eye movement, eye closed position, eye open status or eye movement over a period of time, eye movement, eye closed position, eye open open state, etc. The device receiving the gaze information 150 can analyze the gaze information 150 in real time to determine a command when the eye tracking device 110 dynamically tracks the user's eyes 120 .

The eye tracking device 110 , the image display 130 and the signal processing unit 140 may be independent structures or coupled in a one-piece hardware package. For example, the eye-tracking device 110, the image display 130, and the signal processing unit 140 may be integrated into a single hardware package; the aforementioned display in the image display 130 is opaque and the visual image is transmitted for the generation of visible light An array of light-emitting diodes, liquid crystals that filter out white light, or an array of other light sources is displayed on the display. In some embodiments, the display of image display 130 is at least partially transparent and the visual image can be projected onto the display by optical refraction, diffraction, reflection, directing, or other optical means.

In other examples, the eye tracking device 110 and the signal processing unit 140 can be integrated into a single hardware package, such as a wearable device. A wearable device can be a head-mounted device, a pair of glasses, or other suitable wearable devices. In this case, the wearable device may be mechanically connected with the main stand for embedding the image display 130 . Additionally, the main stand or mechanism for housing the image display 130 can utilize wired or wireless connections to communicate with the wearable device.

In another example, the eye tracking device 110 and the signal processing unit 140 can be integrated into a hardware package, such as a pluggable device. A pluggable device can be a game box, a camcorder, or other pluggable devices. In this case, the pluggable device may be mechanically connected to the main bracket for embedding the image display 130 . Additionally, the stand or mechanism for housing the image display 130 can utilize a wireless or wired connection to communicate with the pluggable device.

FIG. 1B is a schematic diagram of a time difference ranging device. The time difference ranging device can be integrated into the eye tracking device 110 and used to determine the depth map of the user's eyes 120 . The time difference ranging device shown in FIG. 1B includes a time difference ranging pixel (hereinafter referred to as a TOF pixel) 160 and two sets of transistors. As shown in FIG. 1B , each set of transistors includes three switch transistors, that is, a reset transistor 162a or 162b, a source-follower transistor 164a or 164b, and a selection transistor 166a or 166b. In other embodiments, other forms of transistors may also be used to achieve similar functions. TOF pixel 160 is one or more TOF pixels used to detect light. When a TOF pixel detects light, it determines that the charge should be handled by either the first set of transistors or the second set of transistors. In some aspects, a received light signal can be out of phase with the incident light; in this case, the TOF pixel can be designed as a dual-switch TOF pixel, so that one of the switches is adjusted to have the same phase as the incident light signal Phase, the other switch is adjusted to be 180 degrees out of phase with the incident optical signal to accommodate the received inverted optical signal. The aforementioned dual-switch TOF pixel can be disclosed in US Patent Application No. 15/338,660 filed on Oct. 31 and entitled "High-Speed Light Sensing Apparatus", and U.S. Patent Application filed on Aug. 4, 2016 Application No. 15/228,282, patent application under the patent name "GERMANIUM-SILICON LIGHT SENSING APPARATUS".

In some aspects, the aforementioned two sets of transistors can be assembled with TOF pixels on a single wafer; in this case, the aforementioned two sets of transistors can share and occupy the same light-emitting area as the TOF pixel 160, so as to Active fill factor to reduce TOF. The aforementioned two sets of transistors can be implemented by a plurality of NMOS gates; the NMOS gates are used to reduce the size of the transistors and the TOF device. The aforementioned two sets of transistors can also be implemented with a complex number of PMOS gates; the PMOS gates are used to increase certain operating coefficients, such as to provide a higher available voltage headroom. The manner in which the PMOS and NMOS realize the aforementioned transistors will be described in detail later.

FIG. 1C is a schematic diagram of a time difference ranging device. The TOF device shown in FIG. 1C includes a first wafer and a second wafer, and the first wafer and the second wafer are bonded to each other through die or wafer bonding 167 . The first wafer includes a TOF pixel 165 disposed on the first wafer, and the TOF pixel 165 can be used to detect light pulse information. The second wafer may be a circuit wafer 169 including two sets of transistors, and the circuit wafer 169 performs charge processing when the TOF pixel 165 detects a light pulse. In some embodiments, the transistors of the circuit wafer 169 do not occupy the light emitting area, thereby increasing the effective fill factor of the TOF.

The aforementioned two sets of transistors can be implemented with NMOS gates or PMOS gates. For example, each set of transistors can be implemented with NMOS gates having a threshold voltage of 0.7 volts; in this case, a maximum source of about 2.6 volts is obtained when a voltage of 3.3 volts is applied to the gates and the NMOS gate is turned on pole voltage. Therefore, when the NMOS is used as a reset transistor, the reset voltage applied to the TOF pixel is limited to a maximum of 2.6 volts for low voltage margin. Conversely, in another example, each group of transistors can be implemented with a PMOS gate with a threshold voltage of -0.8 volts; in this case, when the gate is applied with a voltage of 0 volts, the PMOS gate turns on, resulting in Maximum source voltage of about 3.3 volts. Therefore, when the PMOS is used as a reset transistor, the reset voltage applied to the TOF pixel is limited to a maximum of 3.3 volts to provide a high voltage margin.

Therefore, when implemented in PMOS gates, the two sets of transistors can provide a high available voltage margin, which is partly due to the negative threshold voltage specification. Furthermore, implementing with PMOS can provide a lower impedance when it is turned on like a switch, allowing a voltage passing through it to be close to a supply circuit; in this way, the two sets of transistors implemented with PMOS provide the operational advantages of TOF devices . However, the actual area of a PMOS gate can be larger than that of an NMOS gate, and a PMOS implementation requires a substantially large TOF device to provide such an implementation. Such a problem can be solved as shown in FIG. 1C, when TOF pixels and PMOS circuits are implemented on two different wafers, as a wafer or die is bonded to be electrically connected to two separate wafers or grains. In some embodiments, the TOF pixel as shown in FIGS. 1B and 1C may include a light absorbing layer containing germanium. In some embodiments, the TOF pixel may further include a demodulation function realized by dual switching transistors or a demodulation function realized by multiple PN junctions as shown in FIG. 1B and FIG. 1C . The aforementioned dual-switch TOF pixel can be disclosed in US Patent Application No. 15/338,660 filed on Oct. 31 with the patent title of "High-Speed Light Sensing Apparatus", and U.S. Patent filed on Aug. 4, 2016 Application No. 15/228,282, patent application under the patent name "GERMANIUM-SILICON LIGHT SENSING APPARATUS".

的反射光脈衝。一光二極體陣列經控制以讓一第一讀出電路讀取經收集的電荷Q1，並使一第二讀出電路讀取經收集電荷Q2；其中，電荷Q1與光脈衝具有相同相位(同步)，電荷Q2與光脈衝具有相反相位。在一些實施方式中，在眼動追蹤裝置110及使用者眼睛120的一點之間的距離D滿足下式︰

；其中c為光速。眼動追蹤裝置110可掃描使用者眼睛120以獲得使用者眼睛120的深度圖。 FIG. 1D shows a schematic diagram of a technique for determining the characteristics of a user's eye 120 . The eye tracking device 110 has a plurality of light pulses, wherein the light pulses are modulated to have a frequency fm and a duty cycle of 50%. The eye tracking device 110 may receive a phase difference

of reflected light pulses. A photodiode array is controlled to cause a first readout circuit to read the collected charge Q1 and a second readout circuit to read the collected charge Q2; wherein the charge Q1 and the light pulse have the same phase (synchronized ), the charge Q2 has the opposite phase to the light pulse. In some embodiments, the distance D between the eye tracking device 110 and a point on the user's eye 120 satisfies the following equation:

; where c is the speed of light. The eye tracking device 110 may scan the user's eye 120 to obtain a depth map of the user's eye 120 .

。光電二極體經控制使第一讀出電路讀取經收集的電荷Q1’，且一第二讀出電路讀取經收集的電荷Q2’；其中，電荷Q1’的相位相同於光脈衝的相位，電荷Q2’的相位不同於光脈衝的相位。在一些實施方式中，在眼動追蹤裝置110及使用者眼睛120之一點之間的距離D滿足下式︰

FIG. 1E is a schematic diagram of another technique for determining the characteristics of the user's eye 120 . The eye tracking device 110 may emit light pulses; wherein the light pulses are modulated to have a frequency fm and a duty cycle of 50%. By reducing the duty cycle of the optical pulse by a factor N, and increasing the intensity of the optical pulse by the factor N, the received reflected optical pulse can be obtained while maintaining substantially the same power consumption of the eye-tracking device 110. signal-to-noise ratio enhancement. This allows the bandwidth of the device to be increased, thereby reducing the duty cycle of the optical pulse without distorting the pulse shape. The eye tracking device 110 can receive reflected light pulses; the reflected light pulses have a phase shift

. The photodiodes are controlled so that a first readout circuit reads the collected charge Q1', and a second readout circuit reads the collected charge Q2'; wherein the phase of the charge Q1' is the same as the phase of the light pulse , the phase of the charge Q2' is different from that of the light pulse. In some embodiments, the distance D between the eye tracking device 110 and a point on the user's eye 120 satisfies the following equation:

。在一些實施方式中，相位差

依使用者眼睛120及眼動追蹤裝置110之間的距離而發生。一個小的相位差能夠讓眼動追蹤裝置110難以有效地偵測一使用者眼睛120的姿勢辨識、定位等。因此，增加一相位偏移φ於收集電荷中有利於眼動辨識更有效地執行。圖1G繪示光檢測及電荷收集之示意圖。光檢測及電荷收集包含如下時間步驟：發光、檢測光及眼動讀取裝置110收集電荷。在任一時間步驟中，收集的資料表示接收光，在0度相位收集的電荷、在90度相位收集的電荷、在180度收集的電荷及在270度收集的電荷。在任一角度收集的電荷能夠指示在各角度收集的電荷的數量。在此狀態下，在任一時間步驟收集的電荷的數量能夠影響眼動追蹤裝置110定位使用者眼睛120。 FIG. 1F shows a schematic diagram of the phase of charge collection. The phase of charge collection represents the phase of the light pulses emitted by the eye tracking device 110 and the collected charge. The charge collection includes a 0-degree phase, a 90-degree phase, a 180-degree phase, and a 270-degree phase, and a control phase offset φ. A phase difference can be observed in the light pulses emitted by the eye-tracking device 110 and the light pulses it receives

. In some embodiments, the phase difference

Occurs depending on the distance between the user's eye 120 and the eye tracking device 110 . A small phase difference can make it difficult for the eye tracking device 110 to effectively detect gesture recognition, positioning, etc. of a user's eyes 120 . Therefore, adding a phase offset φ to the charge collection facilitates the eye movement recognition to be performed more efficiently. FIG. 1G shows a schematic diagram of light detection and charge collection. Light detection and charge collection includes the time steps of emitting light, detecting light, and eye movement reading device 110 collecting charge. At any time step, the data collected represents received light, charge collected at 0 degrees phase, charge collected at 90 degrees phase, charge collected at 180 degrees, and charge collected at 270 degrees. The charge collected at any angle can indicate the amount of charge collected at each angle. In this state, the amount of charge collected at any one time step can affect the positioning of the eye tracking device 110 on the user's eye 120 .

的反射光脈衝。TOF像素經控制以讓眼動追蹤裝置110的第一讀出電路讀取收集電荷Q0，其相位與發射光脈衝的相位相同，即相當於0度相位。眼動追蹤裝置110還能夠包含一第二讀出電路以讀取收集電荷Q180，其相位相反於發射光的相位，例如為180度相位。在另一時間步驟中，TOF像素經控制以使第一讀出電路讀取收集電荷Q90，其相位與發射光相位具90度相位差，例如90度相位。在這個情況下，第二讀出電路能夠讀取收集電荷Q270，其相位與發射光相位具另一90度相位差，例如270度相位。在一些實施方式中，眼動追蹤裝置110與使用者眼睛120的距離可以如下二式表示之︰

，或

。 For example, the eye-tracking device 110 may emit light pulses with a duty cycle of 50% and a modulated frequency _fm . The eye tracking device 110 can receive a phase difference

of reflected light pulses. The TOF pixels are controlled so that the first readout circuit of the eye-tracking device 110 reads the collected charge Q0 in the same phase as the emitted light pulse, ie, equivalent to 0 degree phase. The eye-tracking device 110 can also include a second readout circuit to read the collected charge Q180 whose phase is opposite to the phase of the emitted light, eg, 180 degree phase. In another time step, the TOF pixel is controlled so that the first readout circuit reads the collected charge Q90 with a phase that is 90 degrees out of phase with the emitted light, eg, 90 degrees out of phase. In this case, the second readout circuit is able to read the collected charge Q270 whose phase is another 90 degrees out of phase with the emitted light phase, eg, 270 degrees out of phase. In some embodiments, the distance between the eye tracking device 110 and the user's eyes 120 can be expressed by the following two equations:

,or

.

的條件下，在0度相位收集到的電荷是在整個時間步驟中最大的，且在180度相位收集到的電荷是在整個時間步驟中最小的。如此大的電荷收集差別影響整體電荷收集的準確性。因此，引入相位差Φ可降低在不同相位電荷收集的差異，有助於眼睛姿勢偵測，能夠加強使用者眼睛120的深度圖的準確性。 Referring back to FIG. 1G ; the light pulses emitted by the eye tracking device 110 have a small phase difference with the reflected light pulses received by the eye-tracking device 110

Under the condition of , the charge collected at the 0 degree phase is the largest in the whole time step, and the charge collected at the 180 degree phase is the smallest in the whole time step. Such a large difference in charge collection affects the accuracy of the overall charge collection. Therefore, the introduction of the phase difference Φ can reduce the difference of charge collection in different phases, which is helpful for eye posture detection, and can enhance the accuracy of the depth map of the user's eye 120 .

FIG. 1H shows a schematic diagram of a signal voltage during charge collection. The signal voltage at the time of charge collection shows the variation of the signal voltage at different time points for a plurality of phases. Specifically, FIG. 1H shows the changes of the signal voltage at 0-degree phase, 90-degree phase, 180-degree phase, and 270-degree phase. In each phase, the signal voltage decreases with time, which means that the amount of charge is stored in a particular phase during an interval. As shown in FIG. 1H , the signal voltage of the 180-degree phase is much higher than the signal voltage of the 0-degree phase, so the 180-degree phase has a lower charge storage rate than the 0-degree phase. In this case, the difference in charge storage rates at different phases negatively affects the accuracy of the eye tracking device 110 in detecting the user's eyes 120 . Therefore, introducing a phase shift [phi] into the received light signal facilitates charge collection, which in turn makes the depth map of the user's eye 120 more accurate.

。在一些實施方式中，相位差因使用者眼睛120及眼動追蹤裝置110之間的距離而生。一個小的相位差能夠讓眼動追蹤裝置110有效地檢測使用者眼睛120的一姿勢辨識、定位使用者眼睛120等。45度的一相位偏移φ繪示於圖1I中以收集電荷，故所有的相位可偏移45度的相位偏移φ。 FIG. 1I shows a schematic diagram of shift phase of charge collection. The offsets for charge collection include a 45-degree phase, a 135-degree phase, a 225-degree phase, and a 315-degree phase. A phase shift can be observed in the light pulses emitted and received by the eye-tracking device 110

. In some implementations, the phase difference is due to the distance between the user's eyes 120 and the eye tracking device 110 . A small phase difference enables the eye tracking device 110 to effectively detect a gesture recognition of the user's eyes 120, locate the user's eyes 120, and the like. A phase shift φ of 45 degrees is shown in FIG. 1I to collect charges, so all phases can be shifted by a phase shift φ of 45 degrees.

FIG. 1J shows a schematic diagram of charge collection for photodetection and phase shifting. Charge collection for light detection and phase shifting includes light emission, light detection, and charge collection at the eye tracking device 110 . At each time step, the data collected represents received light, charge collection at 45 degrees phase, charge collection at 135 degrees phase, charge collection at 225 degrees, and charge collection at 315 degrees. Charge collection in any phase can indicate the amount of charge collection in any receive phase. In this case, the amount of charge collection in any phase at any one time step can affect the accuracy of the eye tracking device 110 in locating the user's eye 120 .

的反射光脈衝。TOF像素經控制以讓眼動追蹤裝置110的第一讀出電路讀取收集電荷Q45，其相位與發射光脈衝的相位具有例如45度相位的偏移。眼動追蹤裝置110還能夠包含一第二讀出電路以讀取收集電荷Q225，其相位與發射光的相位具有例如225度相位的偏移。在另一時間步驟中，TOF像素經控制以使第一讀出電路讀取收集電荷Q135，其相位與發射光的相位具有135度相位偏移。在這個情況下，第二讀出電路能夠讀取收集電荷Q315，其相位與發射光的相位具有315度相位偏移。在某些實施方式中，眼動追蹤裝置110與使用者眼睛120的距離可以如下二式表示之︰

For example, the eye-tracking device 110 may emit light pulses with a duty cycle of 50% and a modulated frequency _fm . The eye tracking device 110 can receive a phase difference

of reflected light pulses. The TOF pixels are controlled to cause the first readout circuit of the eye-tracking device 110 to read the collected charge Q45 with a phase offset of, eg, 45 degrees from the phase of the emitted light pulse. The eye-tracking device 110 can also include a second readout circuit to read the collected charge Q225 whose phase is offset from the phase of the emitted light, eg, by 225 degrees. In another time step, the TOF pixel is controlled so that the first readout circuit reads the collected charge Q135 with a phase offset of 135 degrees from the phase of the emitted light. In this case, the second readout circuit is able to read the collected charge Q315 with a phase shift of 315 degrees from the phase of the emitted light. In some embodiments, the distance between the eye tracking device 110 and the user's eyes 120 can be expressed by the following two equations:

的條件下，在45度相位和225度相位收集到的電荷是在整個時間步驟中相近的。相比於圖1G所示不具有相位偏移φ且在0度相位及180度相位收集到的電荷是相當不同的；圖1J式出在不同相位的低差異電荷收集，提供良好的眼睛定位表現。相位收集的差異能夠影響電荷收集的整體準確性，相位差可降低電荷收集在不同相位的差異，有助於眼動偵測並能夠讓使用者眼睛120的深度圖更加準確。 Referring back to FIG. 1J ; the light pulses transmitted and received at the eye tracking device 110 have a small phase difference

Under the conditions of , the charges collected at the 45-degree phase and the 225-degree phase are similar throughout the time step. Compared to that shown in Figure 1G with no phase shift φ and the collected charges at 0 degree phase and 180 degree phase are quite different; Figure 1J shows the low difference charge collection at different phases, providing good eye positioning performance . The difference in phase collection can affect the overall accuracy of charge collection, and the phase difference can reduce the difference in charge collection in different phases, which is helpful for eye movement detection and can make the depth map of the user's eye 120 more accurate.

FIG. 1K shows a schematic diagram of the signal voltage during phase-shifted charge collection. The signal voltage at the time of phase shift charge collection shows the variation of the signal voltage at different times for a plurality of phases. Specifically, FIG. 1H shows the change of the signal voltage at a phase shift of 45 degrees, a phase shift of 135 degrees, a phase shift of 225 degrees, and a phase shift of 315 degrees. In each phase, the decrease in the signal voltage with time indicates that the amount of charge is stored in a particular phase for an interval of time. The phase-shifted signal voltages shown in FIG. 1K contain similar average rates of signal voltages compared to the signal voltages of different average rates shown in FIG. 1H . The similar rate of drop in the phase-shifted signal voltage can then allow for more accurate eye movement detection and positioning of the user's eye 120 . In summary, introducing a phase offset φ into the charge collection can help the charge collection so that the depth map of the user's eye 120 can be more accurate.

FIG. 1L shows a schematic diagram of a TOF device. The TOF device includes a TOF pixel 190 , two capacitors 192 a and 192 b , and two transistor sets 194 and 196 . Each group of transistors contains five switching transistors (5T). In some embodiments, other arrangements of transistors may be used to achieve similar functions. TOF pixel 190 may be one or more TOF pixels used to detect light. The charge generated by the TOF pixel can be controlled by capacitors 192a and 192b. The transistors M1~M4, which can be implemented by NMOS, PMOS, or a combination of NMOS and PMOS, redistribute the collected charge by resetting the common-mode charge and connecting the common-mode voltage to VREF. The voltage VREF may be an operating voltage of the TOF pixel 190 or a predetermined voltage generated by a design requirement. Transistors M5 and M6, which can be implemented by NMOS, PMOS, or a combination of NMOS and PMOS, are used to reset the collected charge and connect it, etc., to voltage VREF2. The voltage VREF2 may be the same as the voltage VREF, which is the operating voltage of the TOF pixel 190 or a predetermined voltage generated by a design requirement.

FIG. 2A shows a schematic diagram of a cross-platform peripheral control system using eye tracking. The cross-platform peripheral control system using eye tracking may include a wearable device and a connected device, such as the wearable device 201, and the connected device such as a phone 220, a tablet computer 230, a computer device 240 and/or Or a television 250 ; the connecting device can communicate with the headset 201 . The headset 201 may be used when the user's eyes 216A, 216B are looking at the phone 220 , the tablet 230 , the computer device 240 and/or the television 250 . The head-mounted device 201 can include an eye movement tracking module 213 for implementing an eye movement tracking device and a signal processing unit, an accelerometer 211, a gyroscope 212, a wireless transmission unit 214 and a transparent lens 218, eye tracking The module 213 is used to track the first and second eyes 216A and 216B of the user, the accelerometer 211 and the gyroscope 212 are used to determine the position of a head of the user; the wireless transmission unit 214 is provided for a connection device, such as a telephone 220 and/or tablet computer 230 and/or computer device 240 and/or television 250 for communication. In some embodiments, the transparent lens 218 may include one or more adjustable elements for adjustment to the user's eyes 216A and/or 216B.

Here, the eye tracking module 213 can illuminate the user's eye 216A with an optical signal, and detect the optical signal reflected by the user's eye 216A. The detected light signal can be used to determine line-of-sight information about the user's eye 216A. The gaze information can include the user's gaze on the display of the connected device. The gaze information also includes instructions regarding the posture of the user's eyes 216A. Eye gesture commands can be used as input commands to the connected device. In some embodiments, the eye tracking module 213 can illuminate the user's eyes (ie, eyes 216A and 216B) with optical signals, and detect the optical signals reflected by the user's eyes 216A and 216B to determine the user's eyes 216A and 216B sight information.

The accelerometer 211 and the gyroscope 212 can be used to detect the positioning of the user's head. The positioning of the user's head can be used to effectively determine the sight line information. In addition, the accelerometer 211 and the gyroscope 212 can be used to track the movement of the user's head; thereby, according to the movement of the user's head, any possible head movement can be judged to avoid misinterpretation of the line-of-sight information.

The wireless transmission unit 214 establishes communication with the headset 201 , the phone 220 , the tablet computer 230 , the computer device 240 and/or the TV 250 through the network. The network includes Wi-Fi, bluetooth, low power bluetooth (BLUETOOTH, BLUTETOOTH LOW ENERGY; BLE for short), local area network, etc.

The transparent lens 218 can be used to assist the user's eyes 216A and 216B to see the display of the phone 220 , tablet 230 , computer device 240 and/or television 250 . The transparent lens 218 includes adjustable optical elements that can be adjusted according to the tracking information represented by the user's eyes 216A and 216B. In some embodiments, the entire transparent lens 218 can be adjusted according to the line-of-sight information. In other embodiments, the line-of-sight information is used to adjust specific portions of the transparent lens; for example, specific portions of the transparent lens 218 can be adjusted to provide gaze point images at specific locations on the display of the phone 220; wherein the specific portion of the phone 220 The location is on the display, where the user's eyes 216A and 216B are looking.

In some embodiments, the phone 220 includes an accelerometer 221 and a gyroscope 222 for determining the location of the phone 220 ; the phone 220 can also include a wireless communication unit 224 for communicating with the headset 201 . The accelerometer 221 and gyroscope 222 of the phone 220 can assist in tracking the location and movement of the phone 220 . By tracking the position and movement of the phone 220 , the headset 201 can effectively determine the gaze information of the user's eyes 216A and 216B by comparing with the user's focus position on the phone 220 . The position and movement of the phone 220 can be communicated to the headset 201 through the wireless communication unit 224 .

In some embodiments, the tablet computer 230 can include an accelerometer 231 and a gyroscope 232 for determining the positioning of the tablet computer 230 ; the tablet computer 230 further includes a wireless communication unit 234 for communicating with the head-mounted device 201 . The accelerometer 231 and the gyroscope 232 of the tablet computer 230 can assist in tracking and the position and movement of the tablet computer 230 . By tracking the position and movement of the tablet computer 230, the headset 201 can effectively determine a sight reference point 236 of the user's eye 216A. The position and movement of the tablet computer 230 can be communicated to the headset 201 via the wireless communication unit 234 .

The computer device 240 can include a wireless communication unit 244 to communicate with the headset 201 . In addition, the television 250 can include a wireless communication unit 254 to communicate with the head-mounted device 201 .

2B shows a schematic diagram of a cross-platform peripheral control system using eye tracking. The cross-platform peripheral control system for eye tracking may include a wearable device, a connecting device, an accelerometer 211 , a gyroscope 212 , a wireless transmission unit 214 , a first transparent lens 218A and a second transparent lens 218B. The wearable device is, for example, the head-mounted device 202 , and the connecting device is, for example, a phone 220 , a tablet computer 230 , a computer device 240 and/or a TV 250 ; the connecting device can communicate with the head-mounted device 201 . The headset 202 may be used when the user's eyes 216A, 216B are looking at the connection device. The headset 202 can include two eye-tracking modules and a pair of signal processing units; wherein the first pair is implemented by the first eye-tracking module 213A, and the second pair is implemented by the second eye-tracking module 213B , to track the user's eyes 216A and 216B. The accelerometer 211 and the gyroscope 212 are used to determine the position of the user's head, the wireless transmission unit 214 is used for communication with the connecting device, the first transparent lens 218A includes one or more adjustable elements, and the second transparent lens 218B includes one or more adjustable element.

The first eye tracking module 213A can be used to illuminate the first user's eye 216A and detect the light signal reflected by the first user's eye 216A. The light signal detected by the first eye tracking module 213A can determine the sight line information related to the first user's eye 216A. The line of sight information includes the line of sight of the first user's eye 216A on the displays of the connected devices (eg, 220, 230, 240, and 250). The gaze information can also contain instructions regarding the posture of the first user's eye 216A; the eye posture instructions can serve as input instructions to the connecting device.

The second eye tracking module 213B can be used to illuminate the second user's eye 216B and detect the light signal reflected by the second user's eye 216B. The light signal detected by the second eye tracking module 213B can determine the sight line information related to the second user's eye 216B. The sight line information includes the sight line of the second user's eye 216B on the display of the connected device. The gaze information can also contain instructions regarding the posture of the second user's eye 216B. Eye gesture commands can be used as input commands to the connected device.

The first transparent lens 218A can be used to assist the first user's eye 216A to view the display of the connected device. The first transparent lens 218A can include adjustable elements that can be adjusted according to the line of sight information representing the first user's eye 216A. In some embodiments, the entire first transparent lens 218A can be adjusted according to the sight line information. In other embodiments, only a specific part of the first transparent lens 218A can be adjusted according to the sight line information. For example, specific portions of the tunable optics can be adjusted to change the image at a specific location of the display of computer device 240; where the specific location of the display of computer device 240 is where the user's eyes 216A and 216B are focused.

The second transparent lens 218B can be used to assist the second user's eye 216B to view the connecting device. The second transparent lens 218B can include adjustable elements that can be adjusted according to the line of sight information representing the second user's eye 216B. In some embodiments, the entire second transparent lens 218B can be adjusted according to the line-of-sight information. In other embodiments, only a specific part of the second transparent lens 218B can be adjusted according to the sight line information. For example, specific portions of the tunable optics can be adjusted to enhance focus at specific locations of the display of computer device 240; where the specific locations of computer device 240 are where the user's eyes 216A and 216B are focused.

In some embodiments, the first user's eye 216A and the second user's eye 216B can focus on a single location. For example, the user's eyes 216A and 216B can include a reference line of sight 246 at the display of a computer device 240, such as a notebook or desktop computer. Although the reference line of sight 246 may be directed to a single point on the display of a laptop or desktop computer, the adjustable optics in the first transparent lens 218A and the second transparent lens 218B can The line-of-sight information of the user's eyes 216B is adjusted independently. FIG. 3A is a schematic diagram of a wearable device 300 using eye tracking. The wearable device 300 includes a single vision wearable device that provides light path adjustment based on eye tracking. The wearable device 300 includes a screen 310, an adjustable optical element 330, a wireless communication unit 340, an image projector 350 and an eye movement tracking module 360; On the opaque screen 310, the adjustable optical element 330 is used to adjust the light path on the transparent or opaque screen 310, the wireless communication unit 340 is used to communicate with the remote device, and the projector is used to project 2D visual data, so that the 2D visual data is The eye tracking module 360 is used to track the movements of the user's eyes 320A, 320B and determine the depth map corresponding to the user's eyes 320A, 320B through the translucent screen 310 or displayed on the opaque screen 310 .

The eye tracking module 360 can determine the line of sight 325 of the user's eyes 320A and 320B. In some embodiments, only parts of the transparent or opaque screen 310 can be adjusted according to the line of sight 325 of the user's eyes 320A and 320B. The eye tracking module 360 can adjust a specific part of the transparent or opaque screen 310 , such as the plurality of adjustable optical elements 330 , using the line of sight information corresponding to the line of sight 325 . Tunable optics 330 can be adjusted to change the focus or defocus of a particular light path through particular portions of the light-transmitting or opaque screen 310 . Tunable optics 330 include complex tunable mirrors, tunable lenses, tunable gratings, or other suitable tunable optics, and combinations thereof. The adjustable optical element 330 can be adjusted according to the sight line information corresponding to the sight line 325 , so the wearable device 300 can provide instant focus/defocus.

When viewing the screen, the instant focus/defocus of the adjustable optical element 330 can be used to solve the problems of focusing and convergence inconsistencies. For example, the feeling of dizziness caused by traditional VR experiences stems from inconsistent depth perception mechanisms. One such depth perception inconsistency mechanism occurs when the focus of the user's eyes (focusing) perceives images at the same distance on a display, but also perceives that the images converge at different depths (convergence) of the user's eyes. The user perceives the inconsistent and contradictory sensations of focusing and vergence, resulting in a feeling of dizziness and nausea.

To address the aforementioned inconsistent depth perception mechanisms, the eye tracking method of the present invention can be implemented in a wearable device, such as the wearable device 300 . The wearable device 300 can refocus the light according to the eye sight information, so as to adjust the angle of the light incident to the eye through the light-transmitting screen 310 or at a specific part of the opaque screen 310 . In this way, the adjustable optical element 330 of the transparent or opaque screen 310 can refocus the light according to the line-of-sight information through the user's eyes 320A, 320B, so as to solve the aforementioned inconsistency problem and improve certain viewing experience. Improve focus and vergence.

3B shows a schematic diagram of an optical image refocusing system using a lens. The optical image refocusing system using a lens shows that a lens is used to refocus a virtual image of an object according to the sight line information of a user's eyes.

In the refocusing system of the optical image system using a lens in Example 1, the user's eyes 320 view an object 370 through a medium, such as air without a screen (eg, a VR screen); the user's eyes 320 may not see through a transparent The lens looks at object 370 . In addition, the user's eyes 320 are viewing a physical object 370 rather than a virtual image representing the object.

In the optical image refocusing system using lenses of Example 2, the user's eyes 320 view the virtual image 375 of the object through the screen 380 . In this state, an image projector can project a virtual image representing the object through a screen 380 as the object virtual image 375 . In this state, the user's eyes 320 may experience inconsistencies in focusing and vergence.

In the optical image refocusing system using lenses of Example 3, the user's eye 320 views the object virtual image 375 through a lens 330 located between the user's eye 320 and the screen 380 . Lens 330 can be a fixed lens used to refocus the virtual image of the object. In other embodiments, the lens 330 can be an adjustable lens for dynamically refocusing the virtual image 375 of the object; in this case, the lens 330 can be adjusted according to the sight line information of the user's eye 320 .

Figure 3C illustrates an optical image refocusing system using a mirror. The optical image refocusing system using a mirror is shown to refocus a virtual image of an object with a mirror according to the sight line information of a user's eyes.

In the optical image refocusing system using mirrors of Example 1, the user's eyes view the object 370 through a medium, such as air, rather than a screen (eg, a VR screen). The user's eye 320 may not view the object 370 through a transparent lens, instead of representing a virtual image of the object.

In the optical image refocusing system using mirrors of Example 2, the user's eyes 320 view a virtual image 376 of an object through a screen 380 . In this case, an image projector can project a virtual image representing the object 370 as the object virtual image 376 through the screen 380 . In this case, the user's eyes 320 may experience inconsistencies between focus and vergence.

In the optical image refocusing system using mirrors in Example 3, the user's eyes 320 view a virtual image 376 of an object through a screen 380 including a mirror 385 . Mirror 385 can be a fixed mirror used to refocus the virtual image of the object. In other embodiments, the mirror 385 can be an adjustable mirror to refocus the dynamic object virtual image 376 on-the-fly through the screen 380 on which the mirror 385 is assembled. In this state, the mirror 385 can be adjusted according to the sight line information of the user's eyes 320 .

FIG. 4 shows a schematic diagram of a wearable device 400 using eye tracking. The wearable device 400 using eye tracking includes a stereo vision wearable device to adjust the light path according to the eye tracking. The wearable device 400 includes a first transparent or opaque screen 410A and a second transparent or opaque screen 410B for viewing by a user, a first set of adjustable optical elements 430A and a second set of adjustable optical elements Optical element 430B; the first group of adjustable optical elements 430A is used to adjust the light path of the first transparent or opaque screen 410A, and the second group of adjustable optical elements 430B is used to adjust the light of the second transparent or opaque screen 410B path.

The wearable device 400 may further include a first wireless communication unit 440A for communicating with a plurality of remote devices or a second wireless communication unit 440B for communicating with the aforementioned remote control device, a first image projector 450A, and a second image projector 450B, a first eye-tracking module 460A, and a second eye-tracking module 460B; the first image projector 450A is used for projecting 2D visual data, so that the 2D visual data can be transmitted or projected on the first transparent or non-transparent light. The light-transmitting screen 410A; the second image projector 450B is used for projecting 2D visual data, so that the 2D visual data can pass through or be projected on the second light-transmitting or non-light-transmitting screen 410B, and the first eye tracking module 460A is used for tracking the first use The posture of the user's eye 420A is used to determine the depth map corresponding to the first user's eye 420A, and the second eye tracking module 460B is used to track the posture of the second user's eye 420B and determine the depth map corresponding to the second user's eye 420B. .

The wearable device 400 may further include a continuous or two separate first translucent or opaque screens 410A and second translucent or opaque screens 410B capable of judging two different sight points 425A and 425B. When either of the user's eyes 420A and 420B is individually tracked by the corresponding first eye tracking module 460A and the second eye tracking module 460B, the first group of adjustable optical elements 430A and the second group of adjustable optical elements 430B can be adjusted individually. In addition, the first image projector 450A and the second image projector 450B can operate independently. In this way, a specific portion of the first transmissive or opaque screen 410A and the second translucent or opaque screen 410B can be selected to refocus light entering the user's eyes 420A and 420B. In this case, the user's eyes 420A and 420B can interpret the 3D projection composed of the plurality of images transmitted through the first transparent or opaque screen 410A and the second transparent or opaque screen 410B simultaneously.

Figure 5A shows a schematic diagram of a separate eye tracking device mounted on a machine. The separate eye tracking device can be a separate peripheral device around the machine (eg, display device 520). The separate eye-tracking device is mounted on a machine that includes a display device 520 to communicate with a separate peripheral device 530 disposed away from the user's eyes 510A and 510B.

The detachable peripheral device 530 includes a mechanical module 532 for controlling the direction of the emitted light and the detection of the eye tracking device 534, so that the user's eyes can always be positioned on the detachable peripheral device. The eye tracking device 534 tracks the posture of the user's eyes 510A and 510B and determines the gaze information corresponding to the user's eyes 510A and 510B. The display device 520 includes a line-of-sight reference point 515 that corresponds to the focal point of the user's eyes 510A and 510B on the display device 520 . The eye tracking device 534 of the separate peripheral device 530 can determine the gaze reference point 515 . In some implementations, the display device 520 can include complex tunable optical elements that adjust according to the line-of-sight reference point 515 . The tunable optics include tunable mirrors on display device 520 . In other embodiments, the display device 520 can include fixed optical elements, such as fixed mirrors, to refocus the light path.

The discrete peripheral device 530 can be used to provide output data to the display device 520 . The output data includes line-of-sight information of the user's eyes 510A and 510B. The display device 520 can use the sight line information to calculate an image corresponding to a specific position of the user's sight line reference point 515 on the screen. The image can be presented on the screen of the display device 520 with an array of light emitting diodes that generate visible light, liquid crystals that filter out white light, or other arrays of light sources located on the screen of the display device 520 . In addition, the computational image can be displayed on the screen of the display device 520 by optical refraction, diffraction, reflection, light guiding or other optical techniques.

5B shows a schematic diagram of an embedded eye tracking device packaged in a machine. The eye-tracking device packaged in the machine includes an embedded peripheral device 545 , and the embedded peripheral device 545 is integrated into a machine, such as a display device 540 . The embedded peripheral device 545 includes a mechanical module 546 to control the direction of light emission and detection from the eye tracking device, whereby the user's eyes can always be positioned by the embedded peripheral device 545 . The eye tracking device 547 tracks the eye postures of the user's eyes 510A and 510B and determines the sight line information according to the user's eyes 510A and 510B.

The display device 540 further includes a line-of-sight reference point 555 , which corresponds to the focus of the user's eyes 510A and 510B on the display device 540 . In some embodiments, the display device 540 includes a plurality of adjustable optical elements, which can be adjusted according to the line-of-sight reference point 555 . Tunable optical elements are included as complex tunable mirrors in display device 540 . In other embodiments, display device 540 includes fixed optical elements, such as fixed mirrors.

In some embodiments, the distance between the eyes 510A, 510B and the eye tracking devices 534, 537 may be determined according to the TOF concept or other methods, such as image processing or triangulation. The optical transmit power can be adjusted depending on the distance between the eye 510A, 510 and the eye tracking device 534, 547. For example, the optical transmit power can be dynamically lowered to reduce the optical emissions seen by the eye 510A or 510B when the distance between it and the eye tracking device 534, 537 is very close.

FIG. 6 shows a procedure 600 of eye movement tracking. The procedure 600 for eye tracking describes a procedure for monitoring an eye movement based on an eye depth map. In step 610, an electronic signal corresponding to measuring an optical signal reflected by an eye is captured, and the optical signal can be provided by an optical source. The light source is an adjustable voltage signal synchronized with a predetermined reference signal, whereby the light source can provide an optical signal that can be reflected by the eye toward the direction of the eye.

The reflected optical signal can be received by one or more photodetectors. In some embodiments, the received optical signal can be filtered to remove certain wavelengths. For example, an or complex filter is used to filter the optical signal, whereby only the wavelengths of interest remain in the filtered optical signal. In some embodiments, one or more lenses are used to focus the optical signal before it is passed to the photodetector. The lens can be a clear lens, a fixed lens, a tunable lens, a photonic barrier, or the like.

In step 620, the depth map is determined according to the phase difference between the received optical signal and the reference signal. The received optical signal can be compared to the reference signal as it is received. In other embodiments, the received optical signal may be filtered and compared to a reference signal. The depth map contains one or more sets of 3D information corresponding to the eyes. In some embodiments, a 3D image of the eye can be generated based on the 3D information of the depth map. The depth map enables immediate and continuous judgment. The depth map can also be judged and updated at a predetermined time point. For example, the depth map can be determined and updated at time intervals such as every microsecond, every millimeter, or every second.

In step 630, the line of sight information is determined according to the depth map. The sight line information can represent the sight line of the eye according to the depth map. In some embodiments, the line-of-sight information can be determined based on the comparison of the reference signal and the reflected light signal. In addition, the sight line information can include one or more of identification for a specific area of the eye, identification for a pupil of the eye, identification for an iris of the eye, and identification for a physiological structure of the eye. In some embodiments, the pose of the eye can be identified from the gaze information. Eye posture can include movement of eyes, rotation of eyes, stable state of eyes, time corresponding to stable state of eyes, closed state of eyes, time corresponding to closed state of eyes, open state of eyes, and corresponding time of open state of eyes. One or more of the corresponding time, the blink state of the eye, the time corresponding to the blink state of the eye, and the frequency corresponding to the blink state.

The depth map can be used to generate a vertical iris vector in a plane tangent to the eye. In such a case, the gaze information can be judged based on the iris vector and the depth map. The depth map can also be used to generate a pupil position in a plane tangent to the eye. In such a situation, the line-of-sight information can be determined based on the pupil position of the eye and the depth map.

In step 640, the line-of-sight information of the eyes is used as the output data. Output data corresponding to line-of-sight information can be communicated to a device, a machine, a system, or other similar elements. In this case, line-of-sight information can be passed as input data to a device, machine, or system. In addition, eye movement judgment from gaze information can be used as output data. Eye movements can give instructions to or interact with a device, machine or system. For example, if the eye blinks three times in rapid succession, it can be used as an instruction to a remote device, such as a television. In this way, if the eye blinks several times in a short period of time is tracked, the TV can be turned off.

FIG. 7 illustrates a procedure 700 for adjusting optics based on eye tracking, according to some exemplary embodiments. Process 700 Adjusting Optics According to Eye Tracking describes monitoring eye movement and adjusting optics according to eye movement. In step 710, an electronic signal corresponding to measuring an optical signal reflected by an eye is captured, and the optical signal can be provided by a light source. The light source is an adjustable voltage signal synchronized with a predetermined reference signal, whereby the light source can provide an optical signal that can be reflected by the eye toward the direction of the eye.

The reflected optical signal can be received by one or more photodetectors. In some embodiments, the received optical signal can be filtered to remove certain wavelengths. For example, an or complex filter is used to filter the optical signal, whereby only the wavelengths of interest remain in the filtered optical signal. In certain embodiments, one or more lenses are used to focus the optical signal before it is passed to the photodetector. The lens can be a transparent lens, a fixed lens, a tunable lens, a photonic fence, or the like.

In step 720, the depth map is determined according to the phase difference between the received optical signal and the reference signal. The received optical signal can be compared with the reference signal as it is received. In other embodiments, the received optical signal may be filtered and then compared to a reference signal. The depth map contains one or more sets of 3D information corresponding to the eyes. In some embodiments, a 3D image of the eye can be generated according to the 3D information of the depth map. The depth map enables instant and continuous judgment. The depth map can also be judged and updated at a predetermined time point. For example, the depth map can be determined and updated at time intervals such as every microsecond, every millimeter, or every second.

In step 730, the line of sight information is determined according to the depth map. The sight line information can represent the sight line of the eye according to the depth map. The sight line information can include one or more of identification for a specific area of the eye, identification for a pupil of the eye, identification for an iris of the eye, and identification for a physiological structure of the eye. In some embodiments, the pose of the eye can be identified from the gaze information. Eye posture can include movement of eyes, rotation of eyes, stable state of eyes, time corresponding to stable state of eyes, closed state of eyes, time corresponding to closed state of eyes, open state of eyes, and corresponding time of open state of eyes. One or more of the corresponding time, the blink state of the eye, the time corresponding to the blink state of the eye, and the frequency corresponding to the blink state.

The depth map can generate a vertical iris vector in a plane tangent to the eye that is perpendicular to a plane tangent to the eye. In such a case, the gaze information can be judged based on the iris vector and the depth map. The depth map can also be used to generate a pupil position in a plane tangent to the eye. In such a situation, the line-of-sight information can be determined based on the pupil position of the eye and the depth map.

In step 740, the tunable optical element is adjusted according to the line-of-sight information. Line-of-sight information can be used to tune specific optics. For example, line-of-sight information can include the focus of the eye at a particular location on the display. The eye of the eye can pass through an adjustable optical element, such as an adjustable lens or mirror. Adjustable lenses or mirrors can be activated or adjusted to refocus the light path. In some embodiments, the tunable optics can be activated or adjusted instantly on the display while the eye is being tracked. Tunable optics can include one or more lenses, mirrors, or the like.

In this way, the tunable optics can adjust according to the movement of the eye being tracked. Tunable optics can be used to provide dynamic focus and defocus on the fly. For example, tunable optics can be used to solve the problem of inconsistency in focusing and vergence when viewing a VR or AR display.

8 shows a schematic diagram of a computer device 800 and a mobile computer device 850 that can be used with the techniques described herein. Computer device 800 is used to represent various forms of digital computers, such as notebook computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The computer device 850 is used to represent various forms of mobile devices, such as personal digital assistants, mobile phones, smart phones, and other similar computer devices. The connections, relationships, and functions of the devices shown herein are illustrative and description only and should be regarded as illustrative in nature and not intended to limit the invention described and/or claimed herein.

The computer device 800 includes a processor 802, memory 804, a storage device 806, a high-speed interface 808 connected to the memory 804 and the high-speed expansion port 810, and a low-speed interface 812 connected to the low-speed bus 814 and the storage device 806 . Each of processor 802, memory 804, storage device 806, high-speed interface 808, high-speed expansion port 810, and low-speed interface 812 are interconnected using various bus bars and may be mounted on a motherboard or in other suitable manners . Processor 802 may process instructions executed within computer device 800, including instructions stored in memory 804 or on storage device 806 to communicate with one of external input/output devices (eg, display 816 coupled to high-speed interface 808). GUI displays image information. In other embodiments, multiple processors and/or multiple bus bars may be used in addition to multiple memories and different kinds of memory. Furthermore, multiple computer devices 800 may be connected to any device that provides the necessary operations (eg, a server bank, a set of blade servers, or a multiprocessor system).

The memory 804 stores information within the computer device 800 . In one embodiment, memory 804 is a volatile memory cell; in another embodiment, memory 804 is a non-volatile memory cell. The memory 804 can also be another form of computer-readable medium, such as a magnetic disk or an optical disk.

The storage device 806 can provide large-capacity storage for the computer device 800 . In one embodiment, the storage device 806 may be a computer-readable medium or may include a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid state storage device, or an array device, including a storage area network or other configuration device. A computer program product can be tangibly embodied in an information carrier. The computer program product may also contain instructions that, when executed, perform one or more of the methods described above. The information carrier is a computer or machine readable medium such as memory 804 , storage device 806 or storage on processor 802 .

The high-speed interface 808 manages the bandwidth-intensive service operations of the computer device 800 , and the low-speed interface 812 manages the low-bandwidth-intensive operations; the function assignments of the high-speed interface 808 and the low-speed interface 812 are merely illustrative. In one embodiment, high-speed interface 808 is coupled to memory 804, display 816 (eg, through a graphics processor or accelerator), and high-speed expansion port 810, which can accept various expansion cards (not shown). In this embodiment, the low-speed interface 812 is coupled to the storage device 806 and the low-speed bus 814 . The low-speed bus 814 includes various communication ports (eg, USB, bluetooth, ethernet, wireless ethernet) and can be coupled to one or more input/output devices through a network adapter, eg, a keyboard, a A pointing device, a scanner, or a network device (eg, a network switch or a router).

As shown, the computer apparatus 800 may be implemented in a number of different forms. For example, computer device 800 may be implemented as standard server 820, or multiple times within a group of such servers. Computer device 800 can also be implemented as part of blade server 824 . Additionally, the computer device 800 may be implemented as a personal computer, such as a notebook computer 822 . The components of the computer device 800 may also be combined with other components of a mobile device (not shown), such as the computer device 850 . These devices may include one or more computer devices 800, 850, and the entire system may be composed of a plurality of computer devices 800, 850 that can communicate with each other.

Computer device 850 includes processor 852, memory 864, an input/output device (eg, display 854), a communication interface, and transceivers between the devices. Computer device 850 may also contain storage devices, such as microdrives or other devices, to provide additional storage space. Each of computer device 850, processor 852, memory 864, and display 854 are interconnected using various busbars, and several of these devices may be mounted on a motherboard or in other suitable manners.

Processor 852 may execute instructions within computer device 850 (including instructions stored in memory 864). The processor may be implemented with a chip set comprising separate multiple analog and digital processors. The processor may be used, for example, for other components of the computer device 850 , such as controlling the user interface, applications executed by the processor 852 , and wireless communication of the computer device 850 .

Processor 852 may communicate with a user through control interface 858 and display interface 856 coupled to display 854 . Display 854 may be, for example, a TFT LCD, an OLED display, or other suitable display technology. Display interface 856 may include appropriate circuitry for driving display 854 to present graphics and other information to a user. The control interface 858 can receive commands from the user and convert them to the processor 852 . In addition, an external interface 862 in communication with the processor 852 may be provided for the computer device 850 to communicate with other adjacent devices. External interface 862 may provide wired communication in some embodiments and wireless communication in other embodiments; of course, multiple interfaces may also be used to provide communication.

Memory 864 stores information within computer device 850 . Memory 864 may be implemented in one or more computer-readable media or media, volatile memory cells, or non-volatile memory cells. The expansion memory 864 may also be connected to the computer device 850 through the expansion processor 852, which may include, for example, a SIMM card port. This expansion memory 864 may provide additional storage space for the computer device 850 , or may also be used by the computer device 850 for applications or other information. Specifically, the expansion memory 864 may contain instructions for executing or supplementing the aforementioned programs, and may also contain security information. Therefore, the expansion memory 864 can be used as a security module of the computer device 850, for example, and can write instructions to enable the device to be used safely. In addition, security applications and additional information can be provided through the SIMM card, such as placing invulnerable identification information on the SIMM card.

The memory may include, for example, flash and/or NVRAM memory as described below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product contains instructions which, when executed, perform one or more of the aforementioned methods. The information carrier is a computer or machine readable medium such as memory 864 , expansion memory 864 , memory on processor 852 , or a propagated signal received by way of a transceiver or external interface 862 , for example.

The computer device 850 can communicate wirelessly through a communication interface, and the communication interface can include a digital signal processing circuit when necessary. The communication interface can provide communication in various modes or protocols such as GSM voice calls, SMS, EMS or MMS messages, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS. The aforementioned communication may take place, for example, by means of a radio frequency transceiver. Additionally, short-range communications, such as using Bluetooth, WiFi or other such transceivers (not shown), can also occur. In addition, the GPS (Global Positioning System) receiver module can provide additional navigation and location-related wireless data to the computer device 850 , which can be used as an application on the computer device 850 .

Computer device 850 may also use audio codec 860 for voice communication, which may receive spoken information from the user and convert it into usable digital information. Audio codec 860 may also produce audible sound to the user, eg, through speakers (eg, in the earpiece of computer device 850). Such sounds may include sounds from voice phone calls, may include recorded sounds (eg, voice messages, music files, etc.), and may include sounds generated by operating applications on computer device 850.

Computer device 850 may be implemented in a number of different forms as shown in the figures. For example, it can be implemented as a mobile phone 880. The computer device 850 may also be implemented as part of a smartphone 882, part of a personal digital assistant or other similar mobile device.

The complex number application can be implemented based on the following concepts. For example, the TOF pixels shown in FIGS. 1B and 1C can also be used to detect the facial features of the user, including eye pose tracking, face recognition or expression detection. In another example, the eye gesture tracking of the present invention can be used to replace or supplement a mouse to locate displayed content on a display that is of interest or focus to a user. In some embodiments, the eye pose tracking of the present invention can be used to more accurately select advertising directions of interest to a user or predict user behavior. For example, the robot learning with artificial neural network can learn and judge the user's behavior according to the user's injection position. According to the behavior of different users, different weights can be given; for example, the weights can be based on (1) not watching at all (2) watching but not choosing (3) watching and choosing from small to large. In addition, the duration of the user's gaze at the specific content can also be used to record the degree of interest of the user, for example, the longer the duration, the higher the weight. In some implementations, displaying advertisements on web pages according to the user's level of interest allows advertisers to pay different fees compared to traditional click or no-click advertisements.

The eye pose tracking of the present invention can also be used in games. For example, a racing or flying game can be played on the user's mobile phone 220 or tablet computer 230 as shown in FIG. 2A . Changes in the user's eye gaze can be used to control the movement (eg, direction) of the vehicle, enabling the vehicle to move in the direction indicated by the user's eye gaze. In another example, the information collected by the accelerometer 211, the gyroscope 212, and the eye tracking module can be used to track the movement of the user's head and eye gaze. In some embodiments, a separate button may be included to control speed, and another optional button may be included to control an additional action, such as shooting or switching tools. Head movement and eye gaze information, alone or in combination, can be used in a game run on a mobile phone or on a tablet to determine the user's actions, and the game can respond accordingly. In this example, a combination of a vector representing the rotational position of the user's head and a line of sight vector representing the user's eyes can be used to determine the angle of the eye's line of sight relative to the head, which can then be used to explain the user's actions or thoughts .

Although the present disclosure has been described herein with reference to specific exemplary embodiments; however, it should be recognized that the present invention is not limited to the described embodiments; Corrections and alterations in the spirit and scope are implemented. For example, the processes previously disclosed may be reordered, steps may be added or removed.

Embodiments of the invention and all functional operations described in this specification may be implemented in digital electronic circuits or in computer software, solid state or firmware, including the structural specifications disclosed herein and their structural equivalents, or one or various combinations. Embodiments of the present invention may be implemented by one or more computer program products, such as one or more modules of computer program instructions encoded on a computer-readable medium, for execution by or to control a data processing device operation. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of things that affect a machine-readable propagated signal, or a combination of one or more of the like. As used herein, the term "data processing apparatus" includes all equipment, devices and machinery for processing data, eg including a programmable processor, a computer or multiple processors or computers. The apparatus may contain, in addition to hardware, other code that creates an execution environment for computer programs, such as one or more of the code making up the processor hardware, a protocol stack, a database management system, an operating system, or the like The combination. A propagated signal is an artificially generated signal, such as a mechanically generated electrical, optical or electromagnetic signal that encodes information for transmission to suitable receiving equipment.

Computer programs (also known as programs, software, software applications, scripting programs, and language or code), including compiled or interpreted languages, may be written in any form of programming language and may include stand-alone programs or as modules suitable for use in a computing environment. Any form of deployment such as groups, devices, subroutines, or other units. Computer programs do not necessarily correspond to files in the file system. A program can be stored in a document that holds a portion of other programs or data (such as one or more scripting programs stored in a markup language document), in a single document dedicated to the program in question, or in multiple documents of the same type (for example: a file that stores one or more modules, subroutines, or code). The computer program can be executed on one computer, and the computer program can also be executed on multiple computers located at the same location; of course, the computer program can also be distributed on multiple computers at different locations and interconnected by a communication network.

Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and one or more processors of any kind in a digital computer. Generally, processors receive instructions and data from at least one of read-only memory and random access memory. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer may also include or be operably coupled to receive data from or transfer data to one or more mass storage devices (eg, magnetic, magneto-optical disks, or optical disks) for storing data. However, a computer does not need such a device. In addition, the computer can be embedded in another device, such as a tablet computer, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, and the like. Computer-readable media suitable for storing computer program instructions and data include non-volatile memory, media, and a form of memory devices, including semiconductor memory devices (eg, EP ROM, EEPROM, and flash memory devices) and magnetic disks (eg internal hard disk or removable disk), magneto-optical disks, CD ROM and DVD-ROM disks. The processor and memory may supplement or be incorporated in special purpose logic circuitry.

In order to provide communication with the user, embodiments of the present invention may be implemented on a computer having a display device (such as a CRT or LCD display), a keyboard, or a pointing device (such as a mouse or trackball) for Displays information to the user, who can provide input to the computer through a keyboard or trackball. Of course, other types of devices can also be used to achieve communication with the user, for example, the feedback provided to the user can be any form of sensory feedback (eg: visual feedback, auditory feedback or tactile feedback), and can receive any form of feedback from User input, including audio, voice, or tactile input.

Embodiments of the present invention may be implemented in a computer system including a back-end device (eg, a data server); embodiments of the present invention may also be implemented in a computer system including an intermediate device (eg, an application server); of course, the implementation of the present invention The approach may include front-end devices, such as client computers with graphical user interfaces or web browsers, for the user to interact with. Embodiments of the present invention may not preclude implementation in a computer system comprising one or more combinations of back-end devices, middle-end devices, and front-end devices. The devices in the system may be interconnected by any form or medium of digital data communication, such as a communication network. Communication networks include, for example, local area networks and wide area networks, such as the Internet.

A computer system may contain clients and servers. In general, clients and servers are remote from each other and interact through a network. A client-server relationship is created by the computer programs running on the respective computers and the client-server relationship between them.

Although many details are contained herein, these are descriptions of specific features of specific embodiments and should not be used to limit the invention. In this document, certain features that are described in a single embodiment can be implemented in only a single embodiment, but features that are described in a single embodiment can also be implemented in multiple embodiments separately or in combination. Conversely, various features that are described above in the context of a single embodiment can also be implemented in multiple embodiments separately or in subcombination. Furthermore, although the features described in the foregoing may function in a certain embodiment and in the originally claimed content, one or more features of the claimed combination may in certain circumstances be removed from the combination and the claimed combination Can be a sub-combination or a variation of a sub-combination.

Similarly, although operations are depicted in a particular order, the operations may be performed in other specific or sequential orders, or in the order shown, to achieve the desired effect. In certain situations, multiplexing or parallel processing can achieve specific advantages. Furthermore, various system elements may be separated from each other in different embodiments, but should not be considered as necessarily separated from each other in all embodiments.

In any of the examples referring to HTML files, other file types or formats may be substituted, eg, HTML files may be replaced by XML, JSON, plain text, or other types of files. In addition, other data structures (eg, spreadsheets, relational databases, or structured documents) can be used as the aforementioned tables or hash tables.

While specific embodiments of the present invention are described, other embodiments are within the scope of the present invention. For example, the operations defined within the scope of the patent may be performed in a different order and still achieve the desired effect.

100…Eye Tracking System

110…Eye Tracking Device

120...User's eyes

130…image display

140…Signal processing unit

150…Sight Information

160, 165, 190...TOF pixels

162a, 162b...reset transistors

164a, 164b...source follower transistors

166a, 166b... select transistors

167…Die/Wafer Bonding

169…Circuit Wafer

170…Console

192a, 192b...Capacitors

194, 196…Transistor group

201, 202...headsets

211, 221, 231...accelerometers

212, 222, 232…Gyroscope

213, 360...Eye Tracking Module

213A, 460A…the first eye tracking module

213B, 460B...Second eye tracking module

214…Wireless Transmission Unit

216A, 216B, 320, 320A, 320B, 420A, 420B, 510A, 510B...eyes

218…Transparent lens

218A...first transparent lens

218B...Second transparent lens

220…Phone

224, 234, 244, 254, 340... wireless communication unit

230…Tablet

236, 555...line of sight reference point

240…Computer devices

246…reference sight

250…TV

300, 400…Wearables

310, 380... screen

325…Sight

330…tunable optics, lenses

350…image projector

370…objects

375, 376...object virtual image

385…Reflector

410A…First light-transmitting or opaque screen

410B…Second transparent or opaque screen

425A, 425B...Sight point

430A…First set of tunable optics

430B…Second set of tunable optics

440A…the first wireless communication unit

440B...Second wireless communication unit

450A…First Image Projector

450B…Second Image Projector

515…Sight reference point

520, 540...display devices

530…Separate Peripherals

532, 546... Mechanical modules

534, 547…Eye Tracking Devices

545…Embedded Peripherals

600, 700... program

610~640, 710~740...steps

800…computer device

802…processor

804...memory

806…Storage device

808…High Speed Interface

810…High Speed Expansion Port

812…Low speed interface

814…Low-speed busbar

820…Standard server

822…Laptop

824…Blade Servo

850…computer unit

852…processor

854…Display

856…Display Interface

858…Control Interface

860…audio codec

864…memory

880…Mobile phone

882…Smartphone

M1~M6…Transistor

VREF, VREF2...Voltage

Q1, Q1, Q1', Q2'...charge

…相位差

...phase difference

816…Display

862…External Interface

FIG. 1A is a schematic diagram of an eye movement tracking system.

FIG. 1B is a schematic diagram of a time difference ranging device.

FIG. 1C is a schematic diagram of a time difference ranging device.

FIG. 1D and FIG. 1E are schematic diagrams of techniques for judging features of a user's eyes, respectively.

FIG. 1F shows a schematic diagram of the phase of charge collection.

Figure 1G shows a schematic diagram of light emission, detection and charge collection.

FIG. 1H shows a schematic diagram of the signal voltage during the charge collection period.

FIG. 1I shows a schematic diagram of the phase change in charge collection.

FIG. 1J shows a schematic diagram of light emission, detection, and charge collection of phase changes.

FIG. 1K shows a schematic diagram of the signal voltage of the charge collection when the phase changes.

FIG. 1L is a schematic diagram of a time difference ranging device.

2A shows a schematic diagram of a cross-platform peripheral control system using eye tracking.

FIG. 2B shows a schematic diagram of a cross-platform control system using eye tracking.

3A shows a schematic diagram of a wearable device using eye tracking.

3B shows a schematic diagram of an optical image refocusing system using a lens.

3C shows a schematic diagram of an optical image refocusing system using a mirror.

FIG. 4 is a schematic diagram of a wearable device utilizing eye tracking.

Figure 5A shows a schematic diagram of a separate eye tracking device mounted on a machine.

5B shows a schematic diagram of an embedded eye tracking device packaged in a machine.

FIG. 6 is a flow chart of the eye movement tracking process.

FIG. 7 is a flowchart illustrating a procedure for adjusting the tunable optical element according to eye tracking.

FIG. 8 is a schematic diagram of a computer device and a mobile computer device.

One or more embodiments of the present disclosure are illustrated by way of example, and not in a limiting sense, in the figures of the accompanying drawings, wherein like reference numerals refer to like elements.

201…headset 211, 221, 231...accelerometers 212, 222, 232…Gyroscope 213…Eye Tracking Module 214…Wireless Transmission Unit 216A, 216B...Eyes 218…Transparent lens 220…Phone 224, 234, 244, 254... wireless communication unit 230…Tablet 236…Sight reference point 240…Computer devices 250…TV

Claims (14)

An eye tracking system comprising: a machine including a display including one or more tunable optical elements; Eye tracking device, including: A single optical transmitter or multiple optical transmitters for emitting one or more specific wavelengths to illuminate a user's eye; one or more light detectors for receiving a light signal reflected from the user's eye; and A circuit for: capturing an electrical signal corresponding to the optical signal reflected from the user's eye measured by the one or more light detectors; determining a depth map of the user's eye according to a phase difference between a reference signal and the electrical signal generated by the photodetector; and determining a line of sight information representing the user's eyes according to the depth map; A signal processing unit, which communicates with the eye tracking device and is disposed separately from the eye tracking device, the signal processing unit is used for: receiving an output data representing the gaze information from the eye tracking device; and The relationship between the display of the machine and the line of sight information representing the user's eyes is determined. The eye-tracking system of claim 1, further comprising an or complex filter for removing a non-target wavelength signal in the optical signal. The eye tracking system of claim 1, further comprising one or more lenses for focusing the light signal on the light detector. The eye tracking system of claim 1, wherein the one or more adjustable optical elements comprise an adjustable lens or an adjustable mirror. The eye-tracking system of claim 1, further comprising a wearable device coupled to the machine and the eye-tracking device to form an integrated hardware package, the display of the machine is Opaque, and a visual image is displayed on a display through an array of one or more light sources. The eye-tracking system of claim 1, further comprising a wearable device coupled to the machine and the eye-tracking device to form an integrated hardware package, the display pair of the machine The image projected toward the display is at least partially transparent, and the characteristics of the image projected toward the display are adjusted by the one or the plurality of adjustable optical elements on the display. The eye tracking system of claim 5 or 6, wherein the signal processing unit communicates with the integrated hardware package through a wireless or wired connection. An eye tracking system comprising: a machine including a display including one or more tunable optical elements; Eye tracking device, including: A single optical transmitter or multiple optical transmitters for emitting one or more specific wavelengths to illuminate a user's eye; one or more light detectors for receiving a light signal reflected from the user's eye; and A circuit for: capturing an electrical signal corresponding to the optical signal reflected from the user's eye measured by the one or more light detectors; and determining a depth map of the user's eye according to a phase difference between a reference signal and the electrical signal generated by the photodetector; A signal processing unit, which communicates with the eye tracking device and is disposed separately from the eye tracking device, the signal processing unit is used for: determining a line of sight information representing the user's eyes according to the depth map; and The relationship between the display of the machine and the line of sight information representing the user's eyes is determined. The eye tracking system of claim 8, further comprising an or complex filter for removing a non-target wavelength signal in the optical signal. The eye tracking system of claim 8, further comprising one or more lenses for focusing the light signal on the light detector. The eye tracking system of claim 8, wherein the one or more adjustable optical elements comprise an adjustable lens or an adjustable mirror. The eye-tracking system of claim 8, further comprising a wearable device coupled to the machine and the eye-tracking device to form an integrated hardware package, the display of the machine is Opaque, and a visual image is displayed on a display through an array of one or more light sources. The eye-tracking system of claim 8, further comprising a wearable device coupled to the machine and the eye-tracking device to form an integrated hardware package, the machine and the display pair The image projected toward the display is at least partially transparent, and the characteristics of the image projected toward the display are adjusted by the one or the plurality of adjustable optical elements on the display. The eye tracking system of claim 12 or 13, wherein the signal processing unit communicates with the all-in-one hardware package through a wireless or wired connection.

Images