US12610036B2 - Head-mounted display and method for eye tracking calibration - Google Patents
Head-mounted display and method for eye tracking calibrationInfo
- Publication number
- US12610036B2 US12610036B2 US18/955,999 US202418955999A US12610036B2 US 12610036 B2 US12610036 B2 US 12610036B2 US 202418955999 A US202418955999 A US 202418955999A US 12610036 B2 US12610036 B2 US 12610036B2
- Authority
- US
- United States
- Prior art keywords
- positions
- display
- calibration
- processor
- head
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/296—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/327—Calibration thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/332—Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
- H04N13/344—Displays for viewing with the aid of special glasses or head-mounted displays [HMD] with head-mounted left-right displays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/30—Image reproducers
- H04N13/366—Image reproducers using viewer tracking
- H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/011—Head-up displays characterised by optical features comprising device for correcting geometrical aberrations, distortion
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Optics & Photonics (AREA)
- Eye Examination Apparatus (AREA)
Abstract
A head-mounted display and a method for eye tracking calibration are provided. The head-mounted display includes a display, an image capture device, and a processor. The image capture device captures an eye image during a display of calibration points. The processor is coupled to the display and the image capture device, wherein the processor is configured to: display the calibration points on positions of a position set sequentially in an order according to a current time and number of positions via the display; and generate calibration data according to the eye image.
Description
This application claims the priority benefit of U.S. provisional application Ser. No. 63/641,940, filed on May 2, 2024. The entirety of the above-mentioned patent application is hereby incorporated by reference herein and made a part of this specification.
The invention relates to an extended reality (XR) technique, and in particular to a head-mounted display and a method for eye tracking calibration.
In the development of metaverse-related techniques, the eye tracking system has become one of the basic equipment of the XR system. The XR system may perform eye tracking on a user wearing a head-mounted display (HMD) to interact with the user. In order to adapt the HMD to users with different eye characteristics, the HMD may perform a calibration function before performing eye tracking. A traditional HMD displays a plurality of calibration points in a fixed pattern. The HMD may perform calibration of eye tracking according to the eye characteristics when the user is looking at the calibration points. However, a user with HMD experience may be able to predict the position, order, or frequency of the calibration points. This anticipatory psychology may affect the eye movements of the user during calibration, causing the data collected by the HMD to be incorrect. Accordingly, the calibration results may not be ideal. Based on the above, how to correctly perform the calibration of the eye tracking system is one of the important issues in the art.
The invention provides a head-mounted display and method for eye tracking calibration that may improve the accuracy of data used for eye tracking calibration.
A head-mounted display for eye tracking calibration includes a display, an image capture device, and a processor. The image capture device captures an eye image during a display of calibration points. The processor is coupled to the display and the image capture device, wherein the processor is configured to: display the calibration points on positions of a position set sequentially in an order according to a current time and number of positions via the display; and generate calibration data according to the eye image.
In an embodiment of the invention, the current time includes a system clock or a gazing time on a previous calibration point.
In an embodiment of the invention, each of the positions of the position set is preset with a value.
In an embodiment of the invention, the processor is further configured to: divide the current time by remaining number of positions to obtain a remainder; and display the calibration point on the position with the value corresponding to the remainder.
In an embodiment of the invention, the processor is further configured to: divide the current time by the remaining number of positions to obtain the remainder in response to the remaining number of positions being greater than zero.
In an embodiment of the invention, the processor is further configured to: stop the display of the calibration point in response to the remaining number of positions equaling zero.
A method for eye tracking calibration of the invention includes: displaying calibration points on positions of a position set sequentially in an order according to a current time and number of positions via a display by a head-mounted display; capturing an eye image by the head-mounted display during a display of the calibration points; and generating calibration data according to the eye image by the head-mounted display.
In an embodiment of the invention, the current time includes a system clock or a gazing time on a previous calibration point.
In an embodiment of the invention, each of the positions of the position set is preset with a value.
In an embodiment of the invention, the step of displaying the calibration points on the positions of the position set sequentially in the order according to the current time and the number of positions includes: dividing the current time by remaining number of positions to obtain a remainder; and displaying the calibration point on the position with the value corresponding to the remainder.
In an embodiment of the invention, the step of dividing the current time by the remaining number of positions to obtain the remainder includes: dividing the current time by the remaining number of positions to obtain the remainder in response to the remaining number of positions being greater than zero.
In an embodiment of the invention, the method further including: stopping displaying the calibration point in response to the remaining number of positions equaling zero.
Based on the above, in the invention, the calibration points may be displayed in a random manner during the calibration of eye tracking. In the invention, the user may be prevented from predicting the position, order, or frequency of the calibration points, making the calibration results of eye tracking more accurate.
The processor 110 is, for example, a central processing unit (CPU), or other programmable general-purpose or special-purpose micro control unit (MCU), microprocessor, digital signal processor (DSP), programmable controller, application-specific integrated circuit (ASIC), graphics processing unit (GPU), image signal processor (ISP), image processing unit (IPU), arithmetic logic unit (ALU), complex programmable logic device (CPLD), field-programmable gate array (FPGA), or other similar elements, or a combination of the above elements. The processor 110 may be coupled to the storage medium 120, the display 130, and the image capture device 140, and access and execute a plurality of modules and various applications in the storage medium 120.
The storage medium 120 is, for example, any type of fixed or removable random-access memory (RAM), read-only memory (ROM), flash memory, hard-disk drive (HDD), solid-state drive (SSD), or similar elements, or a combination of the above elements, and is used to store a plurality of modules or various applications that may be executed by the processor 110.
The display 130 may include a liquid-crystal display (LCD) or an organic light-emitting diode (OLED) display. In an embodiment, the display 130 may provide an image beam to the eyes of the user to form an image on the retina of the user, so that the user may see the XR scene created by the HMD.
The image capture device 140 is, for example, a camera used to capture an image. The image capture device 140 may include a photosensitive element such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD). The image capture device 140 may be used to capture an eye image of the user wearing the head-mounted display 100. The processor 110 may perform eye tracking according to the eye image of the user. The head-mounted display 100 may interact with the user according to the results of eye tracking. For example, when the user looks at a virtual object of a virtual scene, the head-mounted display 100 may determine that the user is looking at the virtual object according to the results of eye tracking, and interact with the user using the virtual object.
In step S202, the processor 110 may select a position from the position set and display the calibration point on the selected position via the display 130. After the calibration point is displayed on the selected position, the processor 110 may remove the selected position from the position set to update the position set. That is, the remaining number of positions in the position set may be reduced after step S202.
During the display of the calibration point by the display 130, the processor 110 may capture the eye image of the user via the image capture device 140.
In an embodiment, the processor 110 may select a selected position from the position set according to the current time and the number of positions of the position set, and display the calibration point on the selected position. Specifically, each of the positions of the position set may be preset with a value by the processor 110, wherein the values of the positions may be different from each other. The processor 110 may obtain a number N of positions in the position set, wherein N is a non-negative integer (i.e., N may be the remaining number of positions of the position set). The processor 110 may divide the current time by N to obtain the remainder (i.e., the processor 110 may perform an operation of dividing the current time by N). The processor 110 may select one of the positions from the position set as the selected position according to the remainder, wherein the selected position is with the value corresponding to the remainder. For example, assume that position 31 is preset with a value ‘1’, position 32 is preset with a value ‘2’, position 33 is preset with a value ‘3’, position 34 is preset with a value ‘4’, and position 35 is preset with a value ‘5’. If the current time is equal to ‘11’, the processor 110 may divide the current time ‘11’ by ‘N=5’ to obtain the remainder ‘1’. Since position 31 is with the value ‘1’ corresponding to the remainder ‘1’, the processor 110 may select position 31 from the position set to be the selected position. Position 31 may be removed from the position set and N may be reduced to 4.
In an embodiment of the present invention, the processor 110 may determine the current time according to the system clock. For example, if the system clock includes ‘11 seconds’, the processor 110 may determine that the current time is ‘11’. In an embodiment of the present invention, the processor 110 may determine the current time according to a gazing time on a previous calibration point. Assume that a previous calibration point was displayed on position 31 during a specific time period. After the previous calibration point on position 31 is vanished and position 31 is removed from the position set, the processor 110 may select one position from the remaining positions of the position set according to the specific time period. For example, if the specific time period is ‘10:50:50 to 10:50:58’, the processor 110 may determine that the current time is equal to ‘58’, which corresponds to the end second of the specific time period.
In step S203, the processor 110 may determine whether the remaining number of positions in the position set is zero. If the remaining number of positions is zero, the processor 110 may perform step S204 and may stop displaying the calibration point. If the remaining number of positions is greater than zero, the processor 110 may perform step S202 again. That is, the processor 110 may select a position from the updated position set as the selected position, and display the calibration point on the selected position via the display 130.
Taking FIG. 3 as an example, the processor 110 may iteratively perform step S202 and step S203 to sequentially display the calibration point on each position according to the order of the position 31, the position 32, the position 33, the position 34, and the position 35.
In step S204, the processor 110 may generate calibration data for eye tracking according to a plurality of eye images corresponding to a plurality of positions (e.g., the position 31, the position 32, the position 33, the position 34, and the position 35) in the initial position set. Then, the processor 110 may determine whether the eye tracking calibration process is successful according to the calibration data. The processor 110 may determine whether the calibration process is successful according to any algorithm or rule, which is not limited by the invention. If the processor 110 determines that the calibration process is successful, step S206 is performed. If the processor 110 determines that the calibration process fails, step S205 is performed.
In step S205, the processor 110 may initialize the position set. For example, the processor 110 may convert the position set from an empty set to a set containing the position 31, the position 32, the position 33, the position 34, and the position 35. After step S205 is completed, the processor 110 may perform step S202 again to attempt the calibration of eye tracking again.
In step S206, the processor 110 may perform eye tracking according to the calibration data.
Based on the above, the head-mounted display of the invention may display calibration points on different positions of the display sequentially in a random order. Users of the head-mounted display are not able to predict the position, order, or frequency of the calibration points according to past experience. Compared with traditional eye tracking calibration methods, the calibration data collected based on the method of the invention is more accurate. The accuracy of eye tracking performed based on the calibration data may be significantly improved.
Claims (8)
1. A head-mounted display for eye tracking calibration, comprising:
a display;
an image capture device, capturing an eye image during a display of calibration points; and
a processor coupled to the display and the image capture device, wherein the processor is configured to:
display the calibration points on positions of a position set sequentially in an order according to a current time and number of positions via the display; and
generate calibration data according to the eye image,
wherein each of the positions of the position set is preset with a value,
wherein the processor is further configured to:
divide the current time by remaining number of positions to obtain a remainder; and
display the calibration point on the position with the value corresponding to the remainder.
2. The head-mounted display of claim 1 , wherein the current time comprises a system clock or a gazing time on a previous calibration point.
3. The head-mounted display of claim 1 , wherein the processor is further configured to:
divide the current time by the remaining number of positions to obtain the remainder in response to the remaining number of positions being greater than zero.
4. The head-mounted display of claim 1 , wherein the processor is further configured to:
stop the display of the calibration point in response to the remaining number of positions equaling zero.
5. A method for eye tracking calibration, comprising:
displaying calibration points on positions of a position set sequentially in an order according to a current time and number of positions via a display by a head-mounted display;
capturing an eye image by the head-mounted display during a display of the calibration points; and
generating calibration data according to the eye image by the head-mounted display,
wherein each of the positions of the position set is preset with a value,
wherein the step of displaying the calibration points on the positions of the position set sequentially in the order according to the current time and the number of positions comprises:
dividing the current time by remaining number of positions to obtain a remainder; and
displaying the calibration point on the position with the value corresponding to the remainder.
6. The method of claim 5 , wherein the current time comprises a system clock or a gazing time on a previous calibration point.
7. The method of claim 5 , wherein the step of dividing the current time by the remaining number of positions to obtain the remainder comprises:
dividing the current time by the remaining number of positions to obtain the remainder in response to the remaining number of positions being greater than zero.
8. The method of claim 5 , further comprising:
stopping displaying the calibration point in response to the remaining number of positions equaling zero.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US18/955,999 US12610036B2 (en) | 2024-05-02 | 2024-11-22 | Head-mounted display and method for eye tracking calibration |
| CN202510127942.4A CN120891641A (en) | 2024-05-02 | 2025-02-05 | Head-mounted display devices and methods for eye-tracking correction |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US202463641940P | 2024-05-02 | 2024-05-02 | |
| US18/955,999 US12610036B2 (en) | 2024-05-02 | 2024-11-22 | Head-mounted display and method for eye tracking calibration |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| US20250343888A1 US20250343888A1 (en) | 2025-11-06 |
| US12610036B2 true US12610036B2 (en) | 2026-04-21 |
Family
ID=97494028
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| US18/955,999 Active US12610036B2 (en) | 2024-05-02 | 2024-11-22 | Head-mounted display and method for eye tracking calibration |
Country Status (2)
| Country | Link |
|---|---|
| US (1) | US12610036B2 (en) |
| CN (1) | CN120891641A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200388054A1 (en) * | 2017-12-19 | 2020-12-10 | Telefonaktiebolaget Lm Ericsson (Publ) | Head-Mounted Display Device and Method Thereof |
| TW202101976A (en) | 2019-05-28 | 2021-01-01 | 見臻科技股份有限公司 | Eye-tracking module with scenario-based mode switching function |
| US20220342478A1 (en) * | 2019-04-01 | 2022-10-27 | Evolution Optiks Limited | User tracking system and method, and digital display device and digital image rendering system and method using same |
| TW202310618A (en) | 2021-08-30 | 2023-03-01 | 美商元平台技術有限公司 | Eye-tracking using embedded electrodes in a wearable device |
| US20230102371A1 (en) * | 2021-09-28 | 2023-03-30 | Ramot At Tel-Aviv University Ltd. | Automatic preference quantification of displayed objects based on eye tracker data |
-
2024
- 2024-11-22 US US18/955,999 patent/US12610036B2/en active Active
-
2025
- 2025-02-05 CN CN202510127942.4A patent/CN120891641A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200388054A1 (en) * | 2017-12-19 | 2020-12-10 | Telefonaktiebolaget Lm Ericsson (Publ) | Head-Mounted Display Device and Method Thereof |
| US20220342478A1 (en) * | 2019-04-01 | 2022-10-27 | Evolution Optiks Limited | User tracking system and method, and digital display device and digital image rendering system and method using same |
| TW202101976A (en) | 2019-05-28 | 2021-01-01 | 見臻科技股份有限公司 | Eye-tracking module with scenario-based mode switching function |
| TW202310618A (en) | 2021-08-30 | 2023-03-01 | 美商元平台技術有限公司 | Eye-tracking using embedded electrodes in a wearable device |
| US20230102371A1 (en) * | 2021-09-28 | 2023-03-30 | Ramot At Tel-Aviv University Ltd. | Automatic preference quantification of displayed objects based on eye tracker data |
Non-Patent Citations (2)
| Title |
|---|
| "Office Action of Taiwan Counterpart Application", issued on Jul. 1, 2025, p. 1-p. 6. |
| "Office Action of Taiwan Counterpart Application", issued on Jul. 1, 2025, p. 1-p. 6. |
Also Published As
| Publication number | Publication date |
|---|---|
| CN120891641A (en) | 2025-11-04 |
| US20250343888A1 (en) | 2025-11-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20200202496A1 (en) | Image processing apparatus, image processing method and storage medium | |
| WO2018019282A1 (en) | Binocular panorama image obtaining method and apparatus, and storage medium | |
| CN112400148B (en) | Method and system for performing eye tracking using an off-axis camera | |
| JP5246078B2 (en) | Object location program and camera | |
| US20210068742A1 (en) | Image processing system, imaging apparatus, electronic device, methods of controlling the system, and the apparatuses, and storage medium | |
| US10078918B2 (en) | Information processing apparatus, information processing method, and storage medium | |
| US12411546B2 (en) | Information processing apparatus, information processing method, and program | |
| EP4425305A1 (en) | Processor, image processing method, and image processing program | |
| CN114415826A (en) | Data processing method and equipment thereof | |
| US12610036B2 (en) | Head-mounted display and method for eye tracking calibration | |
| US10178320B2 (en) | Image display terminal, image display method, and non-transitory computer readable recording medium storing program | |
| US12487670B2 (en) | Information processing device, information processing method, and program | |
| TWI910925B (en) | Head-mounted display and method for eye tracking calibration | |
| KR20150072667A (en) | Apparatus and method for digital picturing image | |
| CN115624315B (en) | Eye movement tracking method and device, electronic equipment, computer storage medium and product | |
| JP7049636B1 (en) | Information processing equipment, programs, systems, and information processing methods | |
| JP2024006315A (en) | display device | |
| US20240385682A1 (en) | Information processing apparatus and information processing method | |
| TW202222073A (en) | Electronic device and method of tracking pose of electronic device | |
| US20260025491A1 (en) | Electronic device, display device and three-dimensional imaging method thereof | |
| US12131720B2 (en) | Transmissive display apparatus and image combining method therein | |
| US12189849B2 (en) | Processor, information processing method, and information processing program | |
| EP2287805A1 (en) | Image processing device, image processing method, and information storage medium | |
| US12548282B2 (en) | Imaging support apparatus, imaging apparatus, imaging support method, and program | |
| US20260030758A1 (en) | Ophthalmic apparatus, method for generating high dynamic-range image for ophthalmic apparatus, and program for generating high dynamic-range image for ophthalmic apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: ALLOWED -- NOTICE OF ALLOWANCE NOT YET MAILED |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
| STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
| STCF | Information on status: patent grant |
Free format text: PATENTED CASE |