TWI738473B - Head mounted display apparatus and eye-tracking apparatus thereof - Google Patents

Abstract

An eye-tracking apparatus includes a first lens, a light splitting device, a display, an image sensor, a second lens and multiple a plurality of light sources. The light splitting device receives a first beam and generates a second beam, and transmits the second beam to the first lens. The display projects a reference mark to the target area through the light splitting device and the first lens. The image sensor captures the detection image on the target area though the first lens, the light splitting device and the second lens. The second lens is disposed between the light splitting device and the image sensor. The plurality of light sources are disposed around the image sensor, and respectively project a plurality of light beams to the target area through the first lens, the light splitting device and the second lens.

Description

Head-mounted display device and eye tracking device

The present invention relates to a head-mounted display device and an eye tracking device, and particularly relates to a head-mounted display device and an eye tracking device that can improve visual comfort.

The problem of dizziness and visual discomfort is the Head Mounted Display (HMD) or smart eyewear with Augmented Reality (AR) or Virtual Reality (VR), etc. One of the major bottlenecks currently faced in optical products. The currently commercially available augmented reality and virtual reality technologies rarely provide the best design configuration or provision for specific users’ head shape, vision conditions, and wearing posture. Effective guidance, unsuitable wear may not only cause physical discomfort for the user, but also seriously affect the user’s immersion and the received information and image quality (blurred image, unnatural image distortion), etc. Consumers shy away when they first encounter augmented reality/virtual reality.

In view of this, the present invention provides a head-mounted display device and an eye tracking device, which can effectively track the eye, so as to reduce dizziness and visual discomfort and improve visual comfort.

The eye tracking device of the present invention includes a first lens group, a light splitting element, a display, an image sensor, a second lens group, and a plurality of light sources. The first lens group has a first surface facing the target area, and has a second surface opposite to the first surface. The light splitting element receives the first light beam and generates a second light beam, so that the second light beam is transmitted to the second surface of the first lens group. The display projects the reference mark to the target area. The image sensor captures the detected image on the target area through the first lens group, the light splitting element and the second lens group. The second lens group is arranged between the light splitting element and the image sensor. A plurality of light sources are arranged around the image sensor, and pass through the first lens group, the light splitting element and the second lens group to respectively project a plurality of light beams to the target area.

The head-mounted display device of the present invention includes a processor and the eye tracking device as described above. The processor is coupled to the image sensor and receives the detected image.

Based on the above, the head-mounted display device proposed by the present invention is arranged around the image sensor by arranging the first lens group, the light splitting element, the image sensor, the second lens group, the multiple light sources, and the display. The multiple light sources transmit multiple light beams to the target area through the first lens group, the light splitting element, and the second lens group, and the display projects a reference mark to the target area, and then the image sensor passes through the first lens group and the light splitter The element and the second lens group capture the detected image on the target area, which can effectively track the eyeball, so as to alleviate dizziness and visual discomfort, thereby improving visual comfort.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

1A and 1B are schematic diagrams of an eye tracking device according to an embodiment of the invention. FIG. 1A and FIG. 1B have the same hardware architecture. The difference is that FIG. 1A also shows the light path projected by multiple light sources, and FIG. 1B also shows the light path for capturing the detected image. In FIG. 1A, the eye tracking device 100 includes lens groups 110 and 120, a beam splitter 130, a display 140, an image sensor 150, and a light source 160. The lens group 110 has a first surface SF1 and a second surface SF2. The first surface SF1 of the lens group 110 faces the target area TG1. The second surface SF2 of the lens group 110 is opposite to the first surface SF1 of the lens group 110. The display 140 is used for projecting the reference mark to the target area TG1 through the light splitting element 130 and the lens group 110. The lens group 120 is disposed between the beam splitting element 130 and the image sensor 150. The light source 160 can be divided into two parts of the light source 160-1 and 160-2, and they are respectively arranged around the image sensor 150. The light sources 160-1 and 160-2 are used to project light beams B1-1 and B1-2 respectively.

Regarding the operation details of the eye tracking device in the embodiment of FIG. 1A of the present invention. In FIG. 1A, the light sources 160_1 and 160_2 respectively project light beams B1_1 and B1_2 to the lens group 120 and pass through the lens group 120 to be directed toward the beam splitting element 130. The beam splitting element 130 receives the beams B1_1 and B1_2 and generates beams B2_1 and B2_2. The beam splitting element 130 transmits the beams B2_1 and B2_2 to the lens group 110 and passes through the lens group 110 to be directed toward the target area TG1. Here, the light splitting element 130 is used to reflect the light beams B1_1 and B1_2 to generate the light beams B2_1 and B2_2. The lens group 110 may be composed of one or more lenses, and there is no fixed limit.

In this embodiment, the target area TG1 may be the position of the user’s eyeballs, and the lens group 120 may be a zoom lens, which allows the light beams B1_1 and B1_2 generated by the light sources 160_1 and 160_2 to pass through the lens group 120. The light beams B2_1 and B2_2 generated by the light splitting element 130 are then distributed on the surface of the eyeball through the lens group 110 to generate multiple light spot images around the pupil of the eyeball. The lens group 120 can be used to focus the detected image on the surface of the eyeball on the image sensor 150.

Next, referring to FIG. 1B, the image sensor 150 captures the detected image on the target area TG1 through the lens group 120, the beam splitting element 130, and the lens group 110, so that the eye tracking device can perform eye tracking actions based on the detected image.

On the other hand, each of the light sources 160_1 and 160_2 may be composed of one or more light emitting diodes. In this embodiment, the above-mentioned light-emitting diode may be an infrared light-emitting diode. The image sensor 150 may be an infrared sensor with a function of receiving infrared rays. The spectroscopic element 130 can reflect the infrared light beams sent by the light sources 160_1 and 160_2. Regarding the material of the light splitting element 130, any light splitting element known to those skilled in the art can be applied, and there is no other limitation in the embodiment of the present invention.

In this embodiment, the eye tracking device 100 can also analyze the target area TG1 including the light spot images corresponding to the beams B2_1 and B2_2, and the detected images of the reference marks projected by the display 140, and use the positional relationship of the multiple images to Detect the deviation information of the user's eyeballs. Generally speaking, the offset includes the offset caused by wearing, or called the left and right eye pupil distance offset (interpupillary distance, IPD), and the other kind of offset is the offset caused by strabismus. Furthermore, the wear deviation can be judged by the positional relationship between the user’s eyeballs and the image sensor 150. In this embodiment, the image sensor 150 is disposed in the center of the multiple light sources 160, so it can be determined based on the multiple The positional relationship between the multiple light spot images projected by the light source 160 and the eyeball is determined. The squint shift depends on the gaze angle of the observing user, and multiple test images are projected through the display 140 according to multiple transmission angles, and judged based on the multiple transmission angles and the user's viewing angle information.

The calculation method of the offset information will be described in detail in subsequent implementations.

Please pay special attention to the fact that the light sources 160_1 and 160_2 based on the embodiment of the present invention are arranged around the image sensor 150. In the detected images captured by the image sensor 150, the light spot images generated by the light sources 160_1 and 160_2, It can be more evenly distributed around the pupil of the eyeball, effectively improving the accuracy of eye-tracking movements. In addition, in this way, the head-mounted display device applied to the eye tracking device 100 of the present invention can further detect whether the head-mounted display device and the user's eyeballs are shifted or squinted. Appropriate warning or guidance can help users adjust the head-mounted display device to a better stereoscopic display position, and greatly improve the dizziness and visual disturbance that consumers are likely to encounter when using augmented reality or virtual reality. Comfort and other issues, thereby enhancing visual comfort.

2A and 2B respectively show schematic diagrams of the configuration of multiple light sources and image sensors according to different embodiments of the present invention. In FIG. 2A, there may be light sources 260_1 to 260_8 around the image sensor 251, and the multiple distances between the light sources 260_1 to 260_8 and the image sensor 251 may all be the same. Of course, the number of light sources 260_1 to 260_8 in FIG. 2A is only an exemplary embodiment for illustration, and does not limit the scope of the present invention. In other embodiments of the present invention, the number of light sources may be at least three.

In another embodiment of FIG. 2B, there may be light sources 260_9 to 260_16 around the image sensor 252, and the light sources 260_9 to 260_16 and the image sensor 252 have multiple distances respectively, and at least two of the light sources and the image sensor 252 have multiple distances. The distance between the detectors 252 is not the same. For example, the light source 260_9 and the image sensor 252 are separated by a distance r1, and the light source 260_11 and the image sensor 252 are separated by another distance r2, wherein the distance r1 is not equal to the distance r2 (r1>r2). Incidentally, when the light source 260_9 and the image sensor 252 have a relatively large distance r1, and the light source 260_11 and the image sensor 252 have a relatively small distance r2, the power of the light source 260_9 may be greater than the power of the light source 260_11. Of course, the number of light sources 260_9 to 260_16 in FIG. 2B is only an example for illustration, and it is not necessary to limit the scope of the present invention.

Incidentally, in other embodiments of the present invention, the light source does not necessarily need to be arranged around the image sensor in a ring-shaped manner. The light source can also be arranged around the image sensor in other shapes.

FIG. 3 is a schematic diagram of a head-mounted display device according to an embodiment of the invention. In this embodiment, the head-mounted display device 300 includes lens groups 310 and 320, a light splitting element 330, a display 340, an image sensor 350, a light source 360, and a processor 370. Different from the embodiments in FIGS. 1A and 1B, this embodiment is additionally provided with a processor 370, and is used to perform actions related to eye tracking and eyeball deviation verification.

The processor 370 may be a processor with computing capability. Alternatively, the processor 370 can be designed through a hardware description language (Hardware Description Language, HDL) or any other digital circuit design method known to those with ordinary knowledge in the art, and through a field programmable logic gate array ( Field Programmable Gate Array (FPGA), complex programmable logic device (Complex Programmable Logic Device, CPLD) or special application integrated circuit (Application-specific Integrated Circuit, ASIC) to implement the hardware circuit.

FIG. 4 is a schematic diagram of offset verification actions in different implementations of an embodiment of the present invention. 3 and 4, the head-mounted display device 300 can perform a wear deviation verification mode through the processor 370. In the detection image TI1 in FIG. 4, the detection image TI1 corresponds to the multiple light spot images LI projected by the light source 360 on the target area TG2 and coincides with the preset reference range RG, which means that the head-mounted display device 300 does not occur. Wearing deviation.

In contrast, in the detection image TI1', the detection image TI1' corresponds to the multiple light spot images LI' projected by the light source 360 on the target area TG2 and does not overlap the preset reference range RG, which represents a head-mounted display device 300 has a case of wearing deviation.

On the other hand, the head-mounted display device 300 can determine the offset information according to the positional relationship between the light spot image LI or LI' and the preset reference range RG through the processor 370, and then feedback the offset information to the user , To guide the user to adjust the left and right pupil distance of the left and right eyes and the wearing posture to the correct direction.

In another embodiment of the present invention, the processor 370 may also first determine whether the image of the reference mark projected by the display 340 onto the target area TG2 is within the preset reference range RG on the target area TG2 for a predetermined time, and then perform the above-mentioned offset Judgment of information is the action of focusing in advance. For example, in this embodiment, the target area TG2 is the position of the user's eyes, and the display 340 can project a logo (for example, a star totem, or any other logo) to the user's eyeballs for the user Focus. At the same time, the user can look directly at the above-mentioned mark for a predetermined time (for example, 1 to 2 seconds), so as to stabilize the eye movement of the user and keep the eyeball as static as possible. Among them, the small-range vibration of the eyeball can be eliminated by subsequent image processing by the processor 370, so as to make the judgment of the deviation verification mode of this embodiment more accurate. Of course, the above-mentioned predetermined time of 1 to 2 seconds is only an example for illustrative purposes, and is not intended to limit the scope of the present invention.

FIG. 5 is a schematic diagram of an offset verification action according to an embodiment of the present invention, and FIG. 6 is a schematic diagram of an offset verification action according to different implementations of an embodiment of the present invention. Referring to FIGS. 5 and 6, the processor in the head-mounted display device of this embodiment may perform the squint deviation verification mode after performing the wear deviation verification mode of FIG. 4. FIG. 5 shows the displays 540_L and 540_R corresponding to the left and right eyes EYEL and EYER, and the image sensors 550_L and 550_R. The display 540_L, 540_R can simultaneously (or time-sharing) project two reference marks (for example, astral totem or other signs of any form) RI_L, RI_R, eyeball EYEL, EYER on the two eyeballs EYEL, EYER respectively according to the sending angle A Looking at the reference marks RI_L and RI_R respectively forming the optical path of the virtual imaging VI of the reference mark, the image sensors 550_L and 550_R can simultaneously (or time-sharing) capture the detected images on the eyeballs EYEL and EYER. The sending angle A can be 0 degrees or multiple angles (10 degrees à 20 degrees... etc.) in order to repeatedly test (multiple test images) and record.

If the user has a problem of squint deviation, the user's eyeball will turn to an angle that is not equal to the transmission angle A, and the head-mounted display device can further display the two eyes alternately through the display 540_L, 540_R, and through processing The device analyzes the detected images of eyeballs EYEL and EYER to obtain the user's perspective information, and then determines whether the user's eyeballs EYEL and EYER are moving based on the viewing angle information and the sending angle A, and then determines whether the user has a squint offset problem. In order to make the judgment of the offset verification mode of this embodiment more accurate.

FIG. 6 is a schematic diagram of the offset verification actions in different implementations of the embodiment of the present invention. In the detection image TI2 in FIG. 6, the detection image TI2 corresponding to the image RI of the reference mark on the target area coincides with the preset reference range RG, which means that the user of the head-mounted display device does not have a squint shift. Condition.

In contrast, in the detection image TI2', the image RI' corresponding to the reference mark on the target area of the detection image TI2' does not coincide with the preset reference range RG, which means that the user of the head-mounted display device has squint Offset situation.

On the other hand, the head-mounted display device can determine the degree of squint according to the positional relationship between the image RI or RI' of the reference mark and the preset reference range RG through the processor, and then feedback the degree of squint to the user.

To sum up, based on the embodiment of the present invention, multiple light sources are arranged around the image sensor. In the detected images captured by the image sensor, the light spot images generated by the multiple light sources can be more evenly distributed Around the pupil of the eyeball, it effectively improves the accuracy of eye-tracking movements. In addition, in this way, the head-mounted display device using the eye tracking device of the present invention can perform the wear deviation verification mode and the squint deviation verification mode through the processor. It can not only detect whether the head-mounted display device is worn incorrectly, whether the left and right eye pupil distance positions are adjusted correctly, and guide the user to adjust the wear of the head-mounted display device, but it can also detect whether the user has squint. It can be combined with medical resources to provide assistance, or to further adjust the display content. Appropriate warning or guidance can help users adjust to a better stereoscopic display position, greatly improving the dizziness and visual discomfort that consumers are likely to encounter when using augmented reality or virtual reality, and then Improve visual comfort.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to those defined by the attached patent application scope.

100: Eye tracking device 110, 120, 310, 320: lens group 130, 330: Spectroscopic element 140, 340, 540_L, 540_R: display 150, 251, 252, 350, 550_L, 550_R: image sensor 160, 160_1, 160_2, 260_1~260_16, 360, 360_1, 360_2: light source 300: Head-mounted display device 370: processor A: Sending angle B1_1, B1_2, B2_1, B2_2, B3_1, B3_2, B4_1, B4_2: beam EYEL, EYER: Eyeball LI, LI’: light spot image r1, r2: distance RG: Reference range RI, RI’: the image of the reference mark RI_L, RI_R: reference mark SF1, SF2, SF3, SF4: surface TG1, TG2: target area TI1, TI1’, TI2, TI2’: detect images VI: Virtual imaging of reference signs

1A and 1B are schematic diagrams of an eye tracking device according to an embodiment of the invention. 2A and 2B respectively show schematic diagrams of the configuration of multiple light sources and image sensors according to different embodiments of the present invention. FIG. 3 is a schematic diagram of a head-mounted display device according to an embodiment of the invention. FIG. 4 is a schematic diagram of offset verification actions in different implementations of an embodiment of the present invention. Fig. 5 is a schematic diagram of an offset verification action according to an embodiment of the present invention. FIG. 6 is a schematic diagram of offset verification actions in different implementations of an embodiment of the present invention.

100: Eye tracking device

110, 120: lens group

130: Spectroscopic element

140: display

150: image sensor

160, 160_1, 160_2: light source

B1_1, B1_2, B2_1, B2_2: beam

SF1, SF2: surface

TG1: target zone

Claims (9)

An eye tracking device includes: a first lens group having a first surface facing a target area and having a second surface opposite to the first surface; a beam splitting element that receives a first light beam and generates a second light beam Light beam, which transmits the second light beam to the second surface of the first lens group; a display, which projects a reference mark to the target area through the beam splitting element and the first lens group; and an image sensor which transmits through the first lens group A lens group, the light-splitting element and a second lens group to capture a detection image on the target area, the second lens group is arranged between the light-splitting element and the image sensor; a plurality of light sources are arranged in Around the image sensor, through the first lens group, the dichroic element and the second lens group to respectively project a plurality of light beams to the target area; a processor, wherein the processor is used to receive the detected image, Analyze the image of the reference mark in the detection image, the multiple light spot images corresponding to the light beams, and the positional relationship of a preset reference range on the target area to determine offset information. The eye tracking device according to claim 1, wherein the number of the light sources is at least three, and the distance between the light sources and the image sensor is the same. The eye tracking device according to claim 1, wherein the light sources and the image sensor respectively have a plurality of distances, and at least two of the distances are different. The eye tracking device according to claim 3, wherein when a first distance is greater than a second distance, the power of a first light source corresponding to the first distance is greater than the power of a second light source corresponding to the second distance. The eye tracking device according to claim 1, wherein the processor is coupled to the image sensor for performing eye tracking actions. A head-mounted display device includes: an eye tracking device, including: a first lens group having a first surface facing a target area and a second surface opposite to the first surface; a light splitting element receiving A first light beam and generating a second light beam, so that the second light beam is transmitted to the second surface of the first lens group; a display, projecting a reference mark to the target area through the light splitting element and the first lens group; An image sensor that captures a detection image on the target area through the first lens group, the light-splitting element and a second lens group, the second lens group is disposed on the light-splitting element and the image sensor And a plurality of light sources arranged around the image sensor, through the first lens group, the light splitting element and the second lens group to respectively project a plurality of light beams to the target area; and a processor, Coupled to the image sensor to receive the detected image, wherein in a first offset verification mode, the processor is used to: parse the image of the reference mark and a plurality of light spot images in the detected image, The light spot images correspond to the light beams respectively; and the offset information is determined according to the positional relationship between the light spot images and a preset reference range on the target area. The head-mounted display device according to claim 6, wherein the processor is further configured to: first determine that the image of the reference mark is located in the reference range for a predetermined time, and then according to the positions of the light spot images and the reference range Relationship to determine the offset information. The head-mounted display device according to claim 6, wherein after the first offset verification mode, a second offset verification mode is performed, wherein the processor is further configured to: according to the image of the reference mark and the reference The positional relationship of the range is used to determine the degree of squint. The head-mounted display device according to claim 8, wherein the processor determines the degree of squint according to the positional relationship between the image of the reference mark and the reference range, and includes: the display is used for sequentially according to a plurality of transmission angles Projecting a plurality of test images to the target area; and the processor parses the detected image to obtain a viewing angle information of a user, and determines the user's squint degree according to the viewing angle information and each transmission angle.

Images