TW201833624A - Head mounted display device, object tracking apparatus and method for tracking object thereof - Google Patents

Abstract

A head mounted display apparatus, an object tracking apparatus and a method for tracking object thereof are provided. The object tracking apparatus includes a lens, a light splitting device, a programmable light source and an image extractor. The lens generates and projects a detection light beam to an object. The programmable light source has a plurality of sub-light sources. The sub-light sources respectively project a plurality of light beams to a plurality positions of the light splitting device. The programmable light source receives a driving signal, and adjusts a light-on status of each of the sub-light sources according to the driving signal. The image extractor extracts a detection image from the object. Wherein, the light splitting device receives at least one of the light beams and generates at least one reflection light beam to the lens, and the lens generates the detection light beam accordingly.

Description

Head-mounted display device, object tracking device, and object tracking method

The invention relates to a head-mounted display device, an object tracking device, and an object tracking method, and more particularly, to an adaptively-adjustable object tracking device and an object tracking method.

In today's technology field, eye tracking technology is often used in electronic products. In virtual reality display technology, eye tracking technology is often cited. Through the eye tracking technology, the virtual reality display can generate Foveated Rendering function to reduce the amount of calculation in the image and reduce the system requirements of the virtual reality display device to provide a better human-computer interaction interface.

In the conventional technical field, since the user keeps closing his eyes, the pupil reflected light spots on the eyeball are blocked. Or, the eyelashes or decorations on the eyes may also cause the pupils' light spots on the eyeballs to be blocked. These conditions may cause errors in eye-tracking actions and reduce the accuracy of eye-tracking.

The invention provides a head-mounted electronic device, an object tracking device and a tracking method thereof, which effectively improve the accuracy of object tracking.

The object tracking device of the present invention includes a lens, a spectroscopic element, a programmable light source, and an image capture device. The lens generates and projects a detection beam onto an object. The spectroscopic element and the lens are arranged on the axis together. The programmable light source has a plurality of sub-light sources. The sub-light sources are used to respectively project a plurality of light beams to a plurality of different positions of the spectroscopic element. The programmable light source receives a driving signal and adjusts the lighting state of each sub-light source according to the driving signal. The image capturer captures a detected image of an object. The beam splitting element receives at least one of the light beams, generates at least one reflected light beam to the lens, and causes the lens to generate a detection light beam.

The head-mounted display device of the present invention includes a display and an object tracking device as described above. The display is arranged on the axis and projects a display image into a lens.

The object tracking method of the present invention includes: providing a lens and a light splitting element to be arranged on the same axis; providing a programmable light source having a plurality of sub-light sources, wherein the sub light sources are respectively used to project a plurality of light beams to a plurality of different positions of the light splitting element And provide driving signals to adjust the lighting status of each sub-light source; enable the beam splitter to receive at least one of the light beams, and generate at least one reflected light beam to the lens, so that the lens generates a detection light beam, and projects the detection light beam to the object ; And, providing an image capturer to capture a detected image of an object.

Based on the above, the present invention provides a programmable light source, and by dynamically adjusting the luminous pattern of the programmable light source, the light beam is projected to multiple positions of the beam splitting element, and the object detection is performed in a coaxial and / or off-axis manner Measure action, effectively improve the accuracy of object detection.

In order to make the above features and advantages of the present invention more comprehensible, embodiments are hereinafter described in detail with reference to the accompanying drawings.

Please refer to FIG. 1, which is a schematic diagram of an object tracking device according to an embodiment of the present invention. In this embodiment, the object tracking device 100 is used to track the position of the eyeball OBJ of the eye 101. The object tracking device 100 includes a lens 110, a spectroscopic element 120, a programmable light source 130, and an image capture device 140. The lens 110 is disposed on the axis AX1 to generate a detection beam DB and project the detection beam DB onto an object (eyeball OBJ). And through the detection beam DB, a corresponding reflection phenomenon is generated on the eyeball OBJ. The spectroscopic element 120 and the lens 110 are disposed on the axis AX1 together. The lens 110 is disposed between the eyeball OBJ and the spectroscopic element 120, and the axis AX1 can pass through the geometric center of the lens 110 and the spectroscopic element 120. In addition, the programmable light source 130 is disposed on one side of the spectroscopic element 120 and is isolated from the axis AX1. The programmable light source 130 may have multiple sub-light sources SLS1 ~ SLSN. The sub-light sources SLS1 to SLSN are respectively disposed at a plurality of different positions of the programmable light source 130 and are used to respectively project a plurality of light beams to a plurality of different positions of the spectroscopic element 120. The spectroscopic element 120 can reflect a plurality of light beams projected by the sub-light sources SLS1 to SLSN, and cause the reflected light beams to be transmitted to the lens 110. The lens 110 focuses light on the reflected light beam and generates a detection light beam DB.

It is worth mentioning that the programmable light source 130 receives the driving signal DRV and adjusts the lighting states of the sub-light sources SLS ~ SLSN according to the driving signal DRV. The sub-light sources SLS ~ SLSN may be fully lit or partially lit, and there are no specific restrictions. Please refer to FIG. 1 and FIG. 2 simultaneously. FIG. 2 is a schematic diagram illustrating an operation manner of a programmable light source according to an embodiment of the present invention. In FIG. 2, the sub-light sources SLS1, SLS3, and SLSM in the programmable light source 130 are taken as examples. The sub-light sources SLS1, SLS3, and SLSM are respectively used to project the light beams LB1, LB2, and LB3 to different positions of the spectroscopic element 120. The spectroscopic element 120 reflects the projected light beams LB1, LB2, and LB3, and generates reflected light beams RLB1, RLB2, and RLB3, respectively. It can be clearly seen from FIG. 2 that the reflected light beams RLB1 and RLB3 are relatively distanced from the axis AX1, and the reflected light beam LB2 travels substantially close to the axis AX1. In other words, when the sub-light source SLS3 is lit, the object tracking device 100 can generate a detection beam DB through a coaxial light detection method to project onto the eyeball OBJ. In addition, when at least one of the sub-light sources SLS1 and SLSM is turned on, the object tracking device 100 can generate the detection beam DB to project onto the eyeball OBJ through the off-axis light detection method.

Please note that in this embodiment, any two of the sub-light sources SLS1, SLS3, and SLSM may be lighted at the same time, or may be lighted in time sharing. That is, according to the driving signal DRV, any of the sub-light sources SLS1 to SLSN in the programmable light source 130 may be independently lit or unlit. Taking FIG. 2 as an example, when the sub-light sources SLS1 and SLSM are lit (when the sub-light source SLS2 is not lit), the programmable light source 130 can provide off-axis light beams LB1 and LB3, thereby generating detection. Beam DB. In contrast, when the sub-light source SLS3 is turned on (when the sub-light sources SLS1 and SLSM are not turned on), the programmable light source 130 can provide a coaxial light beam LB2 and thereby generate a detection beam DB.

Please refer to FIG. 1 again. The image capture device 140 is disposed on the side of the axis AX1 and is used to capture a detection image of an object (eyeball OBJ). When the detection light beam DB is projected onto the eyeball OBJ, the eyeball OBJ may correspond to the phenomenon that the detection light beam DB reflects light. Therefore, the image capture device 140 can obtain a detection image with reflected light (for example, a reflective light spot). Taking the detected object as the eyeball OBJ as an example, when the detection beam DB is projected onto the eyeball OBJ, multiple reflective light spots may appear around the eyeball OBJ. The actual position of the eyeball OBJ can be determined by judging the position of the light spot of the reflection.

Incidentally, in this embodiment, the image capturing device 140 may be an infrared light camera, and the sub-light sources SLS1 to SLSN in the programmable light source 130 may be infrared light sources (such as infrared light emitting diodes).

Please refer to FIG. 3, which is a schematic diagram of an object tracking device according to another embodiment of the present invention. The object tracking device 300 includes a lens 310, a light splitting element 320, a programmable light source 330, an image capture device 340, and a controller 350. In this embodiment, the lens 310 and the spectroscopic element 320 are sequentially arranged at the front end of the eyeball OBJ, and are arranged on the axis AX1. In addition, the geometric center of the programmable light source 330 is arranged on the axis AX2, and the geometric center of the beam splitting element 320 passes through the axis AX2. The programmable light source 330 receives a driving signal DRV to control whether each sub-light source on the programmable light source 330 is lit or not through the driving signal DRV.

In this embodiment, the controller 350 is coupled to the image capturing device 340 and the programmable light source 330. The controller 350 generates and provides a driving signal DRV to drive the programmable light source 330. The controller 350 further receives a detection image generated by the image capture device 340, and determines position information of the eyeball OBJ according to the reflection phenomenon generated by the eyeball OBJ corresponding to the detection light beam in the detection image.

In this embodiment, the controller 350 may generate the driving signal DRV according to a plurality of preset patterns, and sequentially adjust the driving signal DRV according to the preset pattern in different time intervals, and adjust the programmable light source 330. Of the light source. In the following, please refer to FIGS. 4A to 4E when referring to the drawings. FIGS. 4A to 4E respectively illustrate schematic diagrams of adjusting the lighting state of the programmable light source according to different embodiments of the present invention.

In FIG. 4A, the programmable light source 330 may be divided into a plurality of first sub-light sources SLSA1 and a plurality of second sub-light sources SLSB1 according to a range boundary RG. In this embodiment, the sub-light sources outside the range boundary RG are classified as the first sub-light source SLSA1, and the sub-light sources within the range boundary RG are classified as the second sub-light source SLSB1. In this embodiment, the first sub-light source SLSA1 can project the light beam to the surrounding area of the spectroscopic element 320, and the second sub-light source SLSB1 can project the light beam to the central area of the spectroscopic element 320. Moreover, in the present embodiment, the controller 350 may provide a driving signal DRV so that at least one of the first sub-light sources SLSA1 is lit in the first time interval, and at least one of the second sub-light sources SLSB1 is in The second time interval is illuminated, wherein the first time interval and the second time interval do not overlap, partially overlap, or completely overlap.

In an example where the first time interval and the second time interval do not overlap, please refer to FIG. 4B and FIG. 4C. In FIG. 4B, in the first time interval, all the first sub-light sources SLSA1 are turned on, and all the second sub-light sources SLSB1 are not turned on. In FIG. 4C, in the second time interval, all the first sub-light sources SLSA1 are not turned on, and all the second sub-light sources SLSB1 are turned on. In this way, the object tracking device 300 can generate a detection signal by transmitting the coaxial light and the off-axis light in a time-sharing manner.

In the example where the first time interval and the second time interval completely overlap and partially overlap, please refer to FIG. 4D and FIG. 4E. In FIGS. 4D and 4E, in a time interval where the first time interval and the second time interval overlap, part of the first sub-light source SLSA1 and part of the first sub-light source SLSB1 are turned on. In this way, the object tracking device 300 may generate a detection signal by transmitting coaxial light and off-axis light in a time-sharing manner in a non-overlapping time interval between the first time interval and the second time interval, and in the first time interval and the second Overlapping time intervals, the detection signals are generated synchronously through coaxial light and off-axis light, which effectively improves the flexibility of the detection action of the object tracking device 300.

It is worth mentioning that the rectangular programmable light source 330 shown in FIGS. 4A to 4E is only an illustrative example, and is not intended to limit the scope of the present invention. The shape of the programmable light source in the embodiment of the present invention may show any shape, such as a circle, an oval, a polygon, or any irregular shape. In addition, the arrangement of the sub-light sources on the programmable light source is not limited to a matrix form. The sub-light sources on the programmable light source according to the embodiment of the present invention may be arranged according to a preset rule or irregularly, and there is no fixed limit.

Please refer to FIG. 5, which is a schematic diagram of a head-mounted display device according to an embodiment of the present invention. The head-mounted display device 500 includes an object tracking device 510 and a display 520. The object tracking device 510 includes a lens 511 spectroscopic element 512, a programmable light source 513, an image capture device 514, and a controller 515. The implementation details of the object tracking device 510 are similar to the object tracking device of the embodiments described above, and are not repeated here. It is worth mentioning that the display 520 may be disposed on the axis AX1 to generate a display image, and may project and display the display image to the light splitting element 512 along the axis AX1. The spectroscopic element 512 can transmit the display image and cause the display image to be transmitted to the lens 511 and then transmitted through the lens 511 to the user's eyeball OBJ.

Please note that the display 520 and the corresponding object tracking device 510 included in the head-mounted display device 500 may be one group or two groups. The number of dispositions of the display 520 and the corresponding object tracking device 510 may vary according to the application state of the head-mounted display device 500, and there is no specific limitation.

Please refer to FIG. 6, which is a flowchart of an object tracking method according to an embodiment of the present invention. In step S610, a lens and a spectroscopic element are provided to be arranged on the same axis; in step S620, a programmable light source having a plurality of sub-light sources is provided, wherein the sub-light sources are used to respectively project a plurality of light beams to a plurality of the spectroscopic elements. At different positions, driving signals are provided to adjust the lighting status of each sub-light source. In step S630, the spectroscopic element is caused to receive at least one of the light beams, and generate at least one reflected light beam to the lens, so that the lens generates a detection light beam. And project a detection beam onto the object; in step S640, an image capture device is provided to capture a detection image of the object.

Regarding the implementation details of the above steps, detailed descriptions have been given in the foregoing embodiments, and details are not described herein.

In summary, the present invention provides a programmable light source and transmits a plurality of light beams to a plurality of different positions of the spectroscopic element through a plurality of sub-light sources to generate a detection light beam. In this way, the object tracking device can dynamically adjust the generation mechanism of the detection beam to effectively improve the accuracy of object tracking.

Although the present invention has been disclosed as above with the examples, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some modifications and retouching without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be determined by the scope of the attached patent application.

100, 300, 510‧‧‧ object tracking devices

101‧‧‧ eyes

OBJ‧‧‧Eyeball

110, 310, 511‧‧‧ lens

120, 320, 512‧‧‧ Beamsplitters

130, 330, 513‧‧‧programmable light source

140, 340, 514‧‧‧ image capture devices

350, 515‧‧‧ controller

500‧‧‧ head-mounted display

DB‧‧‧detection beam

AX1, AX2‧‧‧ axis

SLS1 ~ SLSN‧‧‧Sub-light source

LB1, LB2, LB3‧‧‧ beam

RLB1 ~ RLB3‧‧‧Reflected beam

SLSA1‧‧‧First Sub-Light Source

SLSB1‧‧‧Second Sub Light Source

RG‧‧‧range boundary

Steps of S610 ~ S640‧‧‧ Object Tracking Method

FIG. 1 is a schematic diagram of an object tracking device according to an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating an operation manner of a programmable light source according to an embodiment of the present invention. FIG. 3 is a schematic diagram of an object tracking device according to another embodiment of the present invention. FIG. 4A to FIG. 4E are schematic diagrams illustrating a lighting state adjustment operation of a programmable light source according to different embodiments of the present invention. FIG. 5 is a schematic diagram of a head-mounted display device according to an embodiment of the invention. FIG. 6 is a flowchart of an object tracking method according to an embodiment of the present invention.

Claims (14)

An object tracking device includes: a lens that generates and projects a detection beam onto the object; a beam splitting element that is arranged on an axis with the lens; a programmable light source having a plurality of sub-light sources for the sub-light sources To project a plurality of light beams to a plurality of different positions of the spectroscopic element, the programmable light source receives a driving signal, and adjusts the lighting state of each of the sub-light sources according to the driving signal; and an image capture device, Capturing a detection image of the object, wherein the spectroscopic element receives at least one of the light beams, generates at least one reflected light beam to the lens, and causes the lens to generate the detection light beam. The object tracking device according to item 1 of the patent application scope further includes: a controller coupled to the programmable light source and the image capture device, generating the driving signal within a detection time interval, and receiving the driving signal The detection image, the controller determines the position information of the object according to the detection image. The object tracking device according to item 2 of the scope of patent application, wherein the sub-light sources include a plurality of first sub-light sources and a plurality of second sub-light sources, and the first sub-light sources are respectively used to provide a plurality of first light beams to A central region of the beam splitting element, and the second sub-light sources are respectively used to provide a plurality of second light beams to a surrounding region of the beam splitting element. The object tracking device according to item 3 of the scope of patent application, wherein the controller causes at least one of the first sub-light sources to be lit in a first time interval according to the driving signal, so that the first At least one of the sub-light sources is lit in a second time interval. The object tracking device according to item 4 of the scope of patent application, wherein the first time interval and the second time interval do not overlap, partially overlap, or completely overlap. The object tracking device according to item 2 of the scope of patent application, wherein the controller generates the driving signals in a plurality of time intervals according to a plurality of preset graphics, respectively. The object tracking device according to item 1 of the scope of patent application, wherein the sub-light sources are infrared light sources. The object tracking device according to item 7 of the patent application scope, wherein the image capturing device is an infrared camera. A head-mounted display device includes: a display; and the object tracking device according to item 1 of the scope of patent application, wherein the display is disposed on the axis and projects a display image to the lens. An object tracking method includes: providing a lens and a spectroscopic element to be arranged on the same axis; providing a programmable light source having a plurality of sub-light sources, wherein the sub-light sources are used to respectively project a plurality of light beams to the spectroscopic element And providing a driving signal to adjust the lighting state of each of the sub-light sources; causing the spectroscopic element to receive at least one of the light beams, and generating at least one reflected light beam to the lens, so that the lens Generating a detection beam and projecting the detection beam onto the object; and providing an image capturer to capture a detection image of the object. The object tracking method according to item 10 of the scope of patent application, further comprising: providing a controller to generate the driving signal in a detection time interval; and causing the controller to receive the detection image in the detection time interval, The location information of the object is determined based on the detected image. The object tracking method according to item 10 of the scope of patent application, further comprising: distinguishing the sub-light sources into a plurality of first sub-light sources and a plurality of second sub-light sources, wherein the first sub-light sources are respectively used to provide a plurality of sub-light sources. The first light beam reaches a central area of the light splitting element, and the second sub-light sources are respectively used to provide a plurality of second light beams to a surrounding area of the light splitting element. The object tracking method according to item 10 of the patent application scope, wherein the step of providing the driving signal to adjust the lighting state of each of the sub-light sources includes: making at least one of the first sub-light sources at a first time The interval is illuminated, and at least one of the first sub-light sources is illuminated during a second time interval. The object tracking method according to item 13 of the scope of patent application, wherein the first time interval and the second time interval do not overlap, partially overlap, or completely overlap.