KR102465800B1 - See-through optics system applied to near eye display device and eye tracking method in see-through optics system - Google Patents

Abstract

A gaze tracking method in a see-through optical system and a near-eye image display device using the same are disclosed. The see-through optical system for matching a virtual image and a real image includes a first reflector that transmits a first image, which is a part of the real image, and changes an optical path of a second image, which is a remaining part; a second reflector for re-changing the optical path of the second image; an image collecting unit in which the second image is collected; a controller that generates the virtual image based on the second image and emits the virtual image from a display panel; an enlargement lens unit for enlarging the virtual image emitted from the display panel; and a third reflector for allowing the virtual image magnified through the magnification lens to be formed on the user's retina, wherein the control unit tracks the user's gaze other than the second image, and based on the tracked gaze, the third reflector. create a virtual image As a result, a component that serves to form the image light of the virtual image on the user's retina and a component that serves to collect the image light of the user's line of sight image are provided separately, or a camera is not separately mounted. In the see-through optical system By allowing the structural design to be integrated in, it is possible to reduce the weight, increase convenience, and prevent the reliability of the product from being deteriorated through the integration of redundant parts.

Description

See-through optics system applied to near eye display device and eye tracking method in see-through optics system

The present invention relates to a see-through optical system applied to a near-eye image display device and a gaze tracking method in the see-through optical system, and more particularly, to VR (Virtual Reality), AR (Augmented Reality) and MR (Mixed Reality) Reality: A see-through optical system for mixed reality and an eye tracking method in the see-through optical system.

In the near-eye image display device for augmented/mixed reality, the see-through optical system allows image rays emitted from displays such as LCD (LCoS), OLED, LED, and laser modules to pass through the designed optical system lens according to the configured optical lens configuration, so that the user's eyes By delivering it to the user, it is possible to watch the video.

At this time, the eyes of the user change in various directions while viewing image information desired by the user or a see-through (see-through) front object. These gazes can be divided into changes in the direction of the gaze and changes in the visual angle when viewed with both eyes as a standard, and tracking the user's gaze by grasping the direction of the gaze or change in the visual angle is VR, It becomes a very important factor in AR and MR services.

In other words, eye tracking not only achieves the secondary purpose of reducing dizziness or motion sickness occurring in the service, but also achieves its original purpose of selecting video content corresponding to the user's gaze and providing it to the user. It is also a matter of necessity.

However, existing services not only add weight by separately mounting a camera for eye tracking, but also seriously reduce convenience, and have problems in that reliability may be deformed due to a lack of integral structural design.

Korean Patent Publication No. 10-2020-0023992 (smart tracker and AR display system for surgery)

The present invention has been made to solve the above problems, and an object of the present invention is to provide an image based on information on the tracked gaze, as well as to enable gaze tracking in a near-eye image display device. It is to provide a see-through optical system applied to a near-eye image display device that allows the see-through optical system to interlock and operate according to the tracked gaze and a gaze tracking method in the see-through optical system.

In order to achieve the above object, a see-through optical system for matching a virtual image and a real image according to an embodiment of the present invention transmits a first image, which is a part of the real image, and changes an optical path of a second image, which is a remaining part. a first reflector; a second reflector for re-changing the optical path of the second image; an image collecting unit in which the second image is collected; a controller that generates the virtual image based on the second image and emits the virtual image from a display panel; an enlargement lens unit for enlarging the virtual image emitted from the display panel; and a third reflector for allowing the virtual image magnified through the magnification lens to be formed on the user's retina, wherein the control unit tracks the user's gaze other than the second image, and based on the tracked gaze, the third reflector. create a virtual image

Here, the first reflector transmits a third image, which is a part of the user's gaze image, and changes an optical path of a fourth image, which is a remaining part, and the third reflector and the second reflector are configured to transmit an optical path of the fourth image. By re-changing, the gaze image may be collected by the image collection unit.

Here, the control unit may track the user's gaze based on the gaze image collected by the image collection unit, and track the user's gaze direction based on the coordinates of the pupil in the eyeball in the gaze image.

Here, the control unit compares the left eye gaze image collected by the left eye image collection unit and the right eye gaze image collected by the right eye image collection unit, analyzes the difference in distance between the pupils in the eyeball, and based on the difference in distance between the pupils in the eyeball. It is possible to track the user's time of day.

Here, the controller determines a subject that the user gazes at based on the tracked gaze direction and the tracked visual angle, and controls the display panel to display a virtual image corresponding to the determined subject on the display panel. can do.

The control unit controls the hinge unit based on the tracked gaze direction and the tracked visual angle, and the hinge unit moves under the control of the control unit so that the see-through optical system rotates and translates at the same time. , the see-through optical system may be located in the direction of the user's line of sight.

In addition, the first reflector, the second reflector, and the third reflector change the optical path of the second image and transmit the image to the image collection unit, and change the optical path of the fourth image. It can also perform a role of transmitting to the image collection unit.

The image collection unit may separate and extract the fourth image from the received image based on a preset sample gaze image, and extract the second image by excluding the fourth image portion from the received image.

In addition, the see-through optical system may be mounted on a head-mounted display device and used for medical purposes.

On the other hand, in order to achieve the above object, according to an embodiment of the present invention, a gaze tracking method in a see-through optical system for matching a virtual image and a real image transmits a first image, which is a part of the real image, and a second image, which is a remaining part. changing an optical path of an image; re-changing the optical path of the second image; collecting the second image; generating the virtual image based on the second image; allowing the virtual image to be emitted from a display panel; enlarging the projected virtual image; and changing a path of the virtual image so that the enlarged virtual image is formed on the user's retina, wherein the generating of the virtual image includes tracking the user's gaze other than the second image, and tracking the tracked gaze. The virtual image may be generated based on.

As a result, it is not necessary to separately prepare a component that serves to form the image light of the virtual image on the retina of the user and a component that serves to collect the image light of the user's line of sight image. By enabling the structural design to be integrated in the see-through optical system without any problems, it is possible to reduce weight, increase convenience, and prevent product reliability from being altered through the integration of redundant parts.

1 is a diagram illustrating an example of a near eye image display device according to an embodiment of the present invention.
2 is a view showing a see-through optical system found on the bottom of the near-eye image display device of the present invention.
3 is a diagram provided to explain the internal configuration of a see-through optical system.
4 is a diagram provided to explain a process in which gaze images are collected by an image collection unit through a see-through optical system.
5 is a diagram provided to explain the vergence angle and the IPD in a state before the gaze is changed.
6 is a diagram provided to explain the vergence angle and the IPD in a state after the gaze is changed.
7 is a diagram illustrating a state in which a see-through optical system is moved as a line of sight is changed.
8 is a view provided to explain a structure for allowing a see-through optical system to move as a line of sight is changed.
9 is a flowchart provided to explain a gaze tracking method in a see-through optical system according to an embodiment of the present invention.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

FIG. 1 is a diagram showing an example of a near eye image display device according to an embodiment of the present invention, and FIG. 2 is a diagram showing a see-through optical system viewed from the bottom of the near eye image display device of the present invention.

1 and 2 show a head-mounted display device as an example of a near-eye image display device. In the case of such a head-mounted display device, a real image is displayed through a see-through optical system inside the head-mounted display device by being mounted on the head by a user. And the virtual image is provided to the user.

As shown in FIG. 2, it can be seen through the lower surface of the head mounted display device that the see-through optical system 100 is designed to be positioned in front of both eyes of the user, and the see-through optical system 100 is The right eye is provided (100-1, 100-2), respectively.

FIG. 3 is a diagram provided to explain the internal configuration of the see-through optical system 100. Referring to FIG.

The see-through optical system 100 according to an embodiment of the present invention refers to a device that generates a virtual image so that a real image viewed by a user and the generated virtual image are matched.

The see-through optical system 100 according to the present embodiment does not block the user's field of view so that the front object (S: subject) that the user is focusing on can be observed, and related to the observed object (S: subject) By generating and outputting a virtual image, it is possible to provide augmented reality to the user.

For this purpose, the see-through optical system 100 according to the present embodiment includes a first reflector 110, a second reflector 120, an image collector 130, a controller 140, a display panel 150, an enlargement It is composed of a lens unit 160, a third reflector 170, and a polarizing film 180.

The first reflector 110 is a polyhedron formed of a glass material or a plastic material having a predetermined transmittance, and partially transmits or partially reflects image light of a real image incident to the first reflector 110 . Accordingly, the image light of the object (S: subject) in the front passes through the first reflector 110 and forms an image on the user's retina.

In particular, some light is transmitted and some light is reflected on the inclined surface of the first reflector 110. In order to enable such partial transmission and partial reflection, the inclined surface distinguishes or Appropriate filters (band pass filter, polarization filter) may be provided to improve visibility.

Like the first reflector 110, the second reflector 120 is formed of a glass material or a plastic material having a predetermined transmittance, and the image light of the real image reflected by the first reflector 110 and incident Partly transmitted or partly reflected.

That is, image light of a real image reflected by the first reflector 110 is reflected by the second reflector 120 and transmitted to the image collector 130 .

Also, in the second reflector 120 , image light of a user's gaze image incident through the first reflector 110 is reflected by the second reflector 120 and transmitted to the image collection unit 130 .

That is, the image collecting unit 130 can collect both the image light of the real image and the image light of the gaze image. The image light of the real image is used to select the contents of the virtual image, and the image light of the gaze image is generated by multiple real images according to the user's gaze direction and visual angle. It is used to enable the content of the corresponding virtual image to be provided.

The image light of the real image and the image light of the line-of-sight image collected together in this way are used after being distinguished/separated from each other by the image collecting unit 130. In particular, the image collecting unit 130 receives based on a preset sample line-of-sight image. Image light of a real image can be obtained by separating and extracting only the gaze image from the received image and excluding the separately extracted gaze image portion from the received image.

In particular, some light is transmitted and some light is reflected on the inclined surface of the second reflector 120. In order to enable such partial transmission and partial reflection, the inclined surface distinguishes or Appropriate filters (band pass filter, polarization filter) may be provided to improve visibility.

The image collection unit 130 is composed of a module capable of collecting images, such as a camera, and serves to collect information about real images and information about the user's gaze, as described above.

In particular, when the image of the user's gaze passes through the first reflector 110 and is reflected and received by the second reflector 120, the image collection unit 130 transmits the image of the user's gaze through the image. When using a monocular as a reference, the direction of the pupil in the user's eyeball is biased with respect to the center, that is, the direction of the line of sight is determined.

For example, when the user looks in the direction of the sky, the pupil will be biased in the northward direction, and the image collection unit 130 tracks the direction of the user's gaze by collecting information indicating that the pupil has moved in the northward direction. do.

Meanwhile, the image collection unit 130 also determines how narrowed the distance between the pupils of both eyes of the user is based on the image of the user's eyes.

For example, as the distance between the user's pupils becomes narrower, the visual angle becomes larger, and it is possible to determine that the user is looking at a closer subject in a situation where two subjects overlap. Through calculation of the viewing angle, it is possible to determine which subject is being watched even when two or more subjects overlap.

In this way, the image collecting unit 130 serves to collect images of the gaze so that the direction of the gaze and the visual angle can be determined.

The control unit 140 serves to receive an image of a front object from the image collection unit 130 in real time and to generate a virtual image corresponding to the image based on the received image.

In addition, the control unit 140 receives images for both eyes of the user from the image collection unit 130 in real time, calculates the direction of gaze and visual angle from the received images for both eyes, and based on the calculated results It tracks the gaze of the user and serves to provide a virtual image matching the gaze.

Then, the control unit 140 transmits the image light of the virtual image to the display panel 150 so that the image light of the virtual image generated above is emitted from the display panel 150 .

The display panel 150 is used to receive and emit image light of a virtual image from the controller 140, such as a liquid crystal display (LCD), organic light emitting diodes (OLED), liquid crystal on display (LCoS), and the like. It may be implemented in various devices or elements.

The magnifying lens unit 160 is used for the purpose of magnifying image light of a virtual image emitted from the small-sized display panel 150 .

When the display panel 150 is provided in a small size, image light of a virtual image emitted from the display panel 150 is smaller in size than image light of a real image viewed by a user.

Therefore, the magnification lens unit 160 is provided so that the image light of the emitted virtual image is magnified to a preset angle of view and provided to the user.

To this end, the magnifying lens unit 160, as shown, may be formed of two or more lens groups, and each lens forming the lens group is a single-sided concave lens, single-sided convex lens, double-sided concave lens, double-convex lens, or the like. It can be done.

The image light of the virtual image magnified by the magnification lens unit 160 is incident to the third reflector.

The third reflector 170 is formed of a glass material or plastic material having a preset reflectance, and the light emitted from the display panel 150 passes through the magnifying lens unit 160 and then enters the third reflector 170. Image light of all virtual images is reflected along a predetermined path.

The image light of the virtual image reflected by the third reflector 170 is re-reflected by the first reflector 110 to form an image on the user's retina.

The polarizing film 180 is used for the purpose of preventing light irradiated from a light source from entering the image collecting unit 130, and for this purpose, the polarizing film 180 is applied to the front surface of the light source (not shown), the image collecting unit 130 ), and between the second reflector 120 and the magnifying lens unit 160.

At this time, in order to prevent the light irradiated from the light source (not shown) from entering the image collecting unit 130, the polarizing film 180 and the front surface of the image collecting unit 130 are provided on the front of the light source (not shown). The polarization film 180 provided on is implemented to absorb different types of polarization components.

Specifically, when one is a P-wave polarizing film, the other may be an S-wave polarizing film, or vice versa.

4 is a diagram provided to explain a process in which gaze images are collected by the image collecting unit 130 through the see-through optical system 100.

As described above, the first reflector 110 is formed of a glass or plastic material having a predetermined transmittance. At this time, the predetermined transmittance is a light reflectance of 50% for partially transmitting or partially reflecting image light incident to the first reflector 110, and the first reflector 110 has a light reflectance of 50%. surface treatment.

The first reflector 110 has a preset transmittance so that a real image of a front object (subject: S) is provided to the user's retina, but also has a reflectance reverse to the transmittance.

Depending on the combination of reflectance and transmittance, the image light of the line of sight image is reflected from the first reflector 110, moves to the third reflector 170 along the path of R10, and is reflected from the third reflector 170 again. It travels the path of R10 in reverse and is transmitted through the first reflector 110 and moves to the magnifying lens unit 160 along the path of R20.

Thereafter, the image light of the user's line of sight image is reflected by the second reflector 120 and transmitted to the image collection unit 130 .

Through this process, the image collection unit 130 can determine the direction of the user's gaze by determining the location of the pupil in the collected image.

In addition, since the see-through optical system 100 is provided for both eyes, information on the left eye gaze image and information on the right eye gaze image collected from the image collection unit 130 provided in each see-through optical system 100 are collected to obtain It is possible to calculate the user's visual angle.

The image collection unit 130 transmits the image light of the left eye gaze image and the image light of the right eye gaze image reached above to the controller 140 . Specifically, the left eye image collection unit 130-1 collects the left eye gaze images and transmits them to the controller 140, and the right eye image collection unit 130-2 collects the right eye gaze images and transmits them to the controller 140. is to do

The controller 140 generates a virtual image to be provided to the user based on the left eye gaze image and the right eye gaze image received from the image collection unit 130 . Of course, the control unit 140 is based on the real image received from the image collection unit 130. That is, a virtual image corresponding to the subject is generated by determining whether an object located at a certain viewing angle in a real image is a subject actually observed by the user.

Then, the controller 140 transmits the generated virtual image to the display panel 150 .

The display panel 150 emits image light of the virtual image so that the image light of the virtual image received from the controller 140 can reach the user.

The emitted image light is partially transmitted from the second reflector 120 and passes through the magnifying lens unit 160, and is magnified to a preset angle of view while passing through the magnifying lens unit 160. Through this, as described above, As described above, image light of a virtual image emitted in a small size from the display panel 150 may be realized to have the same size as image light of a real image viewed by a user.

The image light of the enlarged virtual image passes through the first reflector 110 and reaches the third reflector 170 .

Since the third reflector 170 is a condensing lens having a light reflectance of 100%, the image light of the virtual image that has passed through the first reflector 110 is transmitted to the first reflector 110 at a preset angle of view. will reflect again.

Then, the image light of the virtual image reflected back to the first reflector 110 is reflected once again toward the user's retina from the first reflector 100, and finally forms an image on the user's retina.

In this way, the first reflector 110, the second reflector 120, and the third reflector 170 not only serve to form image light of the virtual image on the user's retina, but also provide a visual image of the user. It serves to collect image light in the image collecting unit 130 in parallel, and allows structural design to be integrated within the see-through optical system 100 without separately mounting a camera, thereby integrating redundant parts. It is possible to reduce weight, increase convenience, and prevent product reliability from being altered.

That is, when a camera is mounted separately to track the user's gaze, since the front of the user's gaze must always be transparent in order to gaze at the subject, various configurations can be accompanied to change the optical path. There is no choice but to use these configurations in a general see-through optical system that does not track the user's gaze. By unifying the configurations for providing virtual images in the see-through optical system and the configurations for tracking gaze, Rather, a more optimized design is possible compared to the case of preparing, and it is possible to improve weight/convenience/reliability.

On the other hand, the above configurations used in the see-through optical system must be changed according to the user's line of sight, and in the past, a method such as a user directly adjusting the distance between the see-through optical systems has been used.

Hereinafter, when a virtual image is formed on the user's retina, a method for more accurately and effectively delivering a virtual image by automatically adjusting a see-through optical system based on information on the tracked gaze will be described.

5 is a diagram provided to explain the vergence angle and IPD in a state before the gaze is changed, and FIG. 6 is a diagram provided to explain the vergence angle and IPD in a state after the gaze is changed. This is a view showing how the see-through optical system moves as the line of sight changes, and FIG. 8 is a view provided to explain a structure for moving the see-through optical system as the line of sight changes.

When the user's gaze gazes at the subject S, when the viewing angle is θ1 °, the angle between the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye is 180-θ1 ° becomes θ2°. At this time, when the inter pupillary distance (IPD) of the user is d1, the distance between both eyes of the user and the subject S is d2 proportional to d1.

In this situation, when the user's gaze moves to a closer subject, the user's pupils are more converged, and accordingly, the IPD is further reduced to d3 as shown in FIG. 6 .

That is, as the distance to the subject decreases to d4, the IPD also decreases to d3 correspondingly. The decrease in the distance between the IPD and the subject in this way means that the viewing angle has increased to θ3 °, and accordingly, the see-through optical system for the left eye (100-1) and the see-through optical system for the right eye (100-2) This means that the angle of the liver should also be changed to 180-θ3°.

That is, the angle θ4° between the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye further decreases when the user's gaze moves to a closer subject.

In this way, in order to reduce the angle between the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye, the see-through optical system 100-1 for the left eye and the see-through optical system 100-1 for the right eye are It is necessary to rotate the see-through optical system 100-2 in opposite directions to each other.

That is, the image collection unit 130 tracks the user's gaze through the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye, thereby determining the user's visual angle along with the direction of the user's gaze. And, by rotating the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye according to the determined viewing angle, the user's left and right eyes are adjusted to the direction in which the user gazes.

At this time, simply by rotating the see-through optical system for the left eye 100-1 and the see-through optical system for the right eye 100-2 based on the center, the see-through optical system for the left eye is perpendicular to the direction in which both eyes of the user gaze. 100-1 and the see-through optical system 100-2 for the right eye cannot be positioned.

That is, when the viewing angle increases as the user looks at a closer subject, the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye rotate in opposite directions and at the same time gather each other more. Only then can the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye be positioned on a straight line distance between both eyes of the user and the subject.

To this end, as shown in FIG. 8 , the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye are connected to each other through a hinge part 190 at the center, and the hinge part 190 ) moves left and right, it is possible for the see-through optical system 100-1 for the left eye and the see-through optical system 100-2 for the right eye to move closer to and then move away from each other simultaneously with rotation.

That is, the control unit 140 receives information about the user's both eyes, determines the user's visual angle, and controls the hinge unit 190 to operate according to the visual angle, thereby obtaining see-through optical for the left eye. The system 100-1 and the see-through optical system 100-2 for the right eye can be controlled.

9 is a flowchart for explaining a gaze tracking method in a see-through optical system for matching a virtual image and a real image according to an embodiment of the present invention.

First, in order to track the user's gaze and provide a virtual image corresponding to the user's gaze, the first image, which is part of the real image, is transmitted, and the optical path of the second image, which is the remaining part, is changed.

Then, when the optical path of the second image is changed again, the second image is transmitted to the image collection unit, and the control unit controls the user's gaze based on the second image collected by the image collection unit and the user's gaze image collected by the image collection unit. After tracking the gaze of the user, a virtual image corresponding to the real image matching the gaze of the user is created.

In addition, the controller changes the path of the virtual image so that the virtual image is projected from the display panel so that the virtual image magnified through the magnifying lens unit is formed on the user's retina.

Such a see-through optical system for matching a real image with a virtual image and a gaze tracking method in the see-through optical system can be used in various fields, and can be used in cases where high concentration is required, such as in medical surgery, When used for medical purposes, it may be used in the form of a head-mounted display worn on the user's head to minimize movement.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is common in the art to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those who have knowledge of, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100: see-through optical system
100-1: See-through optical system for the left eye
100-2 : See-through optical system for the right eye
110: first reflector 120: second reflector
130: image collection unit 140: control unit
150: display panel 160: magnifying lens unit
170: third reflector 180: polarizing film
190: hinge part

Claims (10)

In a see-through optical system for matching a virtual image and a real image,
a first reflector that transmits a first image, which is a part of the real image, and changes an optical path of a second image, which is a remaining part;
a second reflector for re-changing the optical path of the second image;
an image collecting unit in which the second image is collected;
a controller that generates the virtual image based on the second image and emits the virtual image from a display panel;
an enlargement lens unit for enlarging the virtual image emitted from the display panel; and
A third reflector configured to image the virtual image magnified through the magnifying lens unit on the user's retina;
The control unit,
In addition to the second image, the user's gaze is tracked and the virtual image is generated based on the tracked gaze;
The first reflector,
Transmitting a third image, which is a part of the user's gaze image, and changing an optical path of a fourth image, which is a remaining part,
The third reflector and the second reflector,
Re-changing the optical path of the fourth image so that the gaze image is collected by the image collection unit;
The control unit,
Tracking the gaze of the user based on the gaze image collected by the image collection unit,
Tracking the direction of the user's gaze based on the coordinates of the pupil in the eyeball in the gaze image;
The control unit,
The left eye gaze image collected from the left eye image collection unit and the right eye gaze image collected from the right eye image collection unit are compared to analyze the distance difference between intraocular pupils, and based on the distance difference between intraocular pupils, the user's visual angle is determined. tracking,
The control unit,
Determine a subject that the user gazes at based on the tracked gaze direction and the tracked visual angle;
controlling the display panel so that a virtual image corresponding to the determined subject is displayed on the display panel;
The control unit,
Control the hinge based on the tracked gaze direction and the tracked gaze angle;
The hinge part,
The see-through optical system of claim 1 , wherein the see-through optical system is moved under the control of the control unit to rotate and translate the see-through optical system so that the see-through optical system is located in a direction of the user's gaze. delete delete delete delete delete According to claim 1,
The first reflector, the second reflector, and the third reflector,
The see-through optical system characterized in that it performs both a role of changing the optical path of the second image and transmitting it to the image collecting unit and a role of changing the optical path of the fourth image and transmitting it to the image collecting unit. According to claim 7,
The video collection unit,
Separating and extracting the fourth image from the received image based on a preset sample gaze image;
and extracting the second image by excluding the fourth image portion from the received image. According to claim 8,
The see-through optical system,
A see-through optical system mounted on a head-mounted display device and used for medical purposes. delete

Images