KR20230167578A - Target object adaptive depth variable augmented reality apparatus and method - Google Patents

Abstract

A target object adaptive depth variable augmented reality device and method are provided. An augmented reality device according to an embodiment of the present invention includes a display that generates a virtual image, an optical system that combines a virtual image generated by the display and an external image incident in the real world, a depth sensor that generates a depth image for the external image, and It includes a processor that determines the depth of the object of interest from the depth image generated by the depth sensor and adaptively adjusts the depth plane of the virtual image to the determined depth. As a result, it is possible to provide a virtual image with a depth that matches the object of interest observed by the user, thereby relieving the user's visual fatigue and motion sickness caused by convergence-focus mismatch.

Description

Target object adaptive depth variable augmented reality apparatus and method}

The present invention relates to augmented reality technology, and more specifically to an augmented reality image providing device and method worn by a user on the face.

Existing augmented reality image providing devices only provide virtual images at a fixed depth, for example, 2m or 3m in front, or provide two limited virtual depth planes.

At this time, a user who recognizes the depth of the augmented content may perceive the distance to the gaze convergence point according to binocular disparity and the distance to the focal plane where the focus of the actual image is provided differently.

In this case, if the user uses the augmented reality image providing device for a long period of time, visual fatigue and motion sickness may occur due to conflict between perceived distances, and a solution to this problem is needed.

The present invention was created to solve the above problems, and the purpose of the present invention is to provide an object adaptive depth variable augmented reality device that can provide a virtual image with a depth matched to the object of interest observed by the user, and It provides a method.

An augmented reality device according to an embodiment of the present invention for achieving the above object includes a display that generates a virtual image; An optical system that combines a virtual image generated by a display with an external image incident in the real world; A depth sensor that generates a depth image for an external image; and a processor that determines the depth of the object of interest from the depth image generated by the depth sensor and adaptively adjusts the depth plane of the virtual image to the determined depth.

The depth of field of the virtual image can be adjusted by adjusting the separation distance between the display and the optical system.

The augmented reality device according to an embodiment of the present invention further includes a position adjustment unit that changes the position of the display in order to change the separation distance between the display and the optical system, and the processor determines the depth plane of the virtual image at the determined depth. The positioning unit can be controlled to be positioned.

The separation distance between the display and the optical system may be proportional to the depth of the identified object of interest.

The augmented reality device according to an embodiment of the present invention further includes an optical element provided between the display and the optical system, and the separation distance between the display and the optical system is the physical distance between the display and the optical system and the optical separation adjusted by the optical element. It can be determined by distance.

The depth plane of the virtual image may be non-planar. The depth of the virtual image can be adjusted separately for each pixel or pixel group of the virtual image.

The optical element may be an element that distinguishes and changes the refractive index distribution between the display and the optical system.

The processor may extract an object of interest from the depth image, determine the depth of the extracted object of interest, and determine the depth plane of the virtual image based on the identified depth.

Meanwhile, a method of providing an augmented reality image according to another embodiment of the present invention includes the steps of: a display generating a virtual image; A step where the optical system combines a virtual image generated from a display and an external image incident from the real world; A depth sensor generating a depth image for an external image; A step of determining, by a processor, the depth of an object of interest from a depth image generated by a depth sensor; It includes a step of the processor adaptively adjusting the depth plane of the virtual image to the identified depth.

Meanwhile, an augmented reality imaging device according to another embodiment of the present invention includes an optical system that combines a virtual image and an external image incident in the real world; and a processor that determines the depth of the object of interest from the depth image of the external image of the real world and adaptively adjusts the depth plane of the virtual image to the determined depth.

Meanwhile, a method for providing an augmented reality image according to another embodiment of the present invention includes the steps of combining an optical system with a virtual image and an external image incident from the real world; A step of determining, by a processor, the depth of an object of interest from a depth image of an external image of the real world; and, by the processor, adaptively adjusting the depth plane of the virtual image to the identified depth.

As described above, according to embodiments of the present invention, it is possible to provide a virtual image with a depth of field matched to the object of interest observed by the user, thereby relieving the user's visual fatigue and motion sickness caused by convergence-focus mismatch. It becomes possible.

1 is a diagram provided to illustrate convergence-focus mismatch;
Figure 2 is a diagram showing the configuration of an augmented reality device according to an embodiment of the present invention;
Figure 3 is a diagram showing the structure of an augmented reality device according to an embodiment of the present invention for adjusting the depth of field of a virtual image;
Figure 4 is a diagram showing the depth of field of a virtual image when the object of interest is nearby;
Figure 5 is a diagram showing the depth of field of a virtual image when the object of interest is far away;
Figure 6 is a diagram showing the configuration of an augmented reality device according to another embodiment of the present invention;
FIG. 7 is a diagram provided for a detailed description of the optical path length adjusting unit shown in FIG. 6.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

Conventional image provision devices for providing augmented reality mainly adopt a binocular parallax-based method that provides virtual images in a fixed depth plane (2m or 3m in front) and enforces a sense of depth by using only the image difference between both eyes. As a result, as shown in Figure 1, a difference occurs between the focal distance (Accommodation Distance) of the virtual image on which the actual display is imaged and the convergence distance (Vergence Distance) that varies due to binocular parallax, causing the user to experience convergence-focus mismatch. has caused visual fatigue and motion sickness.

Accordingly, an embodiment of the present invention presents an adaptive augmented reality device and method that can change the expression depth of a virtual image adaptively to the target object observed by the user.

In the augmented reality device and method according to an embodiment of the present invention, the distance of the depth of field for which an image is formed is relatively adjusted by physically or optically varying the separation distance between the display and the augmented reality optical element, and thus the distance to the object of interest that is the object of observation. Adaptively synchronizes the depth of field of the augmented reality image and provides it.

Figure 2 is a diagram showing the configuration of an augmented reality device according to an embodiment of the present invention. As shown, the augmented reality device according to an embodiment of the present invention includes a depth sensor 110, a processor 120, a position adjuster 130, a display 140, and an optical system 150.

The depth sensor 110 generates a depth image for an external image incident in the real world. The display 140 generates/displays a virtual image for providing augmented reality. The optical system 150 combines the virtual image generated by the display 140 and the external image incident in the real world and provides the combined image to the user.

The position adjusting unit 130 is configured to change the position of the display 140, and changes the separation distance between the display 140 and the optical system 150 by changing the position.

The processor 120 preprocesses the depth image generated by the depth sensor 110, extracts an object of interest from the depth image, and determines the depth of the extracted object of interest. It is possible to extract the object of interest by determining it to be an object located in the center among multiple objects present in the depth image, but it is also possible to apply other extraction methods.

Then, the processor 120 adjusts the depth plane of the virtual image by adaptively controlling the position adjuster 130 according to the depth of the identified object of interest.

Through control of the position adjustment unit 130 by the processor 120, the depth plane of the virtual image generated by the display 140 is synchronized to the depth of the object of interest.

Figure 3 is a diagram showing the structure of an augmented reality device according to an embodiment of the present invention for adjusting the depth of field of a virtual image. As shown, the depth plane of the virtual image can be adjusted to any one of depth plane #1, depth plane #2, ..., depth plane #N.

As shown in FIG. 3, the optical system 150 includes a lens 151 for imaging a virtual image generated on the display 140, and an AR optical element 152 that combines the virtual image and the external image and transmits them to the user's pupil. It is composed including.

In an embodiment of the present invention, the depth plane in which the virtual image is formed is adjusted by adjusting the separation distance between the display 140 and the optical system 150. As the separation distance becomes longer, the depth plane of the virtual image becomes deeper, and the separation distance increases. The shorter it gets, the shallower the depth of field of the virtual image becomes.

Figure 4 illustrates the case where the object of interest is object of interest #1 (10) in the short distance. Since the depth of object of interest #1 (10) is not deep (the distance from the user to object of interest #1 (10) is not long) ) The depth plane in which the virtual image will be formed should not be as deep as the depth of object of interest #1 (10) (the distance from the user to the depth plane of the virtual image should not be long).

In order to keep the depth of field of the virtual image from being deep, the separation distance between the display 140 and the optical system 150 must be shortened.

Meanwhile, Figure 5 illustrates the case where the object of interest is object of interest #2 (20) located at a distance. Since the depth of object of interest #2 (10) is deep (the distance from the user to object of interest #2 (20) is long) Therefore, the depth plane in which the virtual image will be formed must be as deep as the depth of object of interest #2 (20) (the distance from the user to the depth plane of the virtual image must be long).

In order to deepen the depth of field of the virtual image, the separation distance between the display 140 and the optical system 150 must be adjusted to be long.

In this way, the separation distance between the display 140 and the optical system 150 is controlled to be proportional to the depth of the object of interest, so that the depth plane of the object of interest and the virtual image are synchronized.

Figure 6 is a diagram showing the configuration of an augmented reality device according to another embodiment of the present invention. Unlike the device shown in FIG. 2 in which the separation distance between the display and the optical system is physically varied, the augmented reality device according to an embodiment of the present invention optically varies the separation distance to adapt the depth plane of the virtual image to the distance to the object of interest. synchronize.

As shown, the augmented reality device according to an embodiment of the present invention that performs this function includes a depth sensor 210, a processor 220, a display 230, an optical path length adjuster 240, and an optical system 250. It consists of:

The depth sensor 210 generates a depth image for an external image incident in the real world.

The display 230 generates/displays a virtual image for providing augmented reality, but the virtual image is different from the above-described embodiment in that it is not in a flat shape but in a non-flat shape. The non-planar virtual image generated by display 230 is determined by processor 220.

The optical system 250 combines the virtual image generated by the display 230 and the external image incident in the real world and provides the combined image to the user.

The optical path length adjusting unit 240 is located between the display 230 and the optical system 250, and adjusts the optical path length, that is, the optical separation distance, between the display 230 and the optical system 250 for each pixel or pixel group of the virtual image. It is an element that is classified and adjusted according to each type.

It is different from the above-described embodiment in that the optical separation distance between the display 230 and the optical system 250 is adjusted rather than the physical distance. That is, the positions of the display 230 and the optical system 250 are fixed, and the optical path length adjustment unit 240 located between them locally adjusts the optical path length distribution.

As shown in FIG. 7, the optical path length adjusting unit 240 is a non-homogeneous optical element that adjusts the optical path length by changing the refractive index distribution between the display 230 and the optical system 250 for each pixel or pixel group. It can be implemented as a liquid crystal modulation array element (eg, LCoS spatial light modulator, etc.), EW (Electro-Wetting) lens array element, or liquid type lens array element.

Since the distance felt by light rays is made up of the product of the physical separation distance and the refractive index of the medium within the separation distance, the refractive index distribution between the display 230 and the optical system 250 is locally varied to have different optical path lengths for each pixel. It was done.

The depth plane of the virtual image formed by the optical path length adjusting unit 240 is not a plane perpendicular to the observer's line of sight, but has a locally different depth plane in accordance with the depth distribution of the virtual image to be expressed with the object of interest. . It has the advantage of being able to implement a continuous virtual depth space in a single image frame, avoiding the problem of representing a finite virtual depth space, which is a limitation in the above-described embodiment.

In FIG. 7 , light rays departing from the display 230 may have different optical path distances before reaching the optical system 250. For example, the light ray emitted from the left side of the display 230 passes through a portion of the optical path length adjusting unit 240 with a low refractive index (n1), and the ray emitted from the center passes through a portion with a medium refractive index (n2). , the ray emitted from the right passes through a part with a high refractive index (n3). Therefore, although the physical separation distance is the same, the relative distance felt by each ray is different, so the imaging depth by the optical system 250 is different, and the depth of the virtual image is a non-planar and continuous virtual image that can vary for each pixel or pixel group. It is possible to provide depth of field.

After preprocessing the depth image generated by the depth sensor 210, the processor 220 extracts a number of objects of interest from the depth image and determines the depths of the extracted objects of interest.

And the processor 220 uses the display 230 and the optical path length adjustment unit ( 240).

Through control of the display 230 and the optical path length adjusting unit 240 by the processor 220, the depth plane of the virtual image generated by the display 230 is synchronized to the depth of the objects of interest.

So far, the target object adaptive depth variable augmented reality device and method have been described in detail with preferred embodiments.

In the above embodiment, the user's visual fatigue and motion sickness caused by convergence-focus mismatch were alleviated by providing a virtual image with a depth matched to the object of interest observed by the user through the provision of target object adaptive depth variable augmented reality.

In addition, although preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the invention pertains without departing from the gist of the present invention as claimed in the claims. Of course, various modifications can be made by those of ordinary skill in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

110,210: Depth sensor
120,220: Processor
130: Position adjustment unit
140,230: Display
240: Optical path length adjustment unit
150,250: Optics

Claims (12)

A display that generates a virtual image;
An optical system that combines a virtual image generated by a display with an external image incident in the real world;
A depth sensor that generates a depth image for an external image; and
An augmented reality device comprising a processor that determines the depth of an object of interest from a depth image generated by a depth sensor and adaptively adjusts the depth plane of the virtual image to the determined depth.
In claim 1,
The depth of the virtual image is,
An augmented reality device characterized in that it is adjusted by adjusting the separation distance between the display and the optical system.
In claim 2,
It further includes a position adjustment unit that changes the position of the display to change the separation distance between the display and the optical system,
The processor is,
An augmented reality device characterized in that it controls the position adjustment unit so that the depth plane of the virtual image is located at the determined depth.
In claim 3,
The separation distance between the display and the optical system is,
An augmented reality device characterized in that it is proportional to the depth of the identified object of interest.
In claim 2,
It further includes an optical element provided between the display and the optical system,
The separation distance between the display and the optical system is,
An augmented reality device characterized by being determined by the physical distance between the display and the optical system and the optical separation distance adjusted by the optical element.
In claim 5,
The depth of the virtual image is,
An augmented reality device characterized by being non-flat.
In claim 6,
The depth of the virtual image is,
An augmented reality device characterized in that the virtual image is divided and adjusted by pixel or pixel group.
In claim 7,
The optical element is,
An augmented reality device characterized by an element that distinguishes and changes the refractive index distribution between the display and the optical system.
In claim 1,
The processor is,
An augmented reality device that extracts an object of interest from a depth image, determines the depth of the extracted object of interest, and determines the depth plane of the virtual image with the identified depth.
A display comprising: generating a virtual image;
A step where the optical system combines a virtual image generated from a display and an external image incident from the real world;
A depth sensor generating a depth image for an external image;
A step of determining, by a processor, the depth of an object of interest from a depth image generated by a depth sensor;
A method of providing an augmented reality image comprising: adjusting, by the processor, the depth plane of the virtual image adaptively to the identified depth.
An optical system that combines virtual images with external images incident in the real world; and
An augmented reality device comprising a processor that determines the depth of an object of interest from a depth image of an external image of the real world and adaptively adjusts the depth plane of the virtual image to the determined depth.
A step where the optical system combines a virtual image and an external image incident from the real world;
A step of determining, by a processor, the depth of an object of interest from a depth image of an external image of the real world; and
A method of providing an augmented reality image comprising: adjusting, by the processor, the depth plane of the virtual image adaptively to the identified depth.