KR102244445B1 - Apparatus and method for occlusion capable near-eye display for augmented reality using single dmd - Google Patents

Abstract

Disclosed are an occlusion-possible near-eye display apparatus for augmented reality (AR) using a single digital micromirror device (DMD) to provide a short optical path, and a method thereof. According to one embodiment of the present invention, the occlusion-possible near-eye display apparatus comprises: a DMD adjusting an on/off-state of each mirror and implementing partial masking by adjusting mirrors corresponding to parts to display a virtual image to an off-state; and an light emitting diode (LED) displaying the virtual image on the part where the masking is implemented by the DMD. In the DMD, a part in which the mirrors are the on-state reflect a real image, a part in which the mirrors are the off-state reflect the virtual image, and the real image that exists behind the virtual image is covered to display the virtual image so that the occlusion can be implemented.

Description

A near-eye display device and method for occlusion-capable augmented reality using a single digital micromirror device {APPARATUS AND METHOD FOR OCCLUSION CAPABLE NEAR-EYE DISPLAY FOR AUGMENTED REALITY USING SINGLE DMD}

The following embodiments relate to a near-eye display technology for augmented reality, and more particularly, to an apparatus and method for a near-eye display for augmented reality capable of occlusion using a single digital micromirror device.

The near-eye display is a display that provides virtual reality (VR) and augmented reality (AR), and is a device that is worn on the face and the display is close in front of the eyes. Augmented reality is a technology in which virtual images are added to the real environment.

One of the important research topics in near-eye display research for augmented reality is occlusion. When looking at a real object and a virtual object at the same time, if the virtual object is in front of the real object, the real object should be covered, but it cannot be completely covered, so it overlaps, so the sense of reality decreases. As a technique used at this time, occlusion is a technique that covers objects that should be hidden behind.

One of the ways to implement occlusion is to see the actual environment through the display instead of seeing it directly. This is to make the image of the real environment visible through the masking process on the display. In order to implement this, there are cases where two displays are used for masking and virtual image projection, respectively. However, in this case, there is a problem that the size of the entire module is increased and the optical path is lengthened.

Korean Patent Registration No. 10-1919486 discloses a technology related to such a three-dimensional display device for adjusting the full parallax focus.

Korean Patent Registration No. 10-1919486

The embodiments describe a near-eye display device and method for occlusion-capable augmented reality using a single digital micromirror device, and more specifically, a near-eye display technology for augmented reality using a single digital micro-mirror device capable of occlusion. to provide.

In the embodiments, by synchronizing a Digital Micromirror Device (DMD) and a Light-Emitting Diode (LED) to implement masking and virtual object imaging using time multiplexing, the optical system is minimized to reduce the optical path. An object of the present invention is to provide a near-eye display apparatus and method for occlusion-capable augmented reality using a single digital micromirror device that not only has light and small volume, but also minimizes image distortion.

A near-eye display device for occlusion-capable augmented reality using a single digital micromirror device according to an embodiment controls on/off of each mirror, and a mirror corresponding to a portion to display a virtual image A Digital Micromirror Device (DMD) that partially implements masking by adjusting them to off; And an LED (Light-Emitting Diode) that displays a virtual image on a portion where the masking is implemented by the DMD, wherein an actual image is reflected in a portion where a mirror is on in the DMD, and a portion where the mirror is off is the virtual image. The image is reflected, and occlusion may be implemented by covering the real image behind the virtual image in order to display the virtual image.

The LED is arranged in a direction in which the actual image is reflected from the off-state mirror in the DMD, and when the LED is reflected on the off-state mirror, the LED proceeds at the same angle as the actual image, and light enters the human eye. Can be done. In addition, by synchronizing the DMD and the LED, it is possible to implement occlusion using only the DMD and the LED as a single display using time multiplexing.

The actual image is projected onto the DMD using a polarization beam splitter and sent back to the human eye, and the volume of the device can be reduced by using the DMD and the polarizing beam splitter as a single display.

A near-eye display method for occlusion-capable augmented reality using a single digital micromirror device according to another embodiment controls on/off of each mirror of a digital micromirror device (DMD), and displays a virtual image. Partially implementing masking by adjusting the mirrors corresponding to the portion to be floated off; And displaying a virtual image using an LED (Light-Emitting Diode) in the portion where the masking is implemented by the DMD, wherein the actual image is reflected in the portion where the mirror is on, and the mirror is off. In the portion, the virtual image is reflected, and occlusion may be implemented by covering an actual image present behind the virtual image in order to display the virtual image.

Vertically polarized light received from a real scene through a linear polarizer; Reflecting the polarized light through a polarization beam splitter (PBS) and passing through the first 1/4 wave plate to become circularly polarized light; Reflecting the light passing through the first quarter wave plate to the DMD and then passing through the quarter wave plate to be horizontally polarized; Passing the horizontally polarized light passing through the polarizing beam splitter and passing through a second quarter wave plate; After passing through the second quarter wave plate, it is reflected by a mirror, and when it passes through the second quarter wave plate again, it becomes vertically polarized again, and the light is finally reflected by the polarizing beam splitter and finally enters the eye. It may further include.

According to embodiments, masking and virtual object imaging are implemented using time multiplexing by synchronizing a digital micromirror device (DMD) and an LED, thereby minimizing the optical system to reduce the optical path, resulting in light and small volume, as well as minimizing image distortion. It is possible to provide a near-eye display device and method for occlusion-capable augmented reality using a single digital micromirror device.

According to embodiments, it is possible to provide a near-eye display apparatus and method capable of occluding a small, light, and short optical path by using time multiplexing instead of using two displays.

1 is a diagram illustrating a structure of a near-eye display device for augmented reality according to an exemplary embodiment.
2 is a diagram for explaining a principle in which an image is inverted according to an exemplary embodiment.
3 is a diagram for explaining a time multiplexing method according to an embodiment.
4 is a flowchart illustrating a near-eye display method for augmented reality according to an exemplary embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various different forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more completely describe the present invention to those of ordinary skill in the art. In the drawings, the shapes and sizes of elements may be exaggerated for clearer explanation.

When occlusion is implemented using two displays, the optical path becomes longer and the volume of the entire module becomes larger. The following embodiments may provide a near-eye display device and method capable of occlusion using only one display device. It is possible to provide a near-eye display apparatus and method for augmented reality that is light and has a small volume by minimizing an optical system to reduce an optical path, and is capable of implementing occlusion that minimizes image distortion.

In order to implement occlusion in a near-eye display for augmented reality, a process of masking an actual image and a process of projecting a virtual image are required. We propose a method of implementing these two processes using time multiplexing on a single display.

1 is a diagram illustrating a structure of a near-eye display device for augmented reality according to an exemplary embodiment.

Referring to FIG. 1, a near-eye display device 100 for occlusion-capable augmented reality using a single digital micromirror device according to an embodiment includes a digital micromirror device (DMD) 160 that is a display device. It may include a light emitting device (Light-Emitting Diode) 170. According to an embodiment, the near-eye display device 100 for augmented reality includes a DMD 160, an LED 170, a Pechan prism 110, a linear polarizer 120, a polarization beam splitter (PBS) ( 130), Quarter Wave Plate (QWP) (140, 180) 2, a lens (Lens) 150, and a mirror (190).

The DMD 160 controls on/off of each mirror, and may partially implement masking by adjusting the mirrors corresponding to the portion to be displayed to the virtual image to be off.

The LED 170 may display a virtual image on a portion where masking is implemented by the DMD 160. The LED 170 is arranged in a direction in which the actual image is reflected from the off-state mirror in the DMD 160, and proceeds at the same angle as the actual image when the virtual image raised by the LED 170 is reflected on the off-state mirror. This allows light to enter the human eye.

In the DMD 160, an actual image is reflected in a portion where the mirror is on, and a virtual image is reflected in a portion where the mirror is off, and occlusion may be implemented by covering the actual image behind the virtual image in order to display the virtual image. . More specifically, it is possible to implement both masking and imaging by dividing one frame into subframes by controlling the DMD 160 in units of microseconds. In a frame to be masked among subframes, a portion of the DMD 160 pixel on which a portion to be imaged is projected may be turned off and masked. At this time, the off part appears black because the actual image is not reflected. In a frame for imaging among subframes, a virtual image can be displayed by reflecting the LED on the masked part.

In addition, by synchronizing the DMD 160 and the LED to implement masking and virtual object imaging using time multiplexing, a near-eye display with a small volume capable of occlusion can be implemented using a single display.

In this case, the actual image may be projected onto the DMD 160 by using a polarization beam splitter instead of a plurality of mirrors and sent back to the eye. In other words, the actual image is projected onto the DMD 160 by using the polarization beam splitter and sent back to the human eye, and the volume of the device can be reduced by the single display, the DMD 160 and the polarizing beam splitter.

According to embodiments, instead of using two displays, time multiplexing is used to fabricate an occludeable near-eye display that is small in volume and light, and has a short optical path.

Structure using polarized beam splitter

In order to show the real scene from the same location as the real scene, the light coming into the eyes and the light coming from the real object must exist in a straight line. In the past, four mirrors were used for this, but in this case, in order to reduce the structure, it can be implemented using a polarizing beam splitter. The polarization beam splitter has a property of reflecting light of a specific polarization on a reflective surface and transmitting light of a polarized light perpendicular thereto.

When light from a real scene passes through a linear polarizer, it becomes vertically polarized, and the polarized light is reflected through a polarizing beam splitter (PBS) to form the first quarter wave plate (QWP 1). When it passes, it becomes circularly polarized light. When the passed light reflects the DMD 160 and passes through the quarter wave plate (QWP 1) again, it becomes horizontally polarized, passes through the polarizing beam splitter and passes through the second quarter wave plate (QWP 2). . When passing through the second quarter wave plate (QWP 2) twice, it becomes vertically polarized again and is reflected by the polarizing beam splitter, finally allowing light to enter the eye.

When this structure is used, the light from the real scene and the light coming into the eye exist on the same line, and the position of the actual image that enters the eye becomes the same as the actual scene. This is an important part for realism in a near-eye display for augmented reality.

Use of Pechan Prism

2 is a diagram for explaining a principle in which an image is inverted according to an exemplary embodiment.

Referring to FIG. 2, the structure of FIG. 1 is easily drawn along an optical path, and shows the principle that an image of a real scene is flipped when there is no Pechan prism. Since the real scene is farther from the lens than the focal length of the lens, when passing through the lens for the first time, the real image is turned upside down, left and right. And after the real image is reflected from the DMD, it is formed closer than the focal length of the lens, and when passing through the lens again, a virtual image of the real image is generated. Also, the image is flipped as the light reflects the polarizing beam splitter, but it is neglected because it is flipped left and right twice. Therefore, if there is no Pechan prism, you will see the image upside down, left and right.

The Pechan prism plays the role of flipping the image upside down, left and right in the path of light (in-line). That is, by using the Pechan prism, it is possible to obtain an image that is not inverted.

Occlusion implementation using DMD

Digital micromirror devices (DMDs) and LEDs can be used to add virtual images, which are an important part of augmented reality near-eye displays, to real images.

The DMD is a display device composed of microscopic mirrors, and the angle of each mirror can be adjusted on/off in microseconds. When the angle of the mirror is on, the DMD is tilted to match the angle of the DMD so that the actual image can be reflected vertically after passing through the lens and back vertically. In the part where the angle of the mirror is adjusted to off, the light is not reflected vertically, so the actual image does not enter the eye. Therefore, masking can be implemented partially by adjusting the on/off of each mirror.

In fact, when you see an object with your eyes, objects behind an opaque object are obscured and invisible. It is necessary to implement occlusion in order to give a sense of reality to the near-eye display for augmented reality. That is, in order to display the virtual image, it is necessary to cover the actual image that exists behind the virtual image. In the near-eye display method for augmented reality according to an embodiment, masking may be implemented by turning off mirrors corresponding to a portion to which a virtual image is to be displayed on an image formed on a DMD.

In addition, a virtual image may be displayed on the masked portion by using an LED. The LED is placed in the direction where the actual image is reflected from the off-state mirror in the DMD, and when the LED is reflected on the off-state mirrors, the light goes into the eyes at the same angle as the actual image. That is, in the DMD, the part where the mirror is on reflects the actual image, and the part that is off reflects the virtual image.

DMD and LED control and synchronization

3 is a diagram for explaining a time multiplexing method according to an embodiment.

The DMD and LED can be controlled using time multiplexing. Referring to FIG. 3, a time multiplexing driving method is shown, and a pixel to project a virtual image is driven. Assuming that a virtual image is reproduced at 60Hz, one frame can be divided into subframes that mask the actual image and subframes that project the virtual image and drive it.

In the case of a subframe covering the actual image, the corresponding pixel of the DMD can be simply adjusted to the off state. When Off, the actual image is not reflected at all. The part that projects the virtual image needs to control both the DMD and the LED. To adjust the brightness of the LED, pulse width modulation (PWM) technology can be used. Pulse width modulation technology is a technology that controls a digital signal at a frequency faster than a frequency that can be perceived by the human eye and recognizes it as an analog signal.

마이크로 초인 8 개의 서브프레임으로 다시 나눌 수 있다. 가상 영상 투영 프레임의 서브 프레임에서 LED를 on 또는 off로 조정하여 밝기를 제어할 수 있다. 사람의 눈이 인지할 수 있는 주파수보다 빠르게 구동되기 때문에, LED가 on인 시간에 비례하게 세기를 인지하게 되는 것이다.Assuming that the intensity of the LED is driven by 8 bits, each sub-frame for virtual image projection is

It can be divided into 8 subframes that are microseconds. Brightness can be controlled by turning the LED on or off in the subframe of the virtual image projection frame. Since the human eye is driven faster than the perceptible frequency, the intensity is perceived in proportion to the time the LED is on.

In this embodiment, the on and off of the LED can be controlled by turning on and off each mirror of the DMD capable of fast driving in microseconds. The LED is kept on during the virtual image projection frame, and instead, the on and off of the DMD is adjusted so that the LED is not reflected when the DMD is on, and the LED is not reflected when the DMD is off, thereby turning the LED on and off. It can have the same effect as adjusting with.

In this way, by controlling the on and off of the DMD and LED using PWM technology, occlusion and virtual image projection can be implemented in one frame.

4 is a flowchart illustrating a near-eye display method for augmented reality according to an exemplary embodiment.

Referring to FIG. 4, in a near-eye display method for occlusion-capable augmented reality using a single digital micromirror device according to an embodiment, on/off of each mirror of a digital micromirror device (DMD) is performed. In step S110, partially implementing masking by adjusting the mirrors corresponding to the part where the virtual image is to be displayed (S110), and applying a light-emitting diode (LED) to the part where masking is implemented by the DMD. It may be performed including the step of displaying a virtual image (S120). On the other hand, a near-eye display method for occlusion-capable augmented reality using a single digital micromirror device according to an embodiment includes a near-eye display device for occlusion-capable augmented reality using a single digital micromirror device according to an embodiment described above. It can be done through.

Here, in the DMD, the actual image is reflected in the portion where the mirror is on, the virtual image is reflected in the portion where the mirror is off, and occlusion may be implemented by covering the real image behind the virtual image in order to display the virtual image. In addition, using time multiplexing by synchronizing the DMD and the LED, occlusion can be implemented with only one display, the DMD and the LED.

In addition, according to an embodiment, the step of vertically polarized light from a real scene passing through a linear polarizer, and the polarized light is reflected through a polarizing beam splitter (PBS) to form a first quarter wave plate. The step of becoming circularly polarized by passing through, reflecting the light that has passed through the first 1/4 wave plate to the DMD, and then passing through the 1/4 wave plate again to be horizontally polarized, and the horizontally polarized light passes through the polarization beam splitter. And passing through the second quarter wave plate. After passing through the second quarter wave plate, it is reflected by a mirror, and when passing through the second quarter wave plate again, it becomes vertically polarized again, and is reflected by the polarizing beam splitter It may further include the step of finally entering the light into the eye.

As described above, the present invention relates to a near-eye display device for augmented reality capable of implementing occlusion using a single digital micromirror device (DMD). The volume of the device can be reduced by using a single display, a polarizing beam splitter, and a Pechan prism that can be used in the path of light. By controlling the DMD and the LED using PWM technology, it is possible to implement a virtual image and to cover a real image within one frame. By applying time multiplexing, occlusion can be implemented with only one display and LED.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (5)

In the near-eye display device for occlusion-enabled augmented reality using a single digital micromirror device,
A Digital Micromirror Device (DMD) that controls on/off of each mirror and partially masks by adjusting the mirrors corresponding to a portion to be displayed on a virtual image to be off; And
LED (Light-Emitting Diode) that displays a virtual image on the part where the masking is implemented by the DMD
Including,
In the DMD, the actual image is reflected in the part where the mirror is on, and the virtual image is reflected in the part where the mirror is off. that
Characterized in, a near-eye display device for augmented reality. The method of claim 1,
The LED,
The real image is arranged in the direction reflected from the off-state mirror in the DMD, and when the virtual image floated by the LED is reflected on the off-state mirror, it proceeds at the same angle as the real image to light the human eye. Let this go in,
Synchronizing the DMD and the LED to implement occlusion using only one display, the DMD and the LED, using time multiplexing.
Characterized in, a near-eye display device for augmented reality. The method of claim 1,
Projecting the actual image to the DMD using a polarization beam splitter and sending it back to the human eye, reducing the volume of the device by the DMD and the polarizing beam splitter as a single display
Characterized in, a near-eye display device for augmented reality. In the near-eye display method for occlusion-capable augmented reality using a single digital micromirror device,
Controlling the on/off of each mirror of a digital micromirror device (DMD), and partially implementing masking by adjusting the mirrors corresponding to the portion to be displayed on the virtual image to off; And
Displaying a virtual image using an LED (Light-Emitting Diode) on the portion where the masking is implemented by the DMD
Including,
In the DMD, the actual image is reflected in the part where the mirror is on, and the virtual image is reflected in the part where the mirror is off, and occlusion is implemented by covering the real image behind the virtual image in order to display the virtual image. that
Characterized in, a near-eye display method for augmented reality. The method of claim 4,
Vertically polarized light received from a real scene through a linear polarizer;
Reflecting the polarized light through a polarization beam splitter (PBS) and passing through the first 1/4 wave plate to become circularly polarized light;
Reflecting the light passing through the first quarter wave plate to the DMD and then passing through the quarter wave plate to be horizontally polarized;
Passing the horizontally polarized light passing through the polarizing beam splitter and passing through a second quarter wave plate;
After passing through the second quarter wave plate, it is reflected by a mirror, and when passing through the second quarter wave plate again, it becomes vertically polarized again, and the light is finally reflected by the polarizing beam splitter and finally enters the eye.
Further comprising, a near-eye display method for augmented reality.

Images