KR102055824B1 - Method for displaying augmented reality image and apparatus used therefor - Google Patents

Abstract

The method of displaying an augmented reality image according to an embodiment may be performed by an augmented reality display device including an image display unit, a communication unit, and an area recognition unit, and transmits an image displayed on another display device to the augmented reality display device. And displaying the image, receiving a virtual image related to the image, recognizing a display region in which the image is displayed within a displayable region of the augmented reality display device, and displaying the display region among the displayable regions. And displaying the virtual image in the remaining region except for the following.

Description

TECHNICAL FOR DISPLAYING AUGMENTED REALITY IMAGE AND APPARATUS USED THEREFOR}

The present invention relates to a method for displaying an augmented reality image and a device used therein, and more particularly, to a method for displaying a real image displayed by a display device of the real world and a virtual image related to the real image in an augmented reality method, To the device used.

Recently, a flat panel display method forms a major market for display devices. Accordingly, large-scale technology development and facility investment for flat panel display devices have been made. Flat panel display devices include, for example, Liquid Crystal Display (LCD) or Organic Light Emitting Diodes (OLED). These have advantages in various aspects such as resolution, image quality, commercial convenience or price.

Flat panel display devices typically have a size of about 50 inches, and such flat panel display devices have a field of vision (FOV) of about 30 °. In order to secure a flat panel display device having a viewing angle of 60 ° or more, a large flat panel display device having a size of about 100 inches or more is required.

However, there are limitations in implementing a flat panel display device in a large screen. When the flat panel display device is implemented in a large screen, manufacturing cost and installation cost are high, and the space to be occupied becomes large.

On the other hand, in the case of virtual reality glass (VR glasses), it is possible to secure a viewing angle of 90 ° or more. In other words, unlike a flat panel display, VR glasses are easy to implement in a large screen. However, it is difficult to secure higher resolution than UHD level with VR glasses that can be manufactured.

Korean Patent Registration Publication No. 10-0880544 (January 20, 2009)

Accordingly, an object of the present invention is to provide a method for providing a high resolution image while ensuring a wide viewing angle and a display device implementing the method.

However, the problem to be solved of the present invention is not limited to those mentioned above, another problem to be solved is not mentioned can be clearly understood by those skilled in the art from the following description. will be.

According to an embodiment, a method of displaying an augmented reality image is performed by an augmented reality display device, and transmitting and displaying a reality image displayed by a reality display device so that a wearer of the augmented reality display device may recognize the image. Receiving a virtual image related to the real image, recognizing a display area in which the real image is displayed within a displayable area in the wearer's view, and excluding the display area among the displayable areas Within the area, displaying the virtual image.

According to an embodiment, a method of displaying an augmented reality image is performed by an augmented reality display device, and displaying an image displayed on another display device by transmitting it through the augmented reality display device, and a virtual image related to the image. Receiving a video signal, recognizing a display area in which the image is displayed within a displayable area of the augmented reality display device, and displaying the virtual image in a remaining area other than the display area among the displayable areas. It includes a step.

An augmented reality display device according to an embodiment includes an image display unit for transmitting and displaying an image displayed on another display device, a communication unit receiving a virtual image related to the image, and a displayable area of the image display unit, the image The display unit may include a region recognizing unit that recognizes the displayed display area, and the image display unit displays the virtual image in the remaining area in which the display area is excluded from the displayable area.

According to an embodiment, the user may view a high resolution (Pixel Per Degree (PPD)) reality image provided by the reality display device and simultaneously watch a virtual image related to the reality image in the peripheral area surrounding the reality image. . In this case, the virtual image surrounding the real image may provide a bar immersive display having a lower resolution (PPD) than the real image. In addition, using the augmented reality display device, it is possible to implement such an immersive display at a lower cost than when real image and virtual image are implemented with only one reality display device.

FIG. 1 is a diagram illustrating an augmented reality image recognized in a wearer's view when the wearer wears the augmented reality display device according to an embodiment.
2 is a diagram illustrating an image display system to which an augmented reality display device is applied, according to an exemplary embodiment.
3 is a diagram illustrating a configuration of an augmented reality display device according to an embodiment.
4 is a diagram illustrating an example of a screen shown to a wearer in the augmented reality display device according to an embodiment.
FIG. 5 is a diagram illustrating another example of a screen shown to a wearer in the augmented reality display device according to one embodiment.
6 is a flowchart illustrating a method of displaying an augmented reality image according to an exemplary embodiment.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be implemented in various different forms, and only the embodiments make the disclosure of the present invention complete, and the general knowledge in the art to which the present invention belongs. It is provided to fully inform the person having the scope of the invention, which is defined only by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, terms to be described below are terms defined in consideration of functions in the embodiments of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the contents throughout the specification.

FIG. 1 is a diagram illustrating an augmented reality image recognized in the visual field of the wearer 1 when the wearer 1 wears the augmented reality display device 100 according to an exemplary embodiment. Referring to FIG. 1, the wearer 1 wearing the augmented reality display apparatus 100 is recognized by overlaying the reality image 210 and the virtual image 141 in an augmented reality manner.

The real image 210 refers to an image displayed by the real world display apparatus 200 in a see-through manner. The reality image 210 is recognized by the wearer 1 through the augmented reality display device 100. The real world display device 200 may be, for example, a flat panel display device such as a TV or a monitor.

The virtual image 141 refers to a virtual image displayed on the augmented reality display device 100. The virtual image 141 may be related to the real image 210. For example, if the real image 210 is an image of the main camera in the sports relay broadcast, the virtual image 141 may be an image of the auxiliary camera in the sports relay broadcast.

As shown in FIG. 1, the virtual image 141 may be configured to surround the edge 211 of the real image 210 when viewed from the eye of the wearer 1. That is, the viewing angle of the virtual image 141 may be wider than the viewing angle of the real image 210.

The resolution (Pixel Per Degree, PPD) of the real image 210 may be higher than the resolution PPD of the virtual image 141. The resolution here refers to the number of display pixels per unit viewing angle (Pixel Per Degree PPD). In this regard, since the real image 210 is an image output by the real world display apparatus 200 to be described below, the resolution provided by the real world display apparatus 200 may be reflected as it is. However, the virtual image 141 is a virtual image seen through the augmented reality display device 100. Therefore, when the wearer 1 wears the augmented reality display apparatus 100, the number of display pixels recognizable at the viewing angle at that time is relatively lower than that of the actual image 210.

Based on this, the augmented reality display apparatus 100 according to an embodiment provides a reality image 210 having a relatively high resolution, and a virtual region having a relatively low resolution in an area surrounding the reality image 210. Provide an image 141.

FIG. 2 is a diagram illustrating an image display system 10 to which the augmented reality display device 100 illustrated in FIG. 1 is applied. Referring to FIG. 2, the image display system 10 may include an augmented reality display device 100, a real-world display device 200, an image supply device 300, and an image selection device 400. Can be connected by the network 500. Here, the network 500 may refer to a network connected by wire / wireless. 2 is merely illustrative.

First, the image supply device 300 supplies an image. The supplied image may be provided from a broadcasting station or a provider providing the image. The image supply device 300 may be, for example, a set top box.

The image supply device 300 may divide and supply an image. For example, the image supply device 300 may divide one image into several, and then supply some of the images (eg, one) to the real world display device 200 and supply the rest to the augmented reality display device 100. have. Alternatively, the image supply device 300 may receive a plurality of images from a broadcasting station, some of which may be supplied to the real world display device 200, and others may be supplied to the augmented reality display device 100. have. Here, the image supplied to the real world display apparatus 200 may be, for example, a main image, an image extracted from the middle of the original image, or an image selected by the image selection apparatus 400, which will be described later. The image supplied to the 100 may be an additional image other than the main image or the remaining images except the extracted image. The image supplied from the image supply device 300 to the real world display device 200 and the image supplied to the augmented reality display device 100 may be synchronized with each other.

The image selection device 400 serves to receive an image from the wearer 1. The selected image may be delivered to the real world display apparatus 200 for display. The image selection device 400 may be, for example, a remote controller equipped with a button, a mouse or a keyboard.

The real-world display device 200 may be a liquid crystal display (LCD) or an organic light emitting diodes (OLED) method. These have advantages in various aspects such as resolution, image quality, commercial convenience or price. For example, the real world display apparatus 200 may provide a real image 210 having a higher resolution than the virtual image 141 displayed by the augmented reality display apparatus 100 to be described later. The real world display device 200 may be a flat panel display such as a TV or a monitor. The real world display device 200 may be referred to as another display device 200 in relation to the augmented reality display device 100.

The real world display device 200 may be connected to the image supply device 300 through the network 500. The real image 210 may be an image supplied from the image supply device 300 as described above.

The augmented reality display device 100 is a device that displays a virtual image 141. The virtual image 141 may be an image supplied from the image supply device 300 as described above. The augmented reality display device 100 and the virtual image 141 will be described later.

3 is a diagram illustrating a configuration of the augmented reality display device 100 according to an embodiment. The augmented reality display apparatus 100 may be a wearable virtual reality glass (VR glasses) wearable by the wearer 1, but is not limited thereto.

Referring to FIG. 3, the AR apparatus 100 may include a communication unit 110, an area recognition unit 120, an image processing unit 130, and an image display unit 140, but is not limited thereto. .

The communication unit 110 refers to a module for communicating with an external communication subject, and may be implemented as a wired communication module or a wireless communication module according to an embodiment. By the communication unit 110, the augmented reality display device 100 may receive the virtual image 141 from the image supply device 300, and may also receive information about the virtual image 141.

The image display unit 140 is configured to display and overlay the real image and the virtual image. For example, the image display unit 140 may include a combiner. Since the combiner itself is a known configuration, a description thereof will be omitted.

In some areas of the image display unit 140, the real image 210 displayed by the real world display apparatus 200 may be displayed in a see-through manner. The display of the reality image 210 means that the reality image 210 is incident from the outside toward the image display unit 140, and the incident reality image 210 is transmitted to form an image in the eyes of the wearer 1. can do. That is, the wearer 1 may recognize the real image 210 in the form of see-through by the image display unit 140. The real image 210 displayed in the form of the image display unit 140 may be an image selected by the image selection device 400.

The image display unit 140 may display the virtual image 141. As described above, the virtual image 141 displayed on the image display unit 141 may have a lower resolution than the real image 210, that is, the number of display pixels per unit viewing angle. As described above, the virtual image 141 may be an image related to the real image 210. The image display unit 140 may optically mask an area where the virtual image 141 is displayed in order to display the virtual image 141 more clearly. The image display unit 140 displaying the virtual image 141 will be described later.

The area recognition unit 120 may recognize the position or size of the display area (211 illustrated in FIG. 1) in which the real image 210 is displayed within the displayable area of the image display unit 140. The displayable area may mean all areas where an image can be displayed on the image display unit 140, and may also mean a region recognized by the image display unit 140 in the wearer's view.

Referring to the area recognizer 120 in detail, the area recognizer 120 may recognize the aforementioned display area 211 in the displayable area according to the following procedure. For example, according to the first procedure, the area recognizer 120 may first detect an edge of the real image 210 through image analysis. Next, according to the second procedure, the area recognition unit 120 may extract the closed curved surface by using the edge. The closed curved surface may include a polygon such as a quadrangle. However, a circle or an ellipse may be included in some embodiments. In this case, the edge detection procedure may not be performed in the first procedure. According to the third procedure, the area recognition unit 120 may recognize whether the closed curved surface is the display area 211 of the real world display device 200 based on the brightness of the closed curved surface extracted according to the second procedure. . Here, since the technology for recognizing whether the display area is the same as the edge detection, the closed curve extraction, and the brightness mentioned in the first to third procedures is the same as that already known, a detailed description thereof will be omitted. In addition, since these first to third procedures are merely exemplary, any one of them may not be performed or may be performed in another manner, and other procedures other than the mentioned procedure may be performed.

The image processor 130 may determine and process the virtual image 141 to be displayed on the image display unit 140. The image processor 130 may be implemented by a microprocessor.

In detail, the image processing unit 130 may process the virtual image 141 to be displayed on the image display unit 140. The virtual image 141 to be processed may be supplied from the image supply device 300 through the communication unit 110.

For example, the image processor 130 may process the resolution of the virtual image 141 received from the image supply device 300 to be lower than the resolution of the real image 210. To this end, the image processing unit 130 may receive information about the resolution of the real image from the image supply device 300.

In addition, the image processor 130 may extract the virtual image 141 to be displayed on the image display unit 140 from the virtual image 141 received from the image supply device 300, and then enlarge, reduce or tilt the virtual image 141. . In order to extract, enlarge, reduce, or tilt, the image processing unit 130 may consider the size or position of the display area 211 recognized by the area recognition unit 120. This will be described with reference to FIGS. 4 and 5. Referring to FIG. 4, the real image 210 is displayed in the center and the virtual image 141 is displayed to surround the real image 210. Here, the real image 210 and the virtual image 141 are separated by the boundary 211. In addition, the real image 410 and the virtual image 141 may be displayed in the displayable area 144 of the image display unit 140. As described above, the displayable area 144 refers to an area recognized by the image display unit 140 in the field of view of the wearer 1, which means an area in which the image display unit 140 can display an image. The displayable area 144 may include the remaining area 143 in which the real image 210 and the virtual image 141 are not displayed.

If the wearer 1 of the augmented reality display device 100 faces the real image display device 200 from the right side, the image display unit 140 may have a long right side and a short left side of the display area 211 in FIG. 4. As shown, it may be displayed in a trapezoidal shape. The area recognizer 120 may recognize a size, a position, a shape, or the like in which the display area 211 is viewed in the displayable area 144. The image processor 130 may determine the size, position, or shape of the virtual region 142 on which the virtual image 141 is to be displayed, based on the size, position, or shape recognized as described above. In FIG. 4, the virtual region 142 on which the virtual image 141 is displayed is larger than the trapezoid that is the display region 211.

The same applies to FIG. 5. According to the position or line of sight of the augmented reality display device 100, the center position of the image display unit 140 may be different from the center position of the real image 210, in which case the virtual image 141 or the real image 210 is located. Some may not be displayed in the displayable area 144. In FIG. 5, an area indicated by the identification number 145 refers to an area outside the displayable area 144 of the virtual image 141.

The area recognition unit 120 recognizes the size and position of the display area 211, and the image processing unit 130 displays the virtual area 142 on which the virtual image 141 is to be displayed based on the size and position thus recognized. The size, position or shape of the can be determined. The size, position, or shape of the virtual area 142 may be determined as the virtual area 142 larger than the display area 211 in consideration of the size, location, or shape of the display area 211. Some of the regions 142 145 may not be within the displayable region 144.

The image processor 130 may process the virtual image 141 based on the virtual region 142. The processed image may be an image received by the communication unit 110 from the image supply device 300.

The image display unit 140 may display the virtual image 141 processed by the image processing unit 130 together with the real image 210 in an augmented reality method, that is, in an overlay method. In this case, the image display unit 140 may display the virtual image 141 to surround the real image 210, and the real image 210 may have a higher resolution (PPD) than the virtual image 141. . This is because the real image 210 is recognized by the wearer 1 through which the image provided by the real world display apparatus 200 is transmitted as it is. In this case, an immersive display device can be implemented.

That is, according to an exemplary embodiment, the user may watch a high resolution (Pixel Per Degree (PPD)) reality image provided by the reality display device and simultaneously watch a virtual image related to the reality image in the surrounding area surrounding the reality image. Can be. In this case, the virtual image surrounding the real image may provide a bar immersive display having a lower resolution (PPD) than the real image.

Meanwhile, the augmented reality display apparatus 100 may be implemented as the following embodiments. For example, a multi-display system (connecting a plurality of monitors to one PC main body) currently provided in a PC may be implemented by such an augmented reality display device 100. To this end, a monitor connected to a PC is used as the real image display apparatus 200, a virtual region 142 is set around the real image display apparatus 200, and a plurality of monitors are displayed in the virtual region 142. can do.

In addition, in order to enhance the realism of sports relay, a broadcasting station edits a plurality of images taken by a plurality of cameras and broadcasts them as one image. If the augmented reality display apparatus 100 according to an embodiment is used, the main image selected by the wearer 1 or the viewer may be displayed by the real image display apparatus 100, except for the main image among the plurality of images. The remaining images may be set in the virtual region 142 after setting the virtual region 142 around the real image display apparatus 100.

In addition, in a broadcasting station, a plurality of channels provide broadcasts different from each other. Here, if the augmented reality display device 100 according to an embodiment is used, the image of the channel selected by the wearer 1 or the viewer is displayed on the reality image display apparatus 100, and the image of the remaining channels is reality. The virtual area 142 may be set around the image display apparatus 100 and then displayed on the virtual area 142.

6 is a diagram illustrating a procedure of a method of displaying an augmented reality image, according to an exemplary embodiment. Such a method may be performed by the augmented reality display apparatus 100 illustrated in FIGS. 1 to 5, but at least one of the procedures illustrated in FIG. 6 may not be performed or may be performed in a different order from the illustrated procedure. In addition, other procedures not shown may be performed.

Referring to FIG. 6, step S100 is performed in which the real image 210 displayed by the real world display apparatus 200 is recognized by the wearer 1 wearing the augmented reality display apparatus 100. In this case, the real image 210 is recognized by the wearer 1 in a manner that is projected onto the image display unit 140.

Next, a step (S200) of receiving a virtual image 142 related to the real image 210 is performed. The virtual image 142 may be received through the communication unit 110 included in the augmented reality display device 100.

Next, in operation S300, the display area 211 displaying the real image 210 is recognized in the displayable area 144 of the image display unit 140. This step (S300) may be performed by the area recognition unit 120 included in the augmented reality display device 100.

Next, a step (S500) of determining the size, shape, location, and the like of the virtual area is performed based on the size, shape, location, and the like of the display area 120, and then the virtual image 141 is adjusted to fit the virtual area. Processing step S600 is performed.

Next, the image display unit 140 displays the virtual image 141 processed in step S600 in an overlay manner with the real image in step S100 (S700).

As described above, according to an exemplary embodiment, a high resolution realistic image may be provided in a narrow narrow viewing angle area in which a wearer's vision is maintained. In addition, the peripheral region surrounding the real image is a region in which the wearer's vision is not maintained and a relatively wide viewing angle range, and a high cost real world display device is additionally required to provide a high resolution realistic image to the peripheral region. It must be prepared. However, in one embodiment, a virtual image may be provided for the peripheral area, which may be modified or selected according to a wearer's preference, and has a lower resolution than the actual image.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential quality of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas that fall within the scope of equivalents should be construed as being included in the scope of the present invention.

According to one embodiment, implementation of an immersive display device may be possible at low cost.

100: augmented reality display device
200: reality video display device

Claims (6)

A method of displaying a real image and a virtual image in an augmented reality display device,
Receiving information about the resolution of the real image reproduced on a real world display device;
Displaying the reality image on the augmented reality display device in a see-through manner;
Receiving a virtual image related to the real image;
Recognizing a display area of the reality image within a displayable area of the augmented reality display device;
Adjusting the resolution of the virtual image to be different from the resolution of the real image;
Displaying a virtual image of which the resolution is adjusted in the entire remaining area of the displayable area except the display area;
Display method of augmented reality image. The method of claim 1,
Displaying the virtual image of which the resolution is adjusted,
Displaying the real image to surround the virtual image having the adjusted resolution
Display method of augmented reality image. The method of claim 1,
The adjusting step,
Acquiring information about a resolution (Pixel Per Degree, PPD) of the real image;
Processing the resolution of the virtual image lower than the resolution obtained for the real image;
Displaying the virtual image of which the resolution is adjusted,
Displaying a virtual image processed at a lower resolution
Display method of augmented reality image. The method of claim 1,
Determining the size or position of the virtual area in which the virtual image is to be displayed in the entire remaining area based on the display area;
Display method of augmented reality image. The method of claim 1,
Processing the virtual image to be displayed on the entire remaining region based on the size or position of the virtual region;
Display method of augmented reality image. An image display unit which displays a real image reproduced on a real world display device in a see-through manner,
A communication unit receiving a virtual image related to the real image;
An area recognition unit recognizing a display area of the real image within a displayable area of the image display unit;
And receiving an information on the resolution of the real image, and adjusting the resolution of the virtual image to be different from the resolution provided for the real image.
The video display unit,
Displaying a virtual image of which the resolution is adjusted within the entire remaining area of the displayable area except the display area
Augmented reality display device.

Images