KR102511620B1 - Apparatus and method for displaying augmented reality - Google Patents

Abstract

An apparatus for displaying a virtual object according to an exemplary embodiment includes a photographing unit that captures an object and obtains an RGB image and a depth image, and a posture estimation unit that estimates a pose of the photographing unit based on the obtained RGB image. a virtual object information storage unit configured to store depth information about the virtual object; and a portion of the virtual object viewed from the estimated posture that is covered by the object based on the obtained depth image and the virtual object information. and a rendering unit that determines and renders the rest of the virtual object except for the obscured portion.

Description

Augmented reality display device and method {APPARATUS AND METHOD FOR DISPLAYING AUGMENTED REALITY}

The present invention relates to an augmented reality display device and method.

Augmented reality (AR) is a technology that displays virtual objects in a real space in a see-through manner. Such augmented reality is used in various fields. For example, augmented reality is used in games, navigation, or museums that provide information.

Meanwhile, in augmented reality, a virtual object is rendered and displayed in a real space. A processor such as a GPU is often used for such rendering. However, since rendering consumes a considerable amount of resources, many studies have been conducted to reduce the load associated with rendering. In particular, when augmented reality is implemented in a mobile terminal device such as a smart phone, it can be said that a technique for reducing load according to rendering is further required.

Korea Patent Publication No. 2013-0023433 (2013.04.05)

An object to be solved by the present invention is to provide a technique for reducing the amount of resources consumed in rendering a virtual object.

However, the problem to be solved by the present invention is not limited to those mentioned above, and another problem to be solved that is not mentioned can be clearly understood by those skilled in the art from the description below. will be.

An apparatus for displaying a virtual object according to an exemplary embodiment includes a photographing unit that captures an object and obtains an RGB image and a depth image, and a posture estimation unit that estimates a pose of the photographing unit based on the obtained RGB image. a virtual object information storage unit configured to store depth information about the virtual object; and a portion of the virtual object viewed from the estimated posture that is covered by the object based on the obtained depth image and the virtual object information. and a rendering unit that determines and renders the rest of the virtual object except for the obscured portion.

A method of displaying a virtual object performed by a virtual object display device according to an embodiment includes the steps of acquiring an RGB image and a depth image by photographing an object, and the pose of the photographing unit based on the obtained RGB image. ), determining a portion of the virtual object viewed from the estimated posture based on the obtained depth image and pre-stored depth information about the virtual object, which is covered by the object; and rendering remaining parts of the object except for the occluded part.

According to an embodiment, after determining the portion of the virtual object that is obscured by an object existing in the real space, only the portion other than the obscured portion is rendered. Accordingly, resources consumed in rendering can be reduced.

In addition, since resources consumed for rendering can be reduced, it is easy to calculate occlusion (overlapping phenomenon) between virtual objects and objects existing in real space in real time. Accordingly, even when an object existing in the real space dynamically moves, the display of the virtual object reflecting the aforementioned occlusion can be implemented in real time.

FIG. 1 exemplarily illustrates that a virtual object is augmented and displayed according to an augmented reality display method according to an exemplary embodiment.
2 exemplarily illustrates a configuration of an augmented reality display device according to an exemplary embodiment.
3 illustrates a procedure of an augmented reality display method according to an embodiment.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the embodiment of the present invention, which may vary according to the intention or custom of a user or operator. Therefore, the definition should be made based on the contents throughout this specification.

FIG. 1 exemplarily illustrates that a virtual object is augmented and displayed according to an augmented reality display method according to an exemplary embodiment. However, since FIG. 1 is merely illustrative, the spirit of the present invention is not limited to that shown in FIG. 1 .

Referring to Figure 1, in the picture on the left, the virtual object is displayed without being covered by any object existing in the real space. On the other hand, in the picture in the middle, the virtual object is partially covered by an object existing in the real space (hereinafter referred to as a 'surrounding object') and displayed. At this time, according to an embodiment, a portion of the virtual object covered by a surrounding object is not rendered, and only a portion not covered is rendered and displayed. In the figure on the right, virtual objects are displayed without being obscured by surrounding objects.

That is, according to an embodiment, the virtual object may be dynamically rendered or displayed in real time by reflecting whether or not surrounding objects existing in the real space cover the virtual object.

2 illustrates a configuration of an augmented reality display device 100 according to an exemplary embodiment. Here, the augmented reality display device 100 can be implemented with AR glasses, a head mounted display (HMD), a smart phone, or the like, but is not limited thereto.

Referring to FIG. 2 , the augmented reality display device 100 includes a photographing unit 110, a posture estimation unit 120, a depth image interpolating unit 130, a virtual object information storage unit 140, a rendering unit 150, and It includes the display unit 160, however, the configuration of the augmented reality display device 100 is not limited to that shown in FIG. 2. Among them, the posture estimator 120, the depth image interpolator 130, and the rendering unit 150 can be implemented by a microprocessor.

First, the photographing unit 110 is a means for photographing surrounding objects and can be implemented as an RGB-D camera. The photographed result includes an RGB image or a depth image.

The pose estimation unit 120 estimates the pose of the photographing unit 110 . Here, the posture represents the amount of movement (translation) and gaze direction (rotation) of the photographing unit 110 in real space.

Hereinafter, a process of estimating the posture of the photographing unit 110 by the posture estimating unit 120 will be described in more detail. The posture estimator 120 extracts at least one feature point from each of a plurality of frames constituting the RGB image (obtained by the photographing unit 110). In addition, the attitude estimation unit 120 arranges the above-described plurality of frames in chronological order. After that, the posture estimator 120 analyzes whether the feature points included in any one frame are also included in adjacently aligned frames, and then tracks the feature points included in adjacent frames. carry out In addition, during the tracking process, the posture estimator 120 even analyzes changes in size or shape of feature points included in each frame. As a result of the analysis, the posture of the photographing unit 110 is estimated.

According to an embodiment, the posture estimator 120 may estimate the posture of the photographing unit 110 by using a depth image of a surrounding object captured by the photographing unit 110 . In this case, the posture estimator 120 may estimate the posture of the photographing unit 110 by extracting feature points from each of a plurality of frames constituting the depth image and then performing tracking of the feature points over adjacent frames.

The depth image interpolator 130 interpolates the depth image acquired by the photographing unit 110 . In general, looking at a depth image, a hole exists. A hole refers to a part of an object located far away in real space or a part where light is reflected without a depth value. The depth image interpolator 130 interpolates these holes using various algorithms. For example, the depth image interpolator 130 may interpolate a hole using a push-pull algorithm.

The virtual object information storage unit 140 stores information about a virtual object to be augmented and displayed in real space, and can be implemented as a memory or the like. The type of information to be stored may include, for example, information (including depth information) about a 3D location of a virtual object to be displayed in real space.

The rendering unit 150 renders virtual objects to be displayed in real space. At this time, the rendering unit 150 may perform rendering in consideration of the following.

First, the rendering unit 150 estimates and renders the shape of the virtual object viewed from the photographing unit 110 . To this end, the posture of the photographing unit 110 estimated by the posture estimating unit 120 is used. For example, when the photographing unit 110 views the virtual object from the front and from the side, the shape of the virtual object looks different. Accordingly, the rendering unit 150 estimates the shape of the virtual object viewed from the photographing unit 110 using the posture of the photographing unit 110, and then performs rendering according to this shape.

Second, the rendering unit 150 renders the virtual object in consideration of occlusion (occlusion). Looking more specifically, the rendering unit 150 first determines whether the virtual object viewed from the photographing unit 110 is covered by a surrounding object in a real space. If there is a portion that is partially obscured, the rendering unit 150 excludes the obscured portion and renders only the remaining portion.

That is, according to an embodiment, after determining the portion of the virtual object that is obscured by the surrounding object, only the portion other than the obscured portion is rendered. Accordingly, resources consumed in rendering can be reduced.

Meanwhile, whether the virtual object is occluded by the surrounding object may be determined by comparing depth information between the virtual object and the surrounding object. For example, it may be determined by comparing the depth information of the virtual object obtained from the virtual object information storage unit 140 with the depth information displayed in the depth image of the surrounding object obtained by the photographing unit 110 .

The display unit 160 displays the virtual object rendered by the rendering unit 150 in a real space in a see-through form. Such a display unit 160 can be implemented by various display devices.

3 illustrates a procedure of an augmented reality display method according to an exemplary embodiment. The augmented reality display method shown in FIG. 3 can be performed by the augmented reality display device 100 shown in FIG. 2 . In addition, since the method shown in FIG. 3 is only exemplary, the procedures of the augmented reality display method are not limited to those shown in FIG. 3 .

Referring to FIG. 3 together with FIG. 2 , the photographing unit 110 acquires an RGB image and a depth image by photographing a surrounding object (S100).

Next, the posture estimating unit 120 estimates the posture of the photographing unit 110 (S110). Here, the posture indicates a translation amount and a gaze direction (rotation) of the photographing unit 110 in real space.

Hereinafter, a process of estimating the posture of the photographing unit 110 by the posture estimating unit 120 in step S110 will be described in more detail. The posture estimator 120 extracts at least one feature point from each of a plurality of frames constituting the RGB image acquired by the photographing unit 110 . In addition, the attitude estimation unit 120 arranges the above-described plurality of frames in chronological order. After that, the posture estimator 120 analyzes whether the feature points included in any one frame are also included in adjacently aligned frames, and then tracks the feature points included in adjacent frames. carry out In addition, during the tracking process, the posture estimator 120 even analyzes changes in size or shape of feature points included in each frame. As a result of the analysis, the posture of the photographing unit 110 is estimated.

According to an embodiment, the posture estimator 120 may estimate the posture of the photographing unit 110 by using a depth image of a surrounding object captured by the photographing unit 110 . In this case, the posture estimator 120 may estimate the posture of the photographing unit 110 by extracting feature points from each of a plurality of frames constituting the depth image and then performing tracking of the feature points over adjacent frames.

Next, the rendering unit 150 determines whether or not the virtual object viewed from the photographing unit 110 is obscured by a surrounding object in the real space, or an obscured portion (S120). Step S120 will be described in more detail below.

First, the shape of the virtual object viewed from the photographing unit 110 is estimated. Since the shape of the virtual object looks different when the photographing unit 110 views the virtual object from the front and from the side, the shape of the virtual object seen from the photographing unit 110 must be estimated. To this end, the posture of the photographing unit 110 estimated by the posture estimating unit 120 may be used.

Also, the depth image interpolator 130 interpolates the depth image acquired by the photographing unit 110 . In general, looking at a depth image, a hole exists. A hole refers to a part of an object located far away in real space or a part where light is reflected without a depth value. The depth image interpolator 130 interpolates these holes using various algorithms. For example, the depth image interpolator 130 may interpolate a hole using a push-pull algorithm. However, a process of interpolating the depth image may not be performed according to embodiments.

Next, the rendering unit 150 determines whether or not the virtual object viewed from the photographing unit 110 is obscured by a surrounding object in the real space, or the portion that is obscured. Depth information of the virtual object obtained from the virtual object information storage unit 140 is obtained by the photographing unit 110 to determine whether the virtual object is covered by the surrounding object or which part of the virtual object is covered by the surrounding object. It can be determined by comparing the depth information displayed in the depth image of the surrounding object.

Next, if even a portion of the river object is covered by a surrounding object, the rendering unit 150 excludes the portion covered and renders only the remaining portion (S130).

In addition, the display unit 160 displays the virtual object rendered by the rendering unit 150 in a see-through form in real space (S160). Such a display unit 160 can be implemented by various display devices.

As described above, according to an embodiment, after determining a portion of a virtual object that is obscured by a surrounding object, only the portion other than the obscured portion is rendered. Accordingly, resources consumed in rendering can be reduced.

In addition, since resources consumed in rendering can be reduced, it is easy to calculate occlusion (overlapping phenomenon) between virtual objects and surrounding objects in real time. Accordingly, even when a surrounding object dynamically moves, it is possible to display a virtual object reflecting the aforementioned occlusion.

On the other hand, the virtual object display method includes each step included in the method stored in the form of a computer-readable recording medium storing a computer program programmed to perform including each step included in the method or stored in a computer-readable recording medium. It may be implemented in the form of a computer program programmed to include and perform.

The above description is merely an example of the technical idea of the present invention, and various modifications and variations can be made to those skilled in the art without departing from the essential qualities of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed according to the claims below, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

According to an embodiment, after determining a portion of the virtual object that is obscured by a surrounding object, only the portion other than the obscured portion is rendered. Accordingly, resources consumed in rendering can be reduced.

100: virtual object display device

Claims (7)

A photographing unit that acquires an image and a depth image by photographing an object;
a posture estimator for estimating a pose of the photographing unit based on the acquired image;
a depth image interpolator interpolating a hole existing in the acquired depth image;
a virtual object information storage unit that stores depth information about the virtual object;
Based on the depth image from which the hole is interpolated and the information of the virtual object, a portion of the virtual object viewed from the estimated posture is determined to be obscured by the object, and a portion of the virtual object excluding the obscured portion is determined. Including a rendering unit that renders,
The rendering unit,
Determining whether the virtual object is occluded by the surrounding object based on the depth image interpolated by the depth image interpolator and the stored depth information
Virtual object display device. According to claim 1,
The posture estimation unit,
At least one feature point is extracted from each of a plurality of frames constituting the obtained image, the plurality of frames are arranged in chronological order, and a feature point included in any one of the plurality of frames is assigned to the one frame. It analyzes whether it is included in a plurality of adjacent frames, performs tracking on the feature points included in the plurality of adjacent frames, and during the tracking, the size or shape of the feature points included in each frame. Analyzing changes to estimate the posture of the photographing unit
Virtual object display device. According to claim 1,
The rendering unit,
Determining a portion of the virtual object covered by the object by comparing the depth information of the virtual object with depth information displayed in a depth image of the object
Virtual object display device. According to claim 3,
The depth image interpolator,
Interpolating the hole based on a push-pull algorithm
Virtual object display device. A virtual object display method performed by a virtual object display device,
Acquiring an image and a depth image by photographing the object;
Estimating a pose of a photographing unit that photographs the object based on the obtained image;
Interpolating a hole existing in the obtained depth image;
determining a portion of the virtual object viewed from the estimated posture based on a depth image from which the hole is interpolated and depth information about the previously stored virtual object, which is covered by the object;
Comprising the step of rendering the rest of the virtual object except for the occluded part
How virtual objects are displayed. A computer program included in a computer readable recording medium programmed to perform each step included in the method according to claim 5. A computer readable recording medium storing a computer program programmed to perform each step included in the method according to claim 5.

Images