KR20190043956A - Method and apparatus for displaying augmented reality - Google Patents

Abstract

A method for displaying augmented reality performed by an apparatus for displaying augmented reality comprises the steps of: dividing a background image into a plurality of image sections; extracting each representative color for one or more image sections among the plurality of image sections; and rendering a virtual object in the background image by reflecting the representative color.

Description

TECHNICAL FIELD [0001] The present invention relates to a method and an apparatus for displaying augmented reality,

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an Augmented Reality display method and apparatus, and more particularly, to a method and apparatus for displaying augmented reality in which a virtual object is rendered on a background image.

As is well known, augmented reality technology is a technique of providing a special effect by expressing a virtual object together with a background image of a real world, causing an illusion that a virtual object feels as if it exists in the real world.

In such augmented reality technology field, efforts to improve the reality of the augmented reality image are continuing.

Conventionally, studies have been made mainly on how to display a virtual object on a background image. For example, after estimating the ambient light from the background image, a virtual object is rendered on the background image by reflecting the estimated ambient light .

However, since the entire area of the background image is often embodied in various colors in different areas within the background image than in the case of a single color, the conventional art in which the entire environment light is estimated from the background image and reflected in the rendering of the virtual object There is a limit to improve the reality of the augmented reality image.

Korean Patent Publication No. 10-2011-0107545, published on October 04, 2011.

According to an embodiment of the present invention, there is provided an augmented reality display method and apparatus for rendering a virtual object in a background image by dividing a background image into a plurality of image regions and reflecting representative colors extracted from the divided image regions.

The problems to be solved by the present invention are not limited to those mentioned above, and another problem to be solved can be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, an augmented reality display method performed in an augmented reality display apparatus includes: dividing a background image into a plurality of image regions; extracting representative colors from the plurality of image regions, And rendering the virtual object in the background image by reflecting the representative color.

According to another aspect of the present invention, there is provided an augmented reality display apparatus including a display unit, an information acquiring unit acquiring data on a background image viewed through the display unit, and a control unit, A representative color is extracted for each of at least one of the plurality of image regions, and a virtual object is rendered on the background image by reflecting the representative color, and the background image and the virtual object Generates the augmented reality image, and controls the display unit to output the generated augmented reality image.

According to an embodiment of the present invention, a background image is divided into a plurality of image regions, and a virtual object is rendered on a background image by reflecting a representative color extracted from the divided image regions.

Accordingly, even if the entire area of the background image is not realized as a single color but is implemented in various colors in each area in the background image, the virtual object may be rendered in the background image by reflecting the representative color, There is an effect of increasing the sense of reality.

1 is a block diagram of an augmented reality display apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a method of displaying an augmented reality performed by the augmented reality display apparatus according to an embodiment of the present invention.
3 is a flowchart illustrating an example of a process of rendering a virtual object during a method of displaying an augmented reality according to an embodiment of the present invention.
4 is a flowchart for explaining another example of a process of rendering a virtual object while performing the augmented reality display method according to an embodiment of the present invention.
5 is a flowchart illustrating a method of displaying an augmented reality according to another embodiment of the present invention.
FIG. 6 is a diagram illustrating a process of extracting a representative color during execution of the augmented reality display method according to another embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. To fully disclose the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions in the embodiments of the present invention, which may vary depending on the intention of the user, the intention or the custom of the operator. Therefore, the definition should be based on the contents throughout this specification.

1 is a block diagram of an augmented reality display apparatus according to an embodiment of the present invention.

1, the augmented reality display apparatus 100 includes an information obtaining unit 110, a control unit 120, a storage unit 130, and a display unit 140. As shown in FIG.

The information acquisition unit 110 acquires data on a background image for an augmented reality image. That is, the information obtaining unit 110 obtains data on the background image used when the control unit 120 generates the augmented reality image. For example, the information obtaining unit 110 may include a camera for photographing a real world. In this case, a two-dimensional image information for a background image of an augmented reality image can be directly generated using a camera. Alternatively, the information obtaining unit 110 may include a communication interface for receiving information from a camera provided separately from the augmented reality display device 100. In this case, the two-dimensional image information is generated by a separate camera, and the information obtaining unit 110 can receive and acquire the two-dimensional image information through the communication interface.

The storage unit 130 stores a computer-executable instruction for performing a process of controlling the augmented reality display process at the time of execution by the control unit 120. [ The 3D model data corresponding to the virtual object to be displayed on the augmented reality image may be stored. In addition, the two-dimensional image information on the background image of the augmented reality image acquired by the information acquisition unit 110 can be stored. For example, the storage unit 130 may store at least one of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory, a RAM, and a ROM Readable < / RTI > type of recording medium.

The control unit 120 performs a process of controlling the display process of the augmented reality image by executing computer executable instructions stored in the storage unit 130. [ According to this process, the background image obtained by the information obtaining unit 110 is divided into a plurality of image regions, representative colors are respectively extracted for one or more image regions among the plurality of image regions, and the extracted representative colors are reflected And generates an augmented reality image including a background image and a virtual object, and controls the display unit 140 to output the generated augmented reality image. For example, the control unit 120 may include a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

The display unit 140 displays an augmented reality image in which a background image and a virtual object are output together according to a control signal resulting from a process performed by the control unit 120. [ For example, the output unit 140 may include a flat panel display device such as a liquid crystal display (LCD) or the like.

2 is a flowchart illustrating a method of displaying an augmented reality performed by the augmented reality display apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the method for displaying augmented reality according to an exemplary embodiment includes obtaining data on a background image for an augmented reality image and dividing the acquired background image into a plurality of image regions (S210) . For example, it is possible to divide the background image so as to have a checkerboard shape and divide it into X * Y rectangular image regions.

In addition, the method for displaying an augmented reality according to an exemplary embodiment further includes extracting a representative color for each of at least one image region from among a plurality of image regions (S220). For example, representative colors can be extracted for each of a plurality of image regions. At this time, a color having a redness average value, a greenness average value, and a blueness average value of the pixels forming the image region can be extracted as a representative color. Alternatively, a representative color may be extracted as a representative color having a brightness value average value, a red value average value, an average green value value, and a blue value average value of pixels forming the image region.

According to another aspect of the present invention, there is provided a method of displaying an augmented reality according to an embodiment of the present invention, in which, when representative colors are extracted from at least two or more image regions, And merging the zones (S230). As such, the process of merging adjacent video regions may be omitted.

In addition, the method of displaying an augmented reality according to an exemplary embodiment further includes a step S240 of rendering a virtual object in the background image by reflecting the extracted representative color. As described above, the process of rendering a virtual object in a background image can be implemented in various embodiments. Various embodiments will be described below with reference to FIG. 3 to FIG.

3 is a flowchart illustrating an example of a process of rendering a virtual object during a method of displaying an augmented reality according to an embodiment of the present invention.

Referring to FIG. 3, an example of a step of rendering a virtual object during the augmented reality display method according to an exemplary embodiment of the present invention includes the steps of: And generating and allocating a virtual illumination (S310).

One example of the step of rendering a virtual object during the AR method according to an embodiment further includes a step S320 of discriminating a rendering image region including a position of a virtual object among a plurality of image regions do.

One example of a step of rendering a virtual object during the method of displaying an augmented reality according to an embodiment is to display a virtual image on a background image in an environment where the virtual light is allocated, And rendering the object (S330).

4 is a flowchart for explaining another example of a process of rendering a virtual object while performing the augmented reality display method according to an embodiment of the present invention.

Referring to FIG. 4, another example of a step of rendering a virtual object during performing the augmented reality display method according to an embodiment includes determining a rendering image zone including a position of a virtual object among a plurality of image zones (S410 ).

Another example of the step of rendering a virtual object during the method of displaying the augmented reality according to an embodiment is to render the virtual object in the background image by reflecting the representative color of the determined rendered image area on the surrounding color of the virtual object And further includes step S420.

Hereinafter, a method of displaying an augmented reality performed by the augmented reality display apparatus 100 according to an embodiment of the present invention will be described in more detail with reference to Figs.

The storage unit 130 of the augmented reality display apparatus 100 includes a computer 120 for performing a process of controlling the augmented reality display process at the time of execution by the control unit 120 of the augmented reality display apparatus 100. [ Executable instructions are stored. Then, the 3D model data corresponding to the virtual object to be displayed on the augmented reality image may be stored.

In this state, the controller 120 executes the computer executable instructions stored in the storage unit 130 to perform a process of controlling the augmented reality display process as a whole. When the control unit 120 performs the process, the information obtaining unit 110 of the augmented reality display apparatus 100 obtains a background image to be used for the augmented reality image. For example, the information obtaining unit 110 can directly capture a background image by photographing a real image using a camera. Alternatively, if the information obtaining unit 110 includes a communication interface capable of receiving information from the camera provided separately from the augmented reality display device 100, the information obtaining unit 110 may obtain information from the outside through the communication interface, Images can be input.

Then, the control unit 120 divides the acquired background image into a plurality of image regions. For example, it is possible to divide the background image so as to have a checkerboard shape and divide it into X * Y rectangular image regions. Here, the number of image regions to be divided into the image regions or the types of image regions to be divided may vary depending on the presetting (S210).

Then, the control unit 120 extracts representative colors for at least one of the plurality of image regions obtained by dividing the background image. For example, the control unit 120 may extract a representative color for each of a plurality of image regions. At this time, the color having the average value of the red value, the average value of the green value, and the average value of the blue value of the pixels forming the image region can be extracted as the representative color. Alternatively, when a representative color is extracted, a color having an average of brightness values, an average of red values, an average of green values, and a blue value of pixels forming the image region may be extracted as a representative color (S220).

In addition, when the representative color is extracted from at least two image regions, the control unit 120 may merge adjacent image regions based on a result of comparing representative colors among adjacent image regions among at least two image regions . For example, the control unit 120 compares the representative color of the first image region and the representative color of the second image region, which are adjacent to each other in the plurality of image regions, and determines whether the difference in color, The average values of the two representative colors can be extracted as a new representative color of the merged image region by merging the first image region and the second image region into one image region. Then, when the merged image zone and the third image zone are adjacent to each other, the controller 120 compares the representative color of the merged image zone with the representative color of the third image zone, and if there is a difference in color, If it is within a predetermined threshold value, the merged image area and the third image area can be merged into one image area. At this time, a new representative color of the newly merged image region can be extracted as in the case of merging the first image region and the second image region. For example, the representative color of the first video zone, the representative color of the second video zone, and the representative color of the third video zone may be extracted as a new representative color of the newly merged video zone (S230). However, the merging process of the image region through step S230 may be omitted.

Next, the controller 120 reads the three-dimensional model data stored in the storage unit 130, identifies a corresponding region among a plurality of image regions based on the coordinate information of the read three-dimensional model data, and in Step S210 And arranges the virtual objects corresponding to the three-dimensional model data according to the coordinate information confirmed in the obtained real image. At this time, the controller 120 reflects the representative color extracted in the corresponding image region to render a virtual object in the background image (S240).

3, when the virtual object is rendered on the background image by reflecting the representative color extracted from the corresponding image region, the controller 120 generates and allocates a virtual illumination for each of the extracted image regions S310), a rendering image region including a position of a virtual object among the plurality of image regions is determined (S320), and a virtual object is rendered in the background image in an environment in which virtual light allocated to the determined rendering image region exists (S330). 4, the control unit 120 determines a rendering image zone including a position of a virtual object among the plurality of image zones (S410), and displays the representative color of the determined rendering image zone on the surrounding color of the virtual object To render the virtual object in the background image (S420).

The display unit 140 displays the augmented reality image in which the virtual object is rendered in the background image according to the control signal resulting from the process performed by the controller 120. [ Thus, even if the same virtual object is used, the color of the virtual object can be changed according to which region of the plurality of image regions is rendered.

5 is a flowchart illustrating a method of displaying an augmented reality according to another embodiment of the present invention.

Referring to FIG. 5, a method of displaying an augmented reality according to another embodiment includes a step S510 of acquiring data on a background image for an augmented reality image and extracting an entire representative color from the obtained background image. For example, a color having an average of red values, an average of green values, and an average of blue values of pixels forming a background image can be extracted as a representative color. Alternatively, a color having an average value of brightness values, a red value average value, a green value average value, and a blue value average value of pixels forming a background image may be extracted as an entire representative color.

Next, the augmented reality display method according to another embodiment further includes a step (S520) of dividing the obtained background image into a plurality of image regions. For example, it is possible to divide the background image so as to have a checkerboard shape and divide it into X * Y rectangular image regions.

According to another embodiment, the method for displaying an augmented reality further includes extracting a representative color for each of at least one image region from among a plurality of image regions (S530). For example, a representative color for each zone can be extracted for each of a plurality of video zones. At this time, the color having the red average value, the green average value, and the blue value average value of the pixels forming the image region can be extracted as a representative representative color. Alternatively, when extracting the representative color, the color having the average value of the brightness value, the average value of the red value, the average value of the green value, and the average value of the blue value of the pixels forming the image region may be extracted as a representative representative color.

In addition, the augmented reality display method according to another embodiment further includes a step (S540) of rendering a virtual object on the background image by reflecting the extracted representative representative color and the representative color of each zone.

5 is compared with the augmented reality display method shown in Fig. 2, a process of extracting an entire representative color is added in step S510, and in step S540, a process of rendering a virtual object on a background image Is changed.

The process of extracting the entire representative color of the background image in step S510 is the same as that illustrated in FIG. 6 (a). For example, the controller 120 may extract a color having a red value average value, a green value average value, and a blue value average value of the pixels forming the background image A as a whole representative color. In this case, the background image (A) has a relatively bright color when compared to other areas, such as an area on a desk, but there is also a relatively dark color when compared to other areas such as an area under the desk, Is extracted as the entire representative color 601. [

The process of extracting the representative hue for each of the plurality of image regions in step S530 is the same as that shown in FIG. 6 (B). The control unit 120 may extract a color having a mean value of a red value, an average value of a green value, and an average value of a blue value of the pixels forming the image region B corresponding to a part of the background image A as a representative representative color. In this case, since the image region B includes a large number of areas below the desk, a relatively dark color is a mainstream when compared with other regions, and thus a color having a dark brightness is extracted as a representative representative color 603 . If the image region B is located on the desk in the background image A, the relatively bright color is dominant when compared with the other regions, so that the color having the bright brightness is divided into the representative representative color 602 It would have been extracted.

In step S540, when a virtual object is rendered on the background image, the virtual object is rendered by reflecting both the entire representative color extracted from the background image (A) and the representative color (603) extracted from the image region (B) do.

Meanwhile, although not shown in the flowchart of FIG. 5, when representative colors are extracted in at least two or more image regions between steps S520 and S530, a representative color of adjacent image regions among at least two image regions is compared And merging adjacent video regions based on the merged video region. In this case, since the area of the image area B shown in FIG. 6 (B) changes according to the number of merging of the image areas, in the process of rendering the virtual object on the background image, The color of the virtual object is determined by interpolating the representative colors of the entire representative color and the zone of the rendering image zone according to the ratio of the area of the virtual image. In this case, when the virtual object is rendered in the background image, the interpolation color generated by the virtual illumination of the corresponding region or reflected in the surrounding color of the virtual object can be determined as shown in Equation 1 below.

As described above, according to the embodiment of the present invention, a background image is divided into a plurality of image regions, and a virtual object is rendered on the background image by reflecting representative colors extracted from the divided image regions.

Accordingly, even if the entire area of the background image is not realized as a single color but is implemented in various colors in each area in the background image, the virtual object may be rendered in the background image by reflecting the representative color, The reality is increased.

Combinations of each step of the flowchart and each block of the block diagrams appended to the present invention may be performed by computer program instructions. These computer program instructions may be loaded into a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus so that the instructions, which may be executed by a processor of a computer or other programmable data processing apparatus, And means for performing the functions described in each step are created. These computer program instructions may also be stored in a computer usable or computer readable memory capable of directing a computer or other programmable data processing apparatus to implement the functionality in a particular manner so that the computer usable or computer readable memory It is also possible for the instructions stored in the block diagram to produce a manufacturing item containing instruction means for performing the functions described in each block or flowchart of the block diagram. Computer program instructions may also be stored on a computer or other programmable data processing equipment so that a series of operating steps may be performed on a computer or other programmable data processing equipment to create a computer- It is also possible that the instructions that perform the processing equipment provide the steps for executing the functions described in each block of the block diagram and at each step of the flowchart.

Also, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function (s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps shown in succession may in fact be performed substantially concurrently, or the blocks or steps may sometimes be performed in reverse order according to the corresponding function.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

According to the embodiment of the present invention, even if the entire area of the background image is not realized as a single color but is implemented in various colors in each area in the background image, By rendering, the reality of the augmented reality image is enhanced.

The present invention can be widely used in various augmented reality industrial fields employing a device for augmented reality in which a virtual object is represented together with a background image in a real world to provide a special effect.

100: Augmented reality display
110: Information obtaining unit
120:
130:
140:

Claims (9)

An augmented reality display method performed by an augmented reality display device,
Dividing the background image into a plurality of image regions,
Extracting representative colors for at least one of the plurality of image regions,
And rendering the virtual object in the background image by reflecting the representative color
Augmented reality display method. The method according to claim 1,
The extracting of the representative color may include extracting a color having a red (R) value average value, a green (G) value average, and a blue (B) value average value of the pixels forming the image region as the representative color
Augmented reality display method. The method according to claim 1,
At least two image regions from which the representative color is extracted,
Further comprising merging the adjacent image regions based on a result of comparing representative colors among adjacent image regions of the at least two image regions,
Augmented reality display method. The method according to claim 1,
The rendering of the virtual object in the background image may include:
Generating and assigning a virtual illumination for each of the image regions from which the representative color is extracted by reflecting the representative color;
Determining a rendering image region including a position of the virtual object among the plurality of image regions;
Rendering a virtual object in the background image in an environment in which the virtual light is allocated if the virtual light is allocated to the rendering image area;
Augmented reality display method. The method according to claim 1,
The rendering of the virtual object in the background image may include:
Determining a rendering image region including a position of the virtual object among the plurality of image regions;
And rendering a virtual object in the background image by reflecting a representative color of the rendered image region on a surrounding color of the virtual object
Augmented reality display method. 6. The method of claim 5,
Further comprising extracting an entire representative color from the background image,
The rendering of the virtual object in the background image may include rendering the virtual object in the background image by reflecting the overall representative color and the representative color of the rendering image zone on the surrounding color of the virtual object
Augmented reality display method. The method according to claim 6,
The rendering of the virtual object in the background image may include interpolating the representative color of the entire representative color and the representative color of the rendering image zone according to the ratio of the area of the background image to the area of the rendering image zone
Augmented reality display method. A processor for performing the augmented reality display method according to any one of claims 1 to 7
A computer program stored on a computer readable recording medium. 1. An augmented reality display device for displaying an augmented reality image,
A display section,
An information obtaining unit for obtaining data on a background image viewed through the display unit;
And a control unit,
Wherein,
A computer-implemented method of dividing a background image into a plurality of image regions, extracting a representative color for each of at least one of the plurality of image regions, rendering a virtual object on the background image by reflecting the representative color, Generates the augmented reality image including the virtual object, and controls the display unit to output the generated augmented reality image
Augmented reality display.

Images