KR20230014606A - Method and apparatus for generating mega size augmented reality image information - Google Patents

Abstract

One embodiment of the present invention relates to a method for generating large augmented reality (AR) image information, performed by a user terminal including a photographing device, which comprises the steps of: obtaining information about a virtual object from an external server; obtaining original image information including the sky through the photographing device; deriving a first layer corresponding to a sky area from the original image information based on a pre-trained sky area detection model; deriving a second layer corresponding to the remaining area excluding the sky area from the original image information based on the first layer; deriving a specific virtual lighting among a plurality of virtual lightings from the first layer based on a pre-trained virtual lighting classification model; deriving corrected color information of the virtual object based on the information about the virtual object and information about the specific virtual lighting; and generating information about the corrected virtual object including the corrected color information according to a preset spatial relationship, and generating AR image information in which the virtual object is inserted into the sky based on the first layer and the second layer.

Description

Method and apparatus for generating giant AR image information {METHOD AND APPARATUS FOR GENERATING MEGA SIZE AUGMENTED REALITY IMAGE INFORMATION}

The present invention relates to a method and apparatus for generating augmented reality (AR) image information, and more particularly, to a method for generating AR image information in which a huge virtual object is inserted in a sky space in consideration of a skyline.

AR (Augmented Reality) is a technology that provides additional information based on reality, and is different from VR (Virtual Reality), which provides both backgrounds and objects as virtual images. In other words, AR is to superimpose virtual objects on the real world that users see, and to further increase the effect of reality by combining virtual objects on the environment of the real world.

AR can be provided through various IT (Information Technology) devices such as smartphones, tablets, or smart glasses having a shooting device and display. It is a technology that

However, in general, AR requires an understanding of the environment in which a virtual plane on which a virtual object is placed is constructed in advance by finding a cluster of feature points in space through a photographing device. It is difficult to designate, and virtual objects of a certain size or larger are difficult to implement.

An object of the present invention to solve the above problems is to provide a method for generating large Augmented Reality (AR) image information.

Another object of the present invention to solve the above problems is to provide a device for generating large AR image information.

Another object of the present invention to solve the above problems is to provide a computer readable recording medium recording a program for executing a method for generating large AR image information.

Another object of the present invention to solve the above problems is to provide a program stored in a computer readable recording medium for performing a method for generating large AR image information.

In order to achieve the above object, a method for generating augmented reality (AR) image information performed by a user terminal including a photographing device according to an embodiment of the present invention includes acquiring information on a virtual object from an external server, the Obtaining original image information including the sky through a photographing device, deriving a first layer corresponding to the sky region from the original image information based on a previously learned sky region detection model, based on the first layer deriving a second layer corresponding to an area other than the sky area from the original image information, deriving a specific virtual light among a plurality of virtual lights from the first layer based on a pre-learned virtual light classification model deriving corrected color information of the virtual object based on the information on the virtual object and the information on the specific virtual light, and corrected virtual including the corrected color information according to a preset spatial relationship. The method may include generating AR image information in which the virtual object is inserted into the sky based on object information, the first layer, and the second layer.

Here, the AR image information is synthesized by placing the second layer at the front, the first layer at the back, and the corrected virtual object information between the second layer and the first layer. It can be.

Here, the information on the virtual object includes reference position information of the virtual object, the information on the specific virtual light includes incident angle information and height information of the specific virtual light to be given to the virtual object, and the incident angle Information and the height information may be derived based on current date information, current time information, and the reference position information.

Here, the information on the virtual object includes reference position information of the virtual object, the information on the specific virtual lighting includes distance information to be given to the virtual object, and the distance information includes location information of the user terminal. and contrast control information and saturation control information derived based on distance information between the reference location information, wherein the contrast control information is such that the contrast between the sky area and the virtual object decreases as the distance according to the distance information increases. The saturation adjustment information may be set so that the saturation of the virtual object decreases as the distance according to the distance information increases.

Here, the information on the virtual object includes reference position information and reference size information of the virtual object, and deriving distance information between the current position information of the user terminal and the reference position information; The method may further include deriving corrected size information of the virtual object based on information and the reference size information, and the corrected information on the virtual object may include the corrected size information.

Here, the information on the virtual object includes reference position information of the virtual object and reference direction information of the virtual object, and the user terminal moves from a current position according to the current position information to a reference position according to the reference position information. The method may further include deriving direction information and deriving corrected direction information of the virtual object based on the reference direction information and the direction information.

An electronic device performing a method for generating augmented reality (AR) image information according to an embodiment of the present invention for achieving the other object is a photographing device, at least one processor, and at least one processor executed by the at least one processor. A memory for storing commands of, wherein the at least one command is executed to obtain information on a virtual object from an external server, and is executed to acquire original image information including the sky through the photographing device, and is pre-learned deriving a first layer corresponding to the sky region from the original image information based on the sky region detection model, and a first layer corresponding to the remaining region excluding the sky region from the original image information based on the first layer. Execute to derive two layers, and to derive a specific virtual light among a plurality of virtual lights from the first layer based on a pre-learned virtual light classification model, information about the virtual object and the specific virtual light corrected color information of the virtual object is derived based on information about the corrected virtual object including the corrected color information according to a preset spatial relationship, the first layer and the second layer; It can be executed to generate AR image information in which the virtual object is inserted into the sky based on.

A computer-readable recording medium recording a program for executing a method for generating giant AR (Augmented Reality) image information performed by a user terminal according to an embodiment of the present invention for achieving the above another object, wherein the giant AR image The information generating method includes obtaining information on a virtual object from an external server, obtaining original image information including the sky through a photographing device of the user terminal, and the original image based on a previously learned sky region detection model. Deriving a first layer corresponding to the sky area from information, deriving a second layer corresponding to an area other than the sky area from the original image information based on the first layer, and pre-learned virtual lighting Deriving a specific virtual light from among a plurality of virtual lights from the first layer based on a classification model, correcting color information of the virtual object based on information on the virtual object and the information on the specific virtual light. Information on the corrected virtual object including the corrected color information according to the step of deriving and a preset spatial relationship, AR image information in which the virtual object is inserted into the sky based on the first layer and the second layer It may include the step of generating.

A program stored in a computer readable recording medium for performing a method of generating augmented reality (AR) image information through a user terminal according to an embodiment of the present invention for achieving the another object, wherein the augmented reality image information is generated. The method includes obtaining information on a virtual object from an external server, acquiring original image information including the sky through a photographing device of the user terminal, and obtaining original image information from the original image information based on a previously learned sky region detection model. Deriving a first layer corresponding to the sky area, deriving a second layer corresponding to the remaining area excluding the sky area from the original image information based on the first layer, a pre-learned virtual lighting classification model Deriving a specific virtual light from among a plurality of virtual lights from the first layer based on, deriving corrected color information of the virtual object based on information on the virtual object and information on the specific virtual light generating AR image information in which the virtual object is inserted into the sky based on information about the corrected virtual object including the corrected color information and the first layer and the second layer according to the step and a preset spatial relationship; steps may be included.

According to the present invention, it is possible to overcome the limitations of understanding the environment of the method of deriving a virtual plane in which a virtual object should be placed.

According to the present invention, it is possible to implement a huge virtual object.

According to the present invention, it is possible to provide realistic AR content by providing various views of the same virtual object in consideration of the reference location of the virtual object and the current location of the user.

According to the present invention, by deriving virtual lighting in consideration of the surrounding environment and reflecting the virtual lighting on the virtual object, it is possible to increase the sense of agreement between the virtual object and reality.

According to the present invention, various types of virtual performances can be provided in a vast size.

1 is a block diagram of a device for generating augmented reality (AR) image information according to an embodiment of the present invention.
2 is a diagram for explaining layers used for generating large AR image information according to an embodiment of the present invention.
3 is a diagram for explaining a derivation sequence of layers used to generate giant AR image information according to an embodiment of the present invention.
4 is a diagram for explaining a sky area detection model according to an embodiment of the present invention.
5 is a diagram for explaining a pre-learned virtual light classification model according to an embodiment of the present invention.
6 is a diagram for explaining incident angle information and height information of virtual light for deriving corrected color information of a virtual object according to an embodiment of the present invention.
7 is a diagram for explaining distance information for deriving corrected color information of a virtual object according to an embodiment of the present invention.
8 is a diagram for explaining derivation of corrected size information of a virtual object according to an embodiment of the present invention.
9 is a diagram for explaining derivation of corrected direction information of a virtual object according to an embodiment of the present invention.
10 is a flowchart of a method for generating giant AR image information according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term “and/or” includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

In this document, Mega size AR is a standard size or larger AR virtual reality that calculates horizontal information (x, y) based on GPS (Global Positioning System) and calculates vertical information (z) through machine learning. As an object, it is possible to represent AR implemented in a size of 100 m to 10 km in a tangible way.

In this document, spatial relationship refers to the distance and overlapping state between a virtual object and a real object based on an observer (user) in a virtual space derived from a real space. It can express the superposition state.

In this document, environmental understanding can refer to the process of finding a set of feature points on a typical horizontal or vertical surface such as a table or wall, and locating a plane for positioning a virtual object relative to this surface. Prior to this process, it may be possible to arrange a virtual object having a suitable perspective.

In this document, light estimation is used to infer the lighting of the real environment and apply it to the virtual object in order for the virtual object to harmonize with the surrounding environment because humans are unconsciously aware of the lighting of the object or environment. To this end, it can represent a process of inferring actual lighting using shadows, ambient light, shading, and specular highlights reflections as clues.

In this document, a virtual object is a concept in contrast to a real world object, and can represent all digital content implemented based on real space, and its format can encompass 3D models and 2D images. there is.

In this paper, the aerial perspective is that short-wavelength visible light is scattered farther than longer-wavelength visible light, so distant objects appear bluer than they actually are and have less color contrast. can show how to apply to painting.

In this document, SVM (Support Vector Machine) is a supervised learning model for pattern recognition and data analysis as a field of machine learning, and can be mainly used for classification and regression analysis. A key function of SVM may be a function of deriving a decision boundary on which side a variable belongs to when there is a variable that needs to be classified.

In this document, giant AR image information may be referred to as AR image information for convenience of explanation, and a layer may represent an image corresponding to a partial area or partial information extracted from an original image.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate overall understanding in the description of the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

1 is a block diagram of a device for generating augmented reality (AR) image information according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus for generating giant AR image information includes at least one processor 110, memory 120, transceiver 130, input interface device 140, output interface device 150, storage It may comprise a device 160 and a bus 170.

For example, the at least one processor 110 may be a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. can mean Each of the memory 120 and the storage device 160 may include at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory 120 may be one of a read only memory (ROM) and a random access memory (RAM), and the storage device 160 may be a flash-memory. , a hard disk drive (HDD), a solid state drive (SSD), or various memory cards (eg, a micro SD card).

In addition, the data providing device 100 based on the hierarchical identifier management system may include a transceiver 130 that performs communication through a wireless network. In addition, the data providing device 100 based on the hierarchical identifier management system may further include an input interface device 140, an output interface device 150, a storage device 160, and the like. Each component included in the data providing apparatus 100 based on the hierarchical identifier management system is connected by a bus 170 to communicate with each other.

The giant AR image information generating device may be a user terminal equipped with a photographing device. For example, the user terminal may be a smart phone, tablet PC, mobile phone, or smart glass, but is not limited thereto.

Here, the memory 120 may store at least one command executed by the at least one processor 110 .

For example, the at least one command is executed to obtain information on a virtual object from an external server, is executed to acquire original image information including the sky through the photographing device, and a pre-learned sky area detection model is executed. and derive a first layer corresponding to the sky area from the original image information based on the first layer, and derive a second layer corresponding to the remaining area excluding the sky area from the original image information based on the first layer. and derives a specific virtual light among a plurality of virtual lights from the first layer based on a pre-learned virtual light classification model, and based on the information on the virtual object and the information on the specific virtual light, the It is executed to derive corrected color information of a virtual object, and information on a corrected virtual object including the corrected color information according to a preset spatial relationship, the color of the sky based on the first layer and the second layer. It may be executed to generate AR image information in which the virtual object is inserted.

For example, in the AR image information, the second layer is disposed at the front, the first layer is disposed at the rear, and information on the corrected virtual object is disposed between the second layer and the first layer. and can be synthesized.

For example, the information on the virtual object includes reference position information of the virtual object, and the information on the specific virtual light includes incident angle information and height information of the specific virtual light to be assigned to the virtual object; The incident angle information and the height information may be derived based on current date information, current time information, and reference position information.

For example, the information on the virtual object includes reference position information of the virtual object, the information on the specific virtual lighting includes information on a sense of distance to be given to the virtual object, and the information on the sense of distance includes information on a sense of distance of the user terminal. Contrast control information and saturation control information derived based on distance information between location information and the reference position information, wherein the contrast control information increases the contrast between the sky area and the virtual object as the distance according to the distance information increases. The saturation adjustment information may be set such that the saturation of the virtual object decreases as the distance according to the distance information increases.

For example, the information on the virtual object includes reference position information of the virtual object and reference size information of the virtual object, and the at least one command is a distance between the current position information of the user terminal and the reference position information. information, and derive corrected size information of the virtual object based on the distance information and the reference size information, wherein the corrected information on the virtual object includes the corrected size information. can

For example, the information on the virtual object includes reference position information of the virtual object and reference direction information of the virtual object, and the at least one command includes the reference from a current position according to current position information of the user terminal. Deriving direction information to a reference position according to location information, and deriving corrected direction information of the virtual object based on the reference direction information and the direction information, and information about the corrected virtual object. may include the corrected direction information.

Alternatively, for example, the method for generating giant AR image information according to an embodiment may be executed as a program by a user terminal or a device for generating giant AR image information, and may be stored or recorded in a computer-readable recording medium.

2 is a diagram for explaining layers used for generating large AR image information according to an embodiment of the present invention.

An embodiment of the present invention may generate large AR image information based on information on a plurality of layers or virtual objects. That is, large AR image information may be generated by synthesizing information on a plurality of layers or virtual objects.

Referring to FIG. 2 , A may represent an original image acquired through a photographing device, B may represent an image obtained by extracting a sky region from the original image, and C may represent an image of the remaining region of the original image excluding the sky region. can represent D-1 may represent information on a virtual object, and D-2 may represent information on a virtual object to which corrected color information is reflected. E represents AR image information, and may be image information generated by sequentially combining C, D2, and B among the images described above. In more detail, it will be described later together with FIG. 3 .

3 is a diagram for explaining a derivation sequence of layers used to generate giant AR image information according to an embodiment of the present invention.

Referring to FIG. 3 , an embodiment of the present invention can generate large AR image information by sequentially performing four steps.

A first step may be obtaining an image. For example, since the large AR image information may be provided through a user terminal including a photographing device, an image including the sky may be captured through the photographing device of the user terminal. Here, the captured image may be referred to as an original image or original image information because it is an image in which no virtual object is inserted and a partial area or partial information is not extracted.

A second step may be a step of detecting the sky area. For example, the sky region may be detected or derived from an original image based on a previously learned sky region detection model, and an image corresponding to the detected or derived sky region may be referred to as a first layer. Here, a detailed description of the sky area detection model will be described later with reference to FIG. 4 .

A third step may be a step of detecting an area other than the sky. For example, an area other than the sky area may be detected or derived through the first layer corresponding to the sky area obtained in the previous step. That is, since the non-sky area can be determined as an area excluding the sky area corresponding to the first layer in the original image, if the sky area of the first layer is inverted, the remaining area other than the sky area can be detected or derived, and the remaining area An image corresponding to may be referred to as a second layer. In other words, the second layer may be detected or derived based on original image information or the first layer.

A fourth step may be a step of constructing a spatial relationship. Here, the spatial relationship may represent information about a spatial positional relationship between the first layer, the second layer, and information about the virtual object. In other words, the spatial relationship may indicate an arrangement relationship between images, layers, or information. In the spatial relationship according to an embodiment, a second layer corresponding to a building or objects adjacent to a user is placed at the forefront in addition to the sky area, and the background The first layer corresponding to the sky area may be placed at the rear, and the virtual object to be inserted into the sky area may be set in advance to be placed between the second layer and the first layer.

Since AR image information according to an embodiment must be generated by combining information on a virtual object to be inserted into the sky area in addition to the above-described first layer and second layer, information on the virtual object can be obtained from an external server. .

Here, information on the obtained virtual object may not be directly synthesized with the first layer and the second layer, and after the color, size, direction, etc. are corrected in consideration of the environment and location, the information on the corrected virtual object is It may be combined with the first layer and the second layer. A method of deriving information on the corrected virtual object will be described later in detail together with FIGS. 5 to 9 .

4 is a diagram for explaining a sky area detection model according to an embodiment of the present invention.

Referring to FIG. 4 , the sky area detection model according to an embodiment of the present invention may include a sky related model and a non-sky related model, and the sky area is synthesized by combining two results derived from the two related models. can be detected. Here, both the sky-related model and the non-sky-related model are models learned using SVM. The sky-related model can be referred to as a sky-related SVM (sky concerned SVM), and the non-sky-related model is non-sky-related. It can be expressed as a non-sky concerned SVM (SVM).

The sky-related model and the non-sky-related model may be generated through SVM training. For this purpose, training images for learning including a sky area and a non-sky area are obtained, and each training image is acquired. By segmenting the image, regions in each training image can be divided, classified, or subdivided into similar regions. As for the segmentation method, there are various methods that are obvious to those skilled in the art, such as a sliding window method or a fully convolutional method, so it is sufficient to perform the segmentation through the most effective method among the various methods, and a detailed description thereof will be omitted. . Thereafter, a plurality of samples to be used for learning may be extracted by performing feature extraction on the segmented training image. For example, feature extraction may be a process of deriving a characteristic arrangement between subdivided regions, and feature extraction methods include scale invariant feature transform (SIFT), histogram of oriented gradients (HOG), and local binary pattern (LBP). Since there are various methods obvious to those skilled in the art, it is sufficient to perform the method through the most effective method among the various methods, and a detailed description thereof will be omitted. The extracted plurality of samples (for training) are classified and used for learning a sky-related model or a non-sky-related model using an SVM algorithm.

In one embodiment, when a sky-related model and a non-sky-related model that have been learned are secured, segmentation is performed on a test image or an input image in the same manner as described above, feature extraction is performed, and a plurality of extracted samples are extracted. It can be classified and input into a sky-related model and a non-sky-related model, and the sky-related model and the non-sky-related model can derive image information in which the sky region is primarily detected from a plurality of samples that are classified and input. can That is, two pieces of primarily detected image information can be secured. Thereafter, according to an embodiment, a similarity measure between the two primarily detected images may be performed to finally derive an image in which the sky area is detected. As described above, a model that performs a procedure for deriving an image that finally detects the sky region from an input image may be referred to as a sky region detection model, and an image derived through this may be referred to as a first layer.

5 is a diagram for explaining a pre-learned virtual light classification model according to an embodiment of the present invention.

Since the sky may have various colors such as bright and sunny or sunset depending on the position of the sun, time and date, etc., a virtual object to be inserted in the sky may also reflect this to enhance a sense of reality.

Referring to FIG. 5 , the pre-learned virtual light classification model according to an embodiment of the present invention may derive a specific virtual light corresponding to a color of the sky area from a first layer corresponding to the sky area. In other words, an embodiment may derive a specific virtual light suitable for the first layer from among a plurality of virtual lights predetermined through a virtual light classification model, and correct the color of the virtual object using information on the specific virtual light. can do.

A virtual lighting classification model can be generated through SVM training. To this end, a plurality of training sets including sky areas with various colors are obtained, and after performing feature extraction on them, SVM is performed through machine learning. can define That is, a virtual light classification model may be generated using an SVM algorithm based on a plurality of training sets including sky regions having various colors. Here, the plurality of virtual lights may be virtual lights classified in advance according to the color of the sky area.

In one embodiment, when a trained virtual light classification model is secured, an input image, that is, a sky area corresponding to (color of) of a first layer among a plurality of virtual lights from the first layer based on the virtual light classification model. You can select specific virtual lights.

Thereafter, in an embodiment, color information of the virtual object may be corrected using information on the selected specific virtual light to generate corrected color information giving the same effect as when the specific virtual light illuminates the virtual object.

The virtual object may perform correction on size information and direction information in addition to color information, and a high sensory effect may be provided through the corrected virtual object.

6 is a diagram for explaining incident angle information and height information of virtual light for deriving corrected color information of a virtual object according to an embodiment of the present invention.

Referring to FIG. 6 , information on a specific virtual light according to an embodiment of the present invention may include not only color information of the specific virtual light described with reference to FIG. 5 , but also information on an incidence angle and height of the specific virtual light. That is, in one embodiment, the color of the sky region may be reflected, but the direction and angle of the light may be further considered for a higher sensory effect.

Here, information on the virtual object may include reference position information of the virtual object in order to obtain information on the incident angle and height information. The reference location information of the virtual object may indicate GPS-based coordinate information set as the location of the virtual object.

Incidence angle information and height information may be derived based on current date information, current time information, and reference position information. This is because it has constant azimuth and elevation angles according to the date and time, and the incident angle and height can be derived based on the above information.

7 is a diagram for explaining distance information for deriving corrected color information of a virtual object according to an embodiment of the present invention.

Referring to FIG. 7 , information on a specific virtual light according to an embodiment of the present invention includes information about a sense of distance of a specific virtual light in addition to color information, incident angle information, and height information of the specific virtual light described together with FIGS. 5 and 6 . can do. That is, one embodiment may further consider that the color of an object changes according to the overlapping of the atmosphere by applying the atmospheric perspective method as well as reflecting the color of the hall area for a higher sensory effect.

Here, information on the virtual object may include reference position information of the virtual object in order to secure distance information. The reference location information of the virtual object may indicate GPS-based coordinate information set as the location of the virtual object.

The distance information may include contrast control information and saturation control information derived based on distance information between the location information of the user terminal and the reference location information. The contrast between the virtual objects may be set to decrease, and the saturation adjustment information may be set such that the saturation of the virtual object decreases as the distance according to the distance information increases.

8 is a diagram for explaining derivation of corrected size information of a virtual object according to an embodiment of the present invention.

Referring to FIG. 8 , in addition to correcting the color of a virtual object through information on a specific virtual light according to an embodiment of the present invention, the size of the virtual object is also corrected based on the location of the user terminal to obtain corrected size information. Information on the generated and corrected virtual object may include corrected size information.

Here, in order to derive the corrected size information, information on the virtual object may include reference position information and reference size information of the virtual object. The reference position information of the virtual object may indicate GPS-based coordinate information set as the virtual object is placed, and the reference size information of the virtual object may indicate basic size information of the virtual object positioned at the reference position.

According to an embodiment, distance information between current location information and reference location information of a user terminal may be derived to derive corrected size information, and corrected size information of a virtual object may be derived based on the distance information. That is, when the location of the user terminal is far from the reference location of the virtual object, size information corrected to make the virtual object look small may be derived, and when the location of the user terminal is close to the reference location of the virtual object, the virtual object is relatively Size information corrected to look large can be derived.

9 is a diagram for explaining derivation of corrected direction information of a virtual object according to an embodiment of the present invention.

Referring to FIG. 9 , in addition to correcting the color of a virtual object through information on a specific virtual light according to an embodiment of the present invention, the direction of the virtual object is also corrected based on the location of the user terminal to obtain corrected direction information. Information on the generated and corrected virtual object may include corrected direction information.

Here, information on the virtual object may include reference position information and reference direction information of the virtual object in order to derive corrected direction information. The reference position information of the virtual object may indicate GPS-based coordinate information set as the virtual object is placed, and the reference direction information of the virtual object may indicate information on which front direction the virtual object is facing from the reference position. can

An embodiment may derive direction information about a direction in which the user terminal is located from a virtual object based on current position information and reference position information of the user terminal in order to derive corrected direction information. Corrected direction information may be derived based on the reference direction information of the object. That is, when the virtual object faces north in front according to the reference direction information and the user terminal is located in the south direction from the virtual object according to the direction information, the user terminal provides the back view of the virtual object according to the corrected direction information. It can be.

10 is a flowchart of a method for generating giant AR image information according to an embodiment of the present invention.

Referring to FIG. 10 , in one embodiment, information on a virtual object may be obtained from an external server (S1010). Here, the information on the virtual object may include image information of the virtual object, and may further include at least one of reference position information of the virtual object, reference size information of the virtual object, and reference direction information of the virtual object.

According to an embodiment, original image information including the sky may be acquired through a photographing device (S1020). For example, taking a picture of the user terminal ?遲見? Since the captured image is an image in which a virtual object is not inserted and a partial area or partial information is not extracted, it may be referred to as an original image or original image information.

In one embodiment, a first layer corresponding to the sky area may be derived from original image information (S1030). That is, in an embodiment, a first layer corresponding to the sky region may be derived from original image information based on a previously learned sky region detection model.

According to an embodiment, a second layer corresponding to the remaining area may be derived based on the first layer (S1040). That is, an embodiment may derive a second layer corresponding to the remaining area excluding the sky area from original image information based on the first layer.

In one embodiment, a specific virtual light among a plurality of virtual lights may be derived using the first layer (S1050). That is, in one embodiment, a specific virtual light among a plurality of virtual lights may be derived from the first layer based on a pre-learned virtual light classification model.

For example, the information on the specific virtual light may include information about a color value or a color filter of the specific virtual light.

For example, the information on the specific virtual light may include incident angle information and height information of the specific virtual light to be assigned to the virtual object, and the incident angle information and height information may include current date information, current time information, and It may be derived based on the reference location information.

For example, the information on the specific virtual light may include distance information to be given to the virtual object, and the distance information is a contrast derived based on distance information between the location information of the user terminal and the reference location information. Adjustment information and saturation adjustment information may be included. Here, the contrast adjustment information may be set such that the contrast between the sky area and the virtual object decreases as the distance according to the distance information increases, and the saturation adjustment information increases the saturation of the virtual object as the distance according to the distance information increases. can be set to lower

According to an embodiment, calibrated color information of a virtual object may be derived based on information on the virtual object and information on specific virtual lighting (S1060). For example, the corrected color information may indicate corrected information based on at least one of information about the color filter, information on an incident angle, information on a height, or information on a sense of distance.

According to an embodiment, AR image information in which the virtual object is inserted into the sky may be generated based on information about the virtual object corrected according to a preset spatial relationship, the first layer, and the second layer (S1070). Here, the AR image information is synthesized by placing the second layer at the front, the first layer at the back, and the corrected virtual object information between the second layer and the first layer. It can be.

For example, the information on the virtual object may include at least one of reference position information of the virtual object, reference size information of the virtual object, or reference direction information of the virtual object, and information on the corrected virtual object may include corrected color information. information, corrected size information, or corrected direction information.

Alternatively, although not shown in FIG. 10 , an embodiment may derive distance information between the current location information and the reference location information of the user terminal, and correct the virtual object based on the distance information and the reference size information. Size information can be derived.

Alternatively, an embodiment may derive direction information from a current location according to the current location information of the user terminal to a reference location according to the reference location information, and based on the reference direction information and the direction information, the direction information of the virtual object may be derived. Corrected direction information may be derived.

Operations according to embodiments of the present invention can be implemented as computer-readable programs or codes on a computer-readable recording medium. A computer-readable recording medium includes all types of recording devices in which data that can be read by a computer system is stored. In addition, computer-readable recording media may be distributed to computer systems connected through a network to store and execute computer-readable programs or codes in a distributed manner.

In addition, the computer-readable recording medium may include hardware devices specially configured to store and execute program instructions, such as ROM, RAM, and flash memory. The program command may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine code generated by a compiler.

Although some aspects of the present invention have been described in the context of an apparatus, it may also represent a description according to a corresponding method, where a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method may also be represented by a corresponding block or item or a corresponding feature of a device. Some or all of the method steps may be performed by (or using) a hardware device such as, for example, a microprocessor, programmable computer, or electronic circuitry. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

In embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In embodiments, a field programmable gate array may operate in conjunction with a microprocessor to perform one of the methods described herein. Generally, methods are preferably performed by some hardware device.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

Claims (9)

A method of generating augmented reality (AR) image information performed by a user terminal including a photographing device,
obtaining information about a virtual object from an external server;
acquiring original image information including the sky through the photographing device;
deriving a first layer corresponding to the sky region from the original image information based on a previously learned sky region detection model;
deriving a second layer corresponding to an area other than the sky area from the original image information based on the first layer;
deriving a specific virtual light from among a plurality of virtual lights from the first layer based on a pre-learned virtual light classification model;
deriving corrected color information of the virtual object based on the information about the virtual object and the information about the specific virtual light; and
Generating AR image information in which the virtual object is inserted into the sky based on information about the corrected virtual object including the corrected color information and the first layer and the second layer according to a preset spatial relationship including,
How to generate huge AR image information. The method of claim 1,
In the AR image information, the second layer is disposed at the front, the first layer is disposed at the rear, and information on the corrected virtual object is disposed and synthesized between the second layer and the first layer.
How to generate huge AR image information. The method of claim 1,
The information on the virtual object includes reference position information of the virtual object;
The information on the specific virtual light includes incident angle information and height information of the specific virtual light to be given to the virtual object;
The incident angle information and the height information are derived based on current date information, current time information, and the reference position information,
How to generate huge AR image information. The method of claim 1,
The information on the virtual object includes reference position information of the virtual object;
The information on the specific virtual light includes distance information to be given to the virtual object;
The distance information includes contrast control information and saturation control information derived based on distance information between the location information of the user terminal and the reference location information,
The contrast adjustment information is set so that the contrast between the sky area and the virtual object decreases as the distance according to the distance information increases,
The saturation adjustment information is set so that the saturation of the virtual object decreases as the distance according to the distance information increases.
How to generate huge AR image information. The method of claim 1,
The information on the virtual object includes reference position information of the virtual object and reference size information of the virtual object;
deriving distance information between current location information and the reference location information of the user terminal; and
Deriving corrected size information of the virtual object based on the distance information and the reference size information;
The corrected information on the virtual object includes the corrected size information.
How to generate huge AR image information. The method of claim 1,
The information on the virtual object includes reference position information of the virtual object and reference direction information of the virtual object;
deriving direction information from a current location according to the current location information of the user terminal to a reference location according to the reference location information; and
Further comprising deriving corrected direction information of the virtual object based on the reference direction information and the direction information;
The information on the corrected virtual object includes the corrected direction information.
How to generate huge AR image information. An electronic device that performs a method for generating augmented reality (AR) image information,
filming device;
at least one processor; and
a memory storing at least one instruction executed by the at least one processor;
The at least one command,
Execute to acquire information about a virtual object from an external server;
Execute to obtain original image information including the sky through the photographing device,
Deriving a first layer corresponding to the sky region from the original image information based on a previously learned sky region detection model;
Deriving a second layer corresponding to an area other than the sky area from the original image information based on the first layer;
Execute to derive a specific virtual light from among a plurality of virtual lights from the first layer based on a pre-learned virtual light classification model;
Execute to derive corrected color information of the virtual object based on the information on the virtual object and the information on the specific virtual light;
Execute to generate AR image information in which the virtual object is inserted into the sky based on information about the corrected virtual object including the corrected color information, the first layer, and the second layer according to a preset spatial relationship. felled,
electronic device. A computer-readable recording medium recording a program for executing a method of generating augmented reality (AR) image information performed by a user terminal,
The method for generating the giant AR image information,
obtaining information about a virtual object from an external server;
obtaining original image information including the sky through a photographing device of the user terminal;
deriving a first layer corresponding to the sky region from the original image information based on a previously learned sky region detection model;
deriving a second layer corresponding to an area other than the sky area from the original image information based on the first layer;
deriving a specific virtual light from among a plurality of virtual lights from the first layer based on a pre-learned virtual light classification model;
deriving corrected color information of the virtual object based on the information about the virtual object and the information about the specific virtual light; and
Generating AR image information in which the virtual object is inserted into the sky based on information about the corrected virtual object including the corrected color information and the first layer and the second layer according to a preset spatial relationship including,
A computer-readable recording medium. A program stored in a computer-readable recording medium for performing a method of generating augmented reality (AR) image information through a user terminal,
The method for generating the giant AR image information,
obtaining information about a virtual object from an external server;
obtaining original image information including the sky through a photographing device of the user terminal;
deriving a first layer corresponding to the sky region from the original image information based on a previously learned sky region detection model;
deriving a second layer corresponding to an area other than the sky area from the original image information based on the first layer;
deriving a specific virtual light from among a plurality of virtual lights from the first layer based on a pre-learned virtual light classification model;
deriving corrected color information of the virtual object based on the information about the virtual object and the information about the specific virtual lighting; and
Generating AR image information in which the virtual object is inserted into the sky based on information about the corrected virtual object including the corrected color information and the first layer and the second layer according to a preset spatial relationship including,
program.

Images