WO2018174499A2

WO2018174499A2 - Method for implementing augmented reality image by using virtual marker and vector

Info

Publication number: WO2018174499A2
Application number: PCT/KR2018/003188
Authority: WO
Inventors: 이승학
Original assignee: 주식회사 라이커스게임
Priority date: 2017-03-20
Filing date: 2018-03-19
Publication date: 2018-09-27
Also published as: WO2018174499A3

Abstract

Provided is a computing device-implemented method for implementing an augmented reality image. The method for implementing an augmented reality image comprises the steps of: acquiring a first layer indicating a real world image acquired by a computing device; identifying at least one object contained in the first layer; determining a first marker image on the basis of an image corresponding to the at least one object among previously stored images; matching positions of the first marker image and the at least one object; generating a second layer on the basis of the first marker image; generating an augmented reality image through composition of the first layer and the second layer; and outputting the augmented reality image.

Description

Augmented Reality Image Implementation Using Virtual Markers and Vectors

The present invention relates to a method for implementing augmented reality image using a vector.

Augmented reality refers to computer graphics technology that blends real world images and virtual images that users see with one image. Augmented reality synthesizes an image relating to a virtual object or information to a specific object of the real world image.

Conventionally, physical marker images or location information (eg, GPS location information) are used to identify an object to synthesize a virtual image. In the case of using the physical marker image, the camera of the computing device may not accurately photograph the physical marker image due to the shaking of the user's hand, and thus, the augmented reality image may not be precisely implemented. On the other hand, when using location information, there is a problem that the augmented reality image is not implemented due to the limited or malfunction of the GPS location recognition of the computing device according to the influence of the surrounding environment.

Accordingly, there is a need for a method of implementing an augmented reality image that is independent of physical marker images and location information.

An object of the present invention is to provide a method and program for implementing augmented reality image that can prevent the augmented reality content is not displayed and disconnected continuously as the physical marker image shakes.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

According to an aspect of the present invention, there is provided a method of implementing an augmented reality image implemented by a computing device, the method including: obtaining a first layer indicating an image of a real world acquired by the computing device; Identifying at least one object included in the first layer; Determining a first marker image based on an image corresponding to the at least one object among pre-stored images; Matching a position of the at least one object with the first marker image; Generating a second layer based on the first marker image; Generating an augmented reality image by combining the first layer and the second layer; And outputting the augmented reality image.

According to the present invention, the augmented reality content is not displayed continuously and is cut off as the marker image is shaken in the augmented reality image.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic conceptual view illustrating a virtual reality image implementation method according to an embodiment of the present invention.

2 is a block diagram illustrating the inside of a terminal that provides augmented reality.

3 is a flowchart illustrating a first embodiment of a method for providing augmented reality.

4 is a flowchart illustrating a second embodiment of a method for providing augmented reality.

Here, providing a pre-stored image to the user; Obtaining a user command including image information from the user; And determining a second marker image based on the image information. The second marker image may be further considered in generating the second layer.

Here, the pre-stored image may include an outline vector value.

Here, the user command may include contour vector information of an image to be used as the second marker image.

Here, the user command may include information of an inner point and an outer point of the image to be used as the second marker image.

The first marker image may be transparently generated to be recognized by the computing device and not to be recognized by the user.

Here, the second layer may include augmented reality content corresponding to at least one of the first marker image and the second marker image, and the augmented reality content may mean a virtual image that appears in the augmented reality image.

Here, the object arrangement state of the first layer may be checked based on a vector, and the providing form of the augmented reality content may be determined based on the object arrangement state.

Here, the present invention may include a computer readable medium recording a program for executing the augmented reality image implementation method described.

The application may include an application for a terminal device stored in a medium in order to execute the augmented reality image implementation method described in combination with the computing device, which is hardware.

Other specific details of the invention are included in the detailed description and drawings.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

An augmented reality image implementation method using vectors according to an embodiment of the present invention is realized by a computing device. The augmented reality image implementation method may be implemented as an application, stored in the computing device and performed by the computing device.

For example, the computing device may be provided as a mobile device such as a smart phone, a tablet, and the like, but is not limited thereto. A computing device may be provided with a camera and may process and store data. That is, the computing device may be provided as a wearable device such as glasses or a band equipped with a camera. Any computing device that is not illustrated may be provided.

Although not explicitly shown, the computing device may communicate with other computing devices or servers via a network. In some embodiments, the method of implementing augmented reality images may be realized by linking the computing device with another computing device or server.

Referring to FIG. 1, the computing device 100 captures a space 10 of a real world to obtain a real world image. For example, assume that a plurality of

real objects

11, 12, 13, and 14 exist in real world space 10. The plurality of

real objects

11, 12, 13, and 14 may include any object in two or three dimensions. The plurality of

real objects

11, 12, 13, and 14 may have different or similar shapes. The computing device 100 may distinguish objects based on these morphological differences.

The computing device 100 may identify the plurality of

objects

21, 22, 23, 24 in the real world image. The computing device 100 may extract the outlines of the identified plurality of

objects

21, 22, 23, and 24. The computing device 100 determines an object matching the pre-stored image among the plurality of

objects

21, 22, 23, and 24 by using the vector value of the contour of the pre-stored image.

The computing device 100 may store image samples corresponding to the plurality of

objects

21, 22, 23, and 24 in advance. Data regarding the contour of the image sample corresponding to the plurality of

objects

21, 22, 23, and 24 may be stored in advance.

For example, when the first object 21 is in the shape of a mountain, the computing device 100 may retrieve an image sample similar to the shape of the previously stored first object 21. The computing device 100 may use a pre-stored image sample as a marker image to be described later.

Types of marker images may include a first marker image and a second marker image. The first marker image may indicate a marker image obtained based on the first layer to be described later. That is, the first marker image may indicate the marker image determined based on the actual image, not determined by the user. For example, suppose that the first layer reflecting the real image has a calendar and a frame distinguished from the background, where the first marker image is a transparent marker generated based on the outline and shape of the calendar and the frame in the first layer. Can be. Here, the marker may play a role of generating augmented reality content later.

The second marker image may indicate the marker image obtained based on the information input from the user. For example, the user may arbitrarily set augmented reality content (stars, explosions, characters, etc.) to appear on the display screen. In this case, the second marker image may be used while the user sets the augmented reality content to appear. Here, the second marker image may be a transparent marker previously stored in the first layer based on the outline and shape of the augmented reality content (star, explosion, character, etc.).

In some embodiments, data relating to the contours of the plurality of

objects

21, 22, 23, 24 may be provided in a three-dimensional type. Data relating to the image or contour of the plurality of

objects

21, 22, 23, and 24 may be transmitted to and stored in the computing device 100 from another computing device or server. Meanwhile, images of the plurality of

objects

21, 22, 23, and 24 captured by the user may be stored in the computing device 100 in advance. The data about the contour of the extracted object may be stored in the form of a vector value, that is, as a vector image. Here, the user may indicate a user who implements augmented reality through the computing device 100.

In the augmented reality image implementation method according to an embodiment of the present invention, since a vector image is used instead of a bitmap image, an augmented reality image can be precisely implemented. Depending on the shooting environment of the real world, even if the distance, direction, position, etc. of the object changes from the computing device 100, by appropriately changing the vector image of the object (i.e. any By responding to various forms), it is possible to accurately identify objects within real-world images.

The computing device 100 determines an object 22 that matches the object of the plurality of

objects

21, 22, 23, and 24, and synthesizes the virtual image 40 around the determined object 22 to augmented reality image. Implement

In some embodiments, the user may designate at least one

area

31, 32 in the real world image. The computing device 100 may use the

objects

22 and 24 in the

areas

31 and 32 designated by the user as object candidates, and determine whether the corresponding

objects

22 and 24 match the objects. Or, in substantially the same way, the user may designate at least one

object

22, 24 in the real world image as an object candidate.

Referring to FIG. 2, the computing device 100 may include at least an image acquisition unit 101, a sensor unit 102, an object recognition unit 103, a first layer generator 104, a user command input unit 105, and a user. Command editing unit 106, marker image generation unit 107, image matching unit 108, second layer generation unit 109, second layer storage unit 110, image synthesizing unit 111, display control unit 112 ) May include one of the display 113. Each component may be controlled by a processor (not shown) included in the computing device 100.

The image acquirer 101 may capture a real world image. The image acquirer 101 may acquire a real world image by photographing. The real world image may include a plurality of

real objects

11, 12, 13, and 14. The plurality of

real objects

11, 12, 13, and 14 may have different or similar shapes. The image acquisition unit 101 may be a camera or the like.

The sensor unit 102 may be equipped with devices that support GPS. The sensor unit 102 may recognize a location of the captured image, a direction in which the computing device 100 photographs, a moving speed of the computing device 100, and the like.

The object recognizer 103 may recognize the plurality of

real objects

11, 12, 13, and 14 based on the outlines of the plurality of

real objects

11, 12, 13, and 14 included in the real world image. have. The object recognizer 103 recognizes the plurality of reality objects 11, 12, 13, and 14 based on the contours of the plurality of reality objects 11, 12, 13, and 14, and recognizes the plurality of reality objects ( A plurality of

objects

21, 22, 23, and 24 corresponding to 11, 12, 13, and 14 may be created in the computing device 100.

The first layer generator 104 may generate a first layer that indicates a real image corresponding to the real world image. The augmented reality image may be implemented through the synthesis of the real image and the virtual image. In the present invention, the first layer generator 104 may generate a real image based on the real world image photographed by the image acquirer 101.

The user command input unit 105 may receive a command to output another object distinguished from the plurality of

objects

21, 22, 23, and 24 from a user who uses the computing device 100. For example, the user may recognize the plurality of

objects

21, 22, 23, and 24 from the computing device 100. If the user wants to replace the first object 21 with another object, the user may input a command to the computing device 100 to request that the first object 21 be replaced with another previously stored object. Alternatively, the user may input a command to the computing device 100 requesting to replace the first object 21 with an object that the user directly inputs (or draws) to the computing device 100. The user command may include information about an inner point and an outer point of the image to be used as the marker image.

The user command editing unit 106 may edit at least one of the plurality of

objects

21, 22, 23, and 24 based on the user command obtained from the user command input unit 105.

For example, when the user command input unit 105 receives a command from the user requesting to replace the first object 21 with another previously stored object, the user command editing unit 106 may store the first object 21 in advance. You can edit to change to another object.

The marker image generator 107 may generate a marker image based on the plurality of

objects

21, 22, 23, and 24. The marker image may be an image for generating augmented reality content.

For example, it may be assumed that the computing device 100 provides a virtual reality image of making a stone included in the reality image golden. If it is assumed that the second object 22 is a stone, the marker image generator 107 may generate a marker image capable of generating gold based on the vector value of the second object 22.

The marker image may be recognized by the computing device 100. The marker image may be generated transparently so as not to be recognized by the user.

The image matching unit 108 may correspond to the position between the marker images of the generated plurality of

objects

21, 22, 23, and 24 and the plurality of

objects

21, 22, 23, and 24. The image matching unit 108 moves the marker image to correspond to the plurality of

objects

21, 22, 23, and 24 when the positions of the plurality of

objects

21, 22, 23, and 24 change in real time. You can.

The second layer generator 109 may recognize marker images of the generated plurality of

objects

21, 22, 23, and 24. The second layer generator 109 may generate a second layer in which augmented reality content corresponding to each position of the marker images of the plurality of

objects

21, 22, 23, and 24 generated is combined. Augmented reality content may be identified by the user.

The second layer storage unit 110 may store the second layer generated from the second layer generator 109. The second layer generated based on the marker image may provide the user with a continuous screen that is not broken even when the positions of the plurality of

objects

21, 22, 23, and 24 change in real time.

The image synthesizer 111 may generate an augmented reality image by combining the first layer and the second layer. That is, the augmented reality image may be an image including augmented reality content in a real world image. For example, if there are stones in the real world image acquired through the computing device 100, the image synthesizing unit 111 may generate an image in which only the stones are displayed in gold.

The display controller 112 may control the display 113 to output an augmented reality image. The display unit 113 may output the augmented reality visual screen.

3 may refer to the operation of the computing device 100 when there is no user command. Referring to FIG. 3, the computing device 100 may generate a first layer based on the real world image (S310). The computing device 100 may identify at least one object in the first layer (S320). The computing device 100 may extract the color, resolution, and vector value of the contour of the first layer.

The computing device 100 may identify at least one object in the first layer based on the color, resolution, vector value of the contour, etc. of the first layer. The detailed object identification process may be as follows. The computing device 100 may segment the image based on the resolution of the first layer. The computing device 100 may classify the divided image for each area. When the number of divided images is greater than the preset number, the computing device 100 may merge the hierarchical regions by adjusting the resolution. For example, the computing device 100 may lower the resolution of the first layer so that the computing device 100 may be divided into a plurality of divided images. The computing device 100 may extract an object that can be independently recognized among the divided images.

The computing device 100 may determine the first marker image based on the image corresponding to the identified object among the pre-stored images (S330).

The computing device 100 may match the position of the object included in the first marker image and the first layer (S340). The computing device 100 may generate a second layer including augmented reality content based on the first marker image (S350).

The computing device 100 may generate an augmented reality image through the synthesis of the first layer and the second layer. Since the augmented reality content is generated based on the first marker image formed based on the first layer rather than the first layer, the augmented reality image including the augmented reality content that does not break even when the first layer is shaken due to the hand shake phenomenon. Can be generated. If the position or viewing angle of the first layer is changed, the computing device 100 stores the stored vector using the vector value of any object of the first layer, the position vector value of the object in the real world, the normal vector value of the object, and the like. The vector value compensation corresponding to the change in the position or the viewing angle in one marker image can be performed.

For example, when the computing device 100 photographs a picture frame in the real world and only the position of the computing device 100 is changed, the angle at which the picture frame is viewed may be changed. In this case, the computing device 100 may correct the vector value of the first marker image by using the vector value of the first marker image corresponding to the frame, the position vector value of the frame of the real world, and the normal vector value of the frame. .

The computing device 100 may visually output the augmented reality image through the display 113 (S360).

4 may refer to an operation of the computing device 100 when there is a user command. Referring to FIG. 4, the computing device 100 may generate a first layer based on the real world image (S310).

The computing device 100 may provide the user with at least one object or at least one image stored in advance (S311). The computing device 100 may provide the user with at least one object (or image) stored in advance at the request of the user. Alternatively, the computing device 100 may automatically provide the user with at least one object (image) stored in advance even without a user's request.

A user who has identified at least one object (or image) previously stored through the computing device 100 inputs a command to the computing device 100 requesting to replace at least one object obtained from the real world image with another previously stored object. can do. A user may directly enter (or draw) an object into computing device 100.

The computing device 100 may obtain a command from the user requesting to replace at least one object obtained from the real world image with another object stored in advance (S312). Alternatively, the computing device 100 may obtain a command for requesting the user to replace at least one object obtained from the real world image with an object input (or drawn) directly to the computing device 100.

The computing device 100 may determine a second marker image among pre-stored images based on the command (S313). The computing device 100 may identify at least one object in the first layer (S320). The computing device 100 may extract the color, resolution, and vector value of the contour of the first layer.

The computing device 100 may match the position of the object included in the first marker image and the first layer (S340). The computing device 100 may generate a second layer including augmented reality content based on at least one of the first marker image and the second marker image (S351).

The computing device 100 may generate an augmented reality image through the synthesis of the first layer and the second layer. Since the augmented reality content is generated based on at least one of a first marker image formed based on the first layer and a second marker image formed based on a user command, not the first layer, the first layer may be shaken due to the shaking of the hand. An augmented reality image including an augmented reality content that is not broken even in a case may be generated. If the position or viewing angle of the first layer is changed, the computing device 100 stores the stored vector using the vector value of any object of the first layer, the position vector value of the object in the real world, the normal vector value of the object, and the like. The vector value compensation corresponding to the change in the position or the viewing angle of the first marker image or the second marker image may be performed.

The computing device 100 may generate an augmented reality image through the synthesis of the first layer and the second layer. The computing device 100 may visually output the augmented reality image through the display 113 (S360).

The augmented reality image implementation method according to an embodiment of the present invention described above may be implemented as a program (or an application) and stored in a medium to be executed in combination with a computer which is hardware.

The above-described program includes C, C ++, JAVA, machine language, etc. which can be read by the computer's processor (CPU) through the computer's device interface so that the computer reads the program and executes the methods implemented as the program. Code may be coded in the computer language of. Such code may include functional code associated with a function or the like that defines the necessary functions for executing the methods, and includes control procedures related to execution procedures necessary for the computer's processor to execute the functions according to a predetermined procedure. can do. In addition, the code may further include memory reference code for additional information or media required for the computer's processor to execute the functions at which location (address address) of the computer's internal or external memory should be referenced. have. Also, if the processor of the computer needs to communicate with any other computer or server remotely in order to execute the functions, the code may be used to communicate with any other computer or server remotely using the communication module of the computer. It may further include a communication related code for whether to communicate, what information or media should be transmitted and received during communication.

The stored medium is not a medium for storing data for a short time such as a register, a cache, a memory, but semi-permanently, and means a medium that can be read by the device. Specifically, examples of the storage medium include, but are not limited to, a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like. That is, the program may be stored in various recording media on various servers to which the computer can access or various recording media on the computer of the user. The media may also be distributed over network coupled computer systems so that the computer readable code is stored in a distributed fashion.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims

An augmented reality image implementation method realized by a computing device,

Obtaining a first layer indicating an image of the real world obtained by the computing device;

Identifying at least one object included in the first layer;

Determining a first marker image based on an image corresponding to the at least one object among pre-stored images;

Matching a position of the at least one object with the first marker image;

Generating a second layer based on the first marker image;

Generating an augmented reality image by combining the first layer and the second layer; And

And outputting the augmented reality image.
The method of claim 1,

Providing a user with a pre-stored image;

Obtaining a user command including image information from the user;

Determining a second marker image based on the image information;

And considering the second marker image in the generation of the second layer.
The method of claim 1,

And the prestored image comprises a contour vector value.
The method of claim 2,

The user command includes contour vector information of an image to be used as the second marker image.
The method of claim 2,

The user command may include information about an inner point and an outer point of the image to be used as the second marker image.
The method of claim 1,

The first marker image is recognized by the computing device, transparently generated so as not to be recognized by the user, augmented reality image implementation method.
The method of claim 2,

The second layer may include augmented reality content corresponding to at least one of the first marker image and the second marker image, and the augmented reality content may mean a virtual image that appears in the augmented reality image. Way.
The method of claim 7, wherein

Confirming the object arrangement state of the first layer based on a vector, and determining a form of providing the augmented reality content based on the object arrangement state.
A computer readable medium having recorded thereon a program for executing the augmented reality image implementing method according to any one of claims 1 to 8.
An application for a terminal device, which is combined with the computing device, which is hardware, and stored in a medium for executing the augmented reality image implementing method according to any one of claims 1 to 8.