KR20150125326A - System and method for providing interactive augmented reality service - Google Patents

Abstract

The present invention relates to a system and a method for providing an interactive augmented reality service, and includes an augmented reality processing apparatus which recognizes a printed book which has a page including a marker and enables a user to modify the marker, and a marker in an image obtained by photographing the page of the printed book; and displays a three-dimensional (3D) model corresponding to the modified marker, when the recognized marker is modified, by rendering the model in real time.

Description

TECHNICAL FIELD [0001] The present invention relates to an interactive augmented reality service system,

The present invention relates to an interactive augmented reality service system and method, and more particularly, to a system and method for interactive augmented reality service that recognize a marker in an image acquired through a print book having a page capable of deforming a marker by a user, And more particularly to an interactive augmented reality service system and method in which a 3D model corresponding to a transformation marker is rendered and displayed in real time.

The augmented reality means that a real image and a virtual image are mixed by inserting an image such as a computer graphic into a real environment, and the augmented reality system combines the real world and the virtual world so that the user can interact with the user in real time . It is used for the purpose of interacting with the user 's perception and reality by adding virtual contents to the real environment in a changed form of virtual reality.

Such augmented reality is a form in which the data input by the information provider is exposed according to a specific signal (for example, GPS, image marker, etc.). That is, since the information consumer unilaterally receives the information provided by the information provider, the information consumer could not receive the augment information according to his / her intention.

Korean Patent No. 10-1281314, entitled " Augmented Reality System and Method for Implementing Augmented Reality "

It is an object of the present invention to provide an interactive augmented reality service system and a method for allowing a user who is provided with an augmentation information service based on a marker to receive augmentation information corresponding to his / her intention by modifying a marker.

Another object of the present invention is to provide an interactive augmented reality service system and a method for modifying a 3D model according to a change of a marker or adjusting a scale of each part of a 3D model when a user deforms a marker.

According to an aspect of the present invention, there is provided a method of detecting a marker, comprising the steps of: (a) obtaining an image; (b) recognizing a marker in the obtained image and determining whether the recognized marker is deformed; ) Determining a 3D model according to the marker deformation when the marker is deformed as a result of the determination, (d) masking a border of the marker and extracting an image inside the masking, (e) Determining a real, and (f) applying the determined material to the determined 3D model and displaying the 3D model.

The step (b) includes the steps of extracting a feature point from the acquired image to recognize a border marker, calculating a direction and a position of the virtual space with the recognized marker position, and storing the recognized marker in a marker And judging whether the marker is deformed or not.

If the marker is deformed, the step (c) may include: determining whether the deformation is a predefined deformation, and if the deformation is a predefined deformation, changing the shape of the deformed part in the 3D model corresponding to the marker The scale of the border is determined. If the scale of the border is deformed, the scale of the corresponding part is calculated, and the scale of the 3D model corresponding to the marker is determined And determining the 3D model by applying the calculated scale.

Wherein the step (e) includes the steps of: determining whether the extracted image is a defined image; acquiring a material corresponding to the image from the material information database if the determined result is the determined image; Calculating distance and angle between the image acquiring unit and the marker, processing the extracted image with the calculated distance and angle, and generating material using the processed image.

According to another aspect of the present invention, there is provided an image processing apparatus including an image acquiring unit acquiring an image, a marker recognizing unit recognizing a marker in the acquired image, determining whether the recognized marker is deformed, and determining a 3D model according to the determination result A 3D model determination unit, a material decision unit for masking the edge of the marker, extracting an image of the inside of the masking, and determining materiality in the extracted image; And a rendering unit for applying the material determined in step (a).

The 3D model determination unit compares the recognized marker with the markers stored in the marker information database to determine whether or not the edge of the marker is deformed. If the edge of the marker is deformed, the 3D model determination unit determines whether the deformation is a predefined deformation, In the case of the predefined deformation, the 3D model is determined by replacing the shape of the deformed part in the 3D model corresponding to the marker, and if the deformation is not the predefined deformation, the deformation of the scale of the frame is determined, The 3D model corresponding to the marker can be determined by applying the calculated scale in the 3D model corresponding to the marker.

The material determining unit may mask the edge of the marker, extract an image inside the masking, determine whether the extracted image is a defined image, and if the extracted image is a predefined image, Calculating a distance and an angle between the image acquiring unit and the marker when the image is not a defined image, processing the extracted image at the calculated distance and angle, generating the material using the processed image, can do.

According to another aspect of the present invention, there is provided a method of recognizing a marker, comprising: a board having a page including a marker for augmented reality and capable of deforming the marker by a user; And an augmented reality processing unit for rendering and displaying a 3D model corresponding to a transformation marker when the recognized marker is deformed in real time.

According to the present invention, a user who receives the augmentation information service based on the marker can modify the marker, and the user can receive the augmenting information corresponding to his / her intention.

Further, when the user deforms the marker, the 3D model can be deformed or the scale of each part of the 3D model can be adjusted according to the deformation of the marker.

1 illustrates an interactive augmented reality service system according to an embodiment of the present invention.
2 is a block diagram schematically showing the configuration of an augmented reality processing apparatus according to an embodiment of the present invention;
3 illustrates an interactive augmented reality service method according to an embodiment of the present invention.
4 is a flow diagram illustrating a method for determining a 3D model according to marker deformation according to an embodiment of the present invention.
5 illustrates a method for determining a material by an augmented reality processing apparatus according to an embodiment of the present invention.

The foregoing and other objects, features, and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

Each functional unit represented in the present specification is only an example of implementation of the present invention. Therefore, other functional units may be used in other embodiments of the present invention without departing from the spirit and scope of the present invention. In addition, each functional unit may be implemented solely in hardware or software configuration, but may be implemented in combination of various hardware and software configurations that perform the same function.

1 is a diagram illustrating an interactive augmented reality service system according to an embodiment of the present invention.

1, the augmented reality service system includes a print book 10 having a page including a marker for augmented reality, and a print book 10 for analyzing a photographed image of a page of the print book 10, And an augmented reality processing apparatus 100 that performs a predetermined augmented reality implementing function corresponding to a marker existing in the augmented reality processing apparatus 100 and outputs an augmented reality content to which the execution result is applied to the photographed image.

The printed book 10 is an offline book composed of at least one page including text and / or pictures, and the page may include at least one marker for augmented reality. Here, the marker for the augmented reality is surrounded by the frame, and the image in the frame can be shaped by various images, letters, etc. by the user.

The user can modify the markers displayed on the printed book 10 and the augmented reality processing apparatus 100 can recognize the deformed markers displayed on the printed book 10 and render 3D models corresponding to the recognized deformed markers have. That is, the user can change the border of the marker displayed on the print book 10 to his / her intention. At this time, the user can change the marker displayed on the printed book 10 using a predetermined input device, a writing instrument, or the like, and erase the marker using an eraser or the like. Here, the input device may be a touch pen, a finger or the like in the case of a touch pad, or a whiteboard pen in the case of a white board.

For example, when a user draws a wing of a bird with a larger size on a printing book 10 in which a 2D new figure is indicated by a marker by using an input device, a marker displayed on the printed book 10 is displayed on the display device .

Herein, the device for displaying the marker for the augmented reality has been described as the printing book 10, but it is also possible to use various forms such as a blackboard, a whiteboard, a touchpad, and an electronic board capable of deforming (correcting) a marker while displaying text and / Lt; / RTI > In the concept including all of these various forms, it can be referred to as a board instead of the printed book 10.

The augmented reality processing apparatus 100 photographs an expanded page of the print book 10 located in a specific photographing area and analyzes the photographed image to perform a preset augmented reality implementing function corresponding to a marker existing in the photographed image And outputs the result of the augmented reality applied to the photographed image.

The augmented reality processing apparatus 100 analyzes the photographed image of a page of the printed book 10 and displays it on the external content server 200 based on identification information of the corresponding printed book 10 recognized through a specific marker. It is also possible to obtain the implementation function information for each combination of the markers corresponding to the print book 10. [

Since the process of acquiring the specific information corresponding to the print book 10 from the content server 200 in the augmented reality processing apparatus 100 corresponds to the existing augmented reality service, a detailed description of the acquisition procedure will be omitted.

The augmented reality processing apparatus 100 downloads and installs the augmented reality service application from the content server 200. [

When the augmented reality service application is activated, the augmented reality processing apparatus 100 acquires an image by driving the provided image acquisition unit 110, recognizes the marker in the acquired image, And displays a 3D model corresponding to the marker based on the generated virtual space together with the acquired image to provide an augmented reality. Here, the 3D model may include multimedia contents such as a three-dimensional virtual image, a moving image, and a voice. Or the 3D model may include interactive content that can interact with the user. In addition, the 3D model may include other contents stored in the augmented reality processing apparatus or contents linking other contents stored in an external server connected through a communication network.

The augmented reality processing apparatus 100 recognizes the marker in the image obtained through the image obtaining unit 110 and displays the 3D model corresponding to the deformed marker in real time when the recognized marker is deformed.

The augmented reality processing apparatus 100 includes an image acquisition unit 110 and may be a navigation device, a notebook computer, a mobile communication terminal, a smart phone, a portable media player (PMP) A personal digital assistant (PC), a tablet PC, a set-top box, a smart TV, and the like.

A detailed description of the augmented reality processing apparatus 100 will be given with reference to Fig.

The content server 200 provides the augmented reality service application to the augmented reality processing device 100. [

2 is a block diagram schematically showing the configuration of an augmented reality processing apparatus according to an embodiment of the present invention.

2, the augmented reality processing apparatus 100 includes an image acquisition unit 110, a marker recognition unit 120, a 3D model determination unit 130, a matrial determination unit 140, a rendering unit 150, a display unit 160, and a storage unit 170.

The image obtaining unit 110 is configured to photograph an image according to a user operation, and may include, for example, a camera.

The marker recognition unit 120 recognizes the marker in the image obtained through the image acquisition unit 110. [ The marker recognition unit 120 searches the marker information database 172 stored in the storage unit 170 and recognizes the marker in the acquired image.

The marker recognition unit 120 according to the present invention can recognize the marker included in the external object by comparing the image of the image captured by the image acquisition unit 110 with the image of the previously stored marker.

The marker recognition unit 120 creates a virtual space around the recognized marker. That is, the marker recognition unit 120 calculates the direction and position of the virtual space with the marker position, and displays the 3D model corresponding to the marker based on the calculated virtual space.

The 3D model determination unit 130 determines whether the edge of the marker recognized by the marker recognition unit 120 is deformed, and determines the 3D model according to the determination result.

That is, the 3D model determination unit 130 compares the markers recognized by the marker recognition unit 120 with the markers stored in the marker information database 172, and determines whether the markers are deformed. If the edge of the marker is deformed as a result of the determination, the 3D model determination unit 130 determines whether the deformation is a predefined deformation. As a result of the determination, the 3D model determination unit 130 determines the 3D model by replacing the shape of the deformed portion in the 3D model corresponding to the marker. For example, when a specific portion on the marker is checked or painted with a specific color, the 3D model determination unit 130 replaces only the 3D model type of the portion with the predetermined rule, and determines the replaced 3D model as the final 3D model .

If the marker edge deformation is not a predefined deformation, the 3D model determination unit 130 determines whether or not the edge scale is deformed. If the edge scale is deformed, the 3D model determination unit 130 calculates a scale of the edge scale, The calculated scale is applied to the 3D model to determine the 3D model. At this time, the 3D model determination unit 130 searches the 3D model information database stored in the storage unit 170 to select a 3D model corresponding to the marker, and applies the calculated scale to the selected 3D model. That is, when the 3D model determining unit 130 detects another edge line other than the edge line of the original marker, the 3D model determining unit 130 obtains the distance ratio between the distance of the original edge line from the center of the marker and the distance of the other edge lines sensed, Based on the above, the conversion scale of the corresponding 3D model is determined, and the determined conversion scale is applied to the corresponding 3D model. For example, if a 2D new image is used as a marker, if the user draws the wing on the marker larger, the 3D model is determined by making the wing larger than the wing of the original 3D model.

The material decision unit 140 masks the edge of the recognized marker, extracts the image inside the masking, and determines the material using the extracted image.

That is, the material determining unit 140 masks the edge of the marker, extracts the image of the inside of the masking, and determines whether the extracted image is a defined image. Here, the extracted image may be in various forms such as text, symbols, and images. As a result of the determination, the material determining unit 140 acquires the material corresponding to the image from the material information database 176 when the image is a predefined image. Since the edge of the marker can be composed of a plurality of closed 2D figures, the material decision unit 140 can obtain a pattern inside the pattern by masking each closed 2D figure, and judges whether the obtained pattern is a defined image.

For example, if the extracted image is the text 'Dad', the material decision unit 140 can acquire a material corresponding to 'Dad' from the material information database 176. If the extracted image is a 'star' symbol, the material determining unit 140 may obtain the material corresponding to the 'star' symbol from the material information database 176.

If the extracted image is not a defined image, the material determining unit 140 calculates the distance and angle between the image obtaining unit 110 and the marker, and processes the extracted image at the calculated distance and angle And creates the material with the processed image.

The rendering unit 150 applies the material determined by the material decision unit 140 to the 3D model determined by the 3D model determination unit 130, and causes the display unit 160 to display the 3D model.

The rendering unit 150 may convert the image acquired by the image acquisition unit 110 into an image that can be displayed on the display unit 160. Specifically, the rendering unit 150 may enhance a previously prepared virtual image at a location where the marker is displayed to generate a new image. Herein, the reinforcement includes a rendering function that gives a color or an effect through three-dimensional computer graphics (CG) to each surface of an actual object drawn on the screen, thereby realizing a three-dimensional feeling and a real feeling of an actual object on the screen Can be improved.

The storage unit 170 stores a program necessary for controlling the operation of the augmented reality processing apparatus 100 and data generated during the execution of the program. The storage unit 170 stores an augmented reality service application.

A marker information database 172, a 3D model information database 174, and a material information database 176 are stored in the storage unit 170. The marker information database 172 stores markers according to images and 3D models set for the respective markers. The 3D model information database 174 stores 3D models, 3D models to be transformed according to partial deformations of the respective 3D models, and the like. The material information database 176 stores information on materials for each image.

At this time, the marker information database 172, the 3D model information database 174, and the material information database 176 may exist inside or outside the augmented reality processing apparatus 100.

3 is a diagram illustrating an interactive augmented reality service method according to an embodiment of the present invention.

Referring to FIG. 3, when an image is acquired (S302), the augmented reality processing apparatus recognizes a border marker in the acquired image (S304).

The augmented reality processing apparatus determines whether the recognized border marker is present in the predefined marker information database (S306).

If it is determined in step S306 that the recognized marker exists, the augmented reality processing apparatus calculates the direction and position of the virtual space with the recognized marker position (S308), compares the recognized marker with the markers stored in the marker information database And determines whether or not it is a modification marker (S310).

If it is determined as a result of the determination in step S310, the augmented reality processing apparatus determines a 3D model corresponding to the marker deformation (S312). A detailed description of how the augmented reality processing device determines the 3D model according to the marker deformation will be described with reference to Fig.

When the step S312 is performed, the augmented reality processing apparatus performs masking on the shape of the edge of the deformed marker (S314), and extracts the image inside the masking (S316).

Thereafter, the augmented reality processing apparatus determines the material using the extracted image (S318), and applies the determined material to the determined 3D model and displays it (S320). A detailed description of how the augmented reality processing apparatus determines the material from the extracted image will be described with reference to Fig.

If it is determined as a result of the determination in S310, the augmented reality processing apparatus displays the 3D model corresponding to the recognized marker (S322). At this time, the augmented reality processing apparatus displays a 3D model corresponding to the recognized marker in the direction and position of the calculated virtual space.

4 is a flowchart illustrating a method of determining a 3D model according to marker deformation according to an embodiment of the present invention.

Referring to FIG. 4, the augmented reality processing apparatus determines whether the marker deformed image is a predefined deformed image (S404).

As a result of the determination in step S404, the augmented reality processing apparatus determines the 3D model by replacing the shape of the corresponding part in the 3D model corresponding to the marker (S406).

If it is determined in step S404 that the deformed image is not a deformed image, the augmented reality processing apparatus determines whether the border scale of the deformed marker is deformed (S410).

If it is determined in step S410 that the border scale of the deformed marker is deformed, the augmented reality processing apparatus calculates the scale of the corresponding part (S412) and determines the 3D model by applying the calculated scale to the 3D model (S414) .

5 is a diagram illustrating a method for determining a material by an augmented reality processing apparatus according to an embodiment of the present invention.

Referring to FIG. 5, the augmented reality processing apparatus determines whether an image in the masking is input (S502) and whether the image is a defined image (S504).

If it is determined in step S504 that the augmented reality processing apparatus obtains a material corresponding to the image from the material information database (S506), the augmented reality processing apparatus determines the obtained material as a material of the corresponding 3D model (S508).

If it is determined in step S504 that the image is not a predefined image, the augmented reality processing apparatus calculates the distance and angle between the image obtaining unit and the marker (S510), and processes the extracted image with the calculated distance and angle (S512).

Then, the augmented reality processing apparatus generates a material using the processed image (S514), and determines the generated material as a material of the corresponding 3D model (S508).

Meanwhile, the interactive augmented reality service method can be written as a program, and the codes and code segments constituting the program can be easily deduced by a programmer in the field.

Thus, those skilled in the art will appreciate that the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the embodiments described above are to be considered in all respects only as illustrative and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: augmented reality processing device 110:
120: marker recognition unit 130: 3D model determination unit
140: matrial decision unit 150: rendering unit
160: Display section 170:

Claims (8)

(a) acquiring an image;
(b) recognizing a marker in the obtained image and determining whether the recognized marker is deformed;
(c) if the marker is deformed as a result of the determination, determining a 3D model according to marker deformation;
(d) masking an edge of the marker and extracting an image of the inside of the masking;
(e) determining the material image as the extracted image; And
(f) applying and displaying the determined material on the determined 3D model;
And an interactive augmented reality service method.
The method according to claim 1,
The step (b)
Extracting a feature point from the obtained image and recognizing a border marker; And
Calculating a direction and a position of the virtual space with the recognized marker position and comparing the recognized marker with the markers stored in the marker information database to determine whether the marker is deformed, Service method.
The method according to claim 1,
The step (c)
Determining whether the distortion is a predefined distortion when the marker is deformed; And
As a result of the determination, if the 3D model corresponding to the marker is deformed, the 3D model is determined by replacing the shape of the deformed part. If the 3D model is not the previously defined deformed shape, Calculating a scale of the part when the scale is deformed, and determining the 3D model by applying the calculated scale in the 3D model corresponding to the marker.
The method according to claim 1,
The step (e)
Determining whether the extracted image is a defined image;
Acquiring a material corresponding to the image from the material information database when the determination result is that the image is a defined image, calculating a distance and an angle between the image acquiring unit and the marker when the image is not a defined image, Processing the extracted image with the processed image, and generating material using the processed image.
An image acquiring unit acquiring an image;
A marker recognition unit for recognizing a marker in the acquired image;
A 3D model determination unit for determining whether the recognized marker is deformed and determining a 3D model according to the determination result;
A material decision unit for masking an edge of the marker, extracting an image inside the masking, and determining the material in the extracted image; And
A rendering unit for applying the material determined in the material determining unit to the 3D model determined by the 3D model determining unit;
Wherein the augmented reality processing unit includes:
6. The method of claim 5,
The 3D model determination unit compares the recognized marker with the markers stored in the marker information database to determine whether or not the edge of the marker is deformed. If the edge of the marker is deformed, the 3D model determination unit determines whether the deformation is a predefined deformation, In the case of the predefined deformation, the 3D model is determined by replacing the shape of the deformed part in the 3D model corresponding to the marker, and if the deformation is not the predefined deformation, the deformation of the scale of the frame is determined, Wherein the 3D model is determined by calculating a scale of the portion and applying the calculated scale in the 3D model corresponding to the marker.
6. The method of claim 5,
The material determining unit may mask the edge of the marker, extract an image inside the masking, determine whether the extracted image is a defined image, and if the extracted image is a predefined image, Calculating a distance and an angle between the image acquiring unit and the marker when the image is not a defined image, processing the extracted image at the calculated distance and angle, generating the material using the processed image, Wherein the augmented reality processing unit comprises:
A board having a page including a marker for an augmented reality and capable of deforming the marker by a user; And
An augmented reality processing device for recognizing a marker in an image obtained by photographing a page of the board and rendering and displaying a 3D model corresponding to a deformed marker in real time when the recognized marker is deformed;
And an interactive augmented reality service system.

Images