KR20120122512A - 3D augmented reality photo service system, and service method using thereof - Google Patents

Abstract

PURPOSE: A three-dimensional augmented reality photo service system and a service method using the same are provided to supply a photo image service by utilizing an augmented reality technology. CONSTITUTION: A photo service client(10) generates a depth map about a service location image(S300). The photo service client layers the service location image based on the depth map(S400). The photo service client inserts an virtual object between the layered images(S430). A photo service server(30) matches the images based on context information of a user(S610). The photo service client photographs a subject by using the matched image(S700). The photo service client stores the photographed image(S800). [Reference numerals] (10) Photo service client; (30) Photo service server; (AA) Start; (BB) Context information transmission; (CC) End; (S100) Location information transmission?; (S120) Transmission of service information registered in location information; (S125) Camera preview; (S127) Characteristic points are detected?; (S130) Service information display in camera preview; (S140) Selecting service; (S200) Virtual object exists?; (S210) Require transmission to server; (S215) Searching database; (S217) Transmitting virtual object; (S230) Loading virtual object; (S300) Generating depth map; (S350) Based on characteristic point?; (S400) Layering image based on characteristic point; (S410) Layering image of two or more layers according to depth; (S430) Inserting images of virtual object; (S500) Micro-controlling virtual object; (S600) Requiring work to a server?; (S610) Accordance of images based on context information; (S620) Result video transmission; (S630) Accordance of images based on context information; (S700) Photographing; (S800) Storing images

Description

3D augmented reality photo service system, and service method using equivalent}

The present invention relates to an augmented reality video service system using augmented reality and stereoscopic imaging technology and a method using the same. More specifically, augmented reality is a reality using a method for extracting and tracking location information, depth information and feature points, etc. The present invention relates to various and realistic 3D augmented reality service systems and methods of matching virtual objects to scenes.

Stereoscopic imaging technology uses the principle that the image of an object viewed by human's right eye and left eye from different angles is combined in the brain and appears as one stereoscopic image. It is a technology that makes you feel. With the success of movie avatars, stereoscopic image technology has been rapidly spreading from the movie industry and is being applied to TVs, smartphones and personal cameras.

Augmented reality technology that adds additional images to the double-reality video is widely used. Currently, the technology applied to the widely used smart phone is matched with the real image even if the 2D image is added to the real image or the 3D image is provided. There is a problem that it is unnatural as an additional image that does not become, and takes up a lot of relative space on the screen, and it is difficult to properly adjust its size and position.

In addition, in the conventional technology, when a text or a two-dimensional image is added to a smartphone or a camera and provided, the image displayed on the screen is not only complicated and coarse, but also lacks reality, and the two-dimensional image is displayed on an actual screen. There is a problem in that it cannot provide an additional image suitable for the change of the displayed camera image or the adjustment of the user, and cannot provide various services in that it can be used only in a limited range.

An object of the present invention to solve the above problems is to provide a realistic photo image service system and method utilizing augmented reality technology and stereoscopic image technology. In addition, it is possible to provide a variety of interesting video services according to the user's preferences or selection, to use a client terminal having a low image processing capability using a wireless network, to increase the processing speed and to provide a variety of services to expand For sake.

A first aspect of the present invention for solving the above problems is a method using a system consisting of a client, a wireless network and a photo service server, the method comprising: (a) a service location of the client through service information received through the service server; Selecting a photo service; (b) loading the virtual object on the service location image when the virtual object corresponding to the service exists; (c) generating a depth map of the service location image; (d) layering the service location image based on the depth map; (e) inserting the virtual object between the layered images; (f) registering an image based on context information of the user, and photographing using the matched image; And (g) storing the photographed image.

Here, step (a) may include: transmitting, by the client, location information to the photo service server; Transmitting, by the photo service server, service information registered in the location information to the client; It is preferable to include the step of selecting a photo service through the service information displayed on the client.

Also, preferably, when the location information is not transmitted, searching for a feature point in a camera image of the client; If the feature point is found, it may include selecting the photo service, and the step (b) may include: requesting transmission of the virtual object to the service server when the virtual object does not exist; Searching the database by the service server; The service server may include transmitting the searched virtual object to the client.

In addition, in the step (d), if the feature point exists, the image is layered based on the feature point, and if the feature point is absent, it is preferable that the image is layered into at least two layers according to the depth, and (e) After the step, the client preferably further comprises the step of fine-tuning the virtual object.

In addition, the fine adjustment is preferably to adjust the size, direction and position of the virtual object, the step (f), the step of transmitting the context information to the service server; Preferably, the service server includes the step of registering the context information and a result image in which the virtual object is inserted, and the result image includes at least one of current location information, time information, image information, and sensor information. It is desirable to.

The second aspect of the present invention is a 3D augmented reality photo service system used in any one of the above-described methods, wherein the service system is a service server that provides service information, virtual objects, image processing, and image storage functions. ; And a communication unit wirelessly communicating with the service server, a stereoscopic image acquisition device for acquiring real stereoscopic image information, and extracting feature points from the acquired image information, generating a depth map, and layering the image according to depth by two or more levels. an image preprocessor for separating the virtual object between the separated image layers, and correcting at least one of a position, a size, and a direction of the inserted virtual object, and selecting a user It is preferable to include a client terminal composed of an image post-processing unit to match the image by applying an additional effect to the virtual object using the context information.

According to the present invention, it is differentiated from the augmented reality service that adds conventional 2D information or simply adds a 3D object, and matches the virtual object according to a 3D virtual object depth map. It is possible to provide a natural and realistic photo image service.

In addition, various and interesting photo image services may be provided by adding context information according to a user's preference or selection. In addition, by selectively applying an image processing method through a service server using a wireless network, it is possible to use a client terminal having a low image processing capability, to increase the processing speed, and to provide various services by expanding the service. have.

1 is a flowchart illustrating a 3D augmented reality photo service method according to an embodiment of the present invention.
Figure 2 is a schematic diagram showing a simplified 3D augmented reality photo service method according to the present invention,
3 is a view schematically illustrating the flow of a 3D augmented reality photo service method according to the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and how to accomplish it, will be described with reference to the embodiments described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. The embodiments are provided so that those skilled in the art can easily carry out the technical idea of the present invention to those skilled in the art.

In the drawings, embodiments of the present invention are not limited to the specific forms shown and are exaggerated for clarity. In addition, parts denoted by the same reference numerals throughout the specification represent the same components.

The expression "and / or" is used herein to mean including at least one of the components listed before and after. In addition, the singular also includes the plural unless specifically stated otherwise in the text. Also, components, steps, operations and elements referred to in the specification as " comprises "or" comprising " refer to the presence or addition of one or more other components, steps, operations, elements, and / or devices.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

 1 is a flowchart illustrating a 3D augmented reality photo service method according to an embodiment of the present invention. As shown in Figure 1, the service method according to the present invention is a method using a system consisting of a client, a wireless network and a photo service server, (a) the service location of the client through the service information received through the service server Selecting a photo service at step S140; (b) loading the virtual object on the service location image when the virtual object corresponding to the service exists (S230); (c) generating a depth map of the service location image (S300); (d) layering the service location image based on the depth map (S410, S400); (e) inserting the virtual object into the layered image (S430); (f) registering images based on context information of the user (S610 and S630) and photographing using the matched image (S700); And (g) storing the captured image (S800).

Here, the user terminal as a client is a terminal or system with a built-in camera for implementing and executing the photo service, a virtual object and a server unit for providing image processing and image storage functions; A stereoscopic image acquisition device for acquiring stereoscopic image information of reality; An image preprocessing unit extracting feature points from the acquired image information, generating a depth map to separate the image into two or more layers according to depths, and inserting the virtual object between the separated image layers; The inserted virtual object is configured to include a post-processing unit for correcting a position, a size, a direction, applying an additional effect to the virtual object using context information related to the current environment, and matching the image. Not)

Hereinafter, a preferred 3D augmented reality photo service method of the present invention described below using a system, a wireless network (Internet network, YiFi, etc.) and a service server will be described in detail with reference to the accompanying drawings.

First, the video service client transmits the current location information using the wireless network to the service server in order to search for the video service for the service registered in the current location information (S100). If so, the service information is transmitted to the client (S120). The client displays the received service information and waits for a user's selection (S130), and the user can select a service through the client terminal (S140).

If a QR code or a specific marker is used at the current location without using the service search using the location information, the service may be selected by searching for a feature point in the preview image of the camera (S125). It is also possible to recognize the location through a wireless network and to implement a 3D augmented reality photo service, but if a marker or QR code exists in the service location, the client terminal can recognize it and apply it to the service.

As such, when the user selects a service or finds a feature point in the image on the service information display screen of the client terminal 10, it is checked whether the virtual object exists in the client (S200). Here, the virtual object implements augmented reality. As a 3D stereoscopic image object, the augmented reality photo service is implemented by adding the virtual object to the live screen captured by the camera.

If the client has a virtual object, the client loads itself (S230). If the client does not exist, the client requests a virtual object transmission to the video service server. (S210) The video service server 30 receives a virtual object transmission request. The virtual object is extracted from the object database (S215) and transmitted to the client (S217).

The client generates a depth map of the image of the real scene. (S300) The depth map refers to an image or an image channel according to the height or depth of the object from the viewpoint in 3D computer graphics according to the height, 3D image. In order to implement a specific image, a stereoscopic image can be realized by layering or layering an image based on the depth map as described above, and synthesizing the image.

Then, using the depth map, the client separates the image into hierarchies based on the feature points if the feature points exist in the image (S400), and separates the image into two or more hierarchies according to the depth if the feature points exist (S410).

The virtual object image is inserted into the separated image (S430), and if fine adjustment is required, the user may manually adjust the virtual object with respect to the position, direction, and size (S500). The purpose is to implement a customized service by allowing the virtual object to be selected in size, position, and orientation depending on the selection.

 The matching of the resultant image into which the virtual object is inserted may be performed by the client or by the service server according to a user's selection. When the result is matched by the client, current location information, time information, and image information may be performed. Using the automatically acquired information such as sensor information and the context information provided from the user, the effect is applied to the virtual object according to the current situation and then the separated images are matched.

In addition, the client transmits the user's context information while requesting an operation from the client (S600), and the service server applies the effect to the virtual object using the received context information and matches the separated image (S610). The client transmits the image to the client (S620). The client takes a photograph and stores the resultant image.

Here, the context information refers to various effects of the camera image that can be implemented in the client terminal 10 and can be directly selected by the user. The context information is added to the virtual object and can be configured by adding various effects (snow, rain, fallen leaves, seasonal effect images, etc.), so that the user can be provided with a richer and more interesting photo service.

Figure 2 is a schematic diagram showing a brief 3D augmented reality photo service method according to the present invention. As shown in Figure 2, it shows that the video service to restore the cultural property in the lost cultural heritage site and the video service to augment the pyramid in the school grounds. As described above, the 3D augmented reality photo service of the present invention provides a 3D augmented reality photo service by adding a virtual object to a reality screen image in various places, and is differentiated from the augmented reality service that provides additional 2D information. 3D virtual objects are added and the virtual objects are matched according to a depth map to provide a more natural and realistic photo image service.

In addition, various and interesting photo image services may be provided by adding context information according to a user's preference or selection. In addition, by selectively applying an image processing method through a service server using a wireless network, it is possible to use a client terminal having a low image processing capability, to increase the processing speed, and to provide various services by expanding the service. have.

3 is a view schematically illustrating the flow of a 3D augmented reality photo service method according to the present invention. As shown in FIG. 3, the service method of the present invention separates a scene into layers according to depth, inserts a virtual object image, and applies a sensory effect to the virtual object using context information to generate a result image. Shows.

First, a depth map is generated for a stereoscopic image obtained from a stereoscopic image acquisition apparatus built in a client terminal. If a feature point, such as a QR code or a predetermined marker, exists in the obtained real scene image, the image is separated according to depth into a feature point image layer, a feature point upper image layer, and a feature point lower image layer based on the feature point.

If there are no features in the real scene, it is possible to use a depth map to separate two or more layers. If there is a feature image layer in the separated image, the virtual object image is inserted instead of the feature image. If there is no feature point image layer, the virtual object layer is inserted in the center of the separated image layer.

If necessary, it is possible to correct the position, orientation, and size of the virtual object image. Finally, various effects are applied to the virtual object using location information, time information, or current context information provided from the user. The embodiment shown in FIG. 3 shows that the image is matched by applying the effect of snow accumulation to the virtual object through the context information that snow is accumulated in the real scene.

While the invention has been shown and described with respect to the specific embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined by the appended claims. Anyone with it will know easily.

Claims (11)

In the method using a system consisting of a client, a wireless network and a photo service server,
(a) selecting, by the client, a photo service at a service location through service information received through the service server;
(b) loading the virtual object on the service location image when the virtual object corresponding to the service exists;
(c) generating a depth map of the service location image;
(d) layering the service location image based on the depth map;
(e) inserting the virtual object between the layered images;
(f) registering an image based on context information of the user, and photographing using the matched image; And
3g augmented reality photo service method comprising the step of storing the captured image.
The method of claim 1,
The step (a)
Transmitting, by the client, location information to the photo service server;
Transmitting, by the photo service server, service information registered in the location information to the client;
And selecting a photo service through the service information displayed on the client.
The method of claim 2,
If you do not transmit the location information,
Searching for a feature point in the camera image of the client;
And if the feature point is found, selecting the photo service.
The method of claim 1,
The step (b)
If the virtual object does not exist,
Requesting transmission of a virtual object to the service server;
Searching the database by the service server;
And transmitting, by the service server, the searched virtual object to the client.
The method of claim 3,
The step (d)
If there is the feature point, the image is layered based on the feature point,
3D augmented reality photo service method characterized in that the step of layering the image to at least two layers according to the depth, if there is no feature point.
The method of claim 1,
After the step (e),
And selectively adjusting, by the client, the virtual object.
The method of claim 7, wherein
The fine adjustment is a 3D augmented reality photo service method, characterized in that for adjusting the size, direction and position of the virtual object.
The method of claim 1,
The step (f)
Transmitting the context information to a service server;
And matching the resultant image in which the context information is inserted into the virtual server by the service server.
9. The method of claim 8,
The resulting image,
3D augmented reality photo service method comprising at least one of current location information, time information, image information and sensor information.
A 3D augmented reality photo service system used in the method of any one of claims 1 to 9.
The method of claim 10,
The service system,
A service server providing service information, virtual objects, and image processing and image storage functions; And
A communication unit wirelessly communicating with the service server, a stereoscopic image acquisition device for acquiring actual stereoscopic image information, and extracting feature points from the acquired image information, generating a depth map, and generating the depth map in two or more levels according to depths. An image preprocessor that separates the virtual object between the separated image layers, and corrects at least one of a position, a size, and a direction of the inserted virtual object, and uses the context information selected by the user. 3D augmented reality photo service system comprising a client terminal comprising an image post-processing unit to match the image by applying an additional effect to.

Images