WO2012108721A2

WO2012108721A2 - Device and method for providing augmented reality using image information

Info

Publication number: WO2012108721A2
Application number: PCT/KR2012/001006
Authority: WO
Inventors: 고석필
Original assignee: 팅크웨어(주)
Priority date: 2011-02-11
Filing date: 2012-02-10
Publication date: 2012-08-16
Also published as: WO2012108721A3; KR20120092352A; KR101188105B1

Abstract

Disclosed are a method and a device for providing an augmented reality to an image photographed by a camera on the basis of image information. An augmented reality providing device comprises: a 3D space generation unit which generates a virtual 3D space on the basis of an image photographed by a camera; a plane determination unit which determines a 2D plane on the basis of traffic lane information contained in said image; a mapping unit which maps said 2D plane in said virtual 3D space; and an augmented reality providing unit which provides an augmented reality by synthesizing an information object with said image on the basis of the mapped 2D plane.

Description

Apparatus and method for providing augmented reality using image information

The present invention relates to an apparatus and method for providing augmented reality to an image based on image information, and more particularly, to an augmented reality providing apparatus for synthesizing and providing a virtual object to an image captured by a camera based on the image information. It relates to a method for providing augmented reality.

Navigation route and road information providing method has been developed from the 2D map that provides only the shape of the road to the shape of the building around the road to the user to easily find the destination 3D map.

However, even in the case of a 3D map, the building cannot be displayed in the same shape as the reality due to the limitation of data capacity or modeling time. In particular, when the exterior of a building is changed only for a short period of time due to events or other conditions, it is difficult to reflect the changed information in the 3D map, and thus the state of the building cannot be displayed in real time. In some cases, it may be difficult for a user to match a road displayed on a 2D or 3D map with a road that the user views while driving.

Accordingly, there is a need for a method for allowing a user to easily recognize a real road or building corresponding to the route and road information.

The present invention provides an apparatus and method for providing augmented reality that enhances a user's perception of a corresponding place by displaying information for guiding a route or information related to a road on an image of a real space photographed by a camera.

According to an embodiment of the present invention, an apparatus and method for providing augmented reality for improving mapping accuracy between a 2D plane and a 3D space by mapping a 2D plane to a 3D space based on an image photographed by a camera are provided.

An apparatus for providing augmented reality according to an embodiment of the present invention may include: a 3D space generator configured to generate a virtual 3D space based on an image captured by a camera; A plane determination unit to determine a 2D plane based on the lane information included in the image; A mapping unit which maps the 2D plane to the virtual 3D space; And an augmented reality providing unit configured to provide an augmented reality by synthesizing an information object with the image based on the mapped 2D plane.

Augmented reality providing method according to an embodiment of the present invention comprises the steps of generating a virtual 3D space based on the camera parameters; Determining a 2D plane based on lane information of an image captured by the camera; Mapping the 2D plane to the virtual 3D space; And synthesizing an information object with the image based on the mapped 2D plane to provide augmented reality.

According to an embodiment of the present invention, the user's recognition of the place can be improved by displaying information for guiding a route or information related to a road in an image of the actual space photographed by the camera.

In addition, according to an embodiment of the present invention, the mapping accuracy between the 2D plane and the 3D space can be improved by mapping the 2D plane to the 3D space based on the image photographed by the camera.

1 is a block diagram illustrating an apparatus for providing augmented reality according to an embodiment of the present invention.

2 is an example of a process of determining a 2D plane in an embodiment of the present invention.

3 is an example of an augmented reality image provided in accordance with one embodiment of the present invention.

4 is a flowchart illustrating a method of providing augmented reality according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The augmented reality providing method according to an embodiment of the present invention may be performed by an augmented reality providing apparatus. In this case, the apparatus for providing augmented reality according to an embodiment of the present invention may be one of a mobile terminal, a navigation system, or a black box, and receives an image from a mobile terminal, a navigation system, a black box, or a separate camera to provide augmented reality. It may be a terminal.

Referring to FIG. 1, the apparatus 100 for augmented reality according to an exemplary embodiment of the present invention may include a camera 110, a calibrator 120, a 3D space generator 130, a lane information extractor 140, and a plane. The determination unit 150, the mapping unit 160, and the augmented reality providing unit 170 may be included.

The calibration unit 120 may estimate a camera parameter of the camera from an image photographed by the camera using calibration. In this case, the camera may be a camera 110 embedded in the augmented reality providing apparatus 100 or may be a camera installed separately from the augmented reality providing apparatus 100 to photograph the front of the vehicle. For example, it may be a camera embedded in a mobile terminal, a navigation device, or a black box, or a separate camera installed toward the front of the vehicle. In this case, the camera parameter may be a parameter constituting a camera matrix which is information indicating a relationship between the actual space and the picture.

The 3D space generator 130 may generate a virtual 3D space based on the image photographed by the camera. In detail, the 3D space generator 130 obtains depth information from an image captured by the camera based on the camera parameter estimated by the calibrator 120, and virtually based on the acquired depth information and the image. You can create 3D space.

The lane information extractor 140 may acquire a still image by capturing an image captured by the camera, and extract a plurality of lane information from the still image. Since the video captured by the camera is a video photographing the front of the vehicle in real time, it is difficult to extract lane information because the lane continues to move. Accordingly, the lane information extractor 140 may capture the image to obtain a still image, and extract lane information from a still image having no lane change.

However, when the lane information extractor 140 obtains a still image at a time when the vehicle changes lanes, the lane information extractor 140 may extract only one lane information. Accordingly, when the lane information extractor 140 fails to extract the plurality of lane information from the still image, the lane information extractor 140 may determine that the acquired still image is inappropriate and recapture the image. In this case, the lane information extractor 140 may extract a plurality of lane information from another still image obtained by recapturing.

The plane determiner 150 may determine the 2D plane based on the lane information included in the image photographed by the camera. In this case, the plane determiner 150 may obtain a plurality of reference points based on the lane information extracted by the lane information extractor 140, and determine the 2D plane by using a linear equation connecting the reference points. In this case, the reference point may include a bottom point, which is a point where each lane information and the bottom of the image meet, and a vanishing point.

The process of determining the 2D plane by the plane determination unit 150 will be described in detail with reference to FIG. 2.

The mapping unit 160 may map the 2D plane determined by the plane determination unit 150 to the virtual 3D space generated by the 3D space generation unit 130. In detail, the mapping unit 160 may project the 2D plane determined by the plane determination unit 150 to the virtual 3D space generated by the 3D space generation unit 130 to form a bottom surface of the 3D space.

The augmented reality provider 170 may provide an augmented reality by synthesizing an information object with the image based on the 2D plane mapped by the mapping unit 160. In this case, the information object may be an object that displays information related to a path on which the vehicle is traveling or a road on which the vehicle is driving. For example, the information object may include at least one of information indicating that 000M remains to a specific place, information indicating a place of attention such as a crosswalk, speeding camera position information, guide symbol, and direction information of a route.

The plane determiner 150 according to an exemplary embodiment may determine the 2D plane based on the

lane information

210 and 220 extracted by the lane information extractor 140.

First, the plane determination unit 150 may obtain a vanishing point 230 that is a point at which the lane information 210 and the lane information 220 cross each other. In addition, the plane determination unit 150 may include a bottom point 250, which is a point where the lane information 210 meets the bottom of the image 200, and a bottom point that is a point where the lane information 220 meets the bottom of the image. 240 can be obtained.

Next, the plane determination unit 150 may calculate the equation of the straight line between the vanishing point 230 and the bottom point 250 and the equation of the straight line between the vanishing point 230 and the bottom point 240.

Finally, the plane determination unit 150 is a triangle formed by the equation of the straight line between the vanishing point 230 and the bottom point 250, the equation of the straight line between the vanishing point 230 and the bottom point 240, and the bottom of the image 200. Can be determined as the 2D plane.

The apparatus 100 for augmented reality according to the present invention extracts lane information based on an image photographed by a camera installed in a vehicle, and generates a 2D plane based on the extracted lane information. May not be affected.

The augmented reality providing apparatus 100 according to an embodiment of the present invention may provide an augmented reality by synthesizing an information object with an image photographed by a camera. For example, as shown in FIG. 3, the direction information 310 of the route and the information 320 indicating 500M remaining to a specific place may be displayed on the image 300 captured by the camera.

As shown in FIG. 3, the apparatus 100 for providing augmented reality according to an embodiment of the present invention synthesizes and displays information related to a route guide or a road guide on a real-time image photographed by a camera to display a user's information on a corresponding place. It can increase the recognition.

In operation S410, the calibrator 120 may estimate camera parameters of the camera from an image photographed by the camera using calibration.

In operation S420, the 3D space generator 130 may generate a virtual 3D space based on the camera parameter estimated in operation S410. In detail, the 3D space generator 130 may obtain depth information from an image captured by the camera based on a camera parameter, and generate a virtual 3D space based on the acquired depth information and the image.

In operation S430, the lane information extractor 140 may acquire a still image by capturing an image captured by the camera.

In operation S440, the lane information extractor 140 may determine whether the still image acquired in operation S430 is a still image capable of extracting lane information. In more detail, the lane information extractor 140 may attempt to extract lane information from the still image acquired in operation S430. If the plurality of lane information is not extracted, the lane information extractor 140 may determine that the still image acquired in operation S430 is not a still image from which the lane information may be extracted. In this case, the lane information extractor 140 may return to step S430 to capture another still image from among images captured by the camera to obtain another still image.

In operation S450, the plane determination unit 150 may obtain a plurality of reference points based on the lane information extracted by the lane information extraction unit 140 in operation S440. In this case, the reference point acquired by the plane determination unit 150 may include a bottom point, which is a point where each lane information and the bottom of the image meet, and a vanishing point.

In operation S460, the plane determination unit 150 may determine the 2D plane using the equations of the reference points obtained in operation S440 and a straight line connecting the reference points.

In operation S470, the mapping unit 160 may map the 2D plane determined in operation S460 to the virtual 3D space generated in operation S420. In detail, the mapping unit 160 may project the 2D plane determined in operation S460 to the virtual 3D space generated in operation S460 to configure a bottom surface of the 3D space.

In operation S480, the augmented reality provider 170 may provide an augmented reality by synthesizing an information object with the image based on the 2D plane mapped in operation S470. For example, the augmented reality provider 170 may set the type and position of the information object to be displayed on the 2D plane based on the map data, and map the position of the information object set on the 2D plane to the bottom surface of the 3D space. Can be. Next, the augmented reality providing unit 170 may select a position to display the information object in the image taken by the camera according to the position mapped to the floor surface. Finally, the augmented reality provider 170 may synthesize and display the information object at a selected position in the image photographed by the camera. In this case, the information object may be an object that displays information related to a path on which the vehicle is traveling or a road on which the vehicle is driving.

The present invention can improve the user's perception of a corresponding place by displaying information for guiding a route or information related to a road in an image of a real space photographed by a camera.

In addition, since the 2D plane is mapped to the 3D space based on the image photographed by the camera, mapping accuracy may be improved as compared with the conventional method of mapping the 2D plane to the 3D space based on the location where the sensor is installed.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains various modifications and variations from such descriptions. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined not only by the claims below but also by the equivalents of the claims.

Claims

3D space generating unit for generating a virtual 3D space based on the image taken by the camera;

A plane determination unit to determine a 2D plane based on the lane information included in the image;

A mapping unit which maps the 2D plane to the virtual 3D space; And

Augmented reality providing unit for providing augmented reality by synthesizing the information object to the image based on the mapped 2D plane

Augmented reality providing device comprising a.
The method of claim 1,

The plane determination unit,

And obtaining a plurality of reference points based on the lane information, and determining the 2D plane by connecting the reference points.
The method of claim 2,

A lane information extractor configured to capture the image to obtain a still image and extract a plurality of lane information from the still image

Augmented reality providing device further comprising.
The method of claim 3,

The lane information extraction unit,

And when the plurality of lane information is not extracted from the still image, recapture the image to obtain another still image, and extract the plurality of lane information from the other still image.
The method of claim 2,

The reference point,

And a vanishing point, which is a point where each lane information and the bottom of the image meet, and a vanishing point, where each lane information meets.
The method of claim 1,

The information object,

An apparatus for providing augmented reality, characterized in that the object for displaying information relating to the path to the vehicle or the road on which the vehicle is driving.
The method of claim 1,

Calibration unit for estimating the camera parameters of the camera from the image using the calibration (Calibration)

Augmented reality providing device further comprising.
The method of claim 7, wherein

The 3D space generator,

And obtaining depth information from the image based on the camera parameter, and generating a virtual 3D space based on the acquired depth information and the image.
Generating a virtual 3D space based on an image captured by the camera;

Determining a 2D plane based on lane information included in the image;

Mapping the 2D plane to the virtual 3D space; And

Providing augmented reality by synthesizing an information object on the image based on the mapped 2D plane

Augmented reality providing method comprising a.
The method of claim 9,

Determining the 2D plane,

Obtaining a plurality of reference points based on the lane information; And

Determining the 2D plane by connecting the reference points

Augmented reality providing method comprising a.
The method of claim 10,

Capturing the image to obtain a still image; And

Extracting a plurality of lane information from the still image

Augmented reality providing method further comprising.
The method of claim 11,

Recapturing the image to obtain another still image when the plurality of lane information is not extracted from the still image

More,

Extracting the lane information,

Augmented reality providing method for extracting a plurality of lane information from another still image.
The method of claim 10,

The reference point,

And a vanishing point, which is a point where each lane information and the bottom of the image meet, and a vanishing point, which is a point where each lane information meets.
The method of claim 9,

The information object,

An augmented reality providing method characterized in that the object for displaying information relating to the path to the vehicle or the road on which the vehicle is driving.
The method of claim 9,

Estimating camera parameters of the camera from the image using calibration

Augmented reality providing method further comprising.
The method of claim 15,

Generating the 3D space,

Obtaining depth information from the image based on the camera parameter; And

Generating a virtual 3D space based on the acquired depth information and the image;

Augmented reality providing method comprising a.