KR20110082873A - Image processing apparatus providing distacnce information in a composite image obtained from a plurality of image and method using the same - Google Patents

Abstract

PURPOSE: An image processing apparatus and method thereof are provided to display distance information of a synthesizing image. CONSTITUTION: A plurality of optical cameras(11) is arranged so that an image area is interposed between adjacent optical cameras. An image signal receiving unit(12) receives an input image signal from a plurality of wide angle cameras. An image synthesizing unit(13) receives an input image signal from the image signal receiving unit. An image signal output unit(14) outputs distance information of a feature point.

Description

IMAGE PROCESSING APPARATUS PROVIDING DISTACNCE INFORMATION IN A COMPOSITE IMAGE OBTAINED FROM A PLURALITY OF IMAGE AND METHOD USING THE SAME}

The present invention relates to an image processing apparatus and method, and more particularly, to an image processing apparatus and method having a function of providing distance information in a synthesized image obtained by synthesizing a plurality of images.

Recently, with the development of IT technology, attempts to apply it to vehicles are increasing. For example, a black box device that mounts a camera in a vehicle to record driving conditions or surrounding conditions, or a parking assistance system that installs a camera in the rear of the vehicle to take a rearward image and output it to a display device inside the vehicle when the vehicle is reversed. This trend is reported to be increasing. Meanwhile, among these technologies, wide-angle cameras are installed in front, rear, left, and right sides of the vehicle, and the images captured from these cameras are reconstructed into images of the form directly above the vehicle, that is, viewed from above, and output to the display device of the vehicle. There has also been proposed a system for the driver's convenience. Such a system is called a bird eye vie system or an AVM (around view monitoring) system in that it provides an image as if the bird is looking down from the sky. This technique uses a wide-angle camera with a fish eye lens to secure a wider viewing angle, which includes 1) distortion correction and 2) planarization for a plurality of image signals obtained using such a wide-angle camera. , 3) rearrangement step, and 4) single imaging step to construct and provide a composite image viewed from above the vehicle.

However, although the conventional composite image providing system performs a complex calculation process through several steps, the final composite image provided merely displays images of the vehicle and its surroundings, and thus, various structures appearing in the image around the vehicle It does not provide distance information about people. Accordingly, there is a demand for a method of providing distance information together with surrounding images instead of simply outputting the surrounding image in a conventional composite image providing system.

The present invention has been made to solve the above-mentioned limitations, and when combining and providing a single planar composite image by using a plurality of input image signals obtained from a plurality of wide-angle cameras, the distance information may be displayed together in the composite image. It is an object of the present invention to provide an apparatus and method for enabling the same.

In addition, the present invention is to display the distance information in the composite image to accurately determine the positional relationship of the surrounding structures or people appearing in the composite image to prevent safety accidents and facilitate the operation of the vehicle, especially when applied to the vehicle It is another object of the present invention to provide an apparatus and method.

SUMMARY OF THE INVENTION In order to solve the above problems, the present invention provides an image processing apparatus having a function of providing distance information from a synthesized image obtained by synthesizing a plurality of images, wherein the photographing regions are disposed so as to partially overlap the adjacent wide-angle cameras. At least two or more wide angle cameras; An image signal receiver configured to receive at least two or more input image signals obtained from the plurality of wide-angle cameras; An image synthesizer configured to receive a plurality of input image signals from the image signal receiver and generate a composite image signal; And an image signal output unit configured to generate an output image based on the synthesized image signal generated by the image synthesizer, and output the output image, wherein the image synthesizer includes two input image signals having overlapping regions of the plurality of input image signals. For each, obtaining feature points of overlapping photographing regions, calculating distance information on the feature points, and outputting the distance information display signal based on the calculated distance information together with the output image by the image signal output unit. An image processing apparatus is provided.

The image synthesizing unit may include a distortion correction step of each of the plurality of input image signals obtained from a plurality of wide-angle cameras, a planarization step of each of the input image signals that are distortion-corrected by the distortion correction step, and the planarization step. After each of the rearrangement steps of the planarized input image signals, each of the two rearranged images having overlapping photographing regions among the plurality of rearranged images is obtained, and the feature points of the overlapping photographing regions are respectively acquired. The distance information display signal based on the calculated distance information may be output by the image signal output unit together with the output image.

In addition, the distance information about the feature point,

(Where a is the x-axis coordinate of the first camera which is one of the cameras, b is the y-axis coordinate of the second camera which is the other one of the cameras, α is the angle between the x-axis of the first camera and the feature point, β Is the angle between the y axis of the second camera and the feature point)

It can calculate by the (x, y) coordinate value obtained by the formula of.

The distance information display signal may output a distance value while the distance information calculated for the feature point is displayed by a line connecting the same points to each other.

According to another aspect of the invention, the image processing method by the apparatus as described above, comprising the steps of: receiving a plurality of input image signals; While generating a composite image based on the received input image signal, for each of the two input image signals having overlapping capture regions among the plurality of input image signals, feature points of overlapping capture regions are obtained, respectively, Calculating distance information on the at least one distance information and generating a distance information display signal based on the calculated distance information; An image processing method may include outputting the composite image and a distance information display signal.

According to the present invention, when synthesizing and providing a single planar composite image by using a plurality of input image signals obtained from a plurality of wide-angle cameras, an apparatus and a method for displaying distance information on a composite image can be provided. .

In addition, according to the present invention, by displaying the distance information in the composite image to accurately determine the positional relationship of the surrounding structures, people, etc. appearing in the composite image to prevent safety accidents and facilitate the operation of the vehicle, especially when applied to the vehicle It can work.

1 is a view showing the basic process of the conventional image synthesizing system and the image of each process.
2 is a block diagram illustrating a configuration of an embodiment of an image processing apparatus having a function of providing distance information in a synthesized image obtained by synthesizing a plurality of images according to the present invention.
3 and 4 are diagrams showing images acquired by a camera mounted on a vehicle and their common feature points.
FIG. 5 is a diagram for describing common areas generated when cameras are mounted on the front / rear and left / right sides of a vehicle, respectively.
6 is a diagram for explaining a process of obtaining distance information for one of the feature points in the common area II.
7 is a diagram illustrating an example of a method of displaying a distance information display signal together with a composite image output through an image signal output unit.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

First, prior to explaining the embodiment according to the present invention, the principle of the prior art related to the present invention will be described schematically.

As described above, a wide-angle camera having a plurality of, for example, four fisheye lenses is installed on the front, rear, left, and right sides of the vehicle to take an image in the horizontal direction on the ground, and reconstruct the image into a form looking down from the top of the vehicle (hereinafter, For convenience, such a conventional technique is referred to simply as an "image synthesis system." The basic process of the conventional image synthesizing system and the image for each process are illustrated in FIG. 1.

Referring to FIG. 1, a conventional image synthesizing system is composed of six steps, an image input step S100, a distortion correction step S110, a planarization step S120, a rearrangement step S130, and a single imaging step S140. ) And a single image output step (S150).

The image input step S100 is a process of inputting a plurality of image signals obtained from a plurality of wide-angle cameras. For example, when a plurality of cameras are mounted on an object that is a vehicle, four wide-angle cameras may be mounted on the front / rear / left side / right side, and the captured image is displayed as shown in FIG. 1. As described above, since the wide-angle camera has a fisheye lens, a wide field of view can be secured. In order to construct a single planarized image as described later by an image signal around a target object, a predetermined area is formed between each camera. This is because there must be overlapping parts and each camera must have a wide viewing angle to reconstruct the image with fewer cameras despite the overlapping areas. In the present invention, the camera is used as a concept including not only a fisheye lens but also other electrical devices such as an image sensor. That is, it does not mean simply a mechanism for optically acquiring an image, but an apparatus for converting an optical signal into an electrical signal, for example, as shown in FIG. It is used as a concept that means a means to do.

On the other hand, when using a wide-angle camera equipped with a fisheye lens to use as few cameras as possible as described above, a wide viewing angle can be secured, but the image is distorted radially toward the edge of the acquired image. The distortion correction step S110 is a process for correcting such a distorted image. Fisheye lens distortion correction can be largely divided into two methods: "Equi-solid Angle Projection" and "Orthographic Projection", which define how fisheye lens rearranges the light entering the fisheye lens. Lens manufacturers choose one of two methods to manufacture fisheye lenses. Therefore, if the inverse operation of the distortion operation is obtained according to the distortion method applied to the fisheye lens, and the image converted by the fisheye lens is inversely transformed, a "distortion corrected" image can be obtained, and the converted image is called a "distortion correction image". . The distortion correction image may be displayed as shown in FIG. 1, and an equation for performing distortion correction may use, for example, the following equation.

[Equation 1]

는 카메라의 촛점 거리,

는 카메라의 광중심에서 입력 영상의 (x,y) 좌표 즉,

까지의 거리,

및

은 입력 영상의 (x,y) 좌표값이고,

및

은 왜곡 보정 영상의 (x,y) 좌표값이다.
From here,

The focal length of the camera,

Is the (x, y) coordinate of the input image at the camera's optical center,

Distance to,

And

Is the (x, y) coordinate value of the input image,

And

Is the (x, y) coordinate value of the distortion corrected image.

Next, when the distortion correction image is obtained, a plane (homography) process is performed (S120). The planarization process (S120) is a process of converting an object onto an object, that is, an object on which a camera is mounted, into an image looking downward in a direction toward the ground, that is, vertically. As described above, since the distortion correction image obtained in the distortion correction step S110 corresponds to an image photographed at different viewpoints for each camera, these images are converted into an image of one viewpoint, that is, a vertically viewed viewpoint. The process is a flattening step (S120). An image generated by performing the flattening step S120 is called a flattened image, and an image after performing the flattening step S120 may be displayed as shown in FIG. 1, and the following equation may be used in this flattening step, for example. have.

[Equation 2]

은 상기 수식 1에 의해 획득된 왜곡 보정 영상의 (x,y) 좌표값이고,

은 상기 수식 2에 의해 변환된 평면화영상의 (x,y) 좌표값이며,

은 평면화 변환계수(이를 Perspective Transform이라고 한다)를 의미한다.
here,

Is a (x, y) coordinate value of the distortion correction image obtained by Equation 1,

Is the (x, y) coordinate value of the flattened image converted by Equation 2,

Is the flattening transform coefficient (called Perspective Transform).

Next, when a planarized image is obtained, an rearrangement step (Affine transform) is performed (S130). The rearrangement step (S130) is a step of rearranging the planarized images generated in the planarization step by applying only displacement movement and rotation, wherein the images photographed to surround the target object are reconstructed into surrounding images except for the target object. The rearrangement step S130 may be performed by only moving and rotating the pixel. For this, a method such as an affine transform may be used. An image generated through the rearrangement is called a rearrangement image, and the rearrangement image may be displayed as shown in FIG. 1.

In the rearrangement phase, you can use the following formula:

[Equation 3]

은 수식 2에 의해 변환된 평면화 영상의 (x,y) 좌표값이고,

은 수식 3에 의해 변환될 재배열 영상의 (x,y) 좌표값이고,

는 회전 변환 계수이고,

는 변위 이동 계수(

과

를 합쳐서 affine transform으로 정의한다)를 의미한다.
From here,

Is the (x, y) coordinate value of the flattened image converted by Equation 2,

Is the (x, y) coordinate value of the rearranged image to be converted by Equation 3,

Is the rotation conversion factor,

Is the displacement displacement coefficient (

and

To be defined as an affine transform).

Next, a single imaging step S140 is performed. Since the rearranged images are merely rearranged images, the images photographed around the object have a common area and the common areas are arranged to overlap each other. Therefore, a single imaging step is needed to process overlapping portions of the rearranged image having the common area to obtain one representative image for the common area. Since a single imaging step can use several implementations and can vary depending on the implementation, only the implementation principle is briefly described. When a common area occurs, the single imaging step divides the common area into pixels and analyzes the pixel to form a single image area using only pixels arranged at a more accurate position. The reference for the pixels arranged at the correct position may be various. The simplest reference may be, for example, the distance between the pixel and the optical center of the image to which the pixel belongs. By using this criterion, if the common region is reconstructed using only pixels closer to the optical center among the overlapping pixels of the common region, the rearranged image may be configured as a single image without overlapping regions. An image generated through a single imaging step is called a single image and may be displayed as shown in FIG. 1.

When the single imaging step (S140) is performed as described above, a flattened single image is obtained, and when the output is made (S150), an image of a form in which the object object is looked down from the vertical upper direction as shown in FIG. You will be able to see

On the other hand, the above-described steps and formulas 1, 2, and 3 used in each step are used by the prior art, and are not directly related to the present invention, and thus detailed descriptions thereof are omitted. In particular, each of the coefficients in Equations 2 and 3 may be determined differently according to the method or algorithm used, and the present invention is not related to the method of calculating such coefficients, and thus, detailed description thereof is also omitted.

2 is a block diagram illustrating a configuration of an embodiment of an image processing apparatus having a function of providing distance information in a synthesized image obtained by synthesizing a plurality of images according to the present invention.

Referring to FIG. 2, an image processing apparatus (hereinafter, simply referred to as an image processing apparatus 10) having a function of providing distance information in a synthesized image obtained by synthesizing a plurality of images of the present embodiment may include a plurality of wide-angle cameras 11. And an image signal receiving unit 12, an image synthesizing unit 13, and an image signal output unit 14.

The plurality of wide-angle cameras 11 are disposed such that the photographing areas partially overlap between adjacent wide-angle cameras, and are composed of at least two or more pieces, each of which photographs an image, converts the image into an electrical signal, and transmits the image to the image signal receiving unit 12. As described above, it should be noted that each camera 11 is a concept including not only a simple optical apparatus but also an electrical apparatus such as an image sensor for converting an optical signal into an electrical signal. For example, when the target object is a vehicle, the wide-angle camera 11 may be disposed at the front / rear / left side / right side of the vehicle, and each of these cameras 11 may be located between the cameras 11 whose photographing areas are adjacent to each other. At least a portion is arranged to overlap.

The image signal receiver 12 is a means for receiving at least two or more input image signals obtained from the plurality of wide-angle cameras 11, respectively, and transmits the received plurality of input image signals to the image synthesizer 13. If necessary, the image signal receiver 12 may perform an image preprocessing process using a filter or the like.

The image synthesizing unit 13 receives a plurality of input image signals from the image signal receiving unit 12 and generates a composite image signal, respectively, to each of two input image signals having overlapping photographing regions of the plurality of input image signals. Acquiring feature points of overlapping photographing regions, calculating distance information on the feature points, and outputting the distance information display signal based on the calculated distance information together with the output image by the image signal output unit. do.

Generation of the synthesized image signal by the image synthesizer 13 may be performed by the steps described with reference to FIG. 1. That is, as described with reference to FIG. 1, a composite image signal is generated while sequentially performing a distortion correction step, a planarization step, a rearrangement step, and a single imaging step on a plurality of input image signals obtained from a plurality of cameras. do. During this process, the image synthesizing unit 13 performs two or more rearranged images having overlapping photographing regions among the plurality of rearranged images after the rearrangement step and before the single imaging step. Each of the feature points may be acquired, the distance information of the feature points may be calculated, and the distance information display signal based on the calculated distance information may be output by the image signal output unit together with the output image.

Acquiring a feature point, calculating distance information on the feature point, and outputting a distance information display signal based on the distance information may be performed as follows.

3 and 4 are views showing images acquired by a camera mounted on a vehicle and common feature points thereof, FIG. 3 is an image obtained by a camera mounted on the front of the vehicle, and FIG. 4 is a right side of the vehicle. The image obtained by the mounted camera is shown. 3 and 4 are images of each camera after the rearrangement step and before the single imaging step, as described above.

Referring to FIGS. 3 and 4, since the two images are images obtained by the wide-angle camera as described above, the two images have areas in common with each other, and the feature points can be found as shown in the drawings. The feature point generally refers to a pixel that can represent a feature of an image. The method of finding a pixel that can represent a feature of the image is, for example, a corner detection method such as a Harris corner detector. Conventionally known methods such as detection, scale invariant feature transform (SIFT), speed-up robust feature (SURF) and the like can be used. 3 and 4 apply feature points detection techniques known in the art to obtain feature points common to common areas of both images. 3 and 4, if common feature points are obtained, distance information of the feature points may be obtained by applying the following method to these feature points.

FIG. 5 is a diagram for describing common areas generated when cameras are mounted on the front / rear and left / right sides of a vehicle, respectively. Referring to FIG. 5, four common areas are generated, common area I by front and left cameras, II by front and right cameras, III by rear and right cameras, and IV by rear and left cameras. Each area is generated by the camera.

For each of these common areas, feature points that are common to each other can be obtained for each common area based on the principles described with reference to FIGS. 3 and 4. You can get information.

6 is a diagram for explaining a process of obtaining distance information for one of the feature points in the common area II.

In FIG. 6, P (x, y) is a feature point and the coordinates of this feature point are determined by x, y values.

In FIG. 6, A (a, 0) shows the coordinates where the right side camera is located and B (0, b) shows the coordinates where the front camera is located. At this time, the origin O (0,0) is where the line extending the camera in a straight line intersects. For this purpose, the front and rear cameras, the left and right cameras are preferably placed on the same axis. In addition, α is an angle between the feature point P and the coordinate value A of the right-side camera with respect to the x axis, and β is an angle between the feature point P and the coordinate value B of the front camera with respect to the y axis.

When the coordinate system is constituted as described above, each coordinate of the feature point P (x, y) has the following relationship.

Here, since a, b, α, and β are known values, l, m can be obtained by the following formula using the above equation.

By using this result, the coordinate value of the feature point P (x, y) can be obtained, so that the distance information from the origin can be obtained. If the same process is repeated for all the feature points included in the common area, all the feature points are obtained. Since the coordinate value can be obtained for, similarly, the distance information from the origin can be obtained. This result is performed in the image synthesizing unit 13 as described above. Each feature point of the common area is made before a single imaging step, and distance information on the feature point is made for a single image selected in the single imaging step. .

By repeating this process for all common areas I to IV, coordinate values and distance information for all feature points of each common area can be obtained.

The image synthesizing unit 13 generates the distance information display signal based on the distance information of all the feature points of all the common areas and outputs the distance information display signal together with the output image by the image signal output unit.

A method of displaying the distance information display signal together with the composite image output through the image signal output unit 14 may be, for example, as shown in FIG. 7.

FIG. 7 shows distance information on all of the feature points by the above-described process, grouping feature points having the same distance information from the origin based on them, and connecting them to each other in a line. As shown in FIG. 7, for example, a method of finding feature points located at a distance of 0,5 m from an origin and connecting them to each other, finding feature points located at a distance of 1 m, connecting them to each other, and finding a feature located at a distance of 1.5 m are connected to each other by a line. As a result, the distance information display signal as shown in FIG. 7 can be output. In the case of the display as shown in FIG. 7, the feature points in the common area are areas having the same distance information but are not in the common area, for example, an area between the common areas I and II in FIG. 5 (here is an area photographed only by the front camera). Since the distance information display signal cannot be represented, the same distances in the common areas I and II can be expressed in a straight line. Since areas that are not common in most cases are images of parts with less distortion, even if the same distances are connected in a common area, no big error occurs.

Referring to FIG. 2 again, the image signal output unit 15 generates an output image based on the synthesized image signal configured in the image synthesizer 13 and outputs the output image to a display device external to the device such as an LCD monitor. As described above, the distance information display signal generated by the image synthesizing unit 13 is displayed together with the output image based on the synthesized image signal and displayed on an external display device as shown in FIG. It performs the function.

In the above, the present invention has been described with reference to preferred embodiments, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains may make various modifications and variations from such descriptions. will be. Therefore, the present invention should be construed with reference to the overall description of the appended claims and drawings, and all equivalent or equivalent modifications thereof will belong to the scope of the present invention.

11 camera, 12 video signal receiver,
13 video composite section, 14 video signal output section,
10 ... image processing unit.

Claims (5)

An image processing apparatus having a function of providing distance information in a synthesized image obtained by synthesizing a plurality of images,
A plurality of at least two wide-angle cameras arranged to partially overlap the photographing areas between adjacent wide-angle cameras;
An image signal receiver configured to receive at least two or more input image signals obtained from the plurality of wide-angle cameras;
An image synthesizer configured to receive a plurality of input image signals from the image signal receiver and generate a composite image signal; And
An image signal output unit configured to generate and output an output image based on the synthesized image signal generated by the image synthesizer.
Including,
The image synthesizing unit obtains feature points of overlapping capture regions for each of two input image signals having overlapping capture regions among the plurality of input image signals, calculates distance information on the feature points, and calculates the calculated points. And a distance information display signal based on the distance information is output by the image signal output unit together with the output image.
The method of claim 1,
The image synthesizing unit may include a distortion correction step of each of the plurality of input image signals obtained from a plurality of wide-angle cameras, a planarization step of each of the input image signals distortion-corrected by the distortion correction step, and a planarization step by the planarization step. For each of the two rearranged images having the overlapping photographing regions among the plurality of rearranged images after performing the rearrangement step of the input image signals, feature points of the overlapping photographing regions are respectively obtained, and distance information on the feature points is obtained. And outputting the distance information display signal based on the calculated distance information together with the output image by the image signal output unit.
The method of claim 1,
Distance information about the feature point,

(Where a is the x-axis coordinate of the first camera which is one of the cameras, b is the y-axis coordinate of the second camera which is the other one of the cameras, α is the angle between the x-axis of the first camera and the feature point, β Is the angle between the y axis of the second camera and the feature point)
And an (x, y) coordinate value obtained by the equation of.
The method of claim 1,
And the distance information display signal outputs a distance value while the distance information calculated for the feature point is displayed by lines connected to the same points.
An image processing method by the apparatus of at least one of claims 1 to 4,
Receiving a plurality of input video signals;
While generating a composite image based on the received input image signal, for each of the two input image signals having overlapping capture regions among the plurality of input image signals, feature points of overlapping capture regions are obtained, respectively, Calculating distance information on the at least one distance information and generating a distance information display signal based on the calculated distance information; And
Outputting the composite image and the distance information display signal
Image processing method comprising a.

Images