KR101823657B1 - Stereo image rectification method for mono cameras - Google Patents

Abstract

According to the present invention, a stereo image rectifying method for heterogeneous cameras comprises the following steps: an image processing apparatus receives first, second, and third images which are acquired by three heterogeneous cameras; the image processing apparatus corrects a focal distance between the first and second images and primarily rectifies the first and second images in which the focal distance is corrected; the image processing apparatus corrects a focal distance between the second and third images, and secondarily rectifies the second and third images in which the focal distance is corrected; and the image processing apparatus estimates a relation between the primarily rectified second image and the secondarily rectified second image.

Description

{STEREO IMAGE RECIFICATION METHOD FOR MONO CAMERAS}

The technique described below relates to a stereo image correction technique for an image acquired with a different mono camera.

Stereo cameras are image acquisition devices that are used in various fields of 3D image processing technology. Stereo image rectification is to match the epipolar lines in parallel with each other in the stereo matching process.

Korean Patent No. 10-1457385

Stereo image calibration is performed on an image acquired by a camera including two image sensors in one body or an image acquired by two separated cameras. Conventional stereo image correction assumes the same type of camera.

The technique described below is intended to provide a stereo image correction technique for images acquired by different kinds of cameras.

A stereo image calibrating method for a heterogeneous camera includes the steps of: receiving a first image, a second image, and a third image obtained by the three different cameras of the image processing apparatus; Correcting a focal distance between images and performing a first correction on the first image and the second image with the focal distance corrected, the method comprising the steps of: determining a focal distance between the second image and the third image And performing a second calibration on the second image and the third image on which the focal length is corrected, and correcting the focal distance based on the first corrected second image and the second corrected image Lt; RTI ID = 0.0 > 2 < / RTI > images.

The technique described below can generate a constant stereo image using a plurality of mono cameras installed in a vehicle, for example.

1 is an example of a vehicle using a heterogeneous camera.
Figure 2 is an example of a flowchart for a method of calibrating a stereo image using two heterogeneous cameras.
3 is another example of a flowchart for a method of calibrating a stereo image using two different cameras.
Figure 4 is an example of a flowchart for a method of calibrating a stereo image using three different cameras.
5 is an example of a process of correcting an image acquired by three different cameras.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include "should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms" comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner.

Also, in performing a method or an operation method, each of the processes constituting the method may take place differently from the stated order unless clearly specified in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

The technique described below relates to a stereo image correction technique for an image obtained by a monochrome camera of a different kind. Different mono cameras may have different resolutions, zoom levels, focal lengths, and angle of view.

1 is an example of a vehicle using a heterogeneous camera. 1 is an example in which a camera disposed in the vehicle 10 acquires an image in which the angle of view is superimposed. Fig. 1 is an example in which a plurality of cameras are arranged in the vehicle 10. Fig. A camera 1 is disposed on the front side of the vehicle 10 and acquires an image of the front direction of the vehicle. The camera 2 is disposed on the left side mirror of the vehicle 10 and acquires an image of the rear direction of the vehicle. The camera 3 is disposed on the right side mirror of the vehicle 10 and acquires an image of the rear direction of the vehicle. A camera 4 is disposed on the upper side of the vehicle 10 and acquires an image of the front direction of the vehicle. A camera 5 is disposed on the rear side of the vehicle 10 and acquires an image of the rear direction of the vehicle.

The camera 1 and the camera 4 can acquire an image of the overlapped area A while photographing the front direction of the vehicle 10. [ The camera 2 and the camera 5 can acquire an image of the overlapped area B while photographing the backward direction of the vehicle 10. [ The camera 2 and the camera 3 can acquire an image of the overlapping region C while photographing the backward direction of the vehicle 10. [

The technique described below is for generating a stereo image using an image acquired by a plurality of heterogeneous cameras. Referring to FIG. 1, the positions and heights of the camera 1 and the camera 4 for acquiring images for the region A are different. Further, the hardware specifications and the like of the camera 1 and the camera 4 may be different from each other. Therefore, additional preprocessing or post-processing may be required for processing of conventional stereo cameras.

In the following description, it is assumed that the image processing apparatus corrects the stereo image for convenience of explanation. The image processing apparatus is a device for generating a stereo image using an image acquired by a camera. For example, the image processing apparatus may be a PC, a server, an embedded chip mounted on a vehicle, or the like.

The image processing apparatus generates a stereo image using a plurality of images. For convenience of explanation, it is assumed that the image processing apparatus generates a stereo image using an image acquired by two cameras. It is assumed that the image processing apparatus uses the image A acquired by the first camera and the image B acquired by the second camera. Image A can be the left image, and image B can be the right image.

Figure 2 is an example of a flowchart for a method 100 for calibrating a stereo image using two heterogeneous cameras. FIG. 2 assumes that two cameras are fixed in one frame. That is, two cameras acquire images on the same horizontal line. It is assumed that the focal lengths of the first camera and the second camera are different. The image processing apparatus receives the image A acquired by the first camera 110 and receives the image B acquired by the second camera 120.

The image processing apparatus first compensates the focal length using the image A and the image B (130). The image processing apparatus can compensate the focal length by enlarging or reducing the remaining images based on any one of the images. In some cases, the image processing apparatus may compensate the focal length by enlarging and reducing the two images, respectively. The focal length compensation will first be described.

A. Focal length compensation

The image processing apparatus generates a stereo image using the images A and B having the same focal distance. On the other hand, the heterogeneous cameras may be located at different points and the focal lengths may be different from each other. For example, a plurality of cameras may have different zoom levels. In order to generate a stereo image from an image acquired by a heterogeneous camera, the image processing apparatus needs to constantly compensate the focal length of the image acquired by the heterogeneous camera.

If there is information on the focal length of each camera, the image processing apparatus can compensate the focal length of the image based on the ratio of the focal length. For example, the image A acquired by the first camera having the first focal length and the image B acquired by the second camera having the focal distance longer than the first focal distance will be described as an example. The image processing apparatus can compensate the focal length based on the image B having a relatively long focal length. That is, the image processing apparatus can adjust the focal distance of the image A so that the focal lengths of the image A and the image B can be the same. If the image processing apparatus knows the focal lengths of the first camera and the second camera in advance, the focal length of at least one of the image A and the image B can be corrected using a previously known focal length.

Furthermore, the image processing apparatus may not know the focal distance of the first camera and the second camera in advance. In this case, the image processing apparatus can perform a process of matching the focal distance through image analysis.

The image processing apparatus can detect a predetermined feature point in the image A and the image B and determine a corresponding point. For example, the image processing apparatus may perform a Scale-Invariant Feature Transform (SIFT) on an image to extract feature points. Of course, the image processing apparatus may extract the feature points of the image using another technique.

라고 하고, 영상 B의 특징점 집합을

라고 한다.

과

은 순서에 따라 서로 대응되는 특징점을 포함한다. The feature point set of the image A extracted by the image processing apparatus

And the set of feature points of image B

.

and

Include minutiae corresponding to each other in order.

이고, 영상 A의 특징점 집합은

이라고 할 수 있다.The image processing apparatus may perform the compensation process by identifying the value of the feature point based on the vertical direction. If the feature points are extracted by dividing them into identifiers according to the vertical direction, the feature point set of the image A

, And the feature point set of image A is

.

The focal length generally refers to the distance from the lens to the image sensor. If the focal length is long, the subject appears larger on the screen. If the focal length is short, the range (angle of view) on the screen is widened. For convenience of explanation, it is assumed that the focal length of the first camera acquiring the image A is longer than that of the second camera acquiring the image B. [ Therefore, the size of the object appearing in the image A is larger than that of the image B. The image processing apparatus can reduce the image size of the image A and correct the focal distance to the image B. Alternatively, the image processing apparatus may increase the size of the image B to correct the focal distance from the image A. Since enlarging an image with a small size may result in blurring of resolution, it is described as an example of reducing an image having a smaller size.

To this end, the image processing apparatus determines distances between corresponding feature points in the image A and the image B, respectively. The distance between specific feature points in the image A is referred to as a first distance, and the distance between specific feature points in the image B is referred to as a second distance. The distance between the first distance and the second distance corresponds to a distance between points (locations of the same object) corresponding to each other in the image. The image processing apparatus can compensate the focal length of the image based on the difference between the first distance and the second distance.

및

각각에 포함된 n개의 요소에 대해

개의 수직 거리 차이를 연산할 수 있다. 아래 수학식 1은

에서 수직 거리를 연산하는 수식이고, 수학식 2는

에서 수직 거리를 연산하는 수식이다.The image processing apparatus

And

For each of the n elements contained in each

The vertical distance difference can be calculated. The following equation (1)

And the equation (2)

Is a formula for calculating the vertical distance.

는

에서 i 번째 요소이고,

은

에서 i 번째 요소이다. 영상 처리 장치는 초점 거리의 차이를 보정하기 위하여 두 개의 영상 중 어느 하나의 영상을 줄이기 위한 지수 t를 연산할 수 있다. 아래 수학식 3 내지 수학식 7을 통해 지수 t를 연산할 수 있다.

The

I < / RTI >

silver

Is the i-th element. The image processing apparatus can calculate an index t for reducing any one of the two images in order to correct the difference in focal length. The exponent t can be calculated through the following equations (3) to (7).

Equation (5) is a derivative of (3).

The calculation of t for the value of Equation (5) to be 0 is expressed by Equation (7) below.

The image processing apparatus can reduce the image A having a long focal length by using the exponent t.

The image processing apparatus performs the correction of the focal length compensated image (140). Calibration refers to a process of transforming an image so that the epipolar line is parallel to the image A and the image B, as is well known. The image processing apparatus can perform a correction operation on two images through various algorithms. The calibration process is a well-known process, so a detailed description will be omitted.

Then, the image processing apparatus generates the final stereo image using the corrected images A and B as an input (150).

Figure 3 is another example of a flowchart for a method 200 of calibrating a stereo image using two different cameras. It is assumed that the focal lengths of the first camera and the second camera are different. The image processing apparatus receives the image A acquired by the first camera 210 and receives the image B acquired by the second camera 220.

The image processing apparatus first compensates the focal length using the image A and the image B (230). The focal length compensation is the same as that described above in Fig. Further, the image processing apparatus performs correction (230) so as to be row-aligned on the same horizontal line with respect to the image A and the image B, respectively. If the first camera and the second camera are not located on the same horizontal line in the space, the image A acquired by the first camera may be considered to be rotated constantly compared with the image B acquired by the second camera. For example, different cameras disposed in a vehicle may not be located on the same horizontal line. In this case, a process of matching images acquired by the first camera and the second camera to each other is required.

The image processing apparatus can search for the feature points of the image A and the image B and correct the corresponding feature points to be located on the same horizontal line. Or the image processing apparatus can correct the plurality of feature points corresponding to the image A and the image B to be positioned on the same horizontal line. In other words, the image processing apparatus may correct the image A and the image B so that they are parallel to each other based on information about a specific object appearing on the screen.

On the other hand, the image processing apparatus removes a predetermined region from the image of one of the image A and the image B compensating the focal length (240). Image A and Image B were images with different focal lengths. Therefore, basically, the objects appearing in the image A and the objects appearing in the image B may be different. If the first camera and the second camera shoot images at the same point, the objects appearing in the images A and B are different if the focal distances are different. This is because images taken with a short focal length camera have a wider viewing angle.

For example, when the image processing apparatus reduces the size of the image A, the image B has an area not shown in the image A. The image processing apparatus should crop an area that is not displayed in the image A of the image B in order to generate a stereo image of the same size.

Since the positions of the first camera and the second camera are different, the images A and B basically have different viewpoints. The image processing apparatus generates an image similar to an image acquired by a general stereo camera through region elimination.

The area removed from image B is four zones. The four regions are referred to as d _top , the bottom region d _bottom , the left region d _left, and the right region d _right . Let H _r be the height distance of the image A, and W _r be the width distance of the image A. d _top and d _bottom can be summed and removed by H _r , and d _left and d _right can be summed and removed by W _r .

The image A and the image B obtained by the image processing apparatus for correcting the focal distance and removing a part of the area correspond to the image obtained by a general stereo camera. Therefore, focal length compensation, horizontal line alignment, and constant area removal are preprocessing processes.

The image processing apparatus performs a calibration on the preprocessed image (250). Then, the image processing apparatus generates the final stereo image using the corrected images A and B as inputs (260).

Now we will describe the process of creating a pair of stereo images using three heterogeneous cameras. Figure 4 is an example of a method 300 for calibrating a stereo image using three different cameras.

 The image processing apparatus receives the image A acquired by the first camera 310, receives the image B acquired by the second camera 320, and receives the image C acquired by the third camera 320. It is assumed that the focal lengths of the first camera and the second camera are different and the focal lengths of the second camera and the third camera are different.

The image processing apparatus attempts to generate a stereo image using the image A and the image B, and a stereo image using the image B and the image C. Accordingly, the image processing apparatus individually performs the above-described processes for a pair of image A and image B, image B, and image pair C, respectively.

The image processing apparatus compensates the focal distance using the image A and the image B (340). The focal length compensation is the same as that described above in Fig. The image processing apparatus performs the correction so as to be row-aligned on the same horizontal line with respect to the image A and the image B (350). The image processing apparatus corrects the corrected image A and image B pairs (360).

Meanwhile, the image processing apparatus compensates the focal distance using the image B and the image C (340). The focal length compensation is the same as that described above in Fig. The image processing apparatus performs the correction so as to be row-aligned on the same horizontal line with respect to the image B and the image C (350). The image processing apparatus corrects the corrected image B and the image C pair (360).

When the calibration for the image A and the image B is performed, the first input image A is changed to the image A ¹ , and the input image B is changed to the image B ¹ . Also, when the correction for the image B and the image C is performed, the first input image B is changed to the image B ² , and the input image C is changed to the image C ² . The images B ¹ and B ² are not the same because they are transformed into different processes. Therefore, in order to merge the images A ¹ and the image depth information calculated from the B ¹ and B ² and the image depth information calculated from the image C ² must identify the relationship between the video picture B ¹ and B ² (370).

The relationship between image B ¹ and image B ² can be grasped in various ways. One example is the relationship between image B ¹ and image B ² using an artificial bee colony (ABC) algorithm, an improved form of genetic algorithm.

In the ABC algorithm, it is important to define the structure of the problem and set the cost function. The difference between image B ¹ and image B ² can be defined as an element of rotation and translation. The rotation R and the transformation T can be defined by the following equations (8) and (9), respectively. In the following formulas, x ₁ , x ₂ and x ₃ are variables to be estimated.

(i) the structure of the problem

The rotation matrix R uses a value of x ₁ indicating the degree of rotation. Rotation generally refers to the degree to which another image is rotated based on one image. Therefore, x ₁ has a range of 0 ° ≤ x ₁ ≤ 360 °.

The transformation matrix T uses the values x ₂ representing x-axis shift and x ₃ representing y-axis movement. x ₂ has a range of -W < x ₁ < W. W is the width of the screen. x ₃ has a range of -H? x ₁ ? H. H is the height of the screen.

Therefore, the solution X in the ABC algorithm can be defined as an array of length 3. X = [x ₁ x ₂ x ₃ ]. And the lower threshold value lb = [0-W-H], and the upper threshold value ub = [360 WH].

(ii) cost function

및

라고 한다.The image processing apparatus may calculate the difference between the two images B ¹ and B ^{2 as a} whole. For efficient computation, it is assumed that the image processing apparatus computes the difference between corresponding points in the images B ¹ and B ² . The image processing apparatus can perform SIFT on the images B ¹ and B ² similarly to the calibration process to find corresponding points. A set of corresponding points in image B ¹ and image B ² is denoted by

And

.

은 고정하고,

에 대해서만 회전 및 변환을 수행한다고 가정한다. 집합 B²에서 변환한 집합을

라고 명명한다.

는 아래의 수학식 10 또는 수학식 11로 연산할 수 있다. 수학식 10은 변환 연산 전에 회전 연산이 수행되었다고 가정한 것이고, 수학식 11은 반대로 변환 연산 후에 화전 연산이 수행되었다고 가정한 것이다.It is possible to change the image B ¹ or change the image B ² so that corresponding points in the images B ¹ and B ² have the same coordinates. For convenience of explanation,

Lt; / RTI >

≪ / RTI & Set B ² converted set

.

Can be calculated by the following Equation (10) or (11). Equation (10) assumes that a rotation operation has been performed before the conversion operation, and (11) assumes that the flame operation has been performed after the conversion operation.

및

로부터 오차(또는 비용)를 연산하면 아래의 수학식 12와 같다.E.g

And

(Or cost) is calculated from the following equation (12).

와

사이의 차이 및

와

사이의 차이 중 작은 에러를 갖는 값으로 결정할 수 있다.The final cost is given by Equation 13 below

Wow

Differences between

Wow

Can be determined as a value having a small error.

F determined by Equation (13) becomes a cost function for the ABC algorithm.

The image processing apparatus can derive the parameter values for the rotation and the transformation by applying the structure of the problem and the cost function to the ABC algorithm. That is, estimating the specific parameter value with the smallest defined error (cost).

The image processing apparatus can estimate the relationship between the image B ¹ and the image B ² using the above-described ABC algorithm or the like. The image processing apparatus can then determine the relationship between the first depth information in the stereo image pairs A ¹ and B ¹ and the second depth information in the stereo image pairs B ² and C ¹ . The image processing apparatus can perform integrated processing on the two stereo images using the first depth information and the second depth information.

After estimating the relationship, the image processing apparatus generates a first stereo image using the corrected images A and B (380), and generates a second stereo image using the corrected images B and C (390).

5 is an example of a process of correcting an image acquired by three different cameras. 5 (a) shows an example in which the image processing apparatus receives images acquired by three cameras. It is assumed that the image A obtained by the first camera from the leftmost to the right, the image B acquired by the second camera, and the image C acquired by the third camera. The focal distance of the first camera may be 16 nm, the focal distance of the second camera may be 12 nm, and the focal distance of the third camera may be 16 nm.

5 (b) shows an example in which the image processing apparatus performs focal length and area removal on an image. (a), the red straight line in image A, the green straight line in image B, and the red straight line in image C are connected to each other. The image processing apparatus can correct the focal distances of the image A, the image B, the image B, and the image C through the difference in the lengths of the straight lines. The image processing apparatus can reduce the image A and the image C uniformly and make the image B and the focal distance equal. Further, the image processing apparatus can remove an area not shown in the image A to the image C in the image B. One of the areas to be removed is shown in image B of process (a). The area indicated by the green arrow in the image B corresponds to d _top which is the upper area of the area to be deleted.

5 (c) shows an example in which the corrected image based on the corrected image A and image B is displayed. 5 (d) shows an example of a corrected image based on the corrected image B and the corrected image C, respectively.

The present embodiment and drawings attached hereto are only a part of the technical idea included in the above-described technology, and it is easy for a person skilled in the art to easily understand the technical idea included in the description of the above- It will be appreciated that variations that may be deduced and specific embodiments are included within the scope of the foregoing description.

10: Vehicle

Claims (10)

A stereo image calibrating method for an image processing apparatus including three different cameras,
Receiving a first image, a second image, and a third image acquired by the three different cameras;
Correcting the focal distance between the first image and the second image and correcting the first and second images whose focal lengths have been corrected to be located on the same horizontal line, Generating a second image;
Correcting the focal length between the second image and the third image, correcting the focal distance corrected second and third images so that they are located on the same horizontal line, Generating a transformed third image;
Estimating a relationship between the first converted first image and the second converted second image;
Determining a relationship between the first depth information and the second depth information on the basis of the relationship between the estimated first converted first image and the second converted second image, Is the depth information between the first converted first image and the first converted second image, and the second depth information is the depth information between the second converted first image and the second converted third image Wherein the depth information is depth information between the first image and the second image.
Generating a first stereo image and a second stereo image by integrating the first depth information and the second depth information, wherein the first stereo image includes a first stereo image and a second stereo image, Wherein the second stereo image comprises the second transformed second image and the second transformed third image, wherein the second stereo image comprises the second transformed second image and the second transformed third image. delete delete The method according to claim 1,
Removing an area that is not present in the other image from any one of the first image and the second image whose focal length is corrected; And removing an area not present in the other one of the second image and the third image whose focal length is corrected. The method according to claim 1,
Wherein the image processing apparatus performs a focal distance correction on two images and a relation estimation on two images using a technique of finding a feature point in an image. The method according to claim 1,
The image processing apparatus estimates the relationship by using rotation and positional movement relative to the other image based on any one of the first and second converted images and the second converted second image A stereo image correction method for heterogeneous cameras. The method according to claim 1,
Wherein the image processing apparatus has a problem structure in which the relative degree of rotation and the positional shift between the first converted second image and the second converted second image are in a problem structure, And estimating the relationship through a genetic algorithm for the stereo image. delete delete delete

Images