KR20160098020A - Rectification method for stereo image and apparatus thereof - Google Patents

Abstract

Disclosed is a method for rectifying stereo images. The method according to an embodiment includes a step of receiving images and determining a matrix for rectifying the images based on a cost function. The cost function may include a distortion element. The matrix may include a matrix for the focus of a camera, a matrix for the separation of the camera, and a matrix for the rotation of the camera. So, a distortion generated in an image rectification process can be minimized.

Description

TECHNICAL FIELD [0001] The present invention relates to a method of correcting stereo images, and a device for performing the method.

The following embodiments relate to a technique for correcting stereo images.

Recently, stereo images have been used in various fields. The stereo image may include a left image and a right image. For proper stereo effects, the left and right images should be aligned with each other. For example, corresponding pairs of the left and right images should be located on the same horizontal line, respectively. This requires rectification of the left and right images.

An image correction method according to one aspect, comprising: receiving a plurality of images; And determining a matrix for rectification of the images based on a first cost function associated with a distance of a corresponding pair between the images and a second cost function associated with a distortion of the transformed image . The plurality of images may include a stereo image.

Wherein the second cost function comprises a first distortion component with respect to an aspect ratio of the transformed image; A second distortion component relating to a skewness of the transformed image; A third distortion component related to rotation of the original image-transformed image; And a fourth distortion component related to an area ratio of the original image to the converted image.

The first cost function may include a sum of Sampson distances of corresponding pairs between the images.

The determining may comprise determining values of predetermined parameters included in the matrix such that a sum of the first cost function and the second cost function is minimized.

Wherein the parameters include a first focal distance of a first camera that has captured a first image of the images, a second focal distance of a second camera that captures a second image of the images, a first focal distance of the first camera The second distance of the second camera, the first rotation angles of the first camera, and the second rotation angles of the second camera.

The matrix comprising a first transformation matrix for transforming a first one of the images; And a second transformation matrix for transforming a second one of the images.

Wherein each of the first transformation matrix and the second transformation matrix comprises: a first camera matrix relating to a focus of a camera that captures a corresponding image; A second camera matrix relating to the separation of the camera that captured the image; And a third camera matrix related to the rotation of the camera that captured the image.

Wherein the image correction method comprises: extracting feature points from the images; And extracting corresponding pairs between the images by matching the feature points between the images.

Wherein the image correction method comprises: calculating a homography matrix between the plurality of images based on corresponding pairs between the images; And determining a weight of the second cost function based on the homography matrix.

Wherein the step of determining the weighting step comprises the steps of: determining a weight of a distortion component related to distortion included in the second cost function, based on a distortion component of the homography matrix; Determining a weight of a distortion component associated with rotation included in the second cost function based on the rotational component of the homography matrix; And determining a weight of a distortion component associated with the scale included in the second cost function based on the scale component of the homography matrix.

Wherein the image correction method comprises: transforming the plurality of images based on the matrix; And performing stereo matching based on the transformed images.

An image correction apparatus according to one aspect includes: a receiver for receiving a plurality of images; And a determination unit for determining a matrix for correction of the images based on a first cost function associated with the distance of the corresponding pair between the images and a second cost function associated with the distortion of the transformed image.

An image processing method according to one aspect includes: acquiring a stereo image; Extracting feature points from the stereo image; Extracting corresponding pairs by matching feature points of a left image and feature points of a right image; And determining a basis matrix of the stereo image based on a first cost function associated with the distance of the corresponding pair and a second cost function associated with the distortion of the transformed image.

The image processing method comprising: calculating a homography matrix between the stereo images based on the corresponding pairs; And determining a weight of at least one distortion component included in the second cost function based on the homography matrix.

Wherein the step of determining the weighting step comprises the steps of: determining a weight of a distortion component related to distortion included in the second cost function, based on a distortion component of the homography matrix; Determining a weight of a distortion component associated with rotation included in the second cost function based on the rotational component of the homography matrix; And determining a weight of a distortion component associated with the scale included in the second cost function based on the scale component of the homography matrix.

The distortion component associated with the distortion may include a first distortion component related to the aspect ratio of the transformed image; And a second distortion component related to the skewness of the transformed image. The distortion component associated with the rotation may include a third distortion component related to the rotation of the transformed image relative to the original image. The distortion component associated with the scale may include a fourth distortion component related to an area ratio of the transformed image to the original image.

1A is a view for explaining an image correction apparatus according to an exemplary embodiment;
1B is a view for explaining a lead wire according to an embodiment;
FIG. 2A illustrates a process of acquiring an image through a stereo camera according to an exemplary embodiment; FIG.
FIG. 2B is a diagram illustrating parameters defined for correction of acquired images according to an embodiment; FIG.
3 is a diagram illustrating parameters for determining a degree of distortion of a corrected image according to an exemplary embodiment;
FIG. 4 illustrates a process of obtaining parameters for determining a mattress using a first cost function and a second cost function according to an embodiment; FIG.
5 is a view for explaining an image conversion process through image correction according to an exemplary embodiment;
6 is a view for explaining an image correction method according to an embodiment;
7A to 7C are views showing images before correction and images corrected according to an embodiment;
8 is a view for explaining a basic matrix determination method according to an embodiment;
9 illustrates a method for adaptively determining a weight of a second cost function according to an embodiment;
10A and 10B illustrate a homography matrix according to one embodiment.
11 is a block diagram illustrating an electronic system according to one embodiment in accordance with one embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Like reference symbols in the drawings denote like elements. The embodiments described below can be used to process stereo images. Embodiments may be implemented in various forms of products, such as personal computers, laptop computers, tablet computers, smart phones, televisions, smart home appliances, intelligent cars, kiosks, wearable devices,

FIG. 1A is a view for explaining an image correction apparatus according to an embodiment. Referring to FIG. 1, an image correction apparatus 100 according to an exemplary embodiment includes a receiving unit 110 and a determining unit 120.

The receiving unit 110 receives the plurality of images 131 and 132. The plurality of images 131 and 132 may include a stereo image. A stereo image means an image shot at a plurality of viewpoints. According to one aspect, a stereo image may include a left image photographed at the left viewpoint and a right image photographed at the right viewpoint.

The stereo image includes corresponding pairs. For example, when the same object is photographed at different viewpoints, the positions of points representing the object may differ from each other due to the parallax. The two images can be compared with each other to extract the corresponding pairs from the stereo image. The technique of extracting corresponding pairs from a stereo image may be referred to as stereo matching.

In order to reduce the complexity of stereo matching, the two images can be compared with each other under the assumption that the vertical locations of the two points included in the corresponding pair are the same and only the horizontal location is different. For example, the left camera and the right camera for photographing a stereo image may be spaced apart in the horizontal direction and arranged at the same height in the vertical direction. In this case, in order to extract the corresponding pair in the stereo image, stereo matching may be performed assuming that the corresponding pairs of the left image and the right image exist in the same vertical position.

However, the left camera and the right camera, which photograph stereo images, may not be perfectly aligned due to various reasons. For example, in the manufacturing process, the heights of the left camera and the right camera may be slightly different, and the orientation of the left camera or the right camera may be changed during use. In this case, assuming that the corresponding pairs of the left and right images are in the same vertical position with each other, the accuracy of the stereo matching can be reduced.

The image correction apparatus 100 according to one embodiment can correct each stereo image so that all corresponding pairs of stereo images are in the same vertical position. Hereinafter, stereo correction may mean correcting each stereo image such that all corresponding pairs of stereo images are in the same vertical position.

Referring to FIG. 1A, a plurality of images 131 and 132 may include a first view image 131 and a second view image 132. For example, the image 131 at the first viewpoint corresponds to the left image, and the image 132 at the second viewpoint corresponds to the right image. The image 131 may include a point x and the image 132 may include a point x '. The point (x) and the point (x ') may be the corresponding pair. For convenience of explanation, only the point (x) and the point (x ') are shown as a corresponding pair in FIG. 1, but a plurality of correspondences Pairs can exist.

Of the points included in the image 132, the points corresponding to the point x of the image 131 can be shown as an epipolar line r. Referring to FIG. 1B, a point X on a three-dimensional space is represented by a point x of an image 131 while being projected onto a camera center C at a first point in time. Further, the point X on the three-dimensional space is represented by the point x 'of the image 132 while being projected to the camera center C' of the second view point. In this case, since the point (x) and the point (x ') are the result of projecting the same point (X) in the three-dimensional space on the camera center of each point of view, the point (x) do.

The plane including the camera center C of the first viewpoint, the camera center C 'of the second viewpoint and the point X on the three-dimensional space is an epipolar plane, and the camera center C at the first viewpoint and the plane The straight line connecting camera center C 'is the baseline. The straight line connecting the point (e ') and the point (x') at which the base line and the image 132 meet is the isomorphism line (r). The equipotential line r is a line where the equiphase plane and the image 132 meet.

Referring again to FIG. 1A, the plurality of images 131 and 132 are images before stereo rectification is applied, and the vertical position of the isosaural points of the point x is a point (x) As shown in FIG. The image correction apparatus 100 may convert the image 132 so that the vertical position of the isodose line r points is equal to the vertical position of the point x through stereo correction. For example, in accordance with the stereo correction, the image 132 may be corrected to the image 133. [ Referring to the image 131 and the image 133, the vertical positions of the points x and r _new are the same.

To transform the image 132 such that the vertical position of the isodose (r) vertices is equal to the vertical position of the vertex x, the isodepth line r corresponding to the vertex x must first be derived. For example, if the iso line r is derived, the image 132 can be transformed through a transformation to make the slope of the iso line r zero. The determination unit 120 may determine a matrix for stereo correction of the plurality of images 131 and 312 in order to derive the isodose line r. A more detailed description related to the operation of the determination unit 120 will be described later.

In the above description, the stereo correction is applied to the image 132 for convenience of explanation. However, the stereo correction may be applied to the image 131 as well. For example, after the image correcting device 100 derives the isoelectric line r 'corresponding to the point x' in the image 131, the image correcting device 100 calculates the vertical position of the isoelectric point r ' 'To be equal to the vertical position of the image 131'.

In the above description, the stereo image is composed of the left and right images. However, the relative positions of the plurality of images may vary depending on the viewpoint of the camera or the arrangement of the camera . For example, when a camera for photographing a stereo image is arranged up and down rather than horizontally, a stereo image may include an upper image photographed at an upper point and a lower image photographed at a lower point. In this case, the images can be transformed so that the horizontal positions of the corresponding pairs are equal to each other.

Also, as will be described in detail below, the image correction apparatus 100 can provide a technique for minimizing distortion generated in the correction process of stereo images. Hereinafter, the stereo correction process will be described in detail.

≪ Determination of basic matrix &

The determination unit 120 determines a matrix for stereo correction of the plurality of images 131 and 312. The matrix for stereo correction of images may be a fundamental matrix. The basic matrix F may have the relationship as shown in Equation (1) between the point (x) and the equipotential line (r).

In Equation (1), the basis matrix F may be a 3 x 3 matrix. (x, y, 1) ^T denotes a point (x), (a, b , c) T represents a crowned line (r). According to Equation (1), the point (x) can be transformed into an iso line (r) by the basis matrix F.

The relationship between the equipotential line r and the point x 'can be expressed by Equation (2), and Equation (3) can be obtained from equations (1) and (2).

On the other hand, as the conversion of a stereo image correction for the left H _l, and for the conversion of a stereo image correction for the right to as H _r, and the equation (4) and Equation (5) is satisfied with respect to H and H _{l r.}

Since the heights of the corresponding pairs in the stereo-corrected image are the same, the relation y _new = y ' _new holds, and the equation (6) can be obtained therefrom.

In Equations (4) to (6), 'new' denotes the coordinates of the corrected image. Equation (7) can be obtained from Equations (3) to (6).

According to Equation (7), the basis matrix F can be determined by H _l and H _r . The determination unit 120 can determine the base matrix F by determining H _l and H _r . H _l and H _r can be parameterized by various parameters. The parameters for the basic matrix F will be described in detail with reference to FIGS. 2A and 2B.

≪ Parameters of basic matrix >

2A is a view for explaining a process of acquiring an image through a stereo camera according to an embodiment. Referring to FIG. 2A, the stereo cameras 210 and 220 include a camera 210 at a first viewpoint and a camera 220 at a second viewpoint. The camera 210 and the camera 220 are disposed on the left and right sides. The y-coordinate of the viewpoint of the camera 210 and the viewpoint of the viewpoint of the camera 220 may be the same. Hereinafter, the process of acquiring a stereo image will be described with reference to the stereo cameras 210 and 220 arranged on the left and right sides, but the stereo cameras 210 and 220 may be arranged up and down. When the stereo cameras 210 and 220 are vertically disposed, the x-coordinate of the viewpoint of the camera 210 and the viewpoint of the viewpoint of the camera 220 may be the same.

The camera 210 and the camera 220 have different viewpoints. Therefore, when the camera 210 and the camera 220 photograph the same object, the coordinates of the object in the image photographed by the camera 210 and the coordinates of the object in the image photographed by the camera 220 are different from each other can do. For example, the image 131 of FIG. 1 may be an image captured by the camera 210, and the image 132 of FIG. 1 may be an image captured by the camera 220.

Even when the camera 210 and the camera 220 are arranged side by side, stereo images in an un-aligned state due to various factors can be obtained. For example, various factors may be the focal length of the cameras 210 and 220, the distance of the cameras 210 and 220, and the rotation of the cameras 210 and 220. The image correcting apparatus 100 may perform stereo correction on images obtained by considering various factors.

FIG. 2B is a diagram illustrating parameters defined for correction of acquired images according to an embodiment. Referring to FIG. 2B, the parameters of the cameras 210 and 220 shown in FIG. 2A are shown. The parameters of the camera 210 include a focal length 211, a separation distance 212, and rotation angles 213, 214, 215. The parameters of the camera 220 also include a focal length 221, a separation distance 222, and rotation angles 223, 224, and 225.

The focal lengths 211 and 221 may be defined in a general sense. For example, the focal lengths 211 and 221 may mean distances from the lenses of the respective cameras 210 and 220 to the image sensor. The separation distance 222 refers to a distance that the cameras 210 and 220 are spaced up or down or left and right from the designed position. When the cameras 210 and 220 are disposed to the left and right, the separation distances 212 and 222 may mean distances between the cameras 210 and 220, respectively. When the cameras 210 and 220 are arranged vertically, the separation distances 212 and 222 may mean distances between the cameras 210 and 220, respectively. The rotation angles 213, 214, 215, 223, 224, and 225 indicate degrees of rotation in the direction shown in FIG. 2B.

Referring again to Figure 1, the decision unit 120 may parameterize H _l and H _r through at least some of the parameters described through Figure 2b.

According to one aspect, the decision unit 120 may parameterize H _l and H _r in consideration of the focal distance of the camera, the separation distance of the camera, and the rotation angle of the camera. In this case, H _l and H _r can be expressed by Equations (8) and (9) in consideration of the focal length of the camera, the distance of the camera, and the rotation angle of the camera.

Here, K _ol , K _or , K _nl , and K _nr are camera matrices relating to the focal length of the camera. K _{ol, or} K, K _{nl, nr} K are each left uncorrected image, the pre-compensation of the right image, the corrected left image and the right image correction matrix camera. K _ol and K _or can be expressed by Equations (10) and (11), respectively.

Here, α ₁ denotes the focal length of the left camera, and α _r denotes the focal length of the right camera. w denotes the width of the pre-correction image, and h denotes the height of the pre-correction image. K _n1 and K _nr can be set to Equation 12 so that the focal lengths of the left camera and the right camera are the same in the corrected stereo image.

In Equations (8) and (9), R _l and R _r are camera matrices relating to camera rotation. R _l and R _r are the camera metrics of the left image before correction and the image before correction, respectively. As for the focal length and the rotation angle, the contents described in Fig. 2B can be applied.

In Equations (8) and (9), T _l and T _r are camera matrices relating to camera separation distances. T _l and T _r can be expressed by Equations (13) and (14), respectively.

Here, t _yl represents the vertical separation distance of the left camera and t _yr represents the vertical separation distance of the right camera. If the camera that captures stereo images is composed of an upper camera and a lower camera, the horizontal distance of the upper camera and the horizontal distance of the lower camera can be used as parameters.

The determination unit 120 can determine the basis matrix F by adjusting the values of the parameters described above.

<Cost function>

The determination unit 120 can determine the basic matrix F based on the cost function. The cost function means a function for evaluating the correction performance of the base matrix F determined based on the parameters. The cost function can be variously defined according to the evaluation criteria of the correction performance. The determination unit 120 may determine the values of the parameters of the basic matrix F so that the output of the cost function is minimized.

The cost function may include a first cost function associated with the distance of the corresponding pair. The first cost function can evaluate the correction performance of the basis matrix F based on the Sampson distance. The first cost function can be expressed by Equation (15).

Where E _s is the first cost function, j is the index of the corresponding pairs, and E _j is the Samson distance of the jth corresponding pair. According to equation (15), E _s can be calculated as the sum of the Samson distances of the corresponding pairs. The Samson distance is determined by the distance between the corresponding point of a certain point and the iso line of the point. For example, the Samson distance increases as the distance between the point (x _j ) and the equipotential line (r _j ') in the left image and the distance between the point (x _j ') and the equipotential line (r _j ) in the right image. Embodiments may determine a basis matrix F that derives the isoform lines of the points contained in the image by determining the parameters such that the output of the first cost function is minimized.

In order to calculate the first cost function, the determination unit 120 must know a predetermined number of corresponding pairs. The predetermined number may be a minimum required number to calculate the first cost function. As will be described in detail with reference to FIG. 5, a predetermined number of corresponding pairs may be extracted before stereo matching is performed. For example, feature points can be extracted from a stereo image. A predetermined number of corresponding pairs can be extracted by matching feature points extracted from the left image and feature points extracted from the right image.

The cost function may include a second cost function associated with distortion of the transformed image. The second cost function may include components for evaluating the degree of distortion of the corrected image. For example, the second cost function may include a first distortion component with respect to the aspect ratio of the transformed image, a second distortion component with respect to the skewness of the transformed image, a second distortion component with respect to the original image, ) And a fourth distortion component related to an area ratio of the transformed image to the original image. Embodiments may determine a basis matrix F that minimizes distortion due to stereo correction by determining parameters such that the sum of the output of the first cost function and the output of the second cost function is minimized. The second cost function is described in detail with reference to FIG.

FIG. 3 is a diagram illustrating parameters for determining a degree of distortion of a corrected image according to an embodiment. Referring to FIG. 3, an image 310 before correction and a corrected image 320 are shown. The image 310 includes the corners (a, b, c, d), the centers e, f, g and h of the sides and the center o of the image 310. The image 320 includes the corners (a ', b', c ', d'), the center of the sides (e ', f', g ', h') and the center o 'of the image 320.

The first distortion component with respect to the aspect ratio of the converted image can be determined by Equation (16).

Here, E _AR means a first distortion component. The closer the distances from the center (o ') to the edges (a', b ', c', d ') are, the closer the E _AR is to 1. In other words, the ideal value of E _AR , where no distortion occurs in image 320, is 1. Therefore, it can be determined that the distortion of the image 320 is smaller as the E _AR is closer to 1.

Next, the second distortion component related to the distortion of the transformed image can be determined by Equation (17).

Here, E _SK denotes the second distortion component, and ∠CA _i denotes the angle of the i-th corner in the image 320. E _SK becomes smaller as the internal angle of each of the corners (a ', b', c ', d') of the image 320 is closer to 90 °. Thus, the ideal value of the E _SK to 0, the more E _SK is close to zero can be determined with less distortion of the image (320).

Next, the third distortion component related to the rotation of the image transformed to the original image can be determined by Equation (18). The third distortion component can be calculated using the dot product of the vector as shown in equation (18). G ', h') of the side and the center of the side (f ') of the side, while the expression (18) shows an embodiment relating to the center (f) ). &Lt; / RTI &gt;

Here, E _R means the third distortion component. The ideal value of E _R is 1, and as E _R approaches 1, it can be determined that distortion of image 320 is less.

Next, the fourth distortion component related to the area ratio of the image converted to the original image can be determined by Equation (19).

Here, E _S means the fourth distortion component. Area _rec represents the area of the image 320, and Area _orig represents the area of the image 310. The ideal value of E _S is 1, the more the closer to 1 E _S may be determined to be less distortion of the image (320).

로 정의될 수 있고, 제2 왜곡 성분이 이상적인 값에 가까울수록 감소하는 함수는

로 정의될 수 있고, 제3 왜곡 성분이 이상적인 값에 가까울수록 감소하는 함수는

로 정의될 수 있고, 제4 왜곡 성분이 이상적인 값에 가까울수록 감소하는 함수는

로 정의될 수 있다. 제2 비용 함수는 전술한 함수들에 기초하여 결정될 수 있다.The function that decreases as the first distortion component approaches the ideal value is

And a function that decreases as the second distortion component approaches the ideal value is

, And the function that decreases as the third distortion component approaches the ideal value is

, And the function that decreases as the fourth distortion component approaches the ideal value is

. &Lt; / RTI &gt; The second cost function may be determined based on the functions described above.

Referring again to FIG. 1, the determination unit 120 may determine the basis matrix F based on the first cost function and the second cost function described above. For example, the values of the parameters of the basis matrix F may be determined such that the sum of the first cost function and the second cost function is minimized. Equation (20) represents an example of the sum of the first cost function and the second cost function. The determination unit 120 can determine the basic matrix F based on the expression (20).

,

,

및

는 각각 제1 왜곡 성분, 제2 왜곡 성분, 제3 왜곡 성분, 및 제4 왜곡 성분에 관한 함수들을 나타내며, ρ_AR, ρ_SK, ρ_R 및 ρ_SR는 각각

,

,

및

의 가중치를 나타낸다. 결정부(120)는

가 최소가 되도록 φ를 결정할 수 있다.Where C represents the total cost function for the sum of the first cost function and the second cost function,? Represents the parameters of the base matrix F, E _s represents the first cost function,

,

,

And

Are each the first distortion and the second distortion component, and the third distortion component, and the fourth represents a function of the distortion component, ρ _{AR, SK} ρ, ρ ρ _R and _SR are each

,

,

And

&Lt; / RTI &gt; The determination unit 120 determines

Can be determined to be minimum.

The image correction apparatus 100 may further include a conversion unit. The conversion unit can convert the plurality of images 131 and 132 based on the determined base matrix F. [ For example, the conversion unit derives the iso line r corresponding to the point x by using the base matrix F, and then outputs the image 132 to the image 133 through the conversion to make the inclination of the iso line r to 0, . &Lt; / RTI &gt;

The image correction apparatus 100 may further include a matching unit. The matching unit performs stereo matching based on the images converted by the converting unit.

4 is a diagram illustrating a process of obtaining parameters for determining a mattress using a first cost function and a second cost function according to an embodiment.

Referring to FIG. 4, in step 410, the image correction apparatus determines parameters based on a first cost function. The image correction apparatus can output the parameters? _Init which minimize the first cost function E _s among the input parameters?. In step 410, φ represents the input parameter, Es represents the first cost function, and C (φ) represents the value of the total cost function by the input parameter φ.

In step 420, the image correction device determines the degree of distortion of the converted image. In step 420, r _init refers to the initial weight for the distortion components. For example, ρ _init (0, 0, 0, 0) means that the initial weights of the first to fourth distortion components are all zero. The above-described contents may be applied to the first to fourth distortion components. The image correction apparatus can determine the degree of distortion of the transformed image based on the distortion components. The degree of distortion of the transformed image can be represented by E _x .

In step 430, the image correction apparatus compares the degree of distortion of the converted image with a predetermined threshold value. In step 430, f _x denotes a function for determining the degree of distortion of E _x , and T _x denotes a predetermined threshold value.

In step 440, the image correction device adjusts the weight of the second cost function. The image correction apparatus may adjust the weight by reflecting the comparison result of step 430. [ For example, the image correction apparatus can adjust the weight of the second cost function when the degree of distortion of the converted image is equal to or greater than a preset threshold value. For example, the image correction apparatus may adjust the weights of the first to fourth distortion components to 0.25.

In step 450, the image correction apparatus determines the parameters based on the first cost function and the second cost function. The image correction apparatus can determine the parameters based on Equation (20) described above.

In step 460, the image correction unit compares the value of the total cost function corresponding to the value of the total cost function according to φ _n and φ _{n -1.} The value of the total cost function of the φ _n is calculated as the sum of a first cost function and a second cost function according to φ _n, the value of the total cost function of the φ _{n -1} has a first cost of φ _n-1 Function and a second cost function.

If the value of the total cost function of the φ _n is less than the value of the total cost function of the φ _{n -1,} the image correction device determines that the value of the parameter approaches the optimum value, and step 420 to step 450 Can be repeatedly performed. If step 420 is performed after step 460, then the parameter? _{N +1} may be applied as a parameter. [phi] _{n + 1} may be determined from step 450. [

If the value of the total cost function of the φ _n is greater than the value of the total cost function of the φ _{n -1,} the image correction unit may determine that the values of the parameters away from the optimum value and perform step 470 . In step 470, the image correction device outputs φ _{n -1} . The image correction apparatus can determine the basis matrix F through? _N-1 .

The steps described above with reference to FIGS. 1 to 3 may be applied to each step shown in FIG. 4, so that a detailed description will be omitted.

5 is a view for explaining an image conversion process through image correction according to an embodiment.

In step 510, the image correction device may receive stereo images. In step 520, the image correction device may extract corresponding pairs from the stereo images. For example, although not shown in the drawings, the image correction apparatus 100 of FIG. 1 may further include an extraction unit. The extracting unit may extract the corresponding pairs from the plurality of images 131 and 132. The extraction unit may extract the corresponding pairs by extracting the feature points from the plurality of images 131 and 132, and matching the feature points of the image 131 and the feature points of the image 132. [ According to one aspect, the extracting unit can extract feature points using a Scale Invariant Feature Transform (SIFT) method, and can match feature points using a RANSAC (Consecutive Consensus) method. The extracting unit may transmit the information about the extracted pairs to the determining unit 120.

In step 530, the image correction device may perform parameterization of the basis matrix F. In step 540, the image correction device may determine the parameters of the basis matrix F. The image correction apparatus can determine the parameters of the basis matrix F based on the cost function. In step 550, the image correction apparatus can convert the image.

The steps described above with reference to FIGS. 1 to 4 may be applied to each step shown in FIG. 5, so that a detailed description will be omitted.

6 is a view for explaining an image correction method according to an embodiment.

In step 610, the image correction apparatus receives a plurality of images. In step 620, the image correction apparatus determines a matrix for stereo correction of images based on the first cost function and the second cost function. The steps described above with reference to FIG. 1 through FIG. 5 may be applied to each step shown in FIG. 6, so that a detailed description will be omitted.

7A to 7C are views showing images before correction and images after correction according to an embodiment.

7A shows the images 710 and 720 before the correction, FIG. 7B shows the images 730 and 740 corrected based on the first cost function, FIG. 7C shows the first cost function and FIG. 2 compensated images 730 and 740 based on the cost function. Comparing FIGS. 7B and 7C, it can be seen that the distortion of the images 730 and 740 is reduced compared to the images 730 and 740. Embodiments of the present invention can provide a technique for minimizing distortion caused in the image correction process.

8 is a view for explaining a basic matrix determination method according to an embodiment. Referring to FIG. 8, a processor according to an exemplary embodiment may receive a stereo image and extract a feature point included in a stereo image in step 810. FIG. The processor may extract the first feature points from the left image and extract the second feature points from the right image. For example, the processor can extract feature points using the SIFT method. Of course, the processor can extract feature points using various other known feature point extraction techniques.

The processor can extract the pair of feature points based on the extracted feature points in step 820. [ The processor can extract pairs of feature points between the left image and the right image by matching the first and second feature points with each other. For example, the processor may match the first feature points and the second feature points with each other in the RANSAC scheme. Of course, the processor can match feature points using a variety of other known matching techniques.

The processor may determine the basis matrix F based on the feature point corresponding pairs in step 830. [ In the step 830, since the above-described matters can be applied as they are through FIGS. 1A to 7C, detailed description will be omitted.

9 is a view for explaining a method for adaptively determining a weight of a second cost function according to an embodiment. Referring to FIG. 9, the processor may calculate a homography matrix between stereo images at step 831.

Referring to FIG. 10A, the homography matrix is a matrix that transforms a first ray 1010 to a second ray 1020 at a point X. FIG. The first ray 1010 is a path of light projected from a point X on the three-dimensional space to the camera center C of the first viewpoint and the second ray 1020 is a path of light from the same point X to the camera center C '. Alternatively, the homography matrix may be a matrix that transforms a point (x) on the image 131 to a point (x ') on the image 132.

The point (x) and point (x ') may be one of the extracted pairs of feature points. Although not shown in the figure, the processor can know the pair of feature points other than the point (x) and the point (x '). The processor may calculate the homography matrix using a predetermined number of pairs of feature points.

Referring again to FIG. 9, the processor may use the homography matrix in step 832 to determine the weight of the second cost function. The second cost function may comprise a plurality of distortion components. The processor may use the homography matrix to set the weights of the distortion components included in the second cost function.

Referring to FIG. 10B, the homography matrix 1050 may be a 3 x 3 matrix. The homography matrix 1050 may include a torsion component 1051, a rotational component 1052, a scale component 1053, and a moving component 1054.

Based on the distortion component 1051, the processor can determine the weight of the distortion component associated with the distortion included in the second cost function. The distortion component associated with the distortion included in the second cost function may include a first distortion component related to the aspect ratio of the converted image, a second distortion component related to the distortion of the converted image, and the like. For example, the processor may determine ρ _AR and ρ _SK in equation (20).

Further, the processor may determine a weight of the distortion component associated with the rotation included in the second cost function, based on the rotation component 1052. [ The distortion component related to the rotation included in the second cost function may include a third distortion component related to the rotation of the transformed image relative to the original image. For example, the processor may determine ρ _R in equation (20).

Further, the processor may determine a weight of a distortion component associated with the scale included in the second cost function, based on the scale component 1053. The distortion component associated with the scale included in the second cost function may include a fourth distortion component related to the area ratio of the converted image to the original image. For example, the processor may determine ρ _SR in equation (20).

Referring again to FIG. 9, the processor may determine the basis matrix F at step 833. The processor may determine the basis matrix F such that the sum of the first cost function and the second cost function is minimized. At this time, the processor may weight-sum the first cost function and the second cost function based on the weight of the second cost function determined in step 832. [

11 is a block diagram illustrating an electronic system according to one embodiment in accordance with one embodiment. Referring to FIG. 11, the electronic system includes a sensor 1120, a processor 1110, and a memory 1130. Sensor 1120, processor 1110, and memory 1130 may communicate with each other via bus 1140.

The sensor 1120 may be the stereo camera 210, 220 shown in FIG. 2A. The sensor 1120 can capture a stereo image in a well-known manner (e.g., a manner of converting an optical image into an electrical signal, etc.). The image is output to the processor 1110.

The processor 1110 may include at least one of the modules described above with respect to Figures 1 through 10B, or may perform at least one of the methods described above with respect to Figures 1 through 10B. The memory 1130 may store a stereo image captured by the sensor 1120, feature points extracted by the processor 1110, corresponding pairs, and / or a base matrix computed by the processor 1110. [ Memory 1130 may be volatile memory or non-volatile memory.

The processor 1110 may execute the program and control the electronic system. The program code executed by the processor 1110 may be stored in the memory 1130. [ The electronic system is connected to an external device (e.g., a personal computer or a network) through an input / output device (not shown) and can exchange data.

The electronic system can be used in various electronic systems such as mobile devices, smart phones, PDAs, tablet computers, mobile devices such as laptop computers, computing devices such as personal computers, tablet computers, netbooks, or electronic devices such as televisions, smart televisions, Lt; / RTI &gt;

The embodiments described above may be implemented in hardware components, software components, and / or a combination of hardware components and software components. For example, the devices, methods, and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, such as an array, a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the preferred embodiments of the present invention have been disclosed for illustrative purposes, those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced. Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (29)

Receiving a plurality of images; And
Determining a matrix for rectification of the images based on a first cost function associated with a distance of a corresponding pair between the images and a second cost function associated with a distortion of the transformed image
And correcting the image. The method according to claim 1,
The second cost function
A first distortion component related to the aspect ratio of the converted image;
A second distortion component relating to a skewness of the transformed image;
A third distortion component related to rotation of the original image-transformed image; And
The fourth distortion component with respect to the size ratio of the converted image to the original image
The image correction method comprising: The method according to claim 1,
The first cost function
And a sum of Sampson distances of corresponding pairs between the images. The method according to claim 1,
The step of determining
Determining values of predetermined parameters included in the matrix such that a sum of the first cost function and the second cost function is minimized
And correcting the image. 5. The method of claim 4,
The parameters
A first focal length of a first camera that has captured a first image of the images, a second focal distance of a second camera that captures a second image of the images, a first distance of the first camera, 2 the second distance of the camera, the first rotation angles of the first camera, and the second rotation angles of the second camera
And an image correcting unit for correcting the image. The method according to claim 1,
The matrix
A first transform matrix for transforming a first image of the images; And
A second transform matrix for transforming a second image of the images,
And correcting the image. The method according to claim 6,
Wherein each of the first transformation matrix and the second transformation matrix
A first camera matrix relating to a focus of a camera that captures the image;
A second camera matrix relating to the separation of the camera that captured the image; And
The third camera matrix relating to the rotation of the camera
And correcting the image. The method according to claim 1,
Extracting feature points from the images; And
Extracting corresponding pairs between the images by matching the feature points between the images
Further comprising the steps of: The method according to claim 1,
Calculating a homography matrix between the plurality of images based on corresponding pairs between the images; And
Determining a weight of the second cost function based on the homography matrix
Further comprising the steps of: 10. The method of claim 9,
The step of determining the weights
Determining a weight of a distortion component associated with distortion included in the second cost function based on a distortion component of the homography matrix;
Determining a weight of a distortion component associated with rotation included in the second cost function based on the rotational component of the homography matrix; And
Determining a weight of a distortion component associated with a scale included in the second cost function based on a scale component of the homography matrix
The image correction method comprising: The method according to claim 1,
Transforming the plurality of images based on the matrix; And
Performing stereo matching based on the transformed images
Further comprising the steps of: The method according to claim 1,
Wherein the plurality of images include a stereo image. 12. A computer program stored in a medium for executing the method of any one of claims 1 to 12 in combination with hardware. A receiving unit for receiving a plurality of images; And
Determining a matrix for correction of the images based on a first cost function associated with a distance of a corresponding pair between the images and a second cost function associated with distortion of the transformed image,
And an image correcting unit for correcting the image. 15. The method of claim 14,
The second cost function
A first distortion component relating to the aspect ratio of the converted image;
A second distortion component related to distortion of the converted image;
A third distortion component with respect to the original image and the rotated image is rotated; And
The fourth distortion component with respect to the area ratio of the converted image to the original image
The image correction device comprising: 15. The method of claim 14,
The first cost function
And a sum of the Samson distances of corresponding pairs between the images. 15. The method of claim 14,
The determination unit
And determines values of predetermined parameters included in the matrix such that a sum of the first cost function and the second cost function is minimized. 18. The method of claim 17,
The parameters
A first focal length of a first camera that has captured a first image of the images, a second focal distance of a second camera that captures a second image of the images, a first distance of the first camera, 2 the second distance of the camera, the first rotation angles of the first camera, and the second rotation angles of the second camera
And an image correcting device for correcting the image. 15. The method of claim 14,
The matrix
A first transform matrix for transforming a first image of the images; And
A second transform matrix for transforming a second image of the images,
And an image correcting unit for correcting the image. 20. The method of claim 19,
Wherein each of the first transformation matrix and the second transformation matrix
A first camera matrix relating to a focus of a camera that captures the image;
A second camera matrix relating to the separation of the camera that captured the image; And
The third camera matrix relating to the rotation of the camera
And an image correcting unit for correcting the image. 15. The method of claim 14,
Extracting feature points from the images and extracting corresponding pairs between the images by matching the feature points between the images,
Further comprising: 15. The method of claim 14,
The determination unit
Calculate a homography matrix between the plurality of images based on corresponding pairs between the images, and determine a weight of the second cost function based on the homography matrix. 15. The method of claim 14,
A transform unit for transforming the plurality of images based on the matrix; And
A matching unit for performing stereo matching based on the converted images,
Further comprising: Obtaining a stereo image;
Extracting feature points from the stereo image;
Extracting corresponding pairs by matching feature points of a left image and feature points of a right image; And
Determining a basis matrix of the stereo image based on a first cost function associated with the distance of the corresponding pair and a second cost function associated with the distortion of the transformed image
And a program code for causing the computer to execute the image processing method. 25. The method of claim 24,
The image processing method
Calculating a homography matrix between the stereo images based on the corresponding pairs; And
Determining a weight of at least one distortion component included in the second cost function based on the homography matrix,
The computer program product further comprising: 26. The method of claim 25,
The step of determining the weights
Determining a weight of a distortion component associated with distortion included in the second cost function based on a distortion component of the homography matrix;
Determining a weight of a distortion component associated with rotation included in the second cost function based on the rotational component of the homography matrix; And
Determining a weight of a distortion component associated with a scale included in the second cost function based on a scale component of the homography matrix
&Lt; / RTI &gt; 27. The method of claim 26,
The distortion component associated with the distortion
A first distortion component relating to the aspect ratio of the converted image; And
The second distortion component related to the distortion of the converted image
&Lt; / RTI &gt; 27. The method of claim 26,
The distortion component associated with the rotation
A third distortion component related to the rotation of the converted image relative to the original image
Readable recording medium. 27. The method of claim 26,
The distortion component associated with the scale
The fourth distortion component with respect to the area ratio of the converted image to the original image
Readable recording medium.

Images