KR101558573B1 - Method for compositing stereo camera images - Google Patents

Abstract

An embodiment of the present invention relates to a stereoscopic camera image synthesis method, which includes a feature point detection process for detecting feature points with respect to a reference image captured by a first camera and an extended image captured by a second camera, After all the models having the number of pairs of corresponding points determined are determined, the similarity of the angle information, the length information, and the color information of the models is evaluated, Calculating a projection transformation coefficient to be used in the projection transformation method when the projection transformation method using the determined pair of the corresponding points is performed; The extension image is rear-mapped to the reference image, As such, it includes an image alignment process and a final composite image teuleojim color value, brightness value matches the information crystallized image stabilization process of synthesizing a single image.

Camera, image, synthesis, correspondence, feature point, operation, time

Description

[0001] The present invention relates to a method of composing stereo camera images,

An embodiment of the present invention relates to a stereo camera image synthesis method.

With the development of digital camera devices, various practical image processing techniques have been attempted. In order to overcome the limited field of view of a camera lens, a stereo camera may also be used to acquire a plurality of images acquired from a plurality of cameras having the same viewpoint or different viewpoints as a single image to obtain an expanded field of view angle .

Mosaic technique is one of the most well - known general methods of stereo camera image synthesis. The Mosaic technique is a process of creating a single image with an extended image to be continuously synthesized with an arbitrary reference image. The optimal matching pairs are found in the overlapping regions of two images, The enlarged image to be continued with respect to the image is aligned by geometric transformation according to the derived transformation coefficient and stabilized to obtain one image.

However, a problem that is important in the synthesis of such a stereo camera image is the difficulty in searching for the optimum correspondence point.

In the search for the optimal correspondence between two images, if the extended image F 'to be combined with the reference image F is a stereo camera image taken continuously in any frame, It is possible to accurately and quickly find the position of the pixel information of the overlapping extended image F '.

However, in order for the computer to recognize such a difficult process, if the camera module of the continuous extended image F 'has been perspective transformed by the perspective of the user between the two camera images due to the camera shake of the user or a slight vehicle shake In this case, how the machine automatically recognizes the similarity of the converted image is the biggest problem of the stereo camera.

Traditionally, NCC (Normalized Cross Correlation) is best known as a measure of similarity between two F and F 'images, but the correlation method of color information is captured in non-consecutive frames There is a problem that reliability can not be obtained in the process of determining the correspondence point if the image is different from the image or information due to an external situation such as illuminance, shadows, or the performance difference within the ISP sensor.

On the other hand, another problem that is important in the stereo camera image synthesis is the problem of increasing the computational complexity.

In order to synthesize and align the images of two cameras, M minutiae points that are found and at least M ^N times complexity is required to find the optimal correspondence point through matching of N minutiae points in other images. This computation time complexity can lead to a computation time of several tens of seconds in the worst case in a high-resolution image with many edges.

If we want to control real-time high-resolution video stereo camera images that are scanned at a high speed of several tens of frames per second or more at a scene requiring an extended viewing angle, such as a car camera, an increase in computation time complexity can be a fatal problem.

The embodiment of the present invention is a stereo camera image synthesis method which enables to find an optimal correspondence point in a short time and shortens the calculation time in image synthesis.

The embodiment of the present invention is characterized by a feature point detection process for detecting feature points of a reference image captured by a first camera and an extended image captured by a second camera, a pair of corresponding points for determining the number of corresponding points in the reference image and the extended image, Determining a number of pairs of the corresponding points, and then evaluating the similarity of the angular information, the length information, and the color information of the models, and then filtering out similar models to construct an optimal correspondence point. A projection transformation coefficient computing step of computing a projection transformation coefficient to be used in the projection transformation method when the projection transformation method using the determined pair of corresponding points is performed; By mapping and transforming the projected image, Jung, includes a final composite image teuleojim color value, brightness value information matches crystallized image stabilization process of the.

In the filtering process, all the quadrangle-shaped models are formed, which are four non-collinear points among the extracted feature points of the reference image and the extended image, And an angle between the model of the reference image and the model of the extended image is calculated to obtain a model having an angle difference larger than the angle threshold value, A line length between adjacent models adjacent to the reference line of all the models is calculated and a length difference between the model of the reference image and the model of the extended image is calculated to determine a length difference greater than the length threshold After performing the color-NCC to evaluate the similarity of the color information by normalizing the remaining parts minus the brightness average of the two images, If the color similarity between the information model of the color below the threshold, comprises the step of removing the model.

If the number of pairs of optimal matching points filtered through the optimal correspondence point construction process is four, then the matrix P of the eight projection transformation coefficients is a ₁₁ , a ₁₂ , a ₁₃ , a ₂₁ , a ₂₂ , a ₂₃ , a ₃₁ this, a _32, the optimum corresponding points of the reference image (x _1, y _1), and _{(x 2, y 2),} (x 3, y 3), (x 4, y 4) , and the extended video optimum corresponding point ( _{u 1, v 1), (} u 2, v 2), (u 3, v 3), (u 4, v to ₄₎ d,

The projection transformation coefficient is calculated.

The image alignment process may include generating a dummy image in which a base image and an extended image are matched with each other, mapping and mapping pixels of the reference image to the dummy image, And mapping the extended image back to the reference image to synthesize the expanded image as a single image.

In an embodiment of the present invention, in the stereo camera image synthesis, it is possible to efficiently detect the same position of an optimal pair of matching points among a large number of feature points between an arbitrary image sequence t frame and consecutive t + 1 frames . Further, there is an effect that the amount of calculation of the corresponding point among the minutiae points can be reduced.

Hereinafter, a detailed description of embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the same reference numerals are used to denote the same or similar components in the drawings.

1 is a flowchart illustrating a stereo camera image synthesizing process according to an embodiment of the present invention.

The embodiment of the present invention can reduce the computational complexity by detecting the pair of optimal correspondence by repeating the process of reducing unnecessary feature points to reduce the calculation amount of the corresponding point search among the minutiae points and deriving the accurate projection transformation coefficient.

In order to synthesize the stereo camera image, first, the feature points are detected (S11) on the images captured by the two cameras. The feature point represents a set of corner points in each image, and for the feature point detection S11, a SUSAN method, which is a conventionally known technique for detecting corner points between two images, can be used.

After detecting feature points such as a set of corner points in two images as described above, an optimal correspondence point between the feature points between the two images is searched for.

The process of configuring the optimal correspondence point will be described. First, the number of correspondence pairs to be searched is determined (S12).

[Table 1]

projection parameter least pairs transformations displacement 2 One translation similarity 4 2 translation, rotation, uniform scale affine 6 3 translation, rotation, non-uniform scale perspective 8 4 full motion

Table 1 above shows the pairs of optimal corresponding points according to the projection transformation between the projected 2D images of the two cameras. 'parameter' represents the projection transformation coefficient, and 'least pairs' represents the number of pairs of optimal (minimum) corresponding points.

In the case of a camera movement that preserves translation, if the projection transformation of the two cameras is 'displacement' and if a pair of points corresponding to the minimum is found, the translation displacement between the two cameras is sufficient to calculate the translation distance, Only a pair of optimal correspondence points is required.

If we assume translation, rotation, and uniform scaling between two cameras, the projection transformation of both cameras is 'similarity' and requires at least two pairs of corresponding points.

If we assume translation, rotation, non-uniform scaling, and shear transformation between two cameras, the projection transformation of both cameras is 'affine' Corresponding points are required.

If we want to transform the full motion of both cameras, including the full motion, the projection transformation of both cameras is 'perspective' and requires at least four pairs of corresponding points.

Therefore, in deciding the number of corresponding points, it is determined whether to make at least one pair, two pairs, three pairs, or four pairs depending on the shooting environments of the two cameras.

An embodiment of the present invention will exemplify the case of using a 'perspective' projection transformation scheme which may include a full motion for a transformation between two cameras in any frame, 'Projection transformation method, a search is made for at least four pairs of corresponding points in order to find the projection transformation coefficients of the eight parameters including the entire moving displacement of the camera.

Hereinafter, embodiments of the present invention will be described using four pairs of corresponding points in order to apply the 'perspective' projection transformation method. If another projection transformation method is to be used, the number of pairs of corresponding points is determined, The image synthesis process can be implemented in the same manner.

If four pairs of corresponding points are determined according to the 'perspective' projection transformation method as described above, the quadrangular models having the minutiae points of the determined optimal corresponding points are constructed, and the geometric information (angle information, length information ) Filtering (S13) for eliminating similar non-similar models through similarity comparison and color information similarity comparison.

Referring to FIG. 2, the filtering process is performed as follows.

When the number of corresponding points of four pairs is determined according to the 'perspective' projection equation, all possible quadrangles of four non-collinear points among the extracted feature points of the reference image and the extended image, (S21).

That is, all of the quadrangle-shaped models having four non-collinear points on the extracted feature points and two points adjacent to the reference line between the two points are determined (S21) (S22), the difference between the angle information obtained from the two comparison images is obtained, and the non-similar models larger than the angle threshold value are removed (S22).

Thereafter, the length information for the reference line is obtained again for the remaining models, the difference between the length information is obtained, and the non-similar models larger than the length threshold value are removed (S23).

After eliminating models larger than the angle threshold value and the length threshold value, color-NCC is performed in the local region of the two images for the remaining models to evaluate the color information similarity. After the color-NCC is performed, the color threshold is determined and the remaining models are removed (S24), and the set of uncharacterized feature points is determined as a final correspondence pair (S25).

Since the number of pairs of corresponding points is determined to be four or more pairs, the quadrangular model configuration (S21) must be performed. This is because the feature points extracted from the reference image and the extended image And all the models of a quadrangle that can be made up of four non-collinear points.

Assuming that the feature points extracted from the reference image and the feature points extracted from the extended image are as shown in FIG. 3 (a) and FIG. 3 (b), non-collinear 4 Constructs all possible quadrilateral models that can be made up of four points.

The similarity of the angular information of the constructed models is then evaluated to remove non-similar models. That is, the angles of the baselines of all the constructed models are obtained, and the difference between the angular information obtained from the two comparison images is obtained, thereby removing the model larger than the angle threshold value.

The process of removing a model larger than the angle threshold value is shown in FIG. 4. First, an angle table is created (S41).

The angle table has angle information of each model with respect to a reference point (P ₁ , P ' ₁ ). For example, in the reference image of FIG. 3A, angles between two adjacent points P ₂ and P ₃ are calculated based on the reference lines P ₁ -P ₄ and stored in the angle table. Similarly, the angle between the two points _{(P '2, P' 3} ) which are adjacent to each other based on the reference line (P _'1 -P' ₄₎ from the extracted image in the expanded image of FIG. 3 is to be synthesized (b) the feature points Calculate and store in angle table

An example of the angle table for the models of the reference image and the extended image is as follows. In the following, R_Model [N] refers to the Nth model of the reference image, and A_Model [N] refers to the Nth model of the extended image.

If P ₁ , P ₂ , P ₃ , P ₄ = Non-collinear

R_Model [1] = {baseline (P _1, P _4), between two points (P _2, P _3), the angle information _{(P 3, P 1, P} 2), the angle information _{(P 3, P 4, P} 2) }

If P ₁ , P ₂ , P ₃ , P ₅ = Non-collinear

R_Model [2] = {baseline (P _1, P _5), between two points (P _2, P _3), the angle information _{(P 3, P 1, P} 2), the angle information _{(P 3, P 5, P} 2) }

If P ' ₁ , P' ₂ , P ' ₃ , P' ₄ = Non-collinear

A_Model [1] = {baseline _{(P '1, P' 4} ), between two points _{(P '2, P' 3} ), angle information _{(P '3, P' 1} , P '2), the angle information (P' ₃ , P ' ₄ , P' ₂ )}

If P ' ₁ , P' ₂ , P ' ₃ , P' ₅ = Non-collinear

A_Model [2] = {baseline _{(P '1, P' 5} ), between two points _{(P '2, P' 3} ), angle information _{(P '3, P' 1} , P '2), the angle information (P' ₃ , P ' ₅ , P' ₂ )}

If the angle difference value of the two images is greater than the angle threshold value, the two models are determined not to be the same model (S42) The corresponding model is removed (S45).

In detail, the angle information is calculated for the reference image and the extended image to generate an angle table, and then the angle difference is calculated for each model (S42).

Thereafter, the angle difference between the first model of the reference image and the first model of the extended image is called (S43), and it is determined whether the angle difference is greater than a predetermined angle threshold value (S44).

If the angle difference is greater than the angle threshold, the two models are determined not to be the same model and the corresponding model is removed (S45). Thereafter, the angle difference of the next model is called (S43), and similarly, it is determined whether or not the angle difference is larger than the angle threshold value (S44).

The removal processes are expressed as an algorithm in the following manner.

δ _angle = │R_Model [Angle] - A_Model [Angle] │

If δ _angle > THRESHOLD _angle

R_Model ≠ A_Model

The above processes are repeated until reaching the last models (S46), thereby removing the unequal models.

If the angle difference value is greater than the angle threshold value, the two models are determined not to be the same model, and the model is removed (S22). Similarly, the similarity of the line information of the constructed models is evaluated, The model is removed (S23). That is, if the length difference value is greater than the length threshold value, the two models are determined not to be the same model, and the corresponding model is removed (S23).

The process of removing a model having a length greater than the threshold value is shown in FIG. 5. First, a length table is created (S51).

The length table has line information of each model for reference points (P ₁ , P ' ₁ ). For example, in the reference image of FIG. 3A, the lengths of two adjacent points P ₂ and P ₃ are calculated based on the reference lines P ₁ -P ₄ and stored in the length table. Similarly, the length between the two points _{(P '2, P' 3} ) which are adjacent to each other based on the reference line (P _'1 -P' ₄₎ from the extracted image in the expanded image of FIG. 3 is to be synthesized (b) the feature points And stores them in the length table

An example of the length table for the models of the reference image and the extended image is as follows. In the following, R_Model [N] refers to the Nth model of the reference image, and A_Model [N] refers to the Nth model of the extended image.

If P ₁ , P ₂ , P ₃ , P ₄ = Non-collinear

R_Model [1] = {baseline (P _1, P _4), between two points (P _2, P _3), the length information (P _1, P _2), the length information (P _1, P _3), the length information (P ₄ , P ₂ ), length information (P ₄ , P ₃ )}

If P ₁ , P ₂ , P ₃ , P ₅ = Non-collinear

R_Model [2] = {baseline (P _1, P _5), between two points (P _2, P _3), the length information (P _1, P _2), the length information (P _1, P _3), the length information (P ₅ , P ₂ ), length information (P ₅ , P ₃ )}

If P ' ₁ , P' ₂ , P ' ₃ , P' ₄ = Non-collinear

A_Model [1] = {baseline _{(P '1, P' 4} ), between two points _{(P '2, P' 3} ), the length information _{(P '1, P' 2} ), the length information (P _'1, P' ₃ ), length information (P ' ₄ , P' ₂ ), length information (P ' ₄ , P' ₃ )

If P ' ₁ , P' ₂ , P ' ₃ , P' ₅ = Non-collinear

A_Model [2] = {baseline _{(P '1, P' 5} ), between two points _{(P '2, P' 3} ), the length information _{(P '1, P' 2} ), the length information (P _'1, P' ₃ ), length information (P ' ₅ , P' ₂ ), length information (P ' ₅ , P' ₃ )

As described above, the length information is calculated for the reference image and the extended image, and the length difference values of the two images are compared. If the length difference value of the two images is greater than the length threshold value, the two models are determined not to be the same model. Remove.

In detail, the length information is calculated for the reference image and the extended image to generate a length table, and then the length difference is calculated for each model (S52).

Thereafter, the difference between the length of the first model of the reference image and the length of the first model of the extended image is called (S53), and it is determined whether the length difference is greater than a predetermined length threshold value (S54).

If the length difference is greater than the length threshold, it is determined that the two models are not the same model and the corresponding model is removed (S55). Thereafter, the length difference of the next model is called (S53). Similarly, it is determined whether the length difference is larger than the length threshold value (S54), and if it is larger, the model is removed (S55).

The removal processes are expressed as an algorithm in the following manner.

δ _line = │R_Model [line] - A_Model [line] │

If δ _line > THRESHOLD _line

R_Model ≠ A_Model

The processes are repeated until reaching the last models (S56), thereby removing the unequal models.

By comparing the models of two images with the geometric information (angle information, length information) in the extracted minutiae, the models that are not similar to each other are removed in the memory (S22, S23). In this method, unnecessary feature points are automatically filtered out from a large number of feature points, and the amount of calculation can be remarkably reduced.

As a next step, the similarity of the color information is additionally evaluated for the models filtered by the similarity condition of the geometric information (angle information, length information), and the models that are not similar to each other are removed (S24) I have.

The process of eliminating models whose color information is not similar to each other is performed in such a manner that the color similarity of two images for each pixel point in the vicinity of two comparison images is calculated with respect to the models filtered by similarity conditions of geometric information (angle information, length information) Color-normalized cross correlation (Color-NCC) is used as a rating scale. Color-NCC is a technique for evaluating the similarity by normalizing the remaining part of the comparison image minus the average of brightness in the mask.

For reference, the color similarity evaluation method of Color-NCC will be described together with the following [Equation 1].

[Equation 1]

In Equation 1, f _t0 (x, y) represents a reference image model, and f _t1 (u, v) represents an RGB vector component for pixel coordinates of models of an extended image to be synthesized. m _t0 , m _t1 represents the average of the color values in the f _t0 , f _t1 mask, and "∥∥" represents the norm vector for the RGB components.

를 만들고 확장 영상의 지정된 검색 영역 (i,j) 안에서 마스크

들과 상관계수를 조사한다. 상기 [수식 1]은 정규화 과정을 통해 -1 ∼ 1 사이 값들로 지정되며, 두 영상의 비교 마스크가 일치한다면 1의 값을 갖는다.A mask whose size is s × s based on the coordinate (x, y)

(I, j) of the extended image,

And the correlation coefficient. [Equation 1] is designated as values between -1 and 1 through a normalization process, and has a value of 1 when the comparison masks of two images coincide with each other.

If the δ _c value for two models is less than the threshold value through Color-NCC, the color information of the two models is judged not to be similar and the corresponding model is removed.

As described above, when the similar model is removed using geometric information (angle information, length information) and color information, the final model thus determined is compared with both the geometric information and the color information, , And the pair of corresponding points for the models becomes an optimal pair of corresponding points. 6 is a diagram illustrating a state in which pairs of optimal correspondence points are determined in overlapping regions of two images through the filtering processes.

On the other hand, if the extracted minutiae are filtered using geometric information (angle information, length information) and color information to determine the optimal pair of corresponding points, a projection transformation parameter by pairs of optimal corresponding points is calculated (S14) Process.

In order to synthesize the reference image and the extension image, an image alignment process is performed. Since the image transformation requires a projection transformation coefficient, the projection transformation coefficient is derived before the image alignment.

7, the pixel A of the pixel reference image and the pixel A of the extended image are assumed to be positions of the orthogonal coordinate system in the image captured by the first camera and the image plane captured by the second camera, as shown in FIG. Point R and point A have a homogeneous property with the 3D reality coordinate W.

Therefore, assuming that the relationship between the point R and the point A between the first camera and the second camera image plane is the relation of the real images H ₁ and H ₂ , the 2D projection transformation of the first camera and the second camera The relation H can be obtained.

That is, in summary, the three-dimensional coordinate point W and the two-dimensional coordinate point R have a tangential property H ₁ , and the point A also has a tangential property H ₂ .

(X, y, s) and A (u, v, s) at the transformation ratio s = 1 and for the orthogonal coordinate system R (x / y / s) and A (u / s, v / s).

Finally, the relationship between the two images is as follows.

R (x, y) = H ₁ W, A (u, v) = H ₁ W

A (u, v) = H ₂ H ₁ ^-1 R (x, y), A = HR

If the above relational expression is generalized, a homogeneous determinant such as the following [Expression 2] can be obtained. In the following, a ₁₁ to a ₃₂ are parameters of projection transformation coefficients.

[Equation 2]

Thus, for the set {((x _i , y _i ), (u _i , v _i )) _i = 1,2, ..., n} of pairs of optimal correspondence points detected in step S12, The linear simultaneous equations expressing the projection relation of the projection optical system are represented by the following [Equation 3].

[Equation 3]

In the present invention, an optimum corresponding point and the navigation, and in a search corresponding point [Formula 3] The linear simultaneous equation of the eight projective transformation of the coefficients (a ₁₁ ~ a ₃₂₎ to know able to sort all pixels of all images to be synthesized do.

The equation of Equation (3) is a perspective transformation that transforms almost all full motion (rotation, translation, uniform scaling, shear, perspective projection) between a first camera's image and a successive second camera ) Conversion.

In step S12, at least four optimal corresponding points among pairs of corresponding points of the image projected on the camera are displayed as follows. That is, four points (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₄ , y ₄ ) of the reference image R, The pair of corresponding points of the points u ₁ , v ₁ , u ₂ , v ₂ , u ₃ , v ₃ and u ₄ , v ₄ are as follows, A = WP. Where P is a parameter matrix of projection transformation coefficients.

     Ra

[Equation 4]

If, If, obtain the at least four optimum corresponding points from among the pairs of the corresponding points of the image projected from the S12 step the camera, W is a 4 × 4 has a square matrix, the eight projective transformation coefficients _{a 11, a 12, a 13} , a 21 _{, a 22, a 23, a} 31, and a _32, the optimum corresponding points of the reference image _{(x 1, y 1),} (x 2, y 2), (x 3, y 3), (x 4, y 4 ), the extended video optimum corresponding point (u _1, and _{v 1), (u 2,} v 2), (u 3, v 3), (u 4, v 4) to LA, the eight projective transformation coefficient is the The inverse matrix of [Expression 4] of [Expression 5] can be obtained.

[Equation 5]

Therefore, in order to obtain the eight projection transformation coefficients A (a ₁₁ to a ₃₂ ), the above equation (5) assumes four pairs of the minimum correspondence points, and if it is generalized, it can be obtained as P = W ^-1 A.

However, if the number N of pairs of corresponding points is larger than 4, since W does not have a square matrix and becomes an anisotropic matrix, W ^-1 can not be obtained, and the projection transformation coefficient matrix P can not be obtained.

Therefore, if the number N of pairs of corresponding points is larger than 4, the solution of the projection transformation coefficient that provides the least square error can be obtained as shown in the following [Equation 6]. The projection transformation coefficient matrix P, which is the solution of [Expression 6], is obtained through a pseudo inverse method which is an anisotropic inverse matrix calculation method which is usually obtained by minimizing a square error.

In the case of applying the pseudoinverse calculation method, A is the coordinate matrix of the extended image, W is the linear transformation matrix, P is the projection transformation coefficient matrix, W ^T is the transpose matrix of W, and W ^-1 Is an inverse matrix of W, the projection transformation coefficient matrix can be obtained by the following [Expression 6].

[Equation 6]

A = WP -> P = (W ^T W) ^-1 W ^T A

As a result, the process of deriving the eight projection transformation coefficients (a ₁₁ to a ₃₂ ) is performed in such a manner that i) if the number of pairs of corresponding points R and A searched is four, the eight projection transformation coefficients a ₁₁ to a ₃₂ ) are obtained through the above-mentioned Expression 5, and ii) if the number of pairs of corresponding points R and A found is larger than 4, the eight projection transformation coefficients of Expression 3 are obtained through Expression 6 above.

The algorithm of the program is summarized as follows.

IF correspondence pairs = 4

Find 8 parameters of P using W ^-1

IF correspondence pairs> 4

Find 8 parameters of P using (W ^T W) ^-1 W ^T A

On the other hand, the perspective transformation of [Expression 3] having the eight projection transformation coefficients (a ₁₁ to a ₃₂ ) obtained by Expression 5 or Expression 6 as described above can be applied to image alignment .

The process of image alignment (S15) will be described.

Image alignment refers to arranging arbitrary pixels of the second camera in accordance with the projection transformation of Equation (3) in order to express the arbitrary reference image sequence of the first camera as a single extended image to be displayed to the user The process of aligning the image pixels of the extended image to be synthesized through the projection transformation obtained by the above-mentioned [Expression 3] is performed.

In the embodiment of the present invention, an inverse mapping is performed as shown in FIG. 8 so as not to generate a hole due to the lack of pixel information in the resultant image.

Referring to FIG. 9 illustrating an image alignment process, an image alignment process will be described. First, a dummy image to be matched with the basic image and the extended image is generated (S91).

The dummy image is a virtual image for mapping the reference image of the first camera and the added image of the second camera. The dummy image is an image having no RGB data value

Since the data of the extended image processes the R, G, and B digital codes, the size of the storage space has a size of 24 bits for each pixel, and the total size of data of the extended image is different in the width and the width.

That is, a single extended image obtained by adding the reference image and the extended image is obtained by subtracting the overlapping portion from the width that can represent the field of view (FOV) of the two images, ), And the maximum value (MAX) of the height of the two images is selected in order to secure a sufficient space for the height of the dummy image.

For reference, when the dummy image width is Dummy Image Width, the reference image width is Ref_width, the reference image width is Ref_height, the synthesized image width is Added_Width, and the synthesized image width is Added_Width, Respectively.

Dummy Image Width = Ref_width + Added_Width - (overlap ratio x Ref_width)

Dummy Image Height = Max (Ref_height, Added_height)

Created_Dummy Image = (Dummy Image Width, Dummy Image Height, 24 bit)

When the dummy image is generated as described above, the reference image is mapped to the dummy image created in step S92.

FIG. 10 shows a reference image mapped to the dummy image, and it is understood that the reference image is mapped on the left side of the dummy image.

Each pixel of the reference image is sequentially mapped to the corresponding pixel of the dummy image on the vertical axis and the horizontal axis. For reference, the program algorithm is as follows.

For (y = 0; y < Reference Image_Height; y ++)

For (x = 0; x < Reference Image_Width; x ++)

{

Dummy Image [y] [x] = Reference Image [y] [x];

}

After a reference image is mapped to a dummy image, an added image is inverse-mapped to a part to be synthesized as shown in FIG. 11 (S93). At this time, when the first camera and the second camera are captured, the extended image (Added Image) can be displayed on the display screen by rotating, translating, scaling, shearing, And the model of

The full motion of these cameras can be represented by eight projection transformation coefficients, and the disparity of the two cameras is f _t1 -> R f ' _{t 1} + T.

For reference, a program algorithm in which a (u, v) coordinate of an extended image is mapped backward to an (x, y) coordinate of a reference image is expressed as follows.

In the following, x 'and y' represent the coordinates of the image acquired by the second camera, and u and v represent the coordinates by which the image of the second camera is converted by the projection transformation coefficient.

For (y '= 0; y' < Dummy Image_Height; y '++)

For (x '= 0; x' <Dummy Image_Width; x '++)

If (x '? 0 && x' <Reference_Width) && (y'≥0 && y '<Reference_Heitht)

continue; / * The part where the reference image data is mapped is skipped * /

else then: u = a ₁₁ x x + a ₁₂ x y + a ₁₃ / a ₃₁ x x + a ₃₂ x y + 1;

v = (a ₂₁ x x '+ a ₂₂ x y' + a ₂₃ ) / (a ₃₁ x x '+ a ₃₂ x y' + 1);

If (u <0? U? Added_width? V <0? V? Added_Height)

continue; / * Added image data skip the part outside the dummy image * /

else then: Dummy Image [y '] [x'] = Added Image [v] [u];

After the image alignment process is performed as described above, the reference image and the extended image are synthesized into a single image, and then an image stabilization process is performed.

The image stabilization process is a process for matching the information of the brightness value (Y) of the color values of the final synthesized image. For example, as shown in FIG. 12, clipping or interpolation of a portion where the information representation of the synthesized image does not coincide with each other makes the representation of the image smoother.

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. Accordingly, the scope of the patent of the present invention is not limited by the above-described embodiments, and it will be obvious that the patent scope covers not only the claims but also the equivalents.

1 is a flowchart illustrating a stereo camera image synthesizing process according to an embodiment of the present invention.

2 is a flowchart illustrating a filtering process according to an embodiment of the present invention.

3 is a diagram illustrating feature points extracted from a reference image and feature points extracted from an extended image according to an embodiment of the present invention.

4 is a flowchart illustrating a process of evaluating similarity of angular information and removing a non-similar model according to an embodiment of the present invention.

FIG. 5 is a flowchart illustrating a process of eliminating a non-similar model by evaluating similarity of length information according to an embodiment of the present invention.

FIG. 6 is a diagram illustrating a state in which pairs of optimal matching points are determined in overlapping regions of two images through filtering processes according to an exemplary embodiment of the present invention.

FIG. 7 is a diagram showing a pixel A and a 3D real-world coordinate W and a homogeneous property of the pixel R and the extended image of the pixel reference image on the image captured by the first camera and the image captured by the second camera.

8 is a diagram illustrating a reverse mapping according to an embodiment of the present invention.

9 is a diagram illustrating a reverse mapping process according to an embodiment of the present invention.

10 is a diagram showing a reference image mapped to a dummy image according to an embodiment of the present invention.

11 is a view illustrating a single image obtained by synthesizing a reference image and an extended image according to an embodiment of the present invention.

FIG. 12 is a view illustrating clipping or interpolation of a portion where the information representation of the synthesized image does not match according to an embodiment of the present invention.

Claims (13)

A feature point detection process for detecting feature points for a reference image captured by the first camera and an extended image captured by the second camera; Determining a number of corresponding point pairs for determining the number of corresponding point pairs in the reference image and the extended image; A filtering step of constructing all models having the determined number of pairs of corresponding points and then evaluating the similarity of the angular information, the length information, and the color information of the models, and then removing the dissimilar models to construct an optimal corresponding point; A projection transform coefficient calculation step of calculating a projection transform coefficient to be used in the projection transform method when the projection transform method using the determined pair of corresponding points is performed; An image alignment process for mapping the extended image back to the reference image using the projection transformation coefficients and performing projection transformation on the extended image to synthesize the expanded image into one image; The image stabilization process of matching the color value of the final synthesized image and the luminance value information / RTI &gt; The filtering process includes: A process of constructing all four-cornered models having four non-collinear points on the reference points and two adjacent points on the reference points extracted from the reference image and the extended image, respectively; A step of calculating an angle of the model with respect to neighboring points adjacent to the reference line of all of the constructed models and obtaining an angle difference between the model of the reference image and the model of the extended image and removing the model having the angular difference larger than the angle threshold value ; A process of calculating a line with neighboring points adjacent to the baseline of all configured models and obtaining a length difference between the model of the reference image and the model of the extended image and removing the model having a length difference larger than the length threshold value ; After performing color-NCC to evaluate the color information similarity by normalizing the remainder obtained by subtracting the brightness average of the two images, if the color similarity information between the models of the two images is equal to or less than the color threshold value, And a stereo camera image synthesis method. 2. The method according to claim 1, wherein the feature point detection step uses corner points between two images as feature points. 2. The method of claim 1, wherein the number of pairs of corresponding points to be searched is determined by determining the number of pairs of corresponding points by considering a projection transformation method between the two cameras. 4. The method of claim 3, further comprising the step of performing a full motion transformation including translation, rotation, uniform scaling, non-uniform scaling, shear, A perspective projection transformation method requiring eight projection transformation coefficients is applied, and the number of corresponding points is determined as four pairs. delete The method according to claim 4, wherein the step of calculating the projection transformation coefficient comprises: If the number of pairs of optimal matching points filtered through the optimal correspondence point construction process is four, then the matrix P of the eight projection transformation coefficients is a ₁₁ , a ₁₂ , a ₁₃ , a ₂₁ , a ₂₂ , a ₂₃ , a ₃₁ this, a _32, the optimum corresponding points of the reference image (x _1, y _1), and _{(x 2, y 2),} (x 3, y 3), (x 4, y 4) , and the extended video optimum corresponding point ( _{u 1, v 1), (} u 2, v 2), (u 3, v 3), (u 4, v to ₄₎ d,

And calculating a projection transformation coefficient using the transform coefficient. The method according to claim 4, wherein the step of calculating the projection transformation coefficient comprises: W is a linear transformation matrix, P is a projection transformation coefficient matrix, and W ^T is a transposition matrix of W. In this case, And W ^-1 is an inverse matrix of W, the projection transformation coefficient matrix P, which is made up of the projection transformation coefficients, A = WP -> P = (W ^T W) ^-1 W ^T A And the stereo camera image synthesizing method. A feature point detection process for detecting feature points for a reference image captured by the first camera and an extended image captured by the second camera; Determining a number of corresponding point pairs for determining the number of corresponding point pairs in the reference image and the extended image; A filtering step of constructing all models having the determined number of pairs of corresponding points and then evaluating the similarity of the angular information, the length information, and the color information of the models, and then removing the dissimilar models to construct an optimal corresponding point; A projection transform coefficient calculation step of calculating a projection transform coefficient to be used in the projection transform method when the projection transform method using the determined pair of corresponding points is performed; An image alignment process for mapping the extended image back to the reference image using the projection transformation coefficients and performing projection transformation on the extended image to synthesize the expanded image into one image; The image stabilization process of matching the color value of the final synthesized image and the luminance value information / RTI &gt; The image alignment process includes: A process of generating a dummy image in which a basic image and an extended image are matched and expanded; Mapping and mapping a pixel of the reference image to the dummy image; A step of backward mapping the extended image to the reference image using the projection transformation coefficient and synthesizing the expanded image as a single image And a stereo camera image synthesis method. The method of claim 8, wherein the dummy image has a size of 24 bits for displaying RGB data for each pixel. The method according to claim 9, wherein the dummy image is a width of a dummy image obtained by subtracting the overlapping portion of the basic image and the extended image, and the maximum value of the vertical width of the basic image and the vertical width of the extended image, Width stereo camera image synthesis method. The method of claim 8, wherein it is by using the projection transformation coefficient to the reverse mapping of the expansion image in the reference image, is _{a 11, a 12, a 13} , a 21, a 22, a 23, a 31, a 32 Is a projection transformation coefficient, x ', y' are coordinates of an image acquired by the second camera, and u and v represent coordinates by which the image of the second camera is converted by the projection transformation coefficient, _{u = (a 11 × x '} + a 12 × y' + a 13) / (a 31 × x '+ a 32 × y' + 1) v = (a ₂₁ x x '+ a ₂₂ x y' + a ₂₃ ) / (a ₃₁ x x '+ a ₃₂ x y' + 1) Wherein the backward mapping is performed by the backward mapping. 12. The method of claim 11, wherein a portion of the dummy image to which the reference image is mapped is skipped and the backward mapping is performed. 13. The method of claim 12, wherein the extended image outside the dummy image size is skipped and the backward mapping is performed.

Images