KR20100076351A - Method for compositing stereo camera images - Google Patents

Abstract

PURPOSE: A method for compositing stereo camera images is provided to find an optimal corresponding point in a short time. CONSTITUTION: Feature points are detected for an added image captured through a second camera and a reference image captured through a first camera(S11). The number of correspondence pairs is determined at the reference and added images(S12). All models having the number of correspondence pairs are configured(S13). A perspective transformation parameter used for a perspective transformation mode is calculated(S14).

Description

Method for compositing stereo camera images

An embodiment of the present invention relates to a stereo camera image synthesis method.

With the development of digital camera devices, various practical image processing techniques have been tried. As an example, a stereo camera may acquire a plurality of images acquired from a plurality of cameras having the same viewpoint or different viewpoints as a single image to overcome the limited field of view of the camera lens. How to get.

One of the most well-known methods for synthesizing stereo camera images is Mosaic. The Mosaic technique is a process of creating a single image with an extended image that will be continuously synthesized with an arbitrary reference image. The mosaic technique finds the optimal pair of correspondences in the overlapping regions of the two images and derives the projection transformation coefficient by these pairs. A single image is obtained by aligning and stabilizing the extended image to be successively aligned with geometric transformation according to the derived transformation coefficient.

However, a problem that is important in synthesizing the stereo camera image is a difficulty in searching for the best matching point.

In searching for an optimal correspondence point between two images, if the reference image F and the extended image F 'to be synthesized are stereo camera images taken in succession in an arbitrary frame, the human eye and the brain are determined from the pixel information of the reference image F. It accurately and quickly finds in which area the pixel information of the superimposed extended image F 'is located.

However, it is a very difficult process for the computer to make this recognition. If the camera module of the continuous extended image F 'is converted into perspective between two camera images by the user's hand movement or the shaking of a small vehicle, the perspective is converted. In this case, how the machine automatically recognizes the similarity of the converted images is the biggest problem and problem of the stereo camera.

Traditionally, Normalized Cross Correlation (NCC) is the best known measure of similarity between two F and F 'images.However, this method of comparing color information is captured in non-contiguous frames. If the image is a sudden situation such as external illuminance, shadow, or performance difference inside the ISP sensor, or if the image or information is different, there is a problem that reliability cannot be obtained in the corresponding point determination process.

Meanwhile, another problem that is important in synthesizing stereo camera images is a problem of increasing computational complexity.

In order to synthesize and align images of two cameras, at least M ^N times a time complexity is required to find an optimal correspondence point by matching N feature points found in other images. This computation time complexity can lead to computation times of tens of seconds in the worst case in high pixel images with many edges.

If you want to control a real-time high-pixel video stereo camera image that scans at high speeds of several tens of frames per second or more, where an extended field of view image is required, such as a car camera, increasing computation time complexity can be fatal.

An embodiment of the present invention is a stereo camera image synthesizing method which enables to find an optimum correspondence point in a short time and can reduce the calculation time during image synthesis.

According to an embodiment of the present invention, a feature point detection process of detecting feature points with respect to a reference image captured by a first camera and an extended image captured by a second camera, and a pair of corresponding points that determine the number of pairs of corresponding points in the reference image and the extended image A process of determining the number and forming all models having the determined number of pairs of matching points, and then evaluating the similarity of the angle information, the length information, and the color information of the models, and then removing the dissimilar models to form an optimal matching point. And a projection conversion coefficient calculating step of calculating a projection conversion coefficient to be used for the projection conversion method when the projection conversion method using the determined corresponding point pair is performed, and reversely expanding the extended image to the reference image using the projection conversion coefficient. Image alignment by compositing into one image by mapping and projecting conversion Jung, includes a final composite image teuleojim color value, brightness value information matches crystallized image stabilization process of the.

The filtering process may include configuring all models having two non-collinear points as reference lines among the feature points extracted from the reference image and the extended image, and applying the reference lines to all the models. Calculating an angle between adjacent adjacent points, obtaining an angle difference between a model of the reference image and a model of the extended image, and removing a model having an angle difference larger than an angle threshold; Calculating a length line between adjacent points adjacent to the reference line, obtaining a length difference between the model of the reference image and the model of the extended image, and removing a model having a length difference greater than the length threshold; After color-NCC is performed to normalize the rest of the image by subtracting the brightness average, the color similarity information between the models of two images is color. Or below threshold, it comprises the step of removing the model.

If the number of pairs of optimal matching points filtered through the optimal matching point construction process is four, the matrix of eight projection transformation coefficients is a ₁₁ , a ₁₂ , a ₁₃ , a ₂₁ , a ₂₂ , a ₂₃ , a ₃₁ , a _32, and the optimal correspondence point of the reference image is (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₄ , y ₄ ), u ₁ , v ₁ ), (u ₂ , v ₂ ), (u ₃ , v ₃ ), (u ₄ , v ₄ ),

Calculate the projection transformation coefficient using.

The image aligning process may include generating a dummy image in which a base image and an extended image are matched and extended, mapping and positioning pixels of the reference image on the dummy image, and using the projection conversion coefficient. And backward mapping the extended image to the reference image to synthesize the image as one image.

According to an embodiment of the present invention, in stereo camera image synthesis, an effect of efficiently detecting the same position of an optimal pair of matching points among a myriad of feature points between an arbitrary image sequence t frames and consecutive t + 1 frames is effective. There is. In addition, it is possible to reduce the amount of calculation of the corresponding point search among the feature points.

Hereinafter, a detailed description of embodiments of the present invention will be described with reference to the accompanying drawings. In the following description of the reference numerals to the components of the drawings it should be noted that the same reference numerals as possible even if displayed on different drawings.

1 is a flowchart illustrating a stereo camera image synthesis process according to an exemplary embodiment of the present invention.

The embodiment of the present invention can reduce the computational complexity by detecting an optimal pair of matching points by repeating the process of reducing feature points that are not necessary to reduce the amount of computation of the matching point among the feature points, and inducing an accurate projection transformation coefficient.

In detail, in order to synthesize a stereo camera image, first, feature points of the images captured by the two cameras are detected (S11). The feature point represents a set of corner points in each image, and for this feature point detection (S11), a conventionally known SUSAN method of detecting corner points between two images may be used.

After detecting feature points such as a set of corner points in the two images as described above, an optimum correspondence point between the feature points between the two images is searched and configured.

Referring to the processes of configuring the optimum correspondence point, first, the number of correspondence point pairs to be searched is determined (S12).

TABLE 1

projection parameter least pairs transformations displacement 2 One translation similarity 4 2 translation, rotation, uniform scale affine 6 3 translation, rotation, non-uniform scale perspective 8 4 full motion

[Table 1] shows an optimal pair of correspondence points according to the projection transformation between the projected 2D images of two cameras. 'parameter' represents the projection transformation coefficient, and 'least pairs' represents the number of best (minimum) pairs of matching points.

If the movement of the camera preserves parallel movement, the projection transformation of the two cameras is 'displacement', and if the minimum corresponding pair of points is found, the displacement between the two cameras is enough to calculate the translation distance, Only a pair of optimal matching points are required.

If we assume a transformation of translation, rotation, and uniform scaling between two cameras, the projection transformation of the two cameras is 'similarity' and requires at least two pairs of correspondences.

If we assume translations of translation, rotation, non-uniform scaling, and shear between two cameras, the projection transformation of the two cameras is 'affine' and at least three pairs Requires correspondence.

If you want to convert to include the full motion of both cameras, the projection transformation of the two cameras is 'perspective' and requires at least four pairs of correspondence points.

Therefore, in determining the number of correspondence points, it is determined whether to use at least one pair, two pairs, three pairs or four pairs according to the shooting environment of the two cameras.

An embodiment of the present invention will be described by using an example of a 'perspective' projection transformation method that may include full motion for a conversion between two cameras in an arbitrary frame, and thus, 'perspective' When the projection transformation method is applied, at least four pairs of matching points are searched to find the projection transformation coefficient of 8 parameters including the total displacement of the camera.

Hereinafter, an embodiment of the present invention will be described using four pairs of correspondence points in order to apply a 'perspective' projection conversion method. However, if a different projection conversion method is used, the number of corresponding pairs of corresponding points is determined and accordingly The image synthesis process may be implemented in the same way.

On the other hand, if at least four pairs of corresponding points are determined according to the 'perspective' projection conversion scheme as described above, after configuring the quadrangular models having the determined feature points of the optimal number of corresponding points, the geometric information (angle information, length information) of these models is formed. ) Similarity comparison and color information similarity comparison to remove the remaining similar models have a filtering (S13) process.

Referring to FIG. 2, the filtering is performed as follows.

Once the number of pairs of corresponding points is determined according to the 'perspective' projection equation, all of the possible quadrangles that can be made into four non-collinear points among the feature points extracted from the reference and extended images, respectively Configure the models (S21).

That is, after constructing and determining (S21) all the models of the quadrilateral with two non-collinear points between the two points which are non-collinear among the extracted feature points, the baselines of all the constructed models An angle is obtained, and a difference between angle information obtained from two comparison images is obtained to remove dissimilar models larger than an angle threshold (S22).

Then, after obtaining line information about the baseline for the remaining models, the difference between the pieces of length information is obtained and dissimilar models larger than the length threshold are removed (S23).

After removing the models larger than the angle threshold and the length threshold as described above, color-NCC is performed on the local regions of the two images to evaluate the color information similarity. After performing the color-NCC, a color threshold value is determined to remove the remaining models (S24), thereby determining a set of non-removed feature points as the final corresponding point pairs (S25).

The filtering processes will be described in detail for each step. First, since the number of pairs of corresponding points is determined to be four or more pairs, a quadrangular model configuration S21 must be performed, which is applied to the feature points extracted from the reference image and the extended image, respectively. It consists of constructing all models of a quadrilateral with two non-collinear points as baseline.

Assuming that the feature points extracted from the reference image and the feature points extracted from the extended image are the same as those of FIGS. 3 (a) and 3 (b), respectively, four non-collinear features among the extracted feature points are not collinear. Construct all possible quadratic models that can be made of four points.

Then, the angle information similarity of the constructed models is evaluated to remove dissimilar models. That is, an angle with respect to the reference line of all the configured models is obtained, and the difference between the angle information obtained from the two comparison images is obtained to remove the model larger than the angle threshold.

A process of removing the model larger than the angle threshold is shown in FIG. 4. First, an angle table is generated (S41).

The angle table has angle information of each model with respect to the reference point (P ₁ , P ' ₁ ). For example, in the reference image of FIG. 3A, angles with two adjacent points P ₂ and P ₃ based on the reference lines P ₁ -P ₄ are calculated and stored in the angle table. Similarly, in the extended image of FIG. 3 (b) to be synthesized, the angles between two adjacent points P ' ₂ and P' ₃ based on the reference lines P ' _1- P' ₄ are extracted. Calculate and store in angle table

An example of the angle table for the models of the reference image and the extended image is as follows. In the following, R_Model [N] refers to the Nth model of the reference image, and A_Model [N] refers to the Nth model of the added image.

If P ₁ , P ₂ , P ₃ , P ₄ = Non-collinear

R_Model [1] = {reference line (P ₁ , P ₄ ), adjacent point (P ₂ , P ₃ ), angle information (P ₃ , P ₁ , P ₂ ), angle information (P ₃ , P ₄ , P ₂ ) }

Also, If P ₁ , P ₂ , P ₃ , P ₅ = Non-collinear

R_Model [2] = {reference line (P ₁ , P ₅ ), adjacent point (P ₂ , P ₃ ), angle information (P ₃ , P ₁ , P ₂ ), angle information (P ₃ , P ₅ , P ₂ ) }

If P ' ₁ , P' ₂ , P ' ₃ , P' ₄ = Non-collinear

A_Model [1] = {baseline (P ' ₁ , P' ₄ ), adjacent point (P ' ₂ , P' ₃ ), angle information (P ' ₃ , P' ₁ , P ' ₂ ), angle information (P' ₃ , P ' ₄ , P' ₂ )}

If P ' ₁ , P' ₂ , P ' ₃ , P' ₅ = Non-collinear

A_Model [2] = {baseline (P ' ₁ , P' ₅ ), adjacent point (P ' ₂ , P' ₃ ), angle information (P ' ₃ , P' ₁ , P ' ₂ ), angle information (P' ₃ , P ' ₅ , P' ₂ )}

As described above, the angle information is calculated for the reference image and the extended image (S42), and the angle difference values of the two images are compared, and if the angle difference value of the two images is larger than the angle threshold, it is determined that the two models are not the same model. Remove the model (S45).

In detail, the angle information is calculated from the reference image and the extended image to generate an angle table, and then the angle difference is calculated for each model (S42).

Thereafter, the angle difference between the first model of the reference image and the angle difference between the first model of the extended image is called (S43), and it is determined whether the angle difference is greater than a preset angle threshold (S44).

If the angle difference is greater than the angle threshold, it is determined that the two models are not the same model and remove the corresponding model (S45). Thereafter, the angle difference of the next model is called (S43). Similarly, if the angle difference is larger than the angle threshold (S44), and if large, the model is removed (S45).

The removal processes are expressed as algorithmic expressions as follows.

δ _angle = │R_Model [Angle]-A_Model [Angle] │

If δ _angle > THRESHOLD _angle

R_Model ≠ A_Model

The above steps are repeated until the last models are reached (S46), thereby eliminating models that are not identical.

If the angle difference value is larger than the angle threshold as described above, it is determined that the two models are not the same model, and after removing the corresponding model (S22), similarly, by evaluating the similarity of the length information of the constructed models, the lengths are not similar. The model is removed (S23). That is, if the length difference value is greater than the length threshold by comparing the length difference value, it is determined that the two models are not the same model and has a process of removing the corresponding model (S23).

The process of removing the model larger than the length threshold is shown in FIG. 5. First, a length table is generated (S51).

The length table has line information of each model with respect to the reference point P ₁ , P ′ ₁ . For example, in the reference image of FIG. 3A, the lengths of two adjacent points P ₂ and P ₃ based on the reference lines P _{1 to} P ₄ are calculated and stored in the length table. Similarly, in the extended image of FIG. 3 (b) which is an image to be synthesized, the lengths of the extracted feature points with two adjacent points P ' ₂ and P' ₃ based on the reference lines P ' _1- P' _{4 are} determined. Calculate each and store in length table

An example of the length table for the models of the base view image and the extended image is as follows. In the following, R_Model [N] refers to the Nth model of the reference image, and A_Model [N] refers to the Nth model of the added image.

If P ₁ , P ₂ , P ₃ , P ₄ = Non-collinear

R_Model [1] = {reference line (P ₁ , P ₄ ), adjacent point (P ₂ , P ₃ ), length information (P ₁ , P ₂ ), length information (P ₁ , P ₃ ), length information (P ₄ , P ₂ ), length information (P ₄ , P ₃ )}

If P ₁ , P ₂ , P ₃ , P ₅ = Non-collinear

R_Model [2] = {reference line (P ₁ , P ₅ ), adjacent point (P ₂ , P ₃ ), length information (P ₁ , P ₂ ), length information (P ₁ , P ₃ ), length information (P ₅ , P ₂ ), length information (P ₅ , P ₃ )}

If P ' ₁ , P' ₂ , P ' ₃ , P' ₄ = Non-collinear

A_Model [1] = {reference line (P ' ₁ , P' ₄ ), adjacent point (P ' ₂ , P' ₃ ), length information (P ' ₁ , P' ₂ ), length information (P ' ₁ , P' ₃ ), length information (P ' ₄ , P' ₂ ), length information (P ' ₄ , P' ₃ )}

If P ' ₁ , P' ₂ , P ' ₃ , P' ₅ = Non-collinear

A_Model [2] = {baseline (P ' ₁ , P' ₅ ), adjacent point (P ' ₂ , P' ₃ ), length information (P ' ₁ , P' ₂ ), length information (P ' ₁ , P' ₃ ), length information (P ' ₅ , P' ₂ ), length information (P ' ₅ , P' ₃ )}

As described above, the length information is calculated for the base image and the extended image, and the difference in the length difference between the two images is determined. Remove

In detail, length information is calculated for the reference image and the extended image to generate a length table, and then a length difference for each model is calculated (S52).

Thereafter, the length difference of the first model of the reference image and the length difference of the first model of the extended image are called (S53), and it is determined whether the length difference is greater than a preset length threshold (S54).

If the difference in length is greater than the length threshold, it is determined that the two models are not the same model and the corresponding model is removed (S55). Thereafter, the length difference of the next model is called (S53). Similarly, if the length difference is greater than the length threshold (S54), if the size is large, the model is removed (S55).

The removal processes are expressed as algorithmic expressions as follows.

δ _line = │R_Model [line]-A_Model [line] │

If δ _line > THRESHOLD _line

R_Model ≠ A_Model

The above steps are repeated until the last models are reached (S56), thereby eliminating models that are not identical.

By comparing the models of the two images with geometric information (angle information, length information) in the extracted feature points as described above, dissimilar models are removed from the memory (S22 and S23). This method can automatically filter out unnecessary features among a myriad of feature points and significantly reduce the amount of computation.

As a next step, with respect to the models filtered by the similarity condition of the geometric information (angle information, length information), the process of removing the dissimilar models by evaluating the similarity of the color information (S24) is performed. Have

The process of removing models whose color information is not similar to each other includes color similarity of two images with respect to each pixel point around two comparison images with respect to models filtered by the similarity condition of geometric information (angle information and length information). Color-normalized cross correlation (Color-NCC) is used as an evaluation measure. Color-NCC is a technique to evaluate the similarity by normalizing the remaining parts of the two sub-images minus the average brightness.

For reference, the color similarity evaluation technique of Color-NCC will be described together with Equation 1 below.

[Equation 1]

In Equation 1, f _t0 (x, y) represents a model of a reference image, and f _t1 (u, v) represents an RGB vector component of pixel coordinates of models of the extended image to be synthesized. m _t0 , m _t1 represent the mean for the color values in the f _t0 , f _t1 mask, and “∥∥” represents the norm vector for the RGB component.

를 만들고 확장 영상의 지정된 검색 영역 (i,j) 안에서 마스크

들과 상관계수를 조사한다. 상기 [수식 1]은 정규화 과정을 통해 -1 ∼ 1 사이 값들로 지정되며, 두 영상의 비교 마스크가 일치한다면 1의 값을 갖는다.Mask of size s × s based on the coordinates of the corresponding point (x, y)

And mask within the specified search area (i, j) of the extended image

Examine the correlation coefficients with each other. Equation 1 is designated as values between −1 and 1 through normalization, and has a value of 1 if the comparison masks of the two images match.

If the value of δ _c for the two models is smaller than the threshold value through Color-NCC, it is determined that the color information of the two models is not similar and the corresponding model is removed.

As described above, if removal of dissimilar models is made using geometric information (angle information, length information) and color information, the final model thus determined is the final model filtered by comparing both geometric information and color information. Then, the pair of correspondence points for the models becomes the optimal pair of correspondence points. For reference, FIG. 6 is a diagram illustrating a state where an optimal pair of matching points is determined in an overlapping region of two images through the filtering process.

On the other hand, the optimal feature pairs are finally determined by filtering the extracted feature points using geometric information (angle information, length information) and color information, and calculating (S14) a projection transformation coefficient based on the optimum match point pairs. Have a process.

When the reference image and the extended image are synthesized, the image alignment process is performed. Since the projection transformation coefficient is required for the image alignment, the projection transformation coefficient is derived before the image alignment.

Referring to the process of deriving the projection transformation coefficient, as shown in FIG. 7, the pixel R of the pixel reference image and the pixel A of the extended image are regarded as positions of the Cartesian coordinate system in the image captured by the first camera and the image plane captured by the second camera. If so, points R and A have a homogeneous nature with 3D real coordinates W.

Therefore, if the point R and the point A between the first camera and the second camera image plane assume that the relationship between the real images has a homogeneous matrix H ₁ , H ₂ , the 2D projection transformation of the first camera and the second camera The relationship H can be found.

In other words, the three-dimensional coordinate point W and the two-dimensional coordinate point R have a homogeneous property H ₁ , and the point A also has a homogeneous property H ₂ .

Therefore, for point R, A, the expansion to homogeneous coordinate system can be expressed as R (x, y, s), A (u, v, s) at conversion ratio s = 1, and in Cartesian coordinate system R (x / s, y / s) and A (u / s, v / s).

As a result, the relationship between the two images is as follows.

R (x, y) = H ₁ W, A (u, v) = H ₁ W

A (u, v) = H ₂ H ₁ ^-1 R (x, y), A = HR

By generalizing the above relation, a homogeneous determinant such as the following [Formula 2] can be obtained. In the following, a ₁₁ to a ₃₂ are parameters of the projection transformation coefficient.

[Equation 2]

Therefore, two images of the set {((x _i , y _i ), (u _i , v _i )) │i = 1,2, ...., n} consisting of the optimal pair of matching points N detected in step S12 A linear simultaneous equation representing the projection relationship of is expressed by Equation 3 below.

[Equation 3]

In the present invention, an optimal correspondence point is searched, and if the eight correspondence transformation coefficients (a ₁₁ to a ₃₂ ), which are linear simultaneous equations of [Equation 3], are found, it is possible to align all pixels of all images to be synthesized. do.

Equation (3) is a perspective transformation, and almost all full motion (rotation, translation, uniform scaling, non-uniform scaling, shear, perspective projection) between the image of the first camera and the second camera that is continuous. ) Contains the transformation.

In operation S12, at least four optimum correspondence points among the correspondence point pairs of the image projected on the camera are displayed as follows. That is, four points of the reference image R (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₄ , y ₄ ) and ₄ of the extended image A The pairs of corresponding points of (u ₁ , v ₁ ), (u ₂ , v ₂ ), (u ₃ , v ₃ ), (u ₄ , v ₄ ) are as follows. It has the form A = WP. Where P is a parametric matrix of projection transform coefficients.

     R A

[Equation 4]

If, at step S12, at least four optimal matching points are obtained from the pairs of matching points of the image projected on the camera, W has a 4 × 4 square matrix and eight projection transformation coefficients are a ₁₁ , a ₁₂ , a ₁₃ , and a _21. , a ₂₂ , a ₂₃ , a ₃₁ , a _32, and the optimal correspondence points of the reference image are (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₄ , y ₄ ), And when the extended image optimal correspondence points are (u ₁ , v ₁ ), (u ₂ , v ₂ ), (u ₃ , v ₃ ), (u ₄ , v ₄ ), the eight projection transformation coefficients are The inverse of Equation 4 can be obtained as shown in Equation 5 below.

[Equation 5]

Accordingly, Equation 5 is assuming that at least four pairs of corresponding points are obtained in order to obtain eight projection transformation coefficients A; a ₁₁ to a _32. In general, P = W ⁻¹ A can be obtained.

However, if the number N of the pairs of corresponding points is larger than 4, since W becomes an anisotropic matrix without a square matrix, W ⁻¹ cannot be obtained and the projection transformation coefficient matrix P cannot be obtained.

Therefore, if the number N of pairs of matching points is greater than 4, the solution of the projection transformation coefficient that provides the least square error may be obtained as in Equation 6 below. The projection transformation coefficient matrix P, which is the solution of [Equation 6], is usually obtained through the Pseudoinverse method, which is an anisotropic inverse matrix calculation method obtained by minimizing square error.

Therefore, when the number of pairs of matching points N is greater than 4, when Pseudoinverse calculation is applied, A is the coordinate matrix of the extended image, W is the linear transformation matrix, P is the projection transformation coefficient matrix, W ^T is the transpose matrix of W, and W ^-1. When is an inverse of W, the projection transformation coefficient matrix can be obtained by the following [Equation 6].

[Equation 6]

A = WP-> P = (W ^T W) ^-1 W ^T A

As a result, the process of deriving the eight projection transformation coefficients a ₁₁ to a ₃₂ is performed. I) If the number of the corresponding pairs of matching points R and A is four, the eight projection transformation coefficients (a ₁₁ to a) ₃₂ ) is obtained through Equation 5, and ii) if the number of pairs of matching points R and A found is greater than 4, eight projection transformation coefficients of Equation 3 are obtained through Equation 6.

The algorithm is summarized as follows.

IF correspondence pairs = 4

Find 8 parameters of P using W ^-1 A

IF correspondence pairs> 4

Find 8 parameters of P using (W ^T W) ^-1 W ^T A

On the other hand, the perspective transformation of [Equation 3] having eight projection transformation coefficients (a ₁₁ to a ₃₂ ) obtained by Equation 5 or Equation 6 as described above provides an image alignment in image composition. Need.

The image aligning process (S15) will be described.

Image alignment refers to aligning each pixel of the second camera with the projection transformation of [Equation 3] to express one extended image to be shown to the user for any reference image sequence of the first camera. In other words, a process of aligning the image pixels of the extended image to be synthesized through the projection transformation obtained in Equation 3 is performed.

In the embodiment of the present invention, in order to align the image, inverse mapping is performed as shown in FIG. 8 so as not to generate a hole due to lack of pixel information in the resultant image.

Referring to FIG. 9, which illustrates an image alignment process, first, a process of generating a dummy image in which a base image and an extended image are matched and extended is generated (S91).

The dummy image is one virtual image for mapping a reference image of the first camera and an added image of the second camera. The dummy image is an image having no RGB data value.

Since the data of the extended video processes R, G, and B digital codes, the size of the storage space has a size of 24 bits for each pixel, and the total size of the data of the extended video differs in width and height.

That is, a single extended image, in which a base image and an extended image are added, has a final field of view (FOV) of an image obtained by subtracting an overlapping portion from a width that can represent a field of view (FOV) of the two images. The height of the dummy image is selected as the maximum value MAX of the height of the two images in order to secure sufficient space.

For reference, when the dummy image width is Dummy Image Width, the reference image width is Ref_width, the reference image height is Ref_height, the composite image width is Added_Width, and the composite image height is Added_Width. Same as

Dummy Image Width = Ref_width + Added_Width-(Nested Ratio × Ref_width)

Dummy Image Height = Max (Ref_height, Added_height)

Created_Dummy Image = (Dummy Image Width, Dummy Image Height, 24bit)

When a dummy image is generated as described above, a reference image is mapped to a virtual dummy image (S92).

FIG. 10 shows a state in which the reference image is mapped to the dummy image, and it can be seen that the reference image is mapped and placed on the left side of the dummy image.

Each pixel of the reference image is sequentially mapped to the corresponding pixel of the dummy image in the vertical axis and the horizontal axis. For reference, it is expressed as a program algorithm as follows.

For (y = 0; y <Reference Image_Height; y ++)

For (x = 0; x <Reference Image_Width; x ++)

{

Dummy Image [y] [x] = Reference Image [y] [x];

}

After mapping a reference image to a dummy image, an added image is inversely mapped to the portion to be synthesized as shown in FIG. 11 (S93). At this time, the added image includes two images by rotation, translation, scaling, cutting, and perspective of the camera when the first camera and the second camera are captured. The model of is transformed to death.

The full motion of the camera (full motion) may be represented by the eight projective transformation coefficient, displacement (disparity) of two cameras f _t1 -> Rf 'it is T + _t1.

For reference, a program algorithm in which (u, v) coordinates of an added image are backward mapped to (x, y) coordinates of a reference image may be expressed as follows.

In the following description, x 'and y' represent coordinates of an image acquired by the second camera, and u and v represent coordinates to convert the image of the second camera by the projection conversion coefficient.

For (y '= 0; y' <Dummy Image_Height; y '++)

For (x '= 0; x' <Dummy Image_Width; x '++)

If (x'≥0 && x '<Reference_Width) && (y'≥0 && y' <Reference_Heitht)

continue; / * Skip the part where the reference image data is mapped * /

else then: u = (a ₁₁ x x '+ a ₁₂ x y' + a ₁₃ ) / (a ₃₁ x x '+ a ₃₂ x y' + 1);

v = (a ₂₁ x x '+ a ₂₂ x y' + a ₂₃ ) / (a ₃₁ x x '+ a ₃₂ x y' + 1);

If (u <0∥u ≥ Added_width∥v <0∥v≥Added_Height)

continue; / * Skip the part where the added image data is outside the dummy image * /

else then: Dummy Image [y '] [x'] = Added Image [v] [u];

After combining the base image and the extended image through the image alignment process as described above to form a single image, it has an image stabilization process.

The image stabilization process is a process for matching color information of the final synthesized image and information of luminance value (Y). For example, as shown in FIG. 12, the representation of the synthesized image does not coincide with the clipping or interpolation to smooth the expression of the image.

In the above description of the present invention, specific embodiments have been described, but various modifications may be made without departing from the scope of the present invention. Therefore, the scope of the present invention is not to be determined by the embodiments described above, but will be apparent in the claims as well as equivalent scope.

1 is a flowchart illustrating a stereo camera image synthesis process according to an exemplary embodiment of the present invention.

2 is a flowchart illustrating a filtering process according to an exemplary embodiment of the present invention.

3 is a diagram illustrating feature points extracted from a reference image and feature points extracted from an extended image according to an exemplary embodiment of the present invention.

4 is a flowchart illustrating a process of removing a dissimilar model by evaluating similarity of angle information according to an embodiment of the present invention.

5 is a flowchart illustrating a process of removing a dissimilar model by evaluating similarity of length information according to an embodiment of the present invention.

FIG. 6 is a diagram illustrating an optimal pair of matching points determined in an overlapping region of two images through filtering processes according to an exemplary embodiment of the present invention.

FIG. 7 is a diagram illustrating a homogeneous property of the pixel R of the pixel reference image, the pixel A of the extended reference image, and the 3D real coordinates W on the image captured by the first camera and the image plane captured by the second camera.

8 is a diagram illustrating a reverse mapping state according to an embodiment of the present invention.

9 is a diagram illustrating a reverse mapping process according to an embodiment of the present invention.

FIG. 10 is a diagram illustrating a state where a reference image is mapped to a dummy image according to an embodiment of the present invention.

FIG. 11 is a diagram illustrating a single image by synthesizing a base image and an extended image according to an exemplary embodiment of the present invention.

FIG. 12 is a diagram illustrating a state where clipping or interpolation of a portion where information representations of a synthesized image do not match according to an embodiment of the present invention is performed.

Claims (13)

A feature point detection process of detecting feature points with respect to the reference image captured by the first camera and the extended image captured by the second camera; A corresponding point pair number determination step of determining the number of corresponding point pairs in the reference image and the extended image; A filtering process of constructing all models having the determined number of pairs of matching points, evaluating similarity of angle information, length information, and color information of the models and then removing dissimilar models to form an optimal matching point; A projection transformation coefficient calculating step of calculating a projection transformation coefficient to be used for the projection transformation scheme when the projection transformation scheme using the determined pair of correspondence points is made; An image alignment process of synthesizing the extended image into the one image by performing reverse conversion by mapping the extended image to the reference image using the projection transformation coefficient; Image stabilization process to match color value distortion and luminance value information of final synthesized image Stereo camera image synthesis method comprising a. 2. The method of claim 1, wherein the detecting of the feature point comprises feature points of corner points between two images. The method of claim 1, wherein the determining of the number of pairs of corresponding points to search comprises determining a number of pairs of corresponding points in consideration of a projection conversion method between two cameras. 4. The method of claim 3, further comprising transforming a full motion including translation, rotation, uniform scaling, non-uniform scaling, and shear. A 'perspective' projection conversion method requiring eight projection conversion coefficients is applied, wherein the number of corresponding points is determined as four pairs. The method of claim 1, wherein the filtering process, Constructing all models having two non-collinear points as reference lines among the feature points extracted from the reference image and the extended image, respectively; Calculating an angle between adjacent points adjacent to the reference line of all the constructed models, obtaining an angle difference between the model of the reference image and the model of the extended image, and removing a model having an angle difference larger than an angle threshold ; A process of removing a model having a length difference greater than a length threshold by calculating a length line between adjacent points adjacent to the reference line of all the constructed models, obtaining a length difference between the model of the reference image and the model of the extended image. ; After the color-NCC is performed to normalize the remaining parts of the two images by subtracting the brightness average, the process of removing the model when the color similarity information between the models of the two images is less than or equal to the color threshold value. Stereo camera image synthesis method comprising a. The process of claim 4, wherein the calculating of the projection conversion coefficient is performed. If the number of pairs of optimal matching points filtered through the optimal matching point construction process is four, the matrix of eight projection transformation coefficients is a ₁₁ , a ₁₂ , a ₁₃ , a ₂₁ , a ₂₂ , a ₂₃ , a ₃₁ , a _32, and the optimal correspondence point of the reference image is (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ), (x ₄ , y ₄ ), u ₁ , v ₁ ), (u ₂ , v ₂ ), (u ₃ , v ₃ ), (u ₄ , v ₄ ),

Stereo camera image synthesis method for calculating the projection conversion coefficient using the. The process of claim 4, wherein the calculating of the projection conversion coefficient is performed. If the number of pairs of optimal matching points filtered through the optimal matching point construction process exceeds 4, A is the coordinate matrix of the extended image, W is the linear transformation matrix, P is the projection transformation coefficient matrix, and W ^T is transpose of W. When the matrix, W ⁻¹ is an inverse of W, the projection transformation coefficient matrix P composed of the projection transformation coefficients is A = WP-> P = (W ^T W) ^-1 W ^T A Stereo camera image synthesis method calculated by. The method of claim 1, wherein the image aligning process, Generating a dummy image in which the base image and the extended image are matched and extended; Mapping and positioning pixels of the reference image on the dummy image; A process of backward mapping the extended image to the reference image by using the projection transformation coefficient and synthesizing it as one image Stereo camera image synthesis method comprising a. The method of claim 8, wherein the dummy image has a 24-bit size for displaying RGB data for each pixel. 10. The dummy image of claim 9, wherein the dummy image has a value obtained by subtracting an overlapping portion of the base image and the extended image from the width of the dummy image, and a maximum value of the vertical width of the base image and the vertical width of the extended image is the vertical value of the dummy image. Stereo camera video synthesis method. The method of claim 8, wherein the backward mapping of the extended image to the reference image using the projection transformation coefficient is a ₁₁ , a ₁₂ , a ₁₃ , a ₂₁ , a ₂₂ , a ₂₃ , a ₃₁ , and a ₃₂ . When x 'and y' are coordinates of an image acquired from the second camera, and u and v represent coordinates to convert the image of the second camera by the projection conversion coefficient. u = (a ₁₁ × x '+ a ₁₂ × y' + a ₁₃ ) / (a ₃₁ × x '+ a ₃₂ × y' + 1) v = (a ₂₁ × x '+ a ₂₂ × y' + a ₂₃ ) / (a ₃₁ × x '+ a ₃₂ × y' + 1) Stereo camera image compositing method that is mapped backward. The stereo camera image synthesis method of claim 11, wherein a portion in which the reference image is mapped to the dummy image is skipped and reverse mapping is performed. The method of claim 12, wherein the extended image outside the dummy image size is skipped and reverse mapping is performed.

Images