KR101836238B1 - A Novel Seam Finding Method Using Downscaling and Cost for Image Stitching - Google Patents

Abstract

The present invention relates to a method for estimating an image connection line using a cost function and downscaling. The method for estimating an image connection line according to the present invention includes the steps of: detecting an approximate connection line of images obtained by down-sampling two stitching target images; setting a limited region around the approximate connection line which is interpolated in an overlapped region of images obtained by up-sampling the two images which are down-sampled; and estimating a final connection line of the two stitching target images by using costs based on pixel data corresponding to the images in the limited region and connection line direction information based on the costs. Accordingly, the present invention can reduce a calculation load and an operation time.

Description

[0001] The present invention relates to a method for estimating an image link line using a downscaling function and a cost function,

In particular, the present invention relates to a method of estimating an image link line using downscaling and a cost function in an image matching process for generating a panoramic image or the like.

Panoramic images are applied in various fields such as CCTVB and action camera to provide realistic images with a wider view. In general, cameras equipped with a wide-angle lens are used for capturing panoramic images, and images captured by the same or different cameras are stitched through software or the like to generate panoramic images.

For image stitching, an image registration that detects and matches feature points between images to calculate a focal length, a homography matrix, and the like, and an image represented by a reference coordinate system And performs image warping. After image warping, image blending is performed to remove the stitching traces by eliminating exposure differences in the images as much as possible and removing connection lines and ghost effects.

The panoramic image generated through the image stitching has a large difference in image quality according to various parameter settings, and various efforts are being made to provide a perfect panoramic image. That is, new methods for image registration, image warping, image blending and the like are emerging, and efforts are being made to improve connection line detection and computation speed / accuracy in image blending.

However, in reality, image stitching results are often unsatisfactory in a panoramic image. Therefore, a new method for detecting an accurate connecting line of images to be stitched so as not to leave a stitching trace is needed.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art, and it is an object of the present invention to provide a method and apparatus for downsampling images including overlapping regions, In order to find the actual connecting line from the approximate connecting line, it is necessary to upsample the images again to detect an approximate connecting line in the restored images by detecting the approximate connecting line from the reduced size images, _{( I, j )} representing the gradient of pixel data in an area limited to predetermined right and left pixels (for example, K) in the vertical direction around the connecting line interpolated by the interpolation method, an image capable of detecting an accurate physical connection line based on the calculation of the accumulated cost matrix And a method of estimating a connection line.

According to an aspect of the present invention, there is provided a method for estimating an image line according to one aspect of the present invention, including the steps of: detecting an approximate line of an image obtained by downsampling two images to be stitched; Setting a limited area around the approximate connection line of the overlapping area in the images upsampled from the downsampled two images; And estimating a final connecting line of the two images to be stitched, using the cost based on the corresponding pixel data of the images and the cost-based connecting line direction information in the limited area.

In the estimating step, the cost e _i is calculated by the following equations,

,

,

,

,

,

는 대응되는 픽셀 데이터들,

,

는 대응되는 각 픽셀 데이터의 변화도(gradient)값이다.here,

,

Corresponding pixel data,

,

Is a gradient value of each corresponding pixel data.

In the estimating step, the connecting line direction information is a direction from the position selected to calculate the accumulation cost of each row to the position of each pixel, from three pixels adjacent to the upper row, Direction values.

The line direction information further includes a direction value of the own position at the corresponding pixel position having the minimum cost of the top row.

Calculating the accumulated cost values of each row by using the cost values of the limited area and calculating a line connecting from the minimum accumulated cost value position of the lowest row to the highest row along the connection line direction information inversely It can be estimated as the final connecting line.

The change in the cost value also possible to use a cost function E _{(i, j)} to calculate the (gradient), and calculating the accumulated cost values of each of the rows, the cost function E _{(i, j)} is formula

Expressed and, utilizes cost value at each pixel position (i, j) cost values e _{(i, j),} and close to the adjacent three pixels of the upper line in the left and right of the pixel as a parameter to.

The accumulation cost values are sequentially calculated from the corresponding pixel position having the minimum cost in the uppermost row of the limited region and the accumulation cost values are calculated from the left end to the right direction without reflecting the accumulation cost value of the position yet to be calculated from the next row The accumulation cost values having a smaller value among the accumulation cost values and the accumulation cost values calculated from the right end to the left side are selected.

According to another aspect of the present invention, there is provided a recording medium embodied by computer readable code, the method comprising the steps of: detecting an approximate connecting line of images down-sampling two images to be stitched; Setting a limited area around the approximated connection line of the overlap area in the images upsampled from the downsampled two images; And estimating a final connection line of the two images to be stitched, using the cost based on the corresponding pixel data of the images in the limited area and the connection line direction information based on the cost.

According to the image scaling method using the downscaling and the cost function according to the present invention, in order to find a connecting line for matching images in the overlapping area of two images to be stitched, the image including the overlapping area is down- The calculation load and the calculation time can be reduced.

According to the image scaling method using the downscaling function and the cost function according to the present invention, in order to find an actual connection line from the above approximate connection line, _{( I, j )} representing a gradient of pixel data in an area limited to predetermined left and right pixels (e.g., K) in the vertical direction around a connecting line interpolated by interpolation method accumulated cost matrices are calculated and then lines connected to the highest row along the matrix direction information are reversely detected at the minimum accumulation cost value position of the lowermost row by connecting lines so as to stitch the images to eliminate mismatching, So that a sophisticated connection can be made.

FIG. 1 is a diagram for explaining an approximate connection line estimation process among image connection line estimation methods according to an embodiment of the present invention.
2 is a flowchart illustrating an actual connection line estimation process using cost and direction information among image connection line estimation methods according to an exemplary embodiment of the present invention.
3 is a diagram for explaining an example of the accumulation cost calculation according to an embodiment of the present invention.
4 is a view for explaining connection line direction information according to an embodiment of the present invention.
FIG. 5 is an example of two images to be stitched to be used in connection line estimation comparison between the present invention and an existing method.
FIG. 6 is a result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (a).
FIG. 7 is a result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (b).
FIG. 8 shows the result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (c).
FIG. 9 is a result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (d).
10 is a graph comparing the processing time performance required for image stitching of the present invention and the conventional method.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are denoted by the same reference symbols as possible. In addition, detailed descriptions of known functions and / or configurations are omitted. The following description will focus on the parts necessary for understanding the operation according to various embodiments, and a description of elements that may obscure the gist of the description will be omitted. Also, some of the elements of the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not entirely reflect the actual size, and therefore the contents described herein are not limited by the relative sizes or spacings of the components drawn in the respective drawings.

First, an image connection line estimation method using downscaling and a cost function according to an exemplary embodiment of the present invention can be implemented in a predetermined image processing apparatus. The image processing apparatus can be implemented by hardware such as a semiconductor processor, software such as an application program, Or a combination thereof. The image processing device may be a computing system and may include at least one processor, a memory, a user interface input device, a user interface output device, a storage, and a network interface. A processor may be a central processing unit (CPU) or a semiconductor device that performs processing on instructions stored in memory or storage.

FIG. 1 is a diagram for explaining an approximate connection line estimation process among image connection line estimation methods according to an embodiment of the present invention.

Referring to FIG. 1, in the image connecting line estimation method of the present invention for matching two images to be stitched, an image processing apparatus first detects approximate connecting lines of images down-sampled from two images to be stitched sampled images of the two downsampled images, a predetermined area around the interpolated connection line near the interpolated actual connection line in which the approximate connection line of the overlap area is interpolated (e.g., a predetermined K Pixel range) (Fig. 1 (b)). 2, the image processing apparatus of the present invention uses the cost based on the corresponding pixel data between the upsampled images in such a limited area and the link line direction information based on the cost And estimates a final connection line closer to the actual connection line of the two images to be stitched.

As shown in FIG. 1 (a), the two images to be stitched including the overlap area are down-sampled by 1/2, 1/3, 1/4,... The load and the computation time can be reduced. It is possible to detect an approximate connecting line of downsampled images by using pixel data (gradation value, gradation value change, etc.), and conversely, as shown in (b) of FIG. 1, , 4, ..., and so on to restore the image size, and restoring and filling in the missing parts of the approximate connection line in the reconstructed image by a predetermined interpolation method, thereby performing interpolation close to the actual connection line Can be estimated.

The image processing apparatus of the present invention sets the limited area around the interpolated connection line close to the actual connection line (for example, a predetermined K pixel range in the vertical direction of the connection line) in this way, The closer the final connection line is estimated. Thus, images can be stitched to eliminate mismatches and enable sophisticated connections so that no stitching marks are left. Here, K can be set to a predetermined natural number such as 4, 5, .... In Fig. 3, K = 5 is exemplified.

Hereinafter, an actual connection line estimation process according to an embodiment of the present invention will be described in more detail with reference to the flowchart of FIG.

2 is a flowchart illustrating an actual connection line estimation process using cost and direction information among image connection line estimation methods according to an exemplary embodiment of the present invention.

,

)에 기초한 비용(cost)(e_i)과 비용(e_i)에 기초한 연결선 방향 정보(M_dir)를 이용하여, 상기 스티칭 대상 2 이미지들의 실제 연결선에 더 가까운 최종 연결선을 추정한다.The image processing apparatus of the present invention is capable of processing the corresponding pixel data (e.g., pixel data) between the upsampled images in the limited area (for example, a predetermined K pixel range in the vertical direction of the connecting line)

,

The final connecting line closer to the actual connecting line of the two images to be stitched is estimated by using the connecting line direction information M _dir based on the cost e _i and the cost e _i based on the cost e _i .

,

)을 기초로 비용(cost) e_i을 계산한다(S120). 본 발명의 이미지 처리장치는, [수학식1], [수학식2], [수학식3]을 이용하여 스티칭 대상 2 이미지들의 각 픽셀 위치 (i,j)에 대한 비용(e_i)을 산출하여 도 3의 (a)와 같이 스티칭 대상 2 이미지들에 대한 비용 매트릭스(M_cost)를 생성할 수 있다.

,

는 2 이미지들 사이의 대응되는 각 픽셀 데이터의 변화도(gradient)값이다. Referring to FIG. 2, first, the image processing apparatus of the present invention, as shown in FIG. 1, restores an image size by upsampling and, in the images in which pixel data such as tone values are calculated, , K pixel range) (S110), and the corresponding pixel data between two images (

,

) Cost (cost based on a) Calculate e _i (S120). The image processing apparatus of the present invention calculates the cost (e _i ) for each pixel position (i, j) of two images to be stitched by using [Equation 1], [Equation 2], and [Equation 3] The cost matrix M _cost for the two images to be stitched can be generated as shown in FIG. 3 (a).

,

Is a gradient value of each corresponding pixel data between two images.

[Equation 1]

&Quot; (2) "

&Quot; (3) "

Here, the cost e _i may be energy in another expression and corresponds to the importance value assigned corresponding to each pixel position of the two images to be stitched. The cost e _i of each position may be a value proportional to the gradient value or the intensity / intensity / tone intensity of the pixel data.

The image processing apparatus of the present invention generates the above cost matrix M _cost corresponding to each pixel position of two images to be stitched and then uses the cost function E _{( i, j )} as in Equation (4 ₎ of calculating the accumulated cost value for each pixel position (i, j) of each row of the cost matrix generate accumulation (accumulated) cost matrix (M _{a_cost)} (Fig. 3 (b), (c) , (d) , And a connection line direction information matrix M _dir for each pixel position (i, j) is generated (S 130).

&Quot; (4) "

The connecting line direction information matrix M _dir is calculated by multiplying the three pixels of the adjacent upper row as shown in [Equation 4] with the positions of the selected pixels for the calculation of the accumulation cost for each pixel position (i, j) (Min) pixel positions among the adjacent left and right pixels at the lowest accumulation cost) to each pixel position (i, j). For example, in each pixel position (i, j), directional values (e.g., 1, 2, 3, 4, 4, 6) may be direction information for each pixel position (i, j). In addition, the direction information for each pixel position (i, j) may include a direction value (e.g., 5) for its own position, such as '5' in the top row of FIG. This is because the position of the cost value 2 of the top row of FIG. 3 (b) has the minimum cost and starts to calculate the accumulation cost from this position, so that a direction value ). ≪ / RTI >

In Equation (4), the cost function E _{( i, j )} is a function for calculating a gradient of cost values at each pixel position with respect to the cost matrix M _cost as _shown in FIG. 3, uses the cost value of the pixel position (i, j) cost values e _{(i, j)} and the left and right adjacent to the three pixels of adjacent upper row in the pixel as a parameter. That is, the accumulation cost E _{( i, j )} is calculated by summing three accumulation cost values having the smallest accumulation cost among the three pixels of the upper row adjacent to the cost value e _{( i, j )} and the adjacent left and right pixels.

For example, using the cost matrix (M _cost ) of FIG. 3A, the accumulation cost values of all rows using the cost function E _{( i, j )} are calculated for each row from the first row _, ( _{Ma_cost} ) can be generated. First, as shown in FIG. 3 (b), in the first highest row, accumulation cost values are calculated sequentially from the corresponding pixel position having the minimum cost (position of cost value 2) to the left and right. _{( I, j )} adjacent to the cost value e _{( i, j )} , and the minimum value among the cost values in the adjacent left and right pixels are added to the right side and the left side, The cost value for the three pixels of the upper row is not reflected. In the figure, each arrow indicates each direction information constituting a connection line direction information matrix (M _dir ).

From the next row, as shown in (c) and (d) of FIG. 3, the accumulated cost values calculated from the left end to the right direction and the accumulated cost values calculated from the right end to the left end And accumulation cost values, respectively. In FIG. 3, (c) and (d), the leftmost matrix represents the calculation method of the storage cost values calculated from the left end to the right direction, and the center matrix represents the calculation method of the storage cost values calculated from the right end to the left end . When the accumulation cost values are calculated at the bidirectional ends and the accumulation cost values are selected from among the accumulation cost values at the two ends, the selection is made as shown in the rightmost matrix in FIGS. 3C and 3D. In the figure, each arrow indicates each direction information constituting a connection line direction information matrix (M _dir ).

A In this same way to calculate the accumulated cost value for each row generates the accumulated cost matrix (M _{a_cost)} and the connecting line direction information matrix (M _dir) Then, the image processing apparatus of the present invention, the accumulated cost matrix (M _{a_cost)} The minimum storage cost value position of the lowest row is determined as the final connection line start position (corresponding to the third row and fourth column position in FIG. 4B) (S140).

The image processing apparatus of the present invention is characterized in that a line connecting the minimum accumulated cost value position from the final connection line start position to the most significant row along the connection line direction information according to the connection line direction information matrix (M _dir ) (S150).

FIG. 5 is an example of two images to be stitched to be used in connection line estimation comparison between the present invention and an existing method.

FIG. 6 is a result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (a). As shown in FIG. 6, in conventional methods such as dynamic programming, graph cut, and Voronoi, there is a mismatch in which the back support of the ceiling and the legs of the chair are not connected to each other, In the result (d) according to the image connecting line estimation method of the present invention, it is confirmed that such a mismatching can be eliminated and the images can be stitched with a sophisticated connection so that no stitching trace remains.

FIG. 7 is a result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (b). As shown in FIG. 7, existing methods such as dynamic programming, graph cut, and Voronoi show a mismatch in the connection of the rectangular shape on the column side, but in the image connection line estimation method of the present invention In the result (d), it is confirmed that such a mismatching can be eliminated and the images can be stitched with a sophisticated connection so that no stitching marks are left.

FIG. 8 shows the result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (c). As shown in Fig. 8, in the conventional methods such as dynamic programming, graph cut, and Voronoi, there is a mismatch in the connection between the upper window side and the lower side support for the landscape, In (d), we confirmed that we can stitch images with sophisticated connections so that no such mismatching and stitching marks remain.

FIG. 9 is a result of comparing the image stitching process of the present invention and the conventional method with respect to the image set of FIG. 5 (d). As shown in FIG. 9, due to the existence of fine vertical lines in the images, the existing methods such as dynamic programming, graph cut, Voronoi, and the result according to the image link line estimation method of the present invention d), it was difficult to distinguish between different mismatches.

10 is a graph comparing the processing time performance required for image stitching of the present invention and the conventional method.

As shown in FIG. 10, since the content of the images to be stitched varies with the elapse of the image frame, the processing time is slightly changed. However, the Voronoi method 151 and the dynamic programming 152 appear quickly, and the graph cut method 154 is the slowest. The method 153 of the present invention is faster than the graph cut method 154 and slower than other methods for Gaussian filtering and entropy calculation for stitching but the acquisition quality of the panoramic image is different . Here, a predetermined personal computer (PC) is used for the above-mentioned experiment, and the detailed specification is not mentioned, but the relative comparison as described above is possible.

As described above, the image linking line estimation method using the downscaling and the cost function according to the present invention includes downsampling the images including the overlapping area to search for a connection line for matching images in the overlapping area of two images to be stitched It is possible to reduce computation load and computation time by detecting approximate connecting lines from reduced size images. In order to find the actual connecting line from the above approximate connecting line, the approximate connecting line in the images whose image size is restored by upsampling the images again is divided into predetermined left and right pixels _{( I, j )} representing the gradient of the pixel data in an area limited to the area of the lowest accumulation cost value of the lowest row, and then calculates the accumulated cost matrix using the cost function E Conversely, a line connected to the uppermost row along the matrix direction information is detected as a connection line, thereby stitching the images, thereby eliminating mismatching and enabling a sophisticated connection so that no stitching trace is left.

The function for performing the image link estimation method using the downscaling and the cost function according to an embodiment of the present invention as described above can be implemented as a computer readable code on a recording medium readable by an apparatus such as a computer And may be realized by a combination of such a recording medium and a computer or the like to input or output data and information necessary for performing a function. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, a hard disk, a removable storage device, And the like. In addition, the computer-readable recording medium may be distributed in a computer system connected to a network (e.g., the Internet, a mobile communication network, or the like) and may store computer readable codes in a distributed manner and may be executed through a network.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the essential characteristics of the invention. Therefore, the spirit of the present invention should not be construed as being limited to the embodiments described, and all technical ideas which are equivalent to or equivalent to the claims of the present invention are included in the scope of the present invention .

Claims (14)

delete Detecting connection lines of images down-sampled from two images to be stitched;
Setting a limited area around the interpolated line of the overlapping area in the images upsampled from the downsampled two images; And
Estimating a final connection line of the two images to be stitched using the cost based on the corresponding pixel data of the images in the limited area and the connection line direction information based on the cost,
In the estimating step,
The cost e _i is calculated by the following equations,

,

,

here,

,

Corresponding pixel data,

,

Is a gradient value of each corresponding pixel data. Detecting connection lines of images down-sampled from two images to be stitched;
Setting a limited area around the interpolated line of the overlapping area in the images upsampled from the downsampled two images; And
Estimating a final connection line of the two images to be stitched using the cost based on the corresponding pixel data of the images in the limited area and the connection line direction information based on the cost,
In the estimating step,
The line direction information includes directional values from the three pixels of the adjacent upper row and the neighboring left and right pixels toward itself, as a direction from the position selected to calculate the accumulation cost of each row from the cost to each pixel position A method for estimating an image link line. The method of claim 3,
Wherein the connecting line direction information further includes a direction value of the own position at a corresponding pixel position having a minimum cost of the top row. Detecting connection lines of images down-sampled from two images to be stitched;
Setting a limited area around the interpolated line of the overlapping area in the images upsampled from the downsampled two images; And
Estimating a final connection line of the two images to be stitched using the cost based on the corresponding pixel data of the images in the limited area and the connection line direction information based on the cost,
In the estimating step,
Calculating the accumulated cost values of each row using the cost values of the limited area and estimating a line connected from the minimum accumulated cost value position of the lowest row to the highest row along the connected line direction information as the final connected line The method comprising: 6. The method of claim 5,
The change in the cost value also possible to use a cost function E _{(i, j)} to calculate the (gradient), and calculating the accumulated cost values of each of the rows, the cost function E _{(i, j)} is formula

It is represented by the estimated image connecting line, characterized in that using the cost value at each pixel position (i, j) cost values e _{(i, j),} and the adjacent close to the three pixels in the top row left and right of the pixel by the parameter Way. 6. The method of claim 5,
Calculating accumulation cost values sequentially from the pixel position having a minimum cost to the left and right in the uppermost row of the limited region,
The accumulation cost values calculated from the left end to the right direction and the accumulation cost values calculated from the right end to the left end are selected from the next row without reflecting the accumulation cost value of the position not yet calculated The method comprising the steps of: delete A function of detecting a connection line of images obtained by downsampling two images to be stitched;
A function of setting a limited area around the interpolated line of the overlapping area in the images upsampled of the downsampled two images; And
And estimating a final connection line of the two images to be stitched, using the cost based on the corresponding pixel data of the images in the limited area and the connection line direction information based on the cost,
In the estimating function,
The cost e _i is calculated by the following equations,

,

,

here,

,

Corresponding pixel data,

,

Is a gradient value of each corresponding pixel data. A function of detecting a connection line of images obtained by downsampling two images to be stitched;
A function of setting a limited area around the interpolated line of the overlapping area in the images upsampled of the downsampled two images; And
And estimating a final connection line of the two images to be stitched, using the cost based on the corresponding pixel data of the images in the limited area and the connection line direction information based on the cost,
In the estimating function,
The line direction information includes directional values from the three pixels of the adjacent upper row and the neighboring left and right pixels toward itself, as a direction from the position selected to calculate the accumulation cost of each row from the cost to each pixel position . 11. The method of claim 10,
Wherein the connecting line direction information further includes a direction value of the own position at the corresponding pixel position having the minimum cost of the top row. A function of detecting a connection line of images obtained by downsampling two images to be stitched;
A function of setting a limited area around the interpolated line of the overlapping area in the images upsampled of the downsampled two images; And
And estimating a final connection line of the two images to be stitched, using the cost based on the corresponding pixel data of the images in the limited area and the connection line direction information based on the cost,
In the estimating function,
Calculating the accumulated cost values of each row using the cost values of the limited area and estimating a line connected from the minimum accumulated cost value position of the lowest row to the highest row along the connected line direction information as the final connected line . 13. The method of claim 12,
The change in the cost value also possible to use a cost function E _{(i, j)} to calculate the (gradient), and calculating the accumulated cost values of each of the rows, the cost function E _{(i, j)} is formula

Is represented by, for each pixel position (i, j) cost values e _{(i, j),} and an adjacent recording medium of the cost value in the adjacent left and right pixels, and three pixels in the top row characterized by using as a parameter in the. 13. The method of claim 12,
Calculating accumulation cost values sequentially from the pixel position having a minimum cost to the left and right in the uppermost row of the limited region,
The accumulation cost values calculated from the left end to the right direction and the accumulation cost values calculated from the right end to the left end are selected from the next row without reflecting the accumulation cost value of the position not yet calculated And a recording medium.

Images