JP2006293602A - Image processing method and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain proper corresponding feature points for photographed images of various resolutions without adjusting many parameters in processing wherein a plurality of photographed images obtained by photographing the same subject are associated with each other, and to shorten the processing time. <P>SOLUTION: In the image processing in which feature points are extracted from the plurality of photographed image (image 1 and image 2) obtained by photographing the same subject and feature points corresponding to the same point of the subject are associated with each other, the plurality of photographed images to be processed are reduced at the same reduction ratio (e.g. 1/2), feature points are extracted from the respective reduced photographed images and associated with each other. Then processing for associating positions in the photographed images before reduction with the feature points, to associate the feature points of the photographed images (image 1 and image 2) of the original size with each other. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing method and an image processing apparatus for associating a plurality of captured images.

By associating a plurality of photographed images obtained by photographing the same subject (for example, images obtained by time-series photographing, images obtained by photographing at a plurality of different photographing positions, etc.) Movement, 3D shape, etc. can be determined. For example, in Patent Document 1, feature points that are candidates for association are extracted from each of images obtained by time-series shooting, and after performing preprocessing such as a local spatial filter, the feature points and the feature points are extracted. A technique for associating each captured image with each other by calculating a correlation with neighboring pixel data is disclosed.
JP 2000-259835 A

  Since the feature point extraction process and the feature point association process have many parameters that are affected by the resolution of the captured image, even if these processes are performed on the captured image of various resolutions under the same conditions. There is a problem that an appropriate result cannot be obtained unless a large number of parameters are adjusted. In addition, as the size of the captured image to be processed increases, the processing load related to the feature point extraction processing and the association processing increases, and there is a problem that enormous processing time is required.

  An object of the present invention is to obtain appropriate corresponding feature points for captured images of various resolutions without adjusting a large number of parameters in the process of associating a plurality of captured images obtained by capturing the same subject. At the same time, the processing time is shortened.

In order to solve the above problem, the invention according to claim 1 extracts feature points from each of a plurality of photographed images obtained by photographing the same subject, and features points corresponding to the same point of the subject. In the image processing method for associating
A reduction step of reducing each of the plurality of captured images at the same reduction ratio;
An extraction step of extracting feature points from each reduced photographed image;
An associating step for associating feature points of each of the extracted captured images;
An associating step of associating the associated feature point with which position of the captured image of the original size.

  According to a second aspect of the present invention, in the image processing method according to the first aspect, the reduction processing in the reduction step is composed of a plurality of stages.

  According to a third aspect of the present invention, in the image processing method according to the second aspect, the associating step includes a plurality of steps of associating processing. It is characterized in that it is associated with a photographed image of a size of.

According to a fourth aspect of the present invention, in the image processing method according to any one of the first to third aspects, the reduction ratio of the one-step reduction process in the reduction process is a power of 1/2 or 1/2. Yes,
The associating step is characterized by associating with a photographed image having an image size that is twice or a power of 2 in a one-step association process.

  According to a fifth aspect of the present invention, in the image processing method according to any one of the first to fourth aspects, in the association step, the matching degree of image data between the corresponding corresponding feature point candidates. It is characterized in that corresponding feature points can be obtained by evaluating.

  According to a sixth aspect of the present invention, in the image processing method according to any one of the first to fourth aspects, in the associating step, data correlation of neighboring regions between the corresponding corresponding feature candidates. It is characterized in that corresponding feature points can be obtained by evaluating.

  According to a seventh aspect of the present invention, in the image processing method according to any one of the first to fourth aspects, in the associating step, each of the pixel data is associated between the corresponding corresponding feature point candidates. It is characterized in that corresponding feature points can be obtained by evaluating the specificity with respect to neighboring pixels.

According to an eighth aspect of the present invention, in the image processing method according to any one of the first to seventh aspects, the reduction ratio and / or the number of reductions with respect to the photographed image according to the image size of the photographed image to be processed. Including a determination step of determining
In the reduction process, a reduction process of the determined reduction ratio and / or the number of reductions is performed.

The invention according to claim 9 is the image processing method according to any one of claims 1 to 7, wherein an analysis step of analyzing a spatial frequency component of a captured image to be processed;
Determining a reduction ratio and / or the number of reductions for the captured image based on the analysis result in the analysis step,
In the reduction process, a reduction process of the determined reduction ratio and / or the number of reductions is performed.

The invention according to claim 10 is an image processing device that extracts feature points from each of a plurality of photographed images obtained by photographing the same subject, and associates feature points corresponding to the same point of the subject.
Reduction means for reducing each of the plurality of captured images at the same reduction ratio;
Extraction means for extracting feature points from each reduced photographed image;
Associating means for associating feature points of each of the extracted captured images;
And an associating means for associating with which position of the photographed image of the original size the associated feature point corresponds.

  According to an eleventh aspect of the present invention, in the image processing apparatus according to the tenth aspect, the reduction processing by the reduction means includes a plurality of stages.

  According to a twelfth aspect of the present invention, in the image processing apparatus according to the eleventh aspect, the associating unit includes a plurality of steps of associating processing, and the original processing unit performs processing of associating with a captured image in the middle step by the reducing unit. It is characterized by associating with a photographed image having a size of.

According to a thirteenth aspect of the present invention, in the image processing apparatus according to any one of the tenth to twelfth aspects, the reduction ratio of the one-step reduction processing by the reduction means is a power of 1/2 or 1/2. Yes,
The associating means is characterized by associating with a photographed image having an image size of 2 times or a power of 2 in a one-step associating process.

  According to a fourteenth aspect of the present invention, in the image processing device according to any one of the tenth to thirteenth aspects, the associating means includes a degree of matching of image data between the corresponding corresponding feature point candidates. It is characterized in that corresponding feature points are obtained by evaluating.

  According to a fifteenth aspect of the present invention, in the image processing device according to any one of the tenth to thirteenth aspects, the associating means includes data correlation of neighboring regions between the corresponding corresponding feature candidates. It is characterized in that corresponding feature points are obtained by evaluating.

  According to a sixteenth aspect of the present invention, in the image processing device according to any one of the tenth to thirteenth aspects, the associating means includes a pixel data item between the corresponding corresponding feature point candidates. It is characterized in that corresponding feature points are obtained by evaluating the specificity with respect to neighboring pixels.

According to a seventeenth aspect of the present invention, in the image processing apparatus according to any one of the tenth to sixteenth aspects, the reduction ratio and / or the number of reductions with respect to the photographed image according to the image size of the photographed image to be processed. Determining means for determining
The reduction means performs a reduction process of the determined reduction ratio and / or the number of reductions.

The invention according to claim 18 is the image processing apparatus according to any one of claims 10 to 16, and an analysis means for analyzing a spatial frequency component of a captured image to be processed;
Determination means for determining a reduction ratio and / or the number of reductions for the captured image based on the analysis result by the analysis means,
The reduction means performs a reduction process of the determined reduction ratio and / or the number of reductions.

  According to the present invention, in a process of associating a plurality of photographed images obtained by photographing the same subject, appropriate corresponding feature points are obtained for photographed images with various resolutions without adjusting many parameters. At the same time, the processing time can be shortened.

  FIG. 1 shows a configuration of an image processing apparatus 100 according to an embodiment of the present invention. As shown in FIG. 1, the image processing apparatus 100 includes an image input unit 1, an image reduction unit 2, a feature point extraction unit 3, a corresponding feature point determination unit 4, an inter-resolution association unit 5, a corresponding feature point output unit 6, an image An analysis unit 7 and a condition parameter setting unit 8 are included.

  The image input unit 1 acquires a photographed image obtained by photographing a subject and outputs it to the feature point extraction unit 2 and the image analysis unit 5. In the present embodiment, images input to the image input unit 1 are a plurality of captured images obtained by capturing the same subject. Examples of the plurality of photographed images include images obtained by time-series photographing (for example, moving image photographing and continuous photographing), images obtained by simultaneously photographing at a plurality of different photographing positions, and the like. As a method for acquiring a captured image (input method) in the image input unit 1, a method for acquiring from a device connected to the outside via a predetermined communication network, a photographing device such as a wired or wirelessly connected digital camera, etc. And a method of reading a captured image recorded on a recording medium (CD-R, memory card, etc.).

The image reduction unit 2 performs a reduction process on the captured image input from the image input unit 1 based on the condition parameters (reduction ratio and / or number of reductions) set by the condition parameter setting unit 8. FIG. 2 shows an example of a reduction process with a reduction ratio of 1/2. When a captured image of n _x × n _y pixels is reduced to ½ size, as shown in FIG. 2, an average value of data for 2 × 2 pixels is used as data for one pixel (n _x / 2) × (n _y / 2) pixels are obtained.

  The feature point extraction unit 3 extracts feature points from each captured image reduced by the image reduction unit 2 based on the condition parameters set by the condition parameter setting unit 6, and outputs the extraction results to the corresponding feature point determination unit 4. Output. Specifically, the feature point extraction unit 3 extracts a contour line (edge information) from a captured image by frequency conversion processing, etc., thins the contour line into a line drawing having a thickness of 1, and analyzes the obtained thin line. Then, feature points are extracted by detecting geometric singular points (end points, break points, branch points, intersections (see FIG. 8)).

  The corresponding feature point determination unit 4 determines the feature point corresponding to the same point of the subject for each photographed image subjected to the reduction process by the image reduction unit 2 based on the condition parameter set by the condition parameter setting unit 8. Evaluate the coordinate values, color information, number of connections, and positional relationships between neighboring feature point candidates as candidates, and determine whether there is a corresponding feature point based on these evaluation results (evaluation information) (See FIGS. 6 to 15). If it is determined that there is a corresponding feature point, the corresponding corresponding feature point candidate is determined as the corresponding feature point, and information on the determined corresponding feature point is output to the inter-resolution association unit 5.

  The inter-resolution associating unit 5 performs a process of associating the corresponding feature point determined by the corresponding feature point determining unit 4 with which position of the pre-reduced captured image, thereby corresponding to the captured image of the original size. The feature point is determined (see FIG. 5), and information on the determined corresponding feature point is output to the corresponding feature point output unit 7. For example, when the image reduction unit 2 performs the reduction process n times, if the corresponding feature point association process is performed n times, the corresponding feature point of the captured image of the original size is calculated.

  The corresponding feature point output unit 6 outputs the information about the corresponding feature point input from the resolution association unit 5 to the image condition parameter setting unit 8 and / or an externally connected device (for example, a personal computer).

  The image analysis unit 7 analyzes the image size of the captured image to be processed and the distribution state of the spatial frequency component, and outputs the analysis result to the condition parameter setting unit 8. The distribution of the spatial frequency component of the captured image can be obtained by performing Fourier transform on the captured image. As shown in FIG. 3A, the peak frequency or the center frequency that is the center of gravity of the distribution is determined from the spatial frequency distribution. When the captured image is reduced to 1 / n (n> 1), the frequency becomes n times, so that the spatial frequency distribution shifts as shown in FIG.

  Based on the analysis result of the image analysis unit 7 and / or the corresponding feature point information output from the corresponding feature point output unit 6, the condition parameter setting unit 8 performs image reduction unit 2, feature point extraction unit 3, corresponding feature point Condition parameters necessary for processing in the determination unit 4 and the association unit 5 between resolutions are set. Further, a parameter set by the user may be used as the condition parameter.

  Specifically, the condition parameter setting unit 8 sets a reduction ratio and / or the number of reductions as condition parameters necessary for the reduction process in the image reduction unit 2. The reduction ratio and / or the number of reductions may be fixed values in advance, or may be set according to the characteristics (image size, spatial frequency characteristics, etc.) of the captured image to be processed.

For example, when the reduction ratio is 1/2 and the image size range is 50,000 to 200,000, when a captured image having a size of 2000 × 1500 pixels is input, the reduction processing is performed twice as follows.
Non-reduced state: 2000 × 1500 = 3000000 pixels;
One-stage reduced state: 1000 × 750 = 750,000 pixels;
Two-stage reduction state: 500 × 375 = 187500 pixels.
Further, instead of the image size range, an optimal image size value may be determined, and a condition closest to the optimal value may be selected.

  When setting the condition parameter using the spatial frequency characteristics of the captured image, the condition parameter setting unit 8, as shown in FIG. 3 (b), the spatial frequency distribution shifted by image reduction and a preset allowable frequency range. Alternatively, the reduction ratio and / or the number of reductions are set based on the optimum frequency. Even with the same image size, the edge information differs depending on the shooting situation of the subject (up-shooting, far-field shooting), so it is more appropriate to control the reduction process using spatial frequency information. Can be extracted.

  In addition, the condition parameter setting unit 8 is a condition parameter necessary for the feature point association process in the corresponding feature point determination unit 4 and an allowable value (ε1) necessary for the evaluation of the coordinate value, and an allowable value necessary for the evaluation of the number of connections. A value (ε2), permissible values (ε3, ε4) necessary for evaluating color information, a permissible value (ε5) necessary for evaluating the positional relationship with neighboring feature points, and the like are set.

Next, the operation in this embodiment will be described.
With reference to the flowchart of FIG. 4, the feature point association process executed in the image processing apparatus 100 will be described. In the processing of FIG. 4, processing when the reduction ratio with respect to the captured image is determined in advance (reduction ratio 1/2) will be described.

  First, a plurality of photographed images (for example, photographed images obtained by exposure bracket photography) obtained by photographing the same subject, which is a processing target of the feature point association processing, are input (step A1). Next, each captured image is reduced to a half size (step A2).

  Next, it is determined whether or not the size of the captured image reduced in step A2 is within an appropriate range (within a preset size range) (for example, 50000-200000 pixel size) (step A3). If it is determined in step A3 that the size of the reduced captured image is not within the appropriate range (step A3; NO), the process returns to step A2, and the size of each captured image is further reduced by half. The If it is determined in step A3 that the size of the reduced captured image is within an appropriate range (step A3; YES), feature points are extracted from each reduced captured image (step A4).

  Next, a process of associating feature points corresponding to the same point of the subject is performed on each reduced photographed image (step A5). The association process in step A5 will be described in detail later with reference to FIGS.

  Next, it is associated with which position of the feature point associated with the captured images the position of the pre-reduced captured image (the captured image that is twice the size of the reduced captured image) (step A6). The association processing in step A6 will be described in detail later with reference to FIGS.

  Next, it is determined whether or not the association processing up to the original size of the captured image is completed (step A7). If it is determined in step A7 that the association process up to the original photographed image has not been completed (step A7; NO), the process returns to step A6, and the association process is further performed. If it is determined in step A7 that the association process up to the original photographed image has been completed (step A7; YES), the feature point association process is terminated.

FIG. 5 shows a flow of the feature point association process when the reduction process with the reduction ratio of 1/2 is performed n times on the two captured images (image 1 and image 2). Corresponding feature points are obtained by associating feature points extracted from an image obtained by reducing the size of image 1 to 1/2 ⁿ with feature points extracted from an image obtained by reducing the size of image 2 to 1/2 ⁿ Candidates are determined. Then, when the association process of the corresponding feature point candidates is performed n times, the corresponding feature points of the original size captured images (image 1 and image 2) are determined.

  In the flowchart of FIG. 4, the reduction ratio is set in advance. However, in the condition parameter setting unit 8, the reduction ratio and / or according to the characteristics (image size, spatial frequency characteristics, etc.) of the captured image to be processed. Alternatively, a reduction count may be set, and the reduction process may be performed based on the set reduction ratio and / or the reduction count.

  In the present invention, it is assumed that the sizes of images to be associated are basically the same, and each image may be reduced under the same conditions. For example, when matching between images of different sizes, pre-processing for aligning the image size is performed by enlarging or reducing one of the images, and matching according to the present invention is performed between images of the same size obtained as a result. You can do it. In that case, the reduction process by the pre-processing and the reduction process of the present invention may be integrated. Considering such a case, the reduction conditions of the images to be associated with each other may not necessarily be the same.

<Association process>
Next, the feature point association processing shown in step A5 of FIG. 4 will be described. The corresponding feature point determination unit 4 evaluates the coordinate values, the number of connections, the color information, and the positional relationship with neighboring feature points between the corresponding feature point candidates that are candidate feature points corresponding to the same point of the subject. It is determined whether there is a corresponding feature point based on the result (evaluation information).

そして、距離Δ_ijが許容値ε１以下である特徴点Ｔｉを、対応特徴点候補を順位付けしたテーブルα（図１２参照）への登録候補とする。 First, with reference to FIG. 6, evaluation of coordinate values between corresponding feature point candidates will be described. Hereinafter, the image 2 is set as a reference image. As shown in FIG. 6A, the feature point of the image 1 is Ti (i = 1, 2,..., M), and the feature point of the image 2 is Sj (j = 1, 2,..., N). . In order to evaluate the coordinate value between the feature points, the distance between Ti and Sj may be calculated for all combinations of Ti and Sj as shown in FIG. When the coordinates of Ti are (x _i , y _i ) and the coordinates of Sj are (x ′ _j , y ′ _j ), the distance Δ _ij between Ti and Sj is defined as in equation (1).

Then, feature points Ti whose distance Δ _ij is equal to or smaller than the allowable value ε1 are registered as candidates for registration in the table α (see FIG. 12) in which the corresponding feature point candidates are ranked.

Next, evaluation of the number of connections will be described with reference to FIGS.
As described above, an edge is extracted from the captured image, and a singular point of a thin line obtained by thinning the edge is extracted as a feature point. FIG. 7A shows an example of a photographed image, and FIG. 7B shows an example of fine lines and feature points (white circles in the figure) of the photographed image. The feature points extracted in this way are classified by the number of connections (order) indicating how many thin lines are concentrated on the feature points.

  As shown in FIG. 8, the end point of the thin line is the connection number = 1, the branch point is the connection number = 3, the break (corner) is the connection number = 2, the intersection is the connection number = 4, and the composite point is the connection number> 4. It is. Note that very close feature points are regarded as one feature point. Also, as shown in FIG. 9, an upper limit is provided for the number of connections to be evaluated (evaluation connection number), and when the actual connection number (true connection number) is a certain number or more, the upper limit value (a constant number). May be used as the evaluation connection number. The number of connections between Ti and Sj is compared for all combinations of feature points Ti and Sj, and a feature point Ti whose difference (absolute value of the difference) between both is less than or equal to the allowable value ε2 is represented in table α (see FIG. 12). ) Registration candidates.

Next, evaluation of color information will be described with reference to FIGS.
Assuming an application that aligns captured images with different exposure conditions, it is conceivable that the brightness information between the corresponding feature point candidates has some degree of opening, and as shown in FIG. A method of evaluating only the color (FIG. 10A) or a method of evaluating the hue and saturation between corresponding feature points (FIG. 10B) is adopted.

  When L * a * b * is used as an example of chromaticity coordinates, in the method of evaluating only the hues of the corresponding feature points, as shown in FIG. 10A, for all combinations of Ti and Sj, a * The chromaticity coordinates of the corresponding feature point candidates are plotted on the b * plane, and an angle θ (hue difference) formed by a line connecting each chromaticity coordinate and the origin is calculated. This hue difference θ is an allowable value ε3 or less. Is a candidate for registration in the table α (see FIG. 12). However, if one of the chromaticity coordinates is in a neutral region whose distance from the origin is within a certain value, the hue difference θ is assumed to be insignificant as a neutral color.

  When L * a * b * is used as an example of chromaticity coordinates, in the method of evaluating the hue and saturation between corresponding feature points, as shown in FIG. 10B, for all combinations of Ti and Sj, The distance between the chromaticity coordinates is calculated by plotting the chromaticity coordinates of the corresponding feature point candidates on the a * b * plane, and the feature point Ti whose calculated distance is equal to or less than the allowable value ε4 is stored in the table α ( (See FIG. 12).

FIG. 11 shows a method for calculating color information used for evaluating color information.
Usually, a color space is defined for RGB values of image data photographed and developed by a digital camera or the like. In the case of the most common sRGB color space, the RGB data is converted into chromaticity values in the CIE XYZ space using the conversion formula (10) in FIG. Furthermore, by defining the chromaticity (observation illumination condition) of the white point using the conversion equation (20) in FIG. 11, the white point is converted into a chromaticity value in the CIE L * a * b * space.

  FIG. 10 shows the case where L * a * b * is used as the color information used for the evaluation. However, various other color spaces (L * u * v *, HSV, HLS, YCbCr, etc.) may be used.

When these three evaluation processes (evaluation of coordinate values of corresponding feature point candidates, the number of connections, and color information) are completed, the candidate for registration in the table α is obtained based on the result (evaluation information) in each evaluation process. The feature points Ti are ranked, and the feature points with higher ranks (for example, 1st to 6th) are registered in the table α, thereby creating the table α. For example, the coordinates shift (Equation (1) distance Δ value of _ij) of small, the difference in coupling number is small, as the distance between the hue difference θ or chromaticity coordinates decreases, position on a table α becomes higher.

  FIG. 12 shows an example of the table α in which the corresponding feature point candidates of the images 1 and 2 are ranked. According to FIG. 12, the feature point T2 of the image 1 is positioned at the top as the corresponding feature point candidate of the feature point S1 of the image 2, and the feature point corresponding to the feature point S5 of the image 2 exists in the image 1 Indicates that no.

  In parallel with the three evaluation processes and the table α creation process, for each feature point in each captured image, a neighboring feature point (generally a plurality) located within a certain distance from the feature point is searched. Next, a process of evaluating the positional relationship between the feature points and their neighboring feature points (hereinafter referred to as a positional relationship evaluation process) is performed between the captured images.

  Corresponding feature point candidates (Ti) registered in the table α exist for feature points of the reference image (attention feature point: Sj) and their neighboring feature points (may not exist). In the positional relationship evaluation process, when Sj and Sk are in a close relationship, the positional relationship of all combinations (Ti, Tl) of the corresponding feature point candidate Ti of Sj and the corresponding feature point candidate Tl of Sk is the reference image. It is determined whether or not the current feature point matches the positional relationship between the feature point and its neighboring feature points. In the example of the table α, when the feature points S3 and S4 of the image 2 are in a close relationship, there are corresponding feature point candidates T4 and T2 of S4 with respect to the corresponding feature point candidates T3, T4 and T1 of S3. The positional relationships for the six combinations are evaluated.

ここで、Δx＝x₂−x₁、Δy＝y₂−y₁、Δx’＝x’₂−x’₁、Δy’＝y’₂−y’₁である。又は、デジタル画像の特性に基づき、座標の量子化誤差に考慮して式（２）を調整した式（３）に示す評価値を用いることも可能である。

As shown in FIG. 13, the coordinates of the feature point of interest and its neighboring feature points in the reference image (image 2) are (x ₁ , y ₁ ) and (x ₂ , y ₂ ), respectively (FIG. 13 (a)). The corresponding feature point candidate of the feature point of interest and the coordinates of its neighboring feature points are (x ′ ₁ , y ′ ₁ ) and (x ′ ₂ , y ′ ₂ ), respectively (FIG. 13B). In the positional relationship evaluation process, the evaluation value shown in Expression (2) is used.

Here, Δx = x ₂ −x ₁ , Δy = y ₂ −y ₁ , Δx ′ = x ′ ₂ −x ′ ₁ , and Δy ′ = y ′ ₂ −y ′ ₁ . Alternatively, it is also possible to use an evaluation value represented by Expression (3) obtained by adjusting Expression (2) in consideration of the quantization error of coordinates based on the characteristics of the digital image.

  In the positional relationship evaluation process, when the evaluation value shown in Expression (2) or Expression (3) is the allowable value ε5 or less, the corresponding combination of corresponding feature point candidates is counted up. FIG. 14 shows an example of the table β representing the count information of the corresponding feature point candidates. In the table α in FIG. 12, the evaluation value of the positional relationship between the feature point S3 of the reference image and its neighboring feature point S4 and the positional relationship between the corresponding feature point candidate T3 of S3 and its neighboring feature point T2 is less than or equal to the allowable value ε5. In some cases, the corresponding table information is counted up (+1) as shown in FIG.

  When the tables α and β are created, whether or not there is a corresponding feature point corresponding to the same point of the subject is determined based on the rank information of the table α and the count information of the table β, and there is a corresponding feature point. Is determined, one corresponding feature point is determined. The corresponding feature point determination method is classified into two types depending on which of the evaluation information based on the coordinate shift and the evaluation information based on the positional relationship with the neighboring feature points is used preferentially.

  When the evaluation information based on the coordinate shift is used preferentially, as shown in FIG. 15A, first, a corresponding feature point candidate whose count value is not 0 is selected in the table β, and among the selected corresponding feature point candidates, The corresponding feature point candidate at the top of the table α (that is, the smallest coordinate deviation) is determined as the corresponding feature point.

  When the evaluation information based on the positional relationship with neighboring feature points is used preferentially, as shown in FIG. 15B, among the corresponding feature point candidates registered in the table α, the correspondence with the maximum count value in the table β. Feature point candidates are determined as corresponding feature points.

  Note that the evaluation information used preferentially can be switched according to the setting or situation by the user. For example, the amount of motion and deformation of the subject are estimated by analyzing the distribution of feature points and pattern matching. For example, when the amount of motion is smaller than a predetermined amount, evaluation information based on coordinate deviation (FIG. 12A) ) Is used preferentially.

<Association process>
Next, with reference to FIGS. 16 to 18, association processing for associating which position of the corresponding feature point candidate of the reduced image corresponds to which position of the pre-reduction image will be described.

FIG. 16 shows an example of the association process. As shown in FIG. 16, when the reduction ratio is 1/2, the pre-reduction image has 2 × 2 candidate points corresponding to the feature points of the reduced image, and the most appropriate of the four candidate points. The point will be selected. Therefore, when the association process is performed with n photographed images, there are 4 ⁿ combinations of associations.

When the association processing is performed on two photographed images (image 1 and image 2), four corresponding feature point candidates of the pre-reduction image of image 1 and four corresponding feature point candidates of the pre-reduction image of image 2 are obtained. When matched, there are 16 combinations of associations, and an optimal combination is selected as a corresponding feature point from the 16 combinations. Actually, as shown in FIG. 17, the corresponding feature point candidate pixel data I _{22 of the} pre-reduction image of image 1 and its neighboring pixel (8 pixels) pixel data, and the corresponding feature points of the pre-reduction image of image 2 An optimal combination is selected based on the candidate pixel data I ′ ₂₂ and pixel data of neighboring pixels (eight pixels).

  As a method for selecting the optimum combination from the corresponding feature point candidates of the two pre-reduction images, at least one of the following three methods can be employed.

但し、露出ブラケット撮影の撮影画像間で第１の方法を用いる場合には、対応するデータ補正を行う必要がある。 (A) The first method is a method for evaluating the degree of coincidence of image data between corresponding feature point candidates of two images before reduction. In this first method, the combination that minimizes the data difference between the corresponding feature point candidate of image 1 and its neighboring pixels and the corresponding feature point candidate of image 2 and its neighboring pixels is selected. Specifically, a combination that minimizes the value of Δ ₁ shown in Expression (4) is selected from the 16 combinations.

However, when the first method is used between captured images of exposure bracket shooting, it is necessary to perform corresponding data correction.

ここでＮは、計算範囲の画素数（図１７の例ではＮ＝３×３）である。 (B) The second method is a method for evaluating the data correlation of the neighboring region between the corresponding feature point candidates of the two pre-reduction images. Specifically, among the 16 combinations, the combination that maximizes the value of Δ ₂ shown in Expression (5) is selected.

Here, N is the number of pixels in the calculation range (N = 3 × 3 in the example of FIG. 17).

(C) The third method is a method for evaluating the specificity of each pixel data with respect to neighboring pixels between corresponding feature point candidates of two reduced images. Here, the peculiarity indicates the property of whether or not the image has an edge or a peak characteristic, and can be detected by a convolution operation of a spatial filter or the like. The feature point should have specificity, and as shown in FIG. 18, a spatial filter (for example, Laplacian filter) M (m _ij , i, j = 1, 2, 3) for detecting the specificity is used. By the convolution operation, the corresponding feature point candidate that maximizes the result is selected.

  As described above, according to the image processing apparatus 100 of the present embodiment, in the process of associating a plurality of photographed images obtained by photographing the same subject, photographing with various resolutions without adjusting many parameters. It is possible to obtain an appropriate corresponding feature point for the image and reduce the processing time.

1 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present invention. The figure for demonstrating reduction of an image size. The figure which shows the distribution condition of the spatial frequency component of a picked-up image. The flowchart which shows the feature point matching process performed in an image processing apparatus. The figure which shows the flow of the feature point matching process at the time of performing the reduction process of the reduction ratio 1/2 with respect to two picked-up images n times. The figure which shows the fine line and feature point in the image 1 and the image 2, and the figure which shows the combination of each feature point of the image 1, and each feature point of the image 2. FIG. 4B is a diagram showing a captured image (a) and fine lines and feature points obtained from edge information of the captured image. The figure which shows the positioning of the feature point in a thin line figure. The figure which shows the relationship between the number of true (actual) connections and the number of evaluation connections (number of connections for comparative evaluation). The figure explaining the case where only the hue of the corresponding feature points is evaluated (a), and the case where the hue and saturation are evaluated (b). The figure which shows the calculation method of the color information for evaluation. The figure which shows the table (alpha) which ranked the corresponding feature point candidate. The figure which shows the evaluation method of the positional relationship between neighboring feature points. The figure which shows the table (beta) produced | generated based on the evaluation value of the positional relationship between neighboring feature points. The figure which shows the identification method of a corresponding feature point. The figure which shows the correlation process of the feature point in a reduction image and the image before reduction. The figure which shows the matching process of the feature point in two images before reduction. The figure which shows the determination method of the feature point at the time of using a spatial filter in the matching process of the feature point of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image input part 2 Image reduction part 3 Feature point extraction part 4 Corresponding feature point determination part 5 Inter-resolution correlation part 6 Corresponding feature point output part 7 Image analysis part 8 Condition parameter setting part 100 Image processing apparatus

Claims (18)

In an image processing method for extracting feature points from each of a plurality of captured images obtained by photographing the same subject, and associating feature points corresponding to the same point of the subject,
A reduction step of reducing each of the plurality of captured images at the same reduction ratio;
An extraction step of extracting feature points from each reduced photographed image;
An associating step for associating feature points of each of the extracted captured images;
An associating step for associating which position of the photographed image of the original size corresponds to the associated feature point;
An image processing method comprising:   The image processing method according to claim 1, wherein the reduction process in the reduction process includes a plurality of stages.   3. The association process according to claim 2, wherein the association step includes a plurality of steps of association processing, and the association with the captured image of the original size is performed through the association processing with the captured image in the intermediate stage in the reduction step. The image processing method as described. The reduction ratio of the one-stage reduction process in the reduction process is a power of 1/2 or 1/2,
The image according to any one of claims 1 to 3, wherein in the associating step, associating with a photographed image having an image size of 2 times or a power of 2 is performed in one step of associating processing. Processing method.   5. In the associating step, a corresponding feature point is obtained by evaluating a matching degree of image data between the corresponding corresponding feature point candidates. The image processing method as described.   5. In the associating step, corresponding feature points are obtained by evaluating data correlation of neighboring regions between the corresponding corresponding feature candidates. The image processing method as described.   5. In the associating step, a corresponding feature point is obtained by evaluating the specificity of each pixel data with respect to neighboring pixels between the corresponding corresponding feature point candidates. The image processing method according to any one of the above. Including a determination step of determining a reduction ratio and / or the number of reductions with respect to the photographed image according to the image size of the photographed image to be processed;
The image processing method according to claim 1, wherein in the reduction step, reduction processing of the determined reduction ratio and / or number of reductions is performed. An analysis process for analyzing a spatial frequency component of a captured image to be processed;
Determining a reduction ratio and / or the number of reductions for the captured image based on the analysis result in the analysis step,
The image processing method according to claim 1, wherein in the reduction step, reduction processing of the determined reduction ratio and / or number of reductions is performed. In an image processing apparatus that extracts feature points from each of a plurality of captured images obtained by photographing the same subject, and associates feature points corresponding to the same point of the subject.
Reduction means for reducing each of the plurality of captured images at the same reduction ratio;
Extraction means for extracting feature points from each reduced photographed image;
Associating means for associating feature points of each of the extracted captured images;
Associating means for associating which position of the associated feature point corresponds to the position of the captured image of the original size;
An image processing apparatus comprising:   The image processing apparatus according to claim 10, wherein the reduction processing by the reduction unit includes a plurality of stages.   12. The association unit includes a plurality of steps of association processing, and performs association with a captured image of an original size through an association process with a captured image at an intermediate stage by the reduction unit. Image processing apparatus. The reduction ratio of the one-stage reduction process by the reduction means is a power of 1/2 or 1/2,
The image processing according to any one of claims 10 to 12, wherein the associating means performs associating with a captured image having an image size that is twice or a power of 2 in one-step association processing. apparatus.   The said associating means obtains a corresponding feature point by evaluating a matching degree of image data between the corresponding corresponding feature point candidates. Image processing apparatus.   The association means obtains a corresponding feature point by evaluating data correlation in a neighboring region between the corresponding corresponding feature candidates. Image processing apparatus.   The association means obtains a corresponding feature point by evaluating the specificity of each pixel data with respect to a neighboring pixel between the corresponding corresponding feature point candidates. An image processing apparatus according to claim 1. In accordance with the image size of the captured image to be processed, the image processing apparatus includes a determination unit that determines a reduction ratio and / or the number of reductions for the captured image
The image processing apparatus according to claim 10, wherein the reduction unit performs a reduction process of the determined reduction ratio and / or the number of reductions. An analysis means for analyzing a spatial frequency component of a captured image to be processed;
Determination means for determining a reduction ratio and / or the number of reductions for the captured image based on the analysis result by the analysis means,
The image processing apparatus according to claim 10, wherein the reduction unit performs a reduction process of the determined reduction ratio and / or the number of reductions.

Images