JP2010170482A - Program and apparatus for processing image, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform highly precise image registration while reducing the possibility of falling into a local solution in registration of a plurality of digital images. <P>SOLUTION: An image processing program for registering two images by computer includes: an obtaining step for obtaining two images; a calculation step for calculating an evaluation value about registration by using a plurality of parameters on the basis of the two images; a setting step for setting respective weighting of the plurality of parameters; and a detection step for comparing a plurality of evaluation values calculated at the calculation step by using different parameters set at the setting step to detect a relative position of the two images. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing program for realizing superposition of two images by a computer, an image processing apparatus for superimposing two images, and an imaging apparatus including the image processing apparatus.

Conventionally, a plurality of digital images have been superimposed for the purpose of increasing the resolution. In such superposition, it is important to accurately align a plurality of images. Therefore, various techniques related to alignment, such as a high-speed technique and a high-precision technique, are considered depending on the application contents to image processing. For example, Non-Patent Document 1 discloses a technique for calculating a geometric transformation parameter by a nonlinear least square method as an example of a highly accurate technique.

  The technique described above is very effective because highly accurate alignment is possible. However, there is a possibility of falling into a local solution in the solution calculation process using the nonlinear least square method. In order to solve such a problem, it is necessary to appropriately adjust initial values of parameters and convergence conditions.

  The present invention has been made in view of the above problems, and an object of the present invention is to realize highly accurate alignment while reducing the possibility of falling into a local solution in alignment of a plurality of digital images.

An image processing program of the present invention is an image processing program for realizing alignment of two images by a computer, and relates to the alignment using an acquisition step of acquiring the two images and a plurality of parameters. A calculation step that calculates an evaluation value based on the two images, a setting step that sets each weight of the plurality of parameters, and a calculation that uses the different parameters set in the setting step. And detecting a relative position of the two images by comparing the plurality of evaluation values.

  The plurality of parameters may include parameters having different characteristics with respect to the alignment.

  The plurality of parameters may include at least two parameters of a parameter relating to a change in translation direction in the alignment, a parameter relating to a change in a rotation direction in the alignment, and a parameter relating to enlargement / reduction in the alignment. good.

  The plurality of parameters are parameters for geometrically transforming the two images, and in the calculating step, the evaluation value is obtained after geometrically transforming at least one of the two images using the plurality of parameters. May be calculated.

In the setting step, the weight may be set based on at least one of a photographing condition when the two images are generated and information on a main subject included in the two images.
In addition, an extraction step of extracting an image of a specific region from the two images is provided, and in the calculation step, the evaluation value is calculated based on the image extracted in the extraction step using the plurality of parameters. You may do it.
In addition, what converted and represented the structure regarding the said invention to the image processing apparatus which aligns two images is also effective as a specific aspect of this invention. An imaging device provided with the image processing device is also effective as a specific aspect of the present invention.

  According to the present invention, it is possible to achieve highly accurate alignment while reducing the possibility of falling into a local solution in alignment of a plurality of digital images.

1 is a conceptual diagram when an image processing method according to an embodiment of the present invention is applied to a computer 10; FIG. It is a flowchart which shows operation | movement of CPU1 in one Embodiment of this invention.

  FIG. 1 is a conceptual diagram when an image processing method according to an embodiment of the present invention is applied to a computer 10.

  A computer 10 shown in FIG. 1 includes a CPU 1, a storage unit 2, an input / output interface (input / output I / F) 3 and a bus 4, and the CPU 1, storage unit 2 and input / output I / F 3 are connected via the bus 4. Connected so that information can be transmitted. The computer 10 is connected to an output device 20 that displays the progress of image processing and processing results, and an input device 30 that receives input from the user via the input / output I / F 3. As the output device 20, a general liquid crystal monitor, a printer, or the like can be used. As the input device 30, a keyboard, a mouse, or the like can be appropriately selected and used.

  The CPU 1 reads an image processing program stored in the storage unit 2 based on an instruction from the user received by the input device 30, and performs image processing on the image stored in the storage unit 2. The CPU 1 displays the result of the image processing on the output device 20 as necessary. A general central processing unit can be used for the CPU 1.

  Image data, programs, and the like stored in the storage unit 2 can be referred to as appropriate from the CPU 1 via the bus 4. A storage device such as a general hard disk device or a magneto-optical disk device can be selected and used for the storage unit 2. 1 is incorporated in the computer 10, it may be an external storage device. In this case, the storage unit 2 is connected to the computer 10 via the input / output I / F 3.

  The operation of the CPU 1 when aligning a plurality of images in the computer 10 having the above configuration will be described with reference to the flowchart of FIG. In the following, description will be made by taking as an example the alignment using the nonlinear least square method in the Levenberg-Marquardt method disclosed in Non-Patent Document 1 described above. In addition, it is assumed that the storage unit 2 stores in advance image data of a plurality of images to be aligned.

  In step S <b> 1, the CPU 1 acquires a first image according to a user instruction via the input device 30. The first image is one of the images to be aligned.

  In step S <b> 2, the CPU 1 acquires a second image according to a user instruction via the input device 30. The second image is the other image among the images to be aligned. The first image and the second image are two images that are adjacent in time series. In the case where there are three or more images to be aligned, two of the images that are adjacent in time series are defined as a first image and a second image, respectively.

  In step S3, the CPU 1 extracts an image of a specific area from the first image acquired in step S1. The specific area is an area composed of all the pixels of the first image or a part of the pixels. The specific area may be specified by a user instruction via the input device 30 or may be automatically specified by the CPU 1. For example, the CPU 1 may designate a specific area according to the result of main subject extraction, or may designate an area composed of pixels having a difference value with an adjacent pixel equal to or greater than a predetermined threshold.

  In step S4, the CPU 1 designates a geometric transformation formula. Here, the case where the geometric transformation formula shown in the following formula 1 is specified will be described as an example.

Expression 1 is an affine transformation expression represented by six parameters from p ₀ to p ₅ . Note that the CPU 1 may specify a geometric transformation expression other than Expression 1. For example, a geometric conversion formula indicated by eight parameters may be specified.

In step S5, the CPU 1 determines parameters in the geometric transformation formula designated in step S4. At the first time, the CPU 1 sets the parameters as initial values. For example, the CPU 1 sets all the parameters p ₀ to p ₅ to zero. This initial value is an example. In general, the solution obtained by the nonlinear least square method is affected by the initial parameters. Therefore, when the alignment target is three or more images, a more suitable initial value is determined by determining the initial value in consideration of the alignment result in the previous image, and the efficiency is improved. Good alignment can be achieved.

  The determination (update) of the parameters after the second time will be described later.

  In step S6, the CPU 1 geometrically transforms the image of the specific area extracted in step S3 based on the geometric transformation formula designated in step S4. At this time, the CPU 1 performs geometric transformation according to the parameters determined in step S5.

  In step S7, the CPU 1 calculates an evaluation value ΔE based on the first image subjected to geometric transformation in step S6 and the second image acquired in step S2. The evaluation value ΔE is an evaluation value indicating the alignment state, and is obtained by the following equation 2.

  In Equation 2, A (f (x, y; p)) indicates the first image subjected to the geometric transformation in step S6, and B (x, y) indicates the second image acquired in step S2. Show. N represents the number of pixels included in the specific area extracted in step S3.

  As the evaluation value ΔE calculated in this way is smaller, the matching degree of alignment is higher. Therefore, the alignment process can be replaced with a process for obtaining a parameter that minimizes the evaluation value ΔE.

  In step S8, the CPU 1 calculates a Jacobian matrix based on the parameters determined in step S5. The Jacobian matrix is obtained by the following Equation 3.

  In step S9, the CPU 1 creates a differential image of the second image acquired in step S2. CPU1 produces a differential image by calculating | requiring the difference with an adjacent image about each of x direction and y direction. The differential image ∇B is obtained by the following equation 4.

  In step S10, the CPU 1 calculates a Hessian matrix based on the Jacobian matrix calculated in step S8 and the differential image created in step S9. The Hessian matrix is obtained by the following Equation 5.

  In step S11, the CPU 1 determines whether or not the evaluation value ΔE calculated in step S7 is smaller than the previously calculated evaluation value ΔE. If the CPU 1 determines that the value of the evaluation value ΔE calculated in step S7 is smaller than the value of the evaluation value ΔE calculated last time, the CPU 1 determines that the alignment is proceeding in the convergence direction and proceeds to step S12. . On the other hand, if it is determined that the evaluation value ΔE calculated in step S7 is not smaller than the previously calculated evaluation value ΔE, it is determined that the alignment has not progressed in the convergence direction, and the CPU 1 will be described later. The process proceeds to step S13.

  In step S12, the CPU 1 calculates a parameter correction value Δp based on the assumption that the alignment is proceeding in the convergence direction. The parameter correction value Δp is obtained by the following equations 6 and 7.

Note that the initial value of σ in Equation 6 is approximately 0.01 according to the Levenberg-Marquardt method disclosed in Non-Patent Document 1. When the CPU 1 calculates _HLM according to Equation 6, σ in Equation 6 is multiplied by 1/10. Note that σ in Expression 6 is a coefficient for adjusting the diagonal matrix of the matrix in Expression 6 according to the degree of convergence of the evaluation value ΔE. When this diagonal matrix becomes small, the Levenberg-Marquardt method becomes a general Gauss-Newton method. On the other hand, as the diagonal matrix increases, the solution approaches in the direction of steepest descent.

  Further, the amount of change of σ in Equation 6 may be changed according to the degree of convergence of the evaluation value ΔE. For example, when the evaluation value ΔE becomes the past minimum value, the evaluation value ΔE is changed to 1/5 times instead of the above-mentioned 1/10 times, or the evaluation value ΔE is smaller than the evaluation value ΔE calculated last time. In this case, the convergence accuracy can be further improved by setting the value to 1/2 times instead of 1/10 times described above.

In step S <b> 13, the CPU 1 calculates a parameter correction value Δp based on the assumption that the alignment has not progressed in the convergence direction. The parameter correction value Δp is calculated in the same manner as in step S12 described above. However, when calculating _HLM according to Equation 6, the CPU 1 multiplies σ in Equation 6 by 10 according to the Levenberg-Marquardt method. After calculating the parameter correction value Δp, the CPU 1 returns to step S5 described above.

  As in step S12, the accuracy of convergence is increased by changing the amount of change in σ in Equation 6 to, for example, “2 times” or “5 times” according to the degree of convergence of the evaluation value ΔE. It can be further increased.

In step S14, the CPU 1 determines whether or not the parameter correction value Δp calculated in step S12 is smaller than a predetermined threshold ε. If the CPU 1 determines that the parameter correction value Δp calculated in step S12 is smaller than the predetermined threshold value ε, the CPU 1 determines that the alignment has converged. Then, the CPU 1 stores a parameter corresponding to the evaluation value ΔE calculated in step S11 and ends a series of processes.
On the other hand, if it is determined that the parameter correction value Δp calculated in step S12 is greater than the predetermined threshold value ε, the CPU 1 determines that the alignment has not converged and returns to step S5 described above.

  Next, the determination (update) of parameters for the second and subsequent times in step S5 will be described. Basically, when determining parameters for the second and subsequent times in step S5, the CPU 1 determines a new parameter according to the previously determined parameter and the parameter correction value Δp calculated in step S12 or step S13. To do.

In the second and subsequent parameter determinations, it is possible to arrive at the solution accurately and quickly by appropriately changing the parameters (parameters p ₀ to p ₅ ). Therefore, in the present embodiment, suitable positioning is realized by adjusting the weight of each parameter in consideration of the characteristic of the parameter.

In the present embodiment, since six parameters (parameters p ₀ to p ₅ , see Equation 1) are used, the Jacobian matrix shown in Equation 3 is expressed by Equation 8 below.

Therefore, the parameter affected by σ described in Expression 6 can be divided into a term depending on x and y representing a pixel position and a term independent of the term. Equation 6 is a geometric transformation equation for the purpose of aligning two images. Therefore, the parameters p ₀ to p ₅ can be divided into terms for translation in the x and y directions and terms that include other rotations. More specifically, in Expression 6, parameters p ₀ to p ₃ are terms related to the change in the rotation direction, parameter p ₄ is a term related to translation in the x direction, and parameter p ₅ is a value in the y direction. This is a term related to the translation of. That is, the influence of the variation of each term on the image after geometric transformation is different.

For example, when the parameter p ₄ = 1, which is a term related to translation in the x direction, the image after geometric transformation is shifted by one pixel in the x direction. However, when the parameter p ₀ = 1, which is a term related to the change in the rotation direction, the image after geometric transformation is rotated 90 degrees.
Therefore, it is expected that the solution is more easily converged by changing σ for each of the six parameters (parameters p ₀ to p ₅ ) than changing the value of σ uniformly. In other words, in consideration of the characteristics of each parameter and the characteristics of the image to be aligned, the parameter weight (≈σ) that can be estimated to have a large effect on the image after geometric transformation is emphasized by increasing the weight. By reducing the weight of the parameter (≈σ) that can be estimated to have a small effect on the image after conversion, it is possible to control the solution to arrive at the solution accurately and quickly.

For example, a continuous shooting image generated by a hand-held (no tripod) electronic camera will be described as an example. Normally, hand-held continuous shooting tends to cause camera shake in the x and y directions, but it is difficult for camera shake in the rotating direction to occur. Therefore, when the object of alignment is a continuous shot image generated by a hand-held electronic camera, the weights (≈σ) of parameters p ₀ to p ₃ that are terms related to the change in the rotation direction are made small or 0. The weights (≈σ) of the parameters p ₄ and p ₅ , which are terms related to translation in the x and y directions, are emphasized.

  Such determination of the weight of each parameter may be performed by a user instruction via the input device 30 or may be automatically performed by the CPU 1. For example, the CPU 1 may determine the weight of each parameter based on the shooting conditions when the first image and the second image acquired in step S1 and step S2 are generated. Further, for example, the CPU 1 analyzes the first image and the second image acquired in step S1 and step S2, and determines the weight of each parameter based on the information about the main subject included in each image. good. Information about the main subject can be obtained by, for example, known subject recognition or scene analysis. In any method, in view of the characteristics of each parameter, the weight is increased for those having a large influence of the parameter, and the weight is decreased for those having a small influence.

As described above, according to the present embodiment, an evaluation value related to alignment is calculated based on two images using a plurality of parameters, and weights of the plurality of parameters are set. Then, a plurality of evaluation values calculated using different set parameters are compared to detect the relative positions of the two images. Accordingly, it is possible to exclude in advance parameters that have little influence or parameters that are likely to fall into a local solution. Therefore, in the alignment of a plurality of digital images, it is possible to realize highly accurate alignment while reducing the possibility of falling into a local solution.
Further, according to the present embodiment, the plurality of parameters include parameters having different characteristics with respect to alignment, and include parameters relating to changes in translation direction in alignment and parameters relating to changes in rotation direction in alignment. Therefore, by adjusting the parameters having less influence among these parameters, it is possible to achieve highly accurate alignment while reducing the possibility of falling into a local solution.

  Further, according to the present embodiment, the weight is set based on at least one of the shooting conditions when the two images are generated and the information regarding the main subject included in the two images. Therefore, by optimizing parameters according to the image to be aligned, highly accurate alignment can be realized while reducing the possibility of falling into a local solution.

  Further, according to the present embodiment, when an image of a specific area is extracted from two images and an evaluation value is calculated, the evaluation value is calculated based on the extracted image using a plurality of parameters. . Therefore, the processing load can be reduced and the time required for the processing can be shortened.

Note that the alignment method described in the present embodiment (Levenberg-Marquardt method disclosed in Non-Patent Document 1) is an example, and the present invention is not limited to this example. For example, instead of the six parameters (parameters p ₀ to p ₅ ) described in this embodiment, eight parameters including a parameter relating to a change in translation direction, a parameter relating to a change in rotation direction, and a parameter relating to enlargement / reduction are set. The present invention can be similarly applied to a method of calculating an evaluation value for alignment using the same. Further, the present invention can be similarly applied to other methods for calculating an evaluation value for alignment using a plurality of parameters including parameters having different characteristics regarding alignment.
In the present embodiment, the image to be aligned may be any image such as a RAW image, RGB image, YCbCr image, or JPEG image. For example, in the RGB image, the G component may represent the image structure best, and the above-described processing may be performed only on the G component. Also, for example, in a YCbCr image, the above-described processing may be performed only on the Y component, assuming that the Y component best represents the image structure. Moreover, it is good also as a structure which performs the process mentioned above independently with respect to several components, and integrates the result and performs the process of whole position alignment.
In this embodiment, the computer 10 that performs alignment of two images has been described as an example. However, the present invention is not limited to this example. An image processing program for executing part or all of the processing of the flowchart shown in FIG. 2 is also effective as a specific aspect of the present invention. This image processing program may be recorded on a medium, or may be recorded on a server on the Internet and downloaded via the Internet.

  An imaging device that performs the processing described in this embodiment is also effective as a specific aspect of the present invention. In such an imaging apparatus, the same effect can be obtained by performing the processing described in the present embodiment in the alignment of a plurality of images generated by imaging. Note that when applied to the imaging apparatus, the image to be aligned may be a continuous shot image or a partial image of a moving image (including a so-called through image for composition confirmation).

DESCRIPTION OF SYMBOLS 1 ... CPU, 2 ... Memory | storage part, 10 ... Computer, 20 ... Output device, 30 ... Input device

“Lucas-Kanade 20 Years On: A Unifying Framework: Part 1” Simon Baker and Ian Matthews

Claims (13)

An image processing program for realizing alignment of two images by a computer,
An acquisition step of acquiring the two images;
A calculation step of calculating an evaluation value related to the alignment based on the two images using a plurality of parameters;
A setting step for setting the weight of each of the plurality of parameters;
A detection step of detecting a relative position of the two images by comparing the plurality of evaluation values calculated in the calculation step using the different parameters set in the setting step. An image processing program. In the image processing program according to claim 1,
The image processing program characterized in that the plurality of parameters include parameters having different characteristics with respect to the alignment. The image processing program according to claim 2,
The plurality of parameters include at least two parameters of a parameter relating to a change in a translation direction in the alignment, a parameter relating to a change in a rotation direction in the alignment, and a parameter relating to enlargement / reduction in the alignment. An image processing program. In the image processing program according to claim 1,
The plurality of parameters are parameters for geometrically transforming the two images,
In the calculation step, the evaluation value is calculated after geometrically transforming at least one of the two images using the plurality of parameters. In the image processing program according to claim 1,
In the setting step, the weight is set based on at least one of a photographing condition when the two images are generated and information on a main subject included in the two images. program. In the image processing program according to claim 1,
An extraction step of extracting an image of a specific region from the two images,
In the calculation step, the evaluation value is calculated based on the image extracted in the extraction step using the plurality of parameters. An image processing apparatus for aligning two images,
An acquisition unit for acquiring the two images;
A calculation unit that calculates an evaluation value related to the alignment based on the two images using a plurality of parameters;
A setting unit for setting the weight of each of the plurality of parameters;
A detection unit that detects a relative position of the two images by comparing the plurality of evaluation values calculated by the calculation unit using the different parameters set by the setting unit. An image processing apparatus. The image processing apparatus according to claim 7.
The plurality of parameters include parameters having different characteristics with respect to the alignment. The image processing apparatus according to claim 8.
The plurality of parameters include at least two parameters of a parameter relating to a change in a translation direction in the alignment, a parameter relating to a change in a rotation direction in the alignment, and a parameter relating to enlargement / reduction in the alignment. An image processing apparatus. The image processing apparatus according to claim 7.
The plurality of parameters are parameters for geometrically transforming the two images,
The image processing apparatus, wherein the calculation unit calculates the evaluation value after geometrically converting at least one of the two images using the plurality of parameters. The image processing apparatus according to claim 7.
The setting unit sets the weight based on at least one of a photographing condition when the two images are generated and information on a main subject included in the two images. apparatus. The image processing apparatus according to claim 7.
An extraction unit that extracts an image of a specific region from the two images;
The calculation unit calculates the evaluation value based on the image extracted by the extraction unit using the plurality of parameters. The image processing apparatus according to any one of claims 7 to 12,
An imaging unit that captures a subject image and generates an image,
The said acquisition part acquires the image produced | generated by the said imaging part as said two images. The imaging device characterized by the above-mentioned.

Images