JP2008065530A - Image processor and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To form a composite image with high quality to obtain sufficient composite images. <P>SOLUTION: An image pickup device is provided with an image composition part 134 for operating the projection transformation of the pixels of images to be synthesized into the coordinate system of a composition reference image by using a projection transformation matrix H when it is decided that the reliability of the projection transformation matrix H is high by an H decision part 133 of a video signal processing part 13, and synthesizing the images to be synthesized with the composition reference image by shifting the pixels of the images to be synthesized according to the composition reference image and the deviation value of the mutual low resolution images of the images to be synthesized calculated by global matching when it is decided that the reliability of the projection transformation matrix H is not high. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and a program for synthesizing a plurality of images.

  In recent years, in an imaging apparatus such as a digital camera, continuous imaging is performed with a short exposure time that does not cause camera shake, and a plurality of images acquired by the continuous imaging are superimposed to achieve a clear image with no blurring noise. Images can be acquired.

As a technique for preventing image misalignment in image synthesis, a projective transformation matrix H is calculated from an optical flow between two images, and the misalignment of each pixel between the images is corrected by the projective transform matrix H and superimposed. A method has been developed (see, for example, Patent Document 1).
Specifically, as illustrated in FIG. 9, the optical flow detection unit 501 provided in the imaging apparatus extracts n feature points from the first image and tracks them with the second and subsequent images. Here, the feature points are tracked by sub-sampling each image in a plurality of stages and sequentially tracking from the layer (image) having the lowest resolution toward the layer having the highest resolution.
When the feature point tracking is completed, the RANSAC processing unit 502 (see FIG. 9) randomly selects four points from the acquired n corresponding points, calculates a temporary projective transformation matrix H, and supports it. The projection transformation matrix H obtained by a number is evaluated, and the operation is repeated to calculate a projection transformation matrix H with higher reliability (RANSAC processing). In this way, the projection transformation matrix H is determined using the four pairs of corresponding points with the highest reliability.
Then, the composite image is projectively transformed using the determined projective transformation matrix H, and one composite image is created by adding and compositing.
JP 2005-94614 A

By the way, in the image composition method described above, it is important to obtain a projective transformation matrix H with high reliability. In other words, it is necessary to track feature points with high accuracy.
Here, if the environment in which the image is captured is a sufficiently bright environment, the feature points can be extracted and tracked relatively easily, and the projection transformation matrix H with high reliability can be obtained. However, for example, as shown in FIG. 10B, an image captured in a very dark environment such as 5 lux or less has a very small component for each image, and the feature points are buried in noise. It becomes difficult to extract and track the. Therefore, the projection transformation matrix H itself cannot be calculated in a dark environment, or even if it can be calculated, the projection transformation matrix H becomes low in reliability.

  For example, as illustrated in FIG. 10C, when an image is converted using an incorrect projective transformation matrix H in a case where a continuous image is significantly shifted due to camera shake or the like during continuous imaging. Since there is a possibility that the images will be significantly shifted from each other rather than overlapping, the H determination unit 503 (see FIG. 9) determines the reliability of the projective transformation matrix H before the image synthesis by the image synthesis unit 504, and the reliability In the case of a low projection transformation matrix H, a method of avoiding such as not using it for image synthesis is performed. Similarly, when the projective transformation matrix H itself cannot be calculated, a method is employed in which it is not used for image synthesis.

In the image composition method described above, composition is performed using only images of sufficient quality that can be composed, so that highly accurate alignment can be performed, and as a result, a clear image can be obtained.
However, for example, when a highly reliable projection transformation matrix H is not obtained for most of the captured images (or when the projection transformation matrix H itself is not calculated), these images are used for synthesis. Therefore, the number of synthesized images is smaller than the number of continuously captured images. As a result, the synthesized image may not be reduced in noise as expected or the brightness may be insufficient.

Further, the processing time required for synthesis is the same as when the projection transformation matrix H with high reliability is calculated and synthesized with all the continuous images, and without using the image with the projection transformation matrix H with low reliability. Almost the same as when synthesized.
Therefore, image composition processing that does not simply use unreliable images that are not suitable for image composition requires a long processing time and reduces noise even for composite images that are aligned with high accuracy. There is a risk that low quality images such as unclear images, dark images, or poorly colored images may be synthesized.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an image processing apparatus and a program that can synthesize a high-quality image with a sufficient number of images to be combined.

The image processing apparatus according to the first aspect of the invention (for example, the imaging apparatus 100 of FIG. 1)
Among the plurality of images, the resolution of one image (for example, the composite reference image P0 in FIG. 3) and another image different from the one image (for example, the composite image Pn in FIG. 3) is gradually reduced. Low-resolution image creation means for creating one low-resolution image and another low-resolution image (for example, the optical flow detection unit 131 in FIG. 2);
First shift amount calculation means for calculating a shift amount of a position between the one low resolution image created by the low resolution image creation means and the other low resolution image by a first calculation method (for example, Optical flow detector 131 in FIG. 2),
Second shift amount calculating means (for example, calculating a shift amount of a position between the one low resolution image created by the low resolution image creating means and the other low resolution image by a second calculation method (for example, RANSAC processing unit 132 in FIG.
Reliability determination means for determining whether or not the deviation amount calculated by the second deviation amount calculation means is high (for example, an H determination unit 133 in FIG. 2);
When it is determined by the reliability determination unit that the reliability of the shift amount calculated by the second shift amount calculation unit is high, based on the shift amount calculated by the second shift amount calculation unit, A first image synthesizing unit (for example, the image synthesizing unit 134 shown in FIG. 2) that coordinates the pixels of one of the one image and the other image to synthesize the two images; ,
Based on the deviation amount calculated by the first deviation amount calculating means when the reliability judgment means determines that the reliability of the deviation amount calculated by the second deviation amount calculating means is not high. A second image synthesizing unit (for example, the image synthesizing unit 134 in FIG. 2) that synthesizes the two images by shifting pixels of one of the one image and the other image. It is characterized by providing.

The invention according to claim 2 is the image processing apparatus according to claim 1,
The first deviation amount calculating means is characterized in that the second deviation amount calculating means calculates the deviation amount for an image having a resolution lower than that of an image for which the deviation amount is to be calculated.

The invention according to claim 3 is the image processing apparatus according to claim 1,
The first deviation amount calculation means calculates the deviation amount by searching for global matching between the one low resolution image and the other low resolution image.

According to a fourth aspect of the present invention, in the image processing apparatus according to the first aspect,
The second shift amount calculating means calculates the shift amount by searching the first low resolution image and the other low resolution image by a gradient method.

The invention according to claim 5 is the image processing apparatus according to claim 1,
Deviation amount determination means (for example, determining whether or not the deviation amount of the position of the other image with respect to the one image serving as the composite reference image calculated by the first deviation amount calculation means is a predetermined value or more) Optical flow detector 131 in FIG. 2),
When the deviation amount determination unit determines that the deviation amount is equal to or greater than a predetermined value, the other image is excluded from the target of image synthesis by the first image synthesis unit and the second image synthesis unit. And an image excluding unit (for example, the optical flow detecting unit 131 in FIG. 2).

The program of the invention described in claim 6 is:
In a computer (for example, the imaging apparatus 100 in FIG. 1),
Among the plurality of images, the resolution of one image (for example, the composite reference image P0 in FIG. 3) and another image different from the one image (for example, the composite image Pn in FIG. 3) is gradually reduced. The ability to create one low resolution image and another low resolution image,
A function of calculating a positional shift amount between the one low resolution image and the other low resolution image by a first calculation method;
A function of calculating a positional shift amount between the one low resolution image and the other low resolution image by a second calculation method;
A function of determining whether or not the deviation amount calculated by the second calculation method is highly reliable;
When it is determined that the reliability of the shift amount calculated by the second calculation method is high, the one image and the other image are calculated based on the shift amount calculated by the second calculation method. A function of coordinate-converting pixels of one of the images and synthesizing the two images;
When it is determined that the reliability of the shift amount calculated by the second calculation method is not high, the one image and the other are based on the shift amount calculated by the first calculation method. And a function of synthesizing the two images by shifting pixels of one of the images.

  According to the present invention, it is possible to synthesize images while maintaining a minimum synthesis accuracy even for a synthesized image that is not highly reliable in the amount of deviation with respect to the synthesis reference image, thereby sufficiently securing the number of images to be synthesized. Thus, a high quality image can be synthesized.

Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.
FIG. 1 is a diagram schematically illustrating a schematic configuration of an imaging apparatus 100 according to an embodiment to which the present invention is applied.

The imaging apparatus 100 according to the present embodiment performs, for example, continuous imaging with a short exposure time, and generates a single composite image by superimposing a plurality of image frames acquired by the continuous imaging.
Specifically, for example, as illustrated in FIG. 1, the imaging apparatus 100 includes an imaging unit 1 that images a subject, an imaging auxiliary unit 2 that is driven when the subject is imaged by the imaging unit 1, and an imaging unit 1. The display unit 3 for displaying the image captured by the operation unit 4, the operation unit 4 for performing a predetermined operation of the imaging apparatus 100, the recording medium 5 for recording the captured image, and the USB terminal 6 for connection to an external device. And a control unit 7 for controlling these units.

  The imaging unit 1 has, for example, a focus function and a zoom function. The imaging lens group 11 includes a plurality of imaging lenses, and a CCD (Charge) that converts a subject image that has passed through the imaging lens group 11 into a two-dimensional image signal. An electronic imaging unit 12 made of a coupled device (CMOS) or a complementary metal-oxide semiconductor (CMOS), a video signal processing unit 13 for performing predetermined image processing on an image signal output from the electronic imaging unit 12, and image processing An image memory 14 for temporarily storing subsequent image signals, a photographing control unit 15 for controlling the electronic imaging unit 12 and the video signal processing unit 13 under the control of the CPU 71, and the like are provided.

  Further, for example, when the imaging unit 1 is switched to the continuous imaging mode based on a predetermined operation of the mode setting button 41b of the operation unit 4 by the user, the imaging unit 1 can continuously perform imaging at a high speed. ing.

  For example, as shown in FIG. 2, the video signal processing unit 13 uses a pyramid hierarchical structure of image signals related to a plurality of images to be synthesized, extraction of feature points from a predetermined image, and a gradient method of extracted feature points. An optical flow detection unit 131 that performs tracking (search) and the like, and a predetermined number of feature points are randomly selected from a feature point correspondence table in which feature points are associated with each other, and projective transformation (H Calculation) and a RANSAC processing unit 132 that performs support calculation using other feature points, an H determination unit 133 that determines the reliability of the calculated projective transformation matrix H, and a position where pixels correspond to each other. An image composition unit 134 for adding (combining) image signals is provided.

The imaging auxiliary unit 2 includes, for example, a focus driving unit 21 for driving a focus mechanism unit (not shown) connected to the imaging lens group 11 and a zoom mechanism unit (not shown) connected to the imaging lens group 11. A zoom driving unit 22 for driving, a strobe driving unit 24 for causing the strobe light emitting unit 23 to emit light, and the like are provided.
The focus driving unit 21, zoom driving unit 22, and strobe driving unit 24 are connected to the imaging control unit 15, for example, and are driven under the control of the imaging control unit 15.

  The display unit 3 includes, for example, a display control unit 31 including a video memory (VRAM) that temporarily stores display data that is appropriately output from the CPU 71, and a predetermined image based on an output signal from the display control unit 31. An image display unit 32 such as a liquid crystal monitor for display is provided.

  The operation unit 4 is provided in each unit of the imaging device 100, for example, and includes a shutter button 41a for instructing imaging of a subject by the imaging unit 1, and a mode for switching and setting the imaging mode to a normal imaging mode, a continuous imaging mode, or the like. An operation input unit 41 including various operation switches such as a setting button 41b, and an input circuit 42 for inputting an operation signal input from the operation input unit 41 to the CPU 71 are provided.

  As the recording medium 5, for example, a card-type nonvolatile memory (flash memory), a hard disk, or the like can be applied.

  The control unit 7 includes, for example, a CPU 71 that comprehensively controls each unit of the imaging apparatus 100, a program memory 72 that stores various programs and data necessary for the operation of the CPU 71, and data that is a built-in flash memory for storing image data. A memory 73 and the like are provided.

  The program memory 72 includes, for example, a pyramidization program 72a, a deviation amount calculation program 72b, a deviation amount determination program 72c, an image exclusion program 72d, a coordinate conversion formula calculation program 72e, a reliability determination program 72f, a first image composition program 72g, The second image composition program 72h and the like are stored.

The pyramidization program 72 a is based on a plurality of image signals input to the video signal processing unit 13 by capturing an image by the imaging unit 1 under the control of the imaging control unit 15 driven according to a predetermined control signal from the CPU 71. Thus, for example, the pyramid hierarchical structuring process for making the layers Gm (G′m) to G0 (G′0) into a pyramid hierarchical structure (see FIG. 3) by sequentially halving the number of vertical and horizontal pixels of the image It is a program for realizing such a function.
That is, based on the execution of the pyramidal program 72a by the CPU 71, the optical flow detection unit 131 of the video signal processing unit 13 serves as a low-resolution image creation unit based on a plurality of image signals based on the image signal (for example, A low-resolution image (for example, one low-resolution image, another low-resolution image, etc.) in which the resolution of one image (composite reference image) P0, another image Pn, etc.) is gradually reduced is created. ing.

The shift amount calculation program 72b is one low-resolution image G0 created by the pyramid hierarchical structuring process in the video signal processing unit 13 under the control of the imaging control unit 15 driven according to a predetermined control signal from the CPU 71. Is a program for realizing a function related to a shift amount calculation process of searching for and calculating a shift amount of a position between the image and another low-resolution image G′0 by global matching.
In other words, based on the execution of the shift amount calculation program 72b by the CPU 71, the optical flow detection unit 131 of the video signal processing unit 13 serves as the first shift amount calculation means with the highest resolution of the one image P0 that becomes the composite reference image. A first calculation method for calculating a shift amount between images by performing global matching between the low-resolution image G0 having a low resolution and the low-resolution image G′0 having the lowest resolution among the other images Pn is performed. ing.

Here, the deviation amount calculation processing will be described with reference to FIG.
As shown in FIG. 4, the first image of a plurality of consecutive images is used as a composite reference image P0, and coordinates (x, y) [0 <x <xmax, For 0 <y <ymax], the pixel value is P0 (x, y). Further, the pixel value of the composite image Pn is Pn (x, y) [n is the number of continuous shots].
When coordinates (x + Δx, y + Δy) satisfying P0 (x, y) = Pn (x + Δx, y + Δy) are searched in the composite image with respect to P0 (x, y) of the composite reference image, Δx and Δy are obtained. This is the movement amount (deviation amount) of a pixel (x, y) in the composite reference image.
In the above processing, Δx and Δy are calculated for all the pixels of the composite reference image, and the average is used as the movement amount (Δxn, Δyn) of the composite image Pn, and the size D is calculated from the following equation (1). Is done.

The shift amount determination program 72c is used by the video signal processing unit 13 under the control of the shooting control unit 15 driven according to a predetermined control signal from the CPU 71, and a composite reference image (one image calculated by the shift amount calculation process). ) A program for realizing a function relating to a deviation amount determination process for determining whether or not the deviation amount of the composite image (other image) Pn with respect to P0 is equal to or greater than a predetermined value.
That is, based on the execution of the shift amount determination program 72c by the CPU 71, the optical flow detection unit 131 of the video signal processing unit 13 serves as a shift amount determination unit, and whether or not the movement amount D of the composite image Pn is equal to or greater than the threshold T. It is to judge whether.

The image excluding program 72d applies to the video signal processing unit 13 under the control of the photographing control unit 15 driven according to a predetermined control signal from the CPU 71, with respect to the composite reference image (one image) P0 in the deviation amount determination process. When it is determined that the amount of deviation of the composite image (other image) Pn is equal to or greater than a predetermined value, the image synthesis target in the first image synthesis process and the second image synthesis process (described later) is changed to another image. This is a program for realizing a function related to an image exclusion process for excluding images.
In other words, based on the execution of the image exclusion program 72d by the CPU 71, the optical flow detection unit 131 of the video signal processing unit 13 uses the first image as the synthesized image Pn with the movement amount D equal to or greater than the threshold T as the image exclusion unit. It is excluded from the subject of image composition in the composition processing and the second image composition processing. That is, when the amount of movement D of the synthesized image Pn is equal to or greater than the threshold T, the synthesized image Pn is deviated from P0 so that it is expected that the feature point is difficult to be tracked. The feature point is not tracked, and the composite image is discarded at this point.
When the movement amount D of the combined image Pn is smaller than the threshold value T, it indicates that the displacement of the combined image is small. Therefore, there is a possibility that sufficiently accurate feature point tracking can be performed. The tracking of feature points is continued with (x + Δx, y + Δy) of G1) as a starting point.
As a result, for example, an image that is significantly shifted as shown in FIG. 10C is excluded from the synthesis targets.

The coordinate conversion formula calculation program 72e becomes a composite reference image created by the pyramid hierarchical structuring process in the video signal processing unit 13 under the control of the imaging control unit 15 driven according to a predetermined control signal from the CPU 71. Implements a function related to coordinate conversion formula calculation processing that calculates the coordinate conversion formula (shift amount between images) of a pixel between one low-resolution image and one image that is to be synthesized and another low-resolution image. It is a program to make it.
That is, based on the execution of the coordinate conversion formula calculation program 72e by the CPU 71, the RANSAC processing unit 132 of the video signal processing unit 13 is searched and acquired by the optical flow detection unit 131 as a second deviation amount calculation means by the gradient method. 4 points are randomly selected from a plurality of corresponding points corresponding to the synthesized reference image and the synthesized image, a temporary projective transformation matrix H is calculated, and the projective transformation matrix H obtained by the support number is evaluated. Then, the process is repeated (RANSAC process) to calculate the projection transformation matrix H with higher reliability.

The reliability determination program 72f provides the video signal processing unit 13 with the reliability of the coordinate conversion formula calculated by the coordinate conversion formula calculation process under the control of the imaging control unit 15 driven according to a predetermined control signal from the CPU 71. This is a program for realizing a function related to the reliability determination process for determining whether or not the value is high.
That is, based on the execution of the reliability determination program 72f by the CPU 71, the H determination unit 133 of the video signal processing unit 13 uses, as the reliability determination unit, the projective transformation matrix H calculated by the coordinate conversion formula calculation process. The reliability of the projective transformation matrix H based on the amount of distortion (for example, the length of each side or the angle of each vertex) when a feature point (for example, each vertex) on one image is converted to the coordinate system of another image. It is determined whether or not the property is high.

Here, the reliability determination process will be described with reference to FIGS.
For example, as shown in FIG. 5, if an image in which all the pixels of the composite reference image P0 correspond to the coordinate system of the composite image Pn is P′n, the composite reference image P0 and the composite image Pn are the same coordinate system. If the image P′n is not distorted, the image P′n matches the synthesized image Pn.
However, when the image P′n is distorted (see the broken line), the larger the distortion, the larger the coordinate shift.
Hereinafter, a method of measuring the magnitude of the distortion will be described.

For example, as shown in FIG. 6, the pixel at each vertex of the composite reference image P0 is (a, b, c, d), and the pixel at each vertex of the composite image Pn is (e, f, g, h). The vertex pixels of the image P′n when the vertex pixels of the composite reference image P0 are projectively transformed into the coordinate system of the composite image Pn are defined as (a ′, b ′, c ′, d ′).
Then, the lengths of the four sides (a′−b ′), (b′−c ′), (c′−d ′), and (d′−a ′) of the image P′n after the projective transformation and each vertex The angles of a ′, b ′, c ′, d ′ are calculated. As a result, the positional relationship between the vertex pixels (a ′, b ′, c ′, d ′) of the image P′n is known, and the four sides of the composite reference image P0 before conversion and the vertices (a, b, c, By comparing with the angle d), the distortion amount when the composite reference image P0 is converted into the coordinate system of the composite image Pn can be quantified.
It is determined whether or not the distortion amount value calculated as described above is equal to or greater than a preset threshold value. If it is determined that the distortion amount is within the threshold value, the projection transformation matrix H has high reliability. On the other hand, if it is determined that the distortion amount is larger than the threshold value, it is determined that the reliability of the projective transformation matrix H is low.

The first image composition program 72g is highly reliable in the coordinate conversion formula in the reliability determination process in the video signal processing unit 13 under the control of the imaging control unit 15 driven according to a predetermined control signal from the CPU 71. If it is determined, based on the coordinate conversion formula calculated in the coordinate conversion formula calculation process, the pixel of another image (composite image) is subjected to coordinate conversion and the one image (composite reference) This is a program for realizing a function related to the first image composition processing for composing images.
That is, based on the execution of the first image composition program 72g by the CPU 71, the image composition unit 134 uses the projective transformation matrix H as a first image composition means to convert the pixels of the composition image Pn into the composition reference image P0. Projective transformation is performed on the coordinate system to synthesize the composite image Pn and the composite reference image P0.
In the first image composition process, the pixels of the composition reference image may be projectively transformed into the coordinate system of the composition image.

The second image composition program 72h is highly reliable in the coordinate conversion formula in the reliability determination process in the video signal processing unit 13 under the control of the photographing control unit 15 driven according to a predetermined control signal from the CPU 71. If it is determined that there is not, the pixel of the composite image Pn is shifted according to the shift amount of the composite image (other image) Pn with respect to the composite reference image (one image) P0 calculated by the shift amount calculation process. This is a program for realizing a function related to the second image composition processing for compositing the composite image Pn and the composite reference image P0.
That is, based on the execution of the second image composition program 72h by the CPU 71, the image composition unit 134, as the second image composition means, is the deviation amount of the composition image Pn with respect to the composition reference image P0 calculated by global matching. Using (Δxn, Δyn), the synthesized image Pn and the synthesized reference image P0 are synthesized.

Thereby, for example, an image that is greatly deviated or an image for which the projection transformation matrix H could not be calculated is excluded from the synthesis target in the image exclusion process, as shown in FIG. Even if an image could not be properly tracked due to the dark shooting environment, a high-quality image can be synthesized by securing a sufficient number of images without excluding it from the compositing target. .
In the second image composition process, the pixels of the composition reference image may be shifted to perform projective transformation on the coordinate system of the composition image.

Next, the image composition process will be described with reference to FIGS.
Here, FIG. 7 is a flowchart showing an example of an operation related to the image composition processing. FIG. 8 is a flowchart showing an example of operations related to the main composition process in the image composition process.

As shown in FIG. 7, first, images are continuously captured by the imaging unit 1 based on a pressing operation of the shutter button 41 a by the user in the continuous imaging mode, and the image signal is input to the video signal processing unit 13. (Step S <b> 1), the video signal processing unit 13 performs processing for forming an input image signal into a pyramid hierarchical structure based on the execution of the pyramidal program 72 a by the CPU 71.
Specifically, the video signal processing unit 13 sets “m” as the initial value of the variable i (step S2), and sequentially sub-samples the original image by reducing the resolution by 1/2 times (step S2). S3) and generate layer G (i-1) (step S4).
Subsequently, the video signal processing unit 13 determines whether or not the variable i is 0 or more (step S5). When it is determined that the variable i is 0 or more (step S5; YES), the video signal processing unit 13 sets the variable i to “−”. 1 "After decrementing (step S6), the process proceeds to step S3.
The video signal processing unit 13 repeatedly executes the above processing until it is determined in step S5 that the variable i is not 0 or more (step S5; NO).
Accordingly, a pyramid hierarchical structure is generated in which the lowest resolution image is the layer G0 and the highest resolution image (original image) is the layer Gm.

  If it is determined in step S5 that the variable i is not equal to or greater than 0 (step S5; NO), the video signal processing unit 13 determines the most of the composite reference image P0 based on the execution of the deviation amount calculation program 72b by the CPU 71. Global matching is performed between the low-resolution image G0 and the lowest-resolution image G′0 of the synthesized image Pn, and processing for calculating the movement amount D (shift amount) for all the pixels is performed (step S7).

Subsequently, the video signal processing unit 13 determines whether or not the movement amount D of the composite image Pn with respect to the composite reference image P0 is greater than or equal to the threshold T based on the execution of the shift amount determination program 72c by the CPU 71. Perform (step S8).
Here, if it is determined that the movement amount D is equal to or greater than the threshold T (step S8; YES), the video signal processing unit 13 converts the synthesized image Pn into an image based on the execution of the image exclusion program 72d by the CPU 71. A process of excluding from the synthesis process target is performed (step S9).

On the other hand, when it is determined in step S8 that the movement amount D is smaller than the threshold value T (step S8; NO), the video signal processing unit 13 performs processing for tracking feature points.
Specifically, the video signal processing unit 13 sets “1” as an initial value of the variable i (step S10), and a predetermined feature point in the layer Gi of the composite reference image P0 is, for example, a composite image Pn by a gradient method. Are tracked in the layer G′i (step S11).
Then, after the video signal processing unit 13 passes the tracking result in the layer G′i to the next layer G ′ (i−1) (step S12), the video signal processing unit 13 determines that the variable i is m or less. Is determined (step S13). If it is determined that it is equal to or less than m (step S13; YES), the variable i is incremented by “+1” (step S14), and then the process proceeds to step S11.
The video signal processing unit 13 repeatedly executes the above processing until it is determined in step S13 that the variable i is not less than or equal to m (step S13; NO).
As a result, the feature points are sequentially tracked by the gradient method in order from the image of the layer G1 having the next highest resolution to the layer G1 having the next highest resolution.

  If it is determined in step S13 that the variable i is not less than or equal to m (step S13; NO), the video signal processing unit 13 executes the RANSAC process based on the execution of the coordinate conversion equation calculation program 72e by the CPU 71. Thus, a more reliable projective transformation matrix H is calculated (step S14).

Subsequently, the video signal processing unit 13 performs a synthesis main process for synthesizing the synthesis reference image P0 and the synthesized image Pn (step S15).
Hereinafter, the synthesis main process will be described with reference to FIG.

As shown in FIG. 8, the video signal processing unit 13 determines whether or not the projection transformation matrix H calculated in the coordinate transformation formula calculation process is high based on the execution of the reliability determination program 72 f by the CPU 71. Determination is made (step S151).
Here, when it is determined that the reliability of the projective transformation matrix H is high (step S151; YES), the video signal processing unit 13 performs the projective transformation matrix H based on the execution of the first image composition program 72g by the CPU 71. Is used to projectively transform the pixels of the synthesized image Pn into the coordinate system of the synthesized reference image P0, and the synthesized image Pn and the synthesized reference image P0 are synthesized (step S152).

  On the other hand, if it is determined in step S151 that the reliability of the projective transformation matrix H is not high (step S151; NO), the video signal processing unit 13 is based on the execution of the second image synthesis program 72h by the CPU 71. The synthesized image Pn and the synthesized reference image P0 are synthesized using the shift amount (Δxn, Δyn) of the synthesized image Pn with respect to the synthesized reference image P0 calculated by the global matching of the low-resolution image (step S153).

  Then, the video signal processing unit 13 outputs the combined image to the image memory and the CPU 71 (step S154), and ends the image combining process.

As described above, according to the imaging apparatus 100 of the present embodiment, when the H determination unit 133 determines that the reliability of the projective transformation matrix H is not high, the combined reference image P0 calculated by global matching and The synthesized image Pn and the synthesized reference image P0 can be synthesized by shifting the pixels of the synthesized image Pn according to the shift amount of the low-resolution images of the synthesized image Pn.
That is, conventionally, since the imaging environment is extremely dark, it is difficult to extract and track feature points, and when an incorrect projective transformation matrix H is calculated, the reliability of the projective transformation matrix H becomes low, and thus image synthesis is performed. However, in this case, there is a possibility that there is no deviation between the continuously captured images. Therefore, according to the imaging apparatus 100 of the present embodiment, it is possible to synthesize an image while maintaining a minimum synthesis accuracy even for a synthesized image Pn whose projection transformation matrix H is not highly reliable. A sufficient number of composites can be secured.
Therefore, the synthesized image does not give priority to the synthesis performance or resolution, and it is possible to acquire a high-quality image with little noise, bright and good color development.

  Further, when it is determined that the deviation amount between the low-resolution images of the synthesis reference image P0 and the synthesized image Pn calculated by the global matching is equal to or greater than the threshold T, the synthesized image Pn is excluded from the synthesis target. Therefore, with respect to the composite image Pn, subsequent image processing can be omitted, and the image processing time can be reduced. That is, instead of determining whether or not to use for image synthesis based on the calculated projective transformation matrix H as in the prior art, selecting a composite image in advance before calculating the projective transformation matrix H. Image processing efficiency can be improved.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the above-described embodiment, as a first calculation method of the positional deviation amount between the images, the low-resolution image G0 of the composite reference image and the low-resolution image G′0 of the other image Pn in the deviation amount calculation processing. While the global matching is performed between the images to calculate the shift amount between the images, the calculation method of the shift amount is not limited to this.
Also, the amount of shift between images is calculated between the low-resolution image G0 having the lowest resolution of one image P0 and the low-resolution image G′0 having the lowest resolution of the other image Pn. The image that is the target of deviation amount calculation in the amount calculation processing is not limited to this. That is, the target image in the shift amount calculation process may be any image as long as the image has a lower resolution than the image that is the target of the coordinate conversion formula (shift amount) calculation in the coordinate conversion formula calculation process.

Furthermore, as a second method of calculating the positional deviation amount between images, the coordinate transformation formula of the pixel between the composite reference image and the synthesized image is calculated as the deviation amount in the coordinate transformation formula calculation process. The method for calculating the amount of deviation is not limited to this.
Further, in the coordinate conversion formula calculation process, the feature point is searched by the gradient method to estimate the optical flow and calculate the projective transformation matrix H. However, the method for calculating the projective transformation matrix H is limited to this. It is not a thing.

  In addition, the H determination unit 133 determines whether to use a projection transformation matrix H for each of a plurality of images to be combined or to use a movement amount (motion vector) calculated by global matching. However, the present invention is not limited to this. For example, when the first synthesized image Pn is an extremely dark image or a sufficient number of feature points cannot be obtained, the feature amount of feature points (feature points and If the determination amount is not sufficient), all subsequent synthesis may be performed by global matching.

  In the above embodiment, the imaging apparatus 100 that synthesizes a plurality of captured images is exemplified as the image processing apparatus. However, the imaging apparatus 100 is not limited to this, and a composite image is generated based on at least the plurality of images. Any device can be used.

  In addition, in the above embodiment, the coordinate conversion formula calculation unit, the reliability determination unit, the first image synthesis unit, the low resolution image creation unit, the deviation amount calculation unit, the second image synthesis unit, the deviation amount determination unit, the image The function as the excluding means is realized by executing a predetermined program or the like by the CPU 71, but is not limited to this, for example, a logic circuit or the like for realizing various functions You may do it.

It is a block diagram which shows schematic structure of the imaging device of one Embodiment to which this invention is applied. It is the figure which showed typically the video signal processing part with which the imaging device of FIG. 1 is equipped. It is the figure which showed typically the pyramid hierarchy structuring process which concerns on the image tracking process by the imaging device of FIG. It is a figure for demonstrating the deviation amount calculation processing by the imaging device of FIG. It is a figure for demonstrating the reliability determination process by the imaging device of FIG. It is a figure for demonstrating calculation of the magnitude | size of the distortion in the reliability determination process of FIG. 3 is a flowchart illustrating an example of an operation related to an image composition process performed by the imaging apparatus of FIG. 1. It is a flowchart which shows an example of the operation | movement concerning the synthetic | combination main process in the image synthetic | combination process of FIG. It is the figure which showed typically the video signal processing part with which the conventional image processing apparatus is equipped. It is a figure for demonstrating the propriety of the image composition process by the image processing apparatus of FIG.

Explanation of symbols

100 Imaging device (image processing device)
13 Video signal processing unit 131 Optical flow detection unit (low resolution image creation means, first deviation amount calculation means)
132 RANSAC processing unit (second deviation amount calculating means)
133 H determination part (reliability determination means)
134 Image composition section (first image composition means, second image composition means)
P0 Composite reference image Pn Composite image G0 (G'0) Low resolution image

Claims (6)

Low resolution image creation means for creating one low resolution image and another low resolution image by gradually reducing the resolution of one image and another image different from the one image among the plurality of images;
First shift amount calculation means for calculating a shift amount of the position between the one low resolution image created by the low resolution image creation means and the other low resolution image by a first calculation method;
Second shift amount calculating means for calculating a shift amount of a position between the one low resolution image created by the low resolution image creating means and the other low resolution image by a second calculation method;
Reliability determination means for determining whether or not the reliability of the deviation amount calculated by the second deviation amount calculation means is high;
Based on the deviation amount calculated by the second deviation amount calculating means when the reliability judgment means determines that the reliability of the deviation amount calculated by the second deviation amount calculating means is high. A first image synthesizing means for synthesizing the two images by coordinate-transforming pixels of any one of the one image and the other image;
When it is determined by the reliability determination unit that the reliability of the shift amount calculated by the second shift amount calculation unit is not high, the shift amount calculated by the first shift amount calculation unit is set to An image processing apparatus comprising: a second image composition unit configured to compose the two images by shifting pixels of one of the one image and the other image based on the image.   The first shift amount calculation unit calculates the shift amount for an image having a resolution lower than that of the image for which the second shift amount calculation unit is to calculate the shift amount. An image processing apparatus according to 1.   2. The first shift amount calculation unit calculates the shift amount by searching for global matching between the one low resolution image and the other low resolution image. Image processing device.   The said 2nd deviation | shift amount calculation means calculates the said deviation | shift amount by searching with a gradient method between the said 1 low resolution image and the said other low resolution image. Image processing device. A deviation amount determination unit that determines whether or not a deviation amount of the position of the other image with respect to the one image that is the composite reference image calculated by the first deviation amount calculation unit is a predetermined value or more;
And an image excluding unit for excluding the other image from an image compositing target by the image synthesizing unit when the deviation amount is determined to be greater than or equal to a predetermined value by the deviation amount determining unit. The image processing apparatus according to claim 1. On the computer,
A function of creating one low-resolution image and another low-resolution image by gradually reducing the resolution of one image and another image different from the one image among the plurality of images;
A function of calculating a positional shift amount between the one low resolution image and the other low resolution image by a first calculation method;
A function of calculating a positional shift amount between the one low resolution image and the other low resolution image by a second calculation method;
A function of determining whether or not the deviation amount calculated by the second calculation method is highly reliable;
When it is determined that the reliability of the shift amount calculated by the second calculation method is high, the one image and the other image are calculated based on the shift amount calculated by the second calculation method. A function of coordinate-converting pixels of one of the images and synthesizing the two images;
When it is determined that the reliability of the shift amount calculated by the second calculation method is not high, the one image and the other are based on the shift amount calculated by the first calculation method. A program for realizing a function of shifting the pixels of any one of the images and synthesizing the two images.

Images