JP4919165B2 - Image composition apparatus and program - Google Patents

Description

The present invention relates to an image synthesizing apparatus and program, for example, in the photographing apparatus such as a digital camera, an image synthesizing apparatus and program which can be applied to continuous image synthesis is used as a technology for improving imaging sensitivity of night view, etc. .

  An imaging apparatus such as a digital camera using a semiconductor imaging device such as a CCD or CMOS can change the imaging sensitivity (the imaging sensitivity of the semiconductor imaging device; also referred to as ISO sensitivity) with the camera itself. However, if the ISO sensitivity is increased too much, noise of the semiconductor imaging device increases and the image quality deteriorates.

  Therefore, in the following Patent Document 1, two images are continuously shot at a low shutter speed (for example, 1/60 seconds) and a high shutter speed (for example, 1/1000 seconds), and these images are combined, A technique is described that can substantially expand the dynamic range of a semiconductor imaging device.

  The point of this technique is that an image shot at a low shutter speed (hereinafter referred to as a low-speed image) is an image having excellent black level reproducibility, while an image shot at a high shutter speed (hereinafter referred to as a high-speed image). .), On the other hand, is an image with excellent white level reproducibility. In other words, in a low-speed image, a bright part is likely to be a white-out image, whereas in a high-speed image, a dark part is Although it has the disadvantage that it tends to be a black-out “black-out” image, by combining these two images, it compensates for the shortcomings of both images (out-of-brightness and under-exposure), and over-exposure and under-exposure An image with a wide dynamic range can be obtained.

  In the technique of this Patent Document 1, the image composition is merely detected by detecting a white-out part and a black-out part and applying the corresponding part of the image on the other side to those parts. For example, there is a drawback that it is difficult to perform accurate image composition when there is a moving subject or when there is a camera shake at the time of shooting.

  On the other hand, in Patent Document 2 below, in order to prevent camera shake that occurs in still image shooting, in the image stabilization mode, exposure is performed a plurality of times with a short exposure time, and from the images obtained as a result of these exposures. A technique is disclosed in which feature points composed of pixels with high luminance are extracted, a motion vector is obtained, coordinate transformation is performed based on these data, and an image is synthesized.

Japanese Patent No. 3110797 JP 2004-357202 A

  However, in the technique of Patent Document 2, since the feature point detection target is a high-luminance region, for example, even when the composition of continuously captured images is different, the feature point is one. There has been a problem that improper image composition is performed.

The present invention has been made in view of such a problem, and an image composition device that determines coincidence / non-coincidence between continuously photographed images and performs image synthesis with higher accuracy. And providing a program.

According to the first aspect of the present invention, image acquisition means for continuously acquiring a plurality of image data, and first image data among the plurality of image data continuously acquired by the image acquisition means as reference image data A dividing unit that divides the reference image data into a plurality of areas, a position selecting unit that selects a position of the first pixel block having a large feature amount from each of the areas divided by the dividing unit, and the position Based on the position of the first pixel block selected by the selection means, the position of the second pixel block corresponding to the position of the first block is determined from the second image data of the plurality of image data. a first search means for searching, based on the position of the second pixel block obtained search result by the first search means, a predetermined range of the reference image data Determining means for determining whether the third pixel block corresponding to the second pixel block exists, this determination means, when the third pixel block is determined to exist, the first Motion vector selection means for selecting a predetermined number of sets of motion vectors between the position of the pixel block and the position of the second pixel block corresponding to the position of the first pixel block, and the motion vector selection means based on the set of motion vectors of a predetermined number selected by, characterized in that it comprises a synthesizing means for synthesizing the first image data and the second image data by projective transformation.
According to a second aspect of the present invention, in the first aspect of the invention, the first search means includes a pixel value of the first pixel block and a pixel value in a designated search area in the second image data. The second pixel block corresponding to the first pixel block is searched by calculating a sum of squared differences or a sum of absolute differences between the first and second pixel blocks, and the determination unit includes a pixel of the second pixel block Calculating whether or not the third pixel block exists by calculating a sum of squares of differences or a sum of absolute differences between a value and a pixel value in the designated search area in the reference image data. Features.

The invention described in claim 3 is the invention described in claim 1 or 2, further comprising an imaging unit, wherein the image acquisition unit acquires a plurality of image data by continuously driving the imaging unit. Features.

According to a fourth aspect of the present invention, there is provided a computer based on image acquisition means for continuously acquiring a plurality of image data, and based on the first image data of the plurality of image data continuously acquired by the image acquisition means. As image data, dividing means for dividing the reference image data into a plurality of areas, position selecting means for selecting the position of the first pixel block having a large amount of feature from each area divided by the dividing means, and this position Based on the position of the first block selected by the selection means, the position of the second pixel block corresponding to the position of the first block is searched from the second image data of the plurality of image data. first search means, and the position relative to the first search means search result by the resulting second pixel blocks, given the reference image data to be First determining means for determining whether a third pixel block corresponding to the second pixel block enclosed there by the first determining means, it is determined that the third pixel blocks exist Motion vector selection means for selecting a predetermined number of sets of motion vectors between the position of the first pixel block and the position of the second pixel block corresponding to the position of the first pixel block. and characterized in that the movement on the basis of a predetermined number of sets of motion vectors selected by the vector selecting means, said second image data and projective transformation function as synthesizing means for synthesizing the first image data To do.

According to the present invention, it is possible to determine the coincidence / non-coincidence between continuously captured images, and to perform image synthesis with higher accuracy.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a configuration diagram of a photographing apparatus according to the embodiment. In this figure, a photographing apparatus 1 forms an image of an optical system 2 including a photographing lens and a diaphragm mechanism, a shutter mechanism 3 that can mechanically block an optical axis, and light that has passed through the optical system 2 and the shutter mechanism 3. CCD 4 for converting to an electrical image signal, A / D converter 5 for converting the output of CCD 4 from analog to digital, DRAM 6 for temporarily holding the output of A / D converter 5, and images held in DRAM 6 A liquid crystal display controller 7 that converts the signal into a predetermined display format and outputs it to the liquid crystal display screen 8, a liquid crystal display screen 8 that displays an image output from the liquid crystal display controller 7, and an image displayed on the liquid crystal display screen 8 The shutter button 9 that is operated by the user when a desired composition is obtained, and the mode of the photographing apparatus 1 is set to the normal photographing mode or the continuous photographing mode, or these modes. Mode button 10 for or in or or predetermined system setting mode or the like in the opposite or in the playback mode of recorded images, the integrated circuit 11, and an external storage memory M for storing the captured image.

  The integrated circuit unit 11 includes a shutter control unit 12 that controls the shutter mechanism 3, a light reception control unit 13 that controls the CCD 4 and the A / D converter 5, and image signals stored in the DRAM 6 as YUV signals (color signals) and luminance. A demosaic unit 14 for converting to a signal, a feature amount calculation unit 15 for calculating feature amounts of two images shot in the continuous shooting mode, and an SRAM 16 used as a work memory for feature amount calculation, block matching, and image synthesis The block matching unit 17 that performs block matching of two images shot in the continuous shooting mode, and the image deformation / synthesis adding unit 18 that performs image deformation and synthesis / addition processing of the two images based on the result of the block matching. A CPU core 19 that controls the overall operation of the integrated circuit unit 11 is included.

  2 and 3 are diagrams showing a schematic flow of image composition processing in the present embodiment. In this flow, first, the user operates the mode button 10 to enter the continuous shooting mode, adjusts to a desired composition while viewing the display on the liquid crystal display screen 8, and then operates the shutter button 9 to display two images. When continuous shooting (step S0) is performed, the two pieces of image data are taken from the CCD 4 into the DRAM via the A / D converter 5, and the demosaic unit 14 generates YUV data and luminance data of the two images. . Next, one of the two images is set as a “reference image”, and the other image is set as a “tracked image” (step S1).

  Next, the reference image is divided into a plurality of areas, and a block having a large feature amount is selected from blocks having a predetermined number of pixels (here, 16 × 16 pixels) constituting each area (step S2).

  FIG. 4 is a structural diagram of the reference image. In this figure, the reference image 20 is divided into equal parts except for the upper, lower, left and right edges 20a to 20d of the image, and each divided part is used as an area. Here, 4 × 3 is divided as an example, and each area is defined as areas E0 to E11. Each area E0 to E11 is further equally divided (here, 3 × 3 divisions), and each of the areas E0 to E11 is formed as a block B0 to B8 that is a collection of 16 × 16 pixels.

  Returning to the flow again, if a block with a large amount of features is selected from the blocks of the reference image in step S2, then a block with a large amount of features in the selected reference image and a designated search in the tracked image corresponding to that block The sum of squared differences (or sum of absolute differences) is calculated within the region (step S3).

  FIG. 5 is a conceptual diagram of step S3. (A) is a reference image selection block Bi (i is 1, 2, 3,...) And a search area 22 (designated search area) of the tracked image 21. (B) is a diagram showing the position of the search area 22 (designated search area) of the tracked image where the calculation result (sum of squares of differences or sum of absolute differences) of step S3 is minimized. .

  As shown in (b), when the position at which the calculation result (sum of squared differences or sum of absolute differences) in step S3 is minimized is obtained, the corresponding block Bi (16 × 16) of the tracked image at that position is then obtained. 16 pixels), the sum of squared differences (or sum of absolute differences) is calculated in the designated search area 22 including the corresponding reference image block side and its periphery (step S4).

  FIG. 6 is a conceptual diagram of step S4, where Bi ′ is a corresponding block of the tracked image, and 23 is a designated search area including the corresponding reference image block side and its periphery.

  Then, as a result of the calculation in step S4, a plurality of motion vectors that are “0” or within a certain value are selected for each motion vector group as a search result, and based on these motion vector groups, for example, a reference image A projection transformation matrix representing the relationship between the tracked image and the tracked image is obtained (step S6), and the tracked image is deformed using the transformation matrix so that the tracked image has the same positional relationship as the reference image. After the tracking image is added to the reference image to generate a composite image (step S7), the flow ends.

  Here, RANSAC (RANdom Sample Consensus) is a method of parameter estimation. For a parameter estimation candidate obtained from a small number of points, the number of points suitable for the estimation from a large number of points and the adaptation In this embodiment, the estimation result by the RANSAC, that is, the support is calculated. The tracked image is transformed by the projective transformation matrix in consideration of the rate.

  As described above, in this embodiment, normal template matching is performed on a block having a high feature amount in a predetermined divided area, and the roles of the reference image and the tracked image are reversed from the position once matched. Matching the block periphery of the reference image with the block at the matching position of the tracked image, and if the position fluctuation of the matching is within a certain value even if the role is reversed, it is considered that the matching is successful and the template matching In order to improve the reliability of data, it is possible to detect tracking failure when tracking is impossible in principle due to large camera shake, subject shake, or parallax effect (visible or invisible depending on viewing angle). In addition, it can prevent recognition errors even when tracking in a dark place where the brightness value is low. By better synthesis time, it is possible to suppress the noise generated in the captured image.

  In the above embodiment, after the position where the difference sum value is minimum is obtained, the block on the tracked image side of the obtained position is searched for the reference image side. However, the following improvement is also possible. Good.

  FIG. 7 is a diagram illustrating an improved example (first improved example) of the embodiment. As shown in this figure, after the position where the difference sum value is minimized is obtained, the block in which the block size of the reference image side block Bi is reduced, and a small range (search range on the tracked image side) centered on the obtained position. If the matching position is not shifted, the matching may be successful.

Or you may improve as follows.
FIG. 8 is a diagram showing a second improvement example, and this diagram corresponds to FIGS. 2 and 3 described above. The step numbers (steps S0 to S7) in this figure are the same as those in FIGS. The difference from FIG. 2 and FIG. 3 is that there is a determination (step S8) of “night view mode shooting?” After step S3. Only when this determination result is “YES”, steps S4 to S4 are performed. The point is that step S5 is executed. That is, the process of the above embodiment (steps S4 to S5) is executed only when the night scene mode for forcibly performing projective transformation / image synthesis is selected. By doing this, it is possible to obtain beautiful still image data that is less affected by image blur due to feature point mismatch errors.

  FIG. 9 is a diagram showing a third improvement example, and this diagram is also a diagram corresponding to FIGS. 2 and 3 described above. Similarly, the step numbers (steps S0 to S7) in this figure are the same as those in FIGS. The difference from FIG. 2 and FIG. 3 is that there is a determination (step S9) of “low-contrast image capture?” After step S3. Only when this determination result is “YES”, step The point is that steps S4 to S5 are executed. That is, the process of the above-described embodiment (steps S4 to S5) is executed only when an image with a low contrast of the entire captured image (a feature amount is difficult to generate) is captured and projection conversion / image synthesis is performed. By doing this, it is possible to obtain beautiful still image data that is less affected by image blur due to feature point mismatch errors.

It is a block diagram of the imaging device which concerns on embodiment. It is a figure (1/2) which shows the schematic flow of the image composition process in this embodiment. It is a figure (2/2) which shows the schematic flow of the image composition process in this embodiment. FIG. 6 is a structural diagram of a reference image. It is a conceptual diagram of step S3. It is a conceptual diagram of step S4. It is a figure which shows the 1st example of improvement. It is a figure which shows the 2nd improvement example. It is a figure which shows the 3rd improvement example.

Explanation of symbols

1 Shooting device 4 CCD
15 Feature Quantity Calculation Unit 17 Block Matching Unit 18 Image Deformation Synthesis Addition Unit 19 CPU Core

Claims (4)

Image acquisition means for continuously acquiring a plurality of image data;
Dividing means for dividing the reference image data into a plurality of areas using the first image data of the plurality of image data continuously acquired by the image acquisition means as reference image data;
Position selecting means for selecting the position of the first pixel block having a large amount of features from each area divided by the dividing means;
Based on the position of the first pixel block selected by the position selection means, the second pixel block corresponding to the position of the first block is selected from the second image data of the plurality of image data. First search means for searching for a position;
Whether there is a third pixel block corresponding to the second pixel block in a predetermined range of the reference image data with reference to the position of the second pixel block obtained as a result of the search by the first search means A determination means for determining whether or not,
When the determination means determines that the third pixel block exists, the position of the first pixel block and the position of the second pixel block corresponding to the position of the first pixel block Motion vector selection means for selecting a predetermined number of sets of motion vectors between
Based on the set of motion vectors of a predetermined number selected by the motion vector selecting means, characterized in that it comprises a synthesizing means for synthesizing the first image data and the second image data by projective transformation Image composition device. The first search means calculates a sum of squared differences or a sum of absolute differences between a pixel value of the first pixel block and a pixel value in a designated search area in the second image data, Searching for the second pixel block corresponding to the first pixel block;
The determination means calculates the third sum of squares or the sum of absolute differences between the pixel value of the second pixel block and the pixel value in the designated search area in the reference image data. The image composition apparatus according to claim 1, wherein it is determined whether or not a pixel block exists. Further comprising an imaging means,
It said image acquisition means, the image synthesizing apparatus according to claim 1 or 2, characterized in that to obtain a plurality of image data continuously driven the imaging means. Computer
Image acquisition means for continuously acquiring a plurality of image data;
Dividing means for dividing the reference image data into a plurality of areas, using the first image data of the plurality of image data continuously acquired by the image acquisition means as reference image data;
Position selecting means for selecting the position of the first pixel block having a large amount of features from each area divided by the dividing means;
The position of the second pixel block corresponding to the position of the first block from the second image data of the plurality of image data based on the position of the first block selected by the position selection means A first search means for searching
Whether there is a third pixel block corresponding to the second pixel block in a predetermined range of the reference image data with reference to the position of the second pixel block obtained as a result of the search by the first search means First judging means for judging whether or not,
When the first determination unit determines that the third pixel block exists, the position of the first pixel block and the second pixel block corresponding to the position of the first pixel block Motion vector selection means for selecting a predetermined number of sets of motion vectors between the positions of
Based on the set of motion vectors of a predetermined number selected by the motion vector selection unit, wherein the function as combining means for combining said said second image data and projective transformation first image data program.

Images