JP2008311906A - Image data collating device, image composition device, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform image composition with higher precision by securely deciding whether images which are successively photographed match each other. <P>SOLUTION: An image collating device includes an image acquisition means (4) of acquiring a plurality of images successively, a position detection means (15) of detecting the position of a first pixel block whose feature quantity is large from first image data among the plurality of image data which are acquired successively by the image acquiring means, a first search means (17) of searching for the position of a second pixel block whose feature quantity is large from second image data among the plurality of pixel data based upon the position detected by the position detecting means, and a deciding means (19) of deciding that the first image data and second image data match each other by deciding whether a pixel block whose feature quantity is large is present within a predetermined range including the first pixel block based upon the position of the second pixel block obtained from the result of search by the first search means. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image data collating device, an image synthesizing device, and a program. For example, in a photographing device such as a digital camera, an image that can be applied to continuous-shot image composition used as one of techniques for improving photographing sensitivity such as night scenes The present invention relates to a data collation device, an image composition device, and a program.

  An imaging apparatus such as a digital camera using a semiconductor imaging device such as a CCD or CMOS can change the imaging sensitivity (the imaging sensitivity of the semiconductor imaging device; also referred to as ISO sensitivity) with the camera itself. However, if the ISO sensitivity is increased too much, noise of the semiconductor imaging device increases and the image quality deteriorates.

  Therefore, in the following Patent Document 1, two images are continuously shot at a low shutter speed (for example, 1/60 seconds) and a high shutter speed (for example, 1/1000 seconds), and these images are combined, A technique is described that can substantially expand the dynamic range of a semiconductor imaging device.

  The point of this technique is that an image shot at a low shutter speed (hereinafter referred to as a low-speed image) is an image having excellent black level reproducibility, while an image shot at a high shutter speed (hereinafter referred to as a high-speed image). .), On the other hand, is an image with excellent white level reproducibility. In other words, in a low-speed image, a bright part is likely to be a white-out image, whereas in a high-speed image, a dark part is Although it has the disadvantage that it tends to be a blackened “blackout” image, by combining these two images, it compensates for the shortcomings of both images (out-of-brightness and underexposure). An image with a wide dynamic range can be obtained.

  In the technique of this Patent Document 1, the image composition is merely detected by detecting a white-out part and a black-out part and applying the corresponding part of the image on the other side to those parts. For example, there is a drawback that it is difficult to perform accurate image composition when there is a moving subject or when there is a camera shake at the time of shooting.

  On the other hand, in Patent Document 2 below, in order to prevent camera shake that occurs in still image shooting, in the image stabilization mode, exposure is performed a plurality of times with a short exposure time, and from the images obtained as a result of these exposures. A technique is disclosed in which feature points composed of pixels with high luminance are extracted, a motion vector is obtained, coordinate transformation is performed based on these data, and an image is synthesized.

Japanese Patent No. 3110797 JP 2004-357202 A

  However, in the technique of Patent Document 2, since the feature point detection target is a high-luminance region, for example, even when the composition of continuously captured images is different, the feature point is one. There has been a problem that improper image composition is performed.

  The present invention has been made in view of such a problem, and is an image that can reliably determine match / mismatch between consecutively shot images and perform more accurate image synthesis. The object is to provide a data collation device, an image composition device, and a program.

  According to the first aspect of the present invention, image acquisition means for continuously acquiring a plurality of image data, and first image data among the plurality of image data continuously acquired by the image acquisition means as reference image data As a reference, the position detection means for detecting the position of the first pixel block having a large amount of feature from the reference image data, and the position detected by the position detection means as a reference, the second of the plurality of image data. Obtained as a result of the search by the first search means for searching for the position of the second pixel block having a large amount of features from the tracked image data. Based on the position of the second pixel block as a reference, it is determined whether or not there is a pixel block with a large amount of features in a predetermined range including the first pixel block. Further comprising a determining means for determining a match between the first image data and the second image data, wherein.

  According to a second aspect of the present invention, image acquisition means for continuously acquiring a plurality of image data, and first image data among the plurality of image data continuously acquired by the image acquisition means as reference image data A dividing means for dividing the reference image data into a plurality of areas, a position detecting means for detecting the position of the first pixel block having a large amount of feature from each area divided by the dividing means, and the position Based on the position detected by the detecting means, the second image data of the plurality of image data is set as tracked image data, and the position of the second pixel block having a large amount of feature is determined from the tracked image data. A first search means for searching and a position of the second pixel block obtained as a result of the search by the first search means as a reference, A first determination unit that determines whether or not there is a pixel block with a large amount of feature in a predetermined range; and when the first determination unit determines that a pixel block with a large amount of feature exists, A predetermined number of sets of motion vectors between the position of the first pixel block and the position of the first pixel block and the position of the second pixel block corresponding to the position are selected, and the selected predetermined number And combining means for projectively transforming the second image data based on the position and motion vector of the first pixel block in the set and combining the first image data with the first image data.

  The invention according to claim 3 is the invention according to claim 2, further comprising an imaging unit, wherein the image acquisition unit continuously drives the imaging unit to acquire a plurality of image data. To do.

  According to a fourth aspect of the present invention, there is provided a computer comprising: an image acquisition unit that continuously acquires a plurality of image data; and first image data of the plurality of image data that are continuously acquired by the image acquisition unit. As reference image data, position detection means for detecting the position of the first pixel block having a large amount of feature from the reference image data, and the position detected by the position detection means as a reference, out of the plurality of image data The second image data is the tracked image data, and from this tracked image data, a first search means for searching for the position of the second pixel block having a large amount of features, and the search by the first search means Based on the position of the second pixel block obtained as a reference, it is determined whether or not there is a pixel block having a large amount of feature in a predetermined range including the first pixel block. It allows, characterized in that to function as a determining means for determining a match between said first image data and the second image data.

  According to a fifth aspect of the present invention, the computer is configured to use the image acquisition means for continuously acquiring a plurality of image data, and the first image data of the plurality of image data continuously acquired by the image acquisition means as a reference. Dividing means for dividing the reference image data into a plurality of areas as image data, position detecting means for detecting the position of the first pixel block having a large amount of feature from each area divided by the dividing means, Based on the position detected by the detecting means, the second image data of the plurality of image data is set as tracked image data, and the position of the second pixel block having a large amount of feature is determined from the tracked image data. First search means for searching, and the position of the second pixel block obtained as a result of the search by the first search means is used as a reference, and the first pixel block is First determining means for determining whether or not there is a pixel block having a large amount of feature in a predetermined range including a block, and when the first determining means determines that a pixel block having a large amount of feature exists. A predetermined number of sets of motion vectors between the position of the first pixel block and the position of the first pixel block and the position of the second pixel block corresponding to the position are selected, and the selected The second image data is projected and transformed based on a predetermined number of sets of the first pixel block positions and motion vectors, and is combined with the first image data. To do.

  According to the present invention, it is possible to reliably determine coincidence / non-coincidence between continuously shot images, and to perform image synthesis with higher accuracy.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a configuration diagram of a photographing apparatus according to the embodiment. In this figure, a photographing apparatus 1 forms an image of an optical system 2 including a photographing lens and a diaphragm mechanism, a shutter mechanism 3 that can mechanically block an optical axis, and light that has passed through the optical system 2 and the shutter mechanism 3. CCD 4 for converting to an electrical image signal, A / D converter 5 for converting the output of CCD 4 from analog to digital, DRAM 6 for temporarily holding the output of A / D converter 5, and images held in DRAM 6 A liquid crystal display controller 7 that converts the signal into a predetermined display format and outputs it to the liquid crystal display screen 8, a liquid crystal display screen 8 that displays an image output from the liquid crystal display controller 7, and an image displayed on the liquid crystal display screen 8 The shutter button 9 that is operated by the user when a desired composition is obtained, and the mode of the photographing apparatus 1 is set to the normal photographing mode or the continuous photographing mode, or these modes. Mode button 10 for or in or or predetermined system setting mode or the like in the opposite or in the playback mode of recorded images, the integrated circuit 11, and an external storage memory M for storing the captured image.

  The integrated circuit unit 11 includes a shutter control unit 12 that controls the shutter mechanism 3, a light reception control unit 13 that controls the CCD 4 and the A / D converter 5, and image signals stored in the DRAM 6 as YUV signals (color signals) and luminance. A demosaic unit 14 for converting to a signal, a feature amount calculation unit 15 for calculating feature amounts of two images shot in the continuous shooting mode, and an SRAM 16 used as a work memory for feature amount calculation, block matching, and image synthesis The block matching unit 17 that performs block matching of two images shot in the continuous shooting mode, and the image deformation / synthesis adding unit 18 that performs image deformation and synthesis / addition processing of the two images based on the result of the block matching. A CPU core 19 that controls the overall operation of the integrated circuit unit 11 is included.

  2 and 3 are diagrams showing a schematic flow of image composition processing in the present embodiment. In this flow, first, the user operates the mode button 10 to enter the continuous shooting mode, adjusts to a desired composition while viewing the display on the liquid crystal display screen 8, and then operates the shutter button 9 to display two images. When continuous shooting (step S0) is performed, the two pieces of image data are taken from the CCD 4 into the DRAM via the A / D converter 5, and the demosaic unit 14 generates YUV data and luminance data of the two images. . Next, one of the two images is set as a “reference image”, and the other image is set as a “tracked image” (step S1).

  Next, the reference image is divided into a plurality of areas, and a block having a large feature amount is selected from blocks having a predetermined number of pixels (here, 16 × 16 pixels) constituting each area (step S2).

  FIG. 4 is a structural diagram of the reference image. In this figure, the reference image 20 is divided into equal parts except for the upper, lower, left and right edges 20a to 20d of the image, and each divided part is used as an area. Here, 4 × 3 is divided as an example, and each area is defined as areas E0 to E11. Each area E0 to E11 is further equally divided (here, 3 × 3 divisions), and each of the areas E0 to E11 is formed as a block B0 to B8 that is a collection of 16 × 16 pixels.

  Returning to the flow again, if a block with a large amount of features is selected from the blocks of the reference image in step S2, then a block with a large amount of features in the selected reference image and a designated search in the tracked image corresponding to that block The sum of squared differences (or sum of absolute differences) is calculated within the region (step S3).

  FIG. 5 is a conceptual diagram of step S3. (A) is a reference image selection block Bi (i is 1, 2, 3,...) And a search area 22 (designated search area) of the tracked image 21. (B) is a diagram showing the position of the search area 22 (designated search area) of the tracked image where the calculation result (sum of squares of differences or sum of absolute differences) of step S3 is minimized. .

  As shown in (b), when the position at which the calculation result (sum of squared differences or sum of absolute differences) in step S3 is minimized is obtained, the corresponding block Bi (16 × 16) of the tracked image at that position is then obtained. 16 pixels), the sum of squared differences (or sum of absolute differences) is calculated in the designated search area 22 including the corresponding reference image block side and its periphery (step S4).

  FIG. 6 is a conceptual diagram of step S4, where Bi ′ is a corresponding block of the tracked image, and 23 is a designated search area including the corresponding reference image block side and its periphery.

  Then, as a result of the calculation in step S4, “0” or a plurality of motion vectors within a certain value are selected for each motion vector group as a search result, and based on these motion vector groups, for example, the reference image is obtained by the RANSAC method. A projection transformation matrix representing the relationship between the tracked image and the tracked image is obtained (step S6), and the tracked image is deformed using the transformation matrix so that the tracked image has the same positional relationship as the reference image. After the tracking image is added to the reference image to generate a composite image (step S7), the flow ends.

  Here, RANSAC (RANdom Sample Consensus) is a method of parameter estimation. For a parameter estimation candidate obtained from a small number of points, the number of points suitable for the estimation from a large number of points and the adaptation In this embodiment, the estimation result by the RANSAC, that is, the support is calculated. The tracked image is transformed by the projective transformation matrix in consideration of the rate.

  As described above, in this embodiment, normal template matching is performed on a block having a high feature amount in a predetermined divided area, and the roles of the reference image and the tracked image are reversed from the position once matched. Matching the block periphery of the reference image with the block at the matching position of the tracked image, and if the position fluctuation of the matching is within a certain value even if the role is reversed, it is considered that the matching is successful and the template matching In order to improve the reliability of data, it is possible to detect tracking failure when tracking is impossible in principle due to large camera shake, subject shake, or parallax effect (visible or invisible depending on viewing angle). In addition, it can prevent recognition errors even when tracking in a dark place where the brightness value is low. By better synthesis time, it is possible to suppress the noise generated in the captured image.

  In the above embodiment, after the position where the difference sum value is minimum is obtained, the block on the tracked image side of the obtained position is searched for the reference image side. However, the following improvement is also possible. Good.

  FIG. 7 is a diagram illustrating an improved example (first improved example) of the embodiment. As shown in this figure, after the position where the difference sum value is minimized is obtained, the block in which the block size of the reference image side block Bi is reduced, and a small range (search range on the tracked image side) centered on the obtained position If the matching position is not shifted, the matching may be successful.

Or you may improve as follows.
FIG. 8 is a diagram showing a second improvement example, and this diagram corresponds to FIGS. 2 and 3 described above. The step numbers (steps S0 to S7) in this figure are the same as those in FIGS. The difference from FIG. 2 and FIG. 3 is that there is a determination (step S8) of “night view mode shooting?” After step S3. Only when this determination result is “YES”, steps S4 to S4 are performed. The point is that step S5 is executed. That is, the process of the above embodiment (steps S4 to S5) is executed only when the night scene mode for forcibly performing projective transformation / image synthesis is selected. By doing this, it is possible to obtain beautiful still image data that is less affected by image blur due to feature point mismatch errors.

  FIG. 9 is a diagram showing a third improvement example, and this diagram is also a diagram corresponding to FIGS. 2 and 3 described above. Similarly, the step numbers (steps S0 to S7) in this figure are the same as those in FIGS. The difference from FIG. 2 and FIG. 3 is that there is a determination (step S9) of “low-contrast image capture?” After step S3. Only when this determination result is “YES”, step The point is that steps S4 to S5 are executed. That is, the process of the above-described embodiment (steps S4 to S5) is executed only when an image with a low contrast of the entire captured image (a feature amount is difficult to generate) is captured and projection conversion / image synthesis is performed. By doing this, it is possible to obtain beautiful still image data that is less affected by image blur due to feature point mismatch errors.

It is a block diagram of the imaging device which concerns on embodiment. It is a figure (1/2) which shows the schematic flow of the image composition process in this embodiment. It is a figure (2/2) which shows the schematic flow of the image composition process in this embodiment. FIG. 6 is a structural diagram of a reference image. It is a conceptual diagram of step S3. It is a conceptual diagram of step S4. It is a figure which shows the 1st example of improvement. It is a figure which shows the 2nd improvement example. It is a figure which shows the 3rd improvement example.

Explanation of symbols

1 Shooting device 4 CCD
15 Feature Quantity Calculation Unit 17 Block Matching Unit 18 Image Deformation Synthesis Addition Unit 19 CPU Core

Claims (5)

Image acquisition means for continuously acquiring a plurality of image data;
Position for detecting the position of the first pixel block having a large amount of feature from the reference image data using the first image data of the plurality of image data continuously acquired by the image acquisition means as the reference image data. Detection means;
Based on the position detected by the position detecting means, the second image data of the plurality of image data is used as tracked image data, and the second pixel block having a large amount of features is detected from the tracked image data. First search means for searching for a position;
Based on the position of the second pixel block obtained as a result of the search by the first search means, it is determined whether or not there is a pixel block with a large amount of feature in a predetermined range including the first pixel block. Thus, an image data collating apparatus comprising: a determination unit that determines whether the first image data matches the second image data. Image acquisition means for continuously acquiring a plurality of image data;
Dividing means for dividing the reference image data into a plurality of areas using the first image data of the plurality of image data continuously acquired by the image acquisition means as reference image data;
Position detecting means for detecting the position of the first pixel block having a large amount of features from each area divided by the dividing means;
Based on the position detected by the position detecting means, the second image data of the plurality of image data is used as tracked image data, and the second pixel block having a large amount of features is detected from the tracked image data. First search means for searching for a position;
Based on the position of the second pixel block obtained as a result of the search by the first search means, it is determined whether or not there is a pixel block with a large amount of feature in a predetermined range including the first pixel block. A first determination means;
When it is determined by the first determination means that a pixel block with a large amount of features exists, the position of the first pixel block, the position of the first pixel block, and the second pixel block corresponding to the position. A predetermined number of sets of motion vectors between the positions of the pixel blocks are selected, and the second image data is selected based on the positions and motion vectors of the first pixel blocks of the selected predetermined number of sets. An image synthesizing apparatus comprising: a synthesizing unit that performs projective transformation and synthesizes the first image data. Further comprising an imaging means,
The image synthesizing apparatus according to claim 2, wherein the image acquiring unit acquires a plurality of image data by continuously driving the imaging unit. Computer
Image acquisition means for continuously acquiring a plurality of image data;
Position for detecting the position of the first pixel block having a large amount of feature from the reference image data using the first image data of the plurality of image data continuously acquired by the image acquisition means as the reference image data. Detection means,
Based on the position detected by the position detecting means, the second image data of the plurality of image data is used as tracked image data, and the second pixel block having a large amount of features is detected from the tracked image data. First search means for searching for a position;
Based on the position of the second pixel block obtained as a result of the search by the first search means, it is determined whether or not there is a pixel block with a large amount of feature in a predetermined range including the first pixel block. Accordingly, a program that causes the first image data and the second image data to function as determination means. Computer
Image acquisition means for continuously acquiring a plurality of image data;
Dividing means for dividing the reference image data into a plurality of areas, using the first image data of the plurality of image data continuously acquired by the image acquisition means as reference image data;
Position detecting means for detecting the position of the first pixel block having a large amount of features from each area divided by the dividing means;
Based on the position detected by the position detecting means, the second image data of the plurality of image data is used as tracked image data, and the second pixel block having a large amount of features is detected from the tracked image data. First search means for searching for a position;
Based on the position of the second pixel block obtained as a result of the search by the first search means, it is determined whether or not there is a pixel block with a large amount of feature in a predetermined range including the first pixel block. First determination means,
When it is determined by the first determination means that a pixel block with a large amount of features exists, the position of the first pixel block, the position of the first pixel block, and the second pixel block corresponding to the position. A predetermined number of sets of motion vectors between the positions of the pixel blocks are selected, and the second image data is selected based on the positions and motion vectors of the first pixel blocks of the selected predetermined number of sets. A program that functions as a combining unit that performs projective conversion and combines with the first image data.

Images