JP2006229868A - Image processing apparatus and method, and motion detection apparatus and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To realize improvement in image quality when generating one image by combining a plurality of images. <P>SOLUTION: In an image processor and an image processing method for detecting motions among the plurality of images with respect to the plurality of images outputted from an imaging part and combining the plurality of images on the basis of the detected motions, white balance compensation under the condition almost same as the optimal light exposure condition is performed to the plurality of images by including at least one image with light exposure different from the proper one. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method, and more particularly to an image processing apparatus and method for combining a plurality of images.

  The present invention also relates to a motion detection apparatus and a motion detection method for obtaining a movement parameter for aligning a plurality of images.

  When taking a picture with a digital still camera, the photographed image may be blurred due to movement of the camera or subject during the taking. In order to prevent such blurring, it is desirable to perform shooting at a high shutter speed. The fastest shutter speed that can be set is determined from the relationship between the brightness of the subject, the aperture value, and the sensitivity of the image sensor. Therefore, for example, when the brightness of the subject is low, the shutter speed may have to be slowed down in order to obtain a necessary exposure amount even if the aperture value is opened. In such a case, in order to prevent blurring, measures such as photographing with a strobe, fixing the camera with a tripod, and increasing the sensitivity of the image sensor can be considered.

  However, when the sensitivity of the image sensor is increased, noise in the captured image becomes conspicuous. In addition, the effect of using a strobe is limited to the range where the strobe light can reach, and cannot be used for shooting a subject at a long distance. Also, shooting using a tripod is inferior in mobility, and the situation that can be handled is limited.

JP-A-3-224372

  In view of these problems, a method has been proposed in which a plurality of images are taken at a short shutter interval and at a high shutter speed, and a plurality of images thus taken are combined to ensure a substantially necessary exposure amount (patent). Reference 1).

  However, in the conventional method for acquiring a proper exposure image by combining multiple images, there is no mention of color reproduction, and an unnatural color is obtained when image data output by an existing digital still camera is simply combined. There is a problem that a reproduced image is generated.

  In addition, in such image composition, if the alignment between a plurality of images is not performed accurately, the synthesized image becomes like a blurred image, and the image quality is significantly lowered. Usually, alignment between a plurality of images is performed using camera movement parameters estimated from the association between images using the least square method. However, if the data that correlates images includes data that is not correctly associated, or data that is not related to camera movement due to movement of a part of the subject, camera movement The parameter estimation error increases, and the alignment accuracy decreases.

  In view of these problems, the LMedS method for estimating camera movement parameters based on the criterion for minimizing the median square error, data with large errors based on movement parameters estimated from randomly selected associations The RANSAC method, etc., has been proposed in which parameters with the highest reliability are selected.

  However, the LMedS method has high convergence when the ratio of data including an error is small, but is not stable when the ratio of data including an error is large. In the RANSAC method, stability is good even when the ratio of data containing errors is large, but in order to improve reproducibility, it is necessary to perform processing in enormous combinations.

  An object of the present invention is to solve such problems of the prior art.

  The above-described object is to detect a motion between a plurality of images having overlapping portions, which are photographed under an exposure condition different from the proper exposure condition, and to align the plurality of images based on the detected motion. A white balance adjustment coefficient based on a color temperature of a light source estimated using a pixel having a predetermined luminance value, with each of the plurality of images as a target image. And a white balance adjustment unit that adjusts the white balance of the target image using the coefficient, and the white balance adjustment unit has a predetermined luminance value according to a relationship between different exposure conditions and appropriate exposure conditions. This is achieved by an image processing apparatus characterized in that the color temperature of the light source is estimated using the corrected value, and the motion detection unit and the synthesis unit perform motion detection and synthesis on the image after white balance adjustment. .

  In addition, the above-described object is to detect a motion between a plurality of images having overlapping portions, which are photographed under an exposure condition different from the proper exposure condition, and position the plurality of images based on the detected motion. An image processing apparatus having a combining unit that combines and combines the white balance based on a color temperature of a light source estimated using a pixel having a predetermined luminance value, with the image combined by the combining unit as a target image A white balance adjusting unit that calculates a coefficient for adjustment and adjusts the white balance of the target image using the coefficient, and the white balance adjusting unit applies a predetermined luminance value to an image shot under appropriate exposure conditions. The image processing apparatus is also characterized in that the color temperature of the light source is estimated using the luminance value used for the light source.

  The above-described object is a motion detection device for obtaining a movement parameter for aligning the first and second images having overlapping portions, and corresponding points between the first image and the second image are obtained. Corresponding point detection means to detect; determination means for determining reliability of corresponding points; first estimation means for estimating a movement parameter based on a combination of corresponding points determined to be highly reliable by the determination means; For each of the movement parameters estimated by the first estimation means, a detected corresponding point is detected from the detected corresponding point, with an error corresponding to the corresponding point estimated using the estimated movement parameter being equal to or less than a predetermined value. And a second estimation unit that estimates a movement parameter using the number of acceptable corresponding points when the number of acceptable corresponding points is the maximum and uses the estimation result as a final movement parameter. Also achieved by the detection device.

  The above-described object can also be achieved by an imaging apparatus comprising the image processing apparatus of the present invention having the motion detection apparatus of the present invention as motion detection means, and imaging means for capturing a plurality of images. .

  With such a configuration, according to the present invention, color reproducibility in an image generated by combining a plurality of images can be improved.

  Further, according to another aspect of the present invention, it is possible to improve the stability and accuracy of alignment of a plurality of images.

Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.
(First embodiment)
FIG. 1 is a diagram showing a configuration example of a digital still camera 100 as an image processing apparatus according to the first embodiment of the present invention.

  The digital still camera 100 can use a removable recording medium 200 (for example, a semiconductor memory card, an optical disk, etc.). The imaging block 10, the image processing block 2, the image memory 30, the image synthesis block 40, and the encoding / recording block 50 are provided, and the function of the digital still camera 100 is realized by the control unit 60 controlling these blocks. .

  In the imaging block 10, the imaging unit 11 includes an imaging lens (not shown), an imaging element such as a CCD, a gain adjustment circuit that adjusts the gain of the output signal of the imaging element, and A / D conversion that converts the signal after gain adjustment into a digital signal. It consists of a circuit. As described above, the imaging unit 11 converts the analog signal read from the imaging device into digital data and outputs the digital data.

  The exposure adjustment unit 12 determines a shutter speed and an aperture value according to a predetermined program diagram based on the luminance of a predetermined area of the digital image data output from the imaging unit 11, and exposes and shoots the image sensor of the imaging unit 11. Control the aperture of the lens.

  In the image processing block 20, the white balance detection unit 21 outputs a parameter value for adjusting the white balance from the image data output from the imaging unit 11 and temporarily stored in the image memory 30.

  The pixel interpolation unit 22 interpolates the RGB value of each pixel from the luminance value obtained from each pixel of the image sensor, the luminance value of the surrounding pixels, and the color array of the color filter used for the image sensor, Output color image data.

  The image correction unit 23 performs color correction on the color image data output from the pixel interpolation unit 22 so that the color reproduction characteristic becomes a desired characteristic. Further, edge emphasis processing for emphasizing the contour portion of the subject is performed, and γ correction conversion is performed by referring to a table so that desired gradation characteristics are obtained for each of the RGB channels of the image data.

In the image composition block 40, the motion detector 41 detects a motion between a plurality of image data stored in the image memory 30.
The image composition unit 42 aligns and synthesizes a plurality of image data so as to correct the motion detected by the motion detection unit 41.
In the encoding / recording block 50, the image compression unit 51 compresses the image data stored in the image memory 30 by a predetermined method such as the JPEG method.

  The recording / playback unit 52 adds predetermined incidental information such as shooting date / time, aperture, shutter speed, camera model information, and the like to the image data compressed by the image compression unit 51, and creates an image data file according to a predetermined format. Form. As such a file format of photographed image data, the Exif format is generally used. Then, the image data file is recorded on the recording medium 200, or the image data file already recorded on the recording medium 200 is read, and an image reproduction process for displaying it on a liquid crystal monitor (not shown) or the like is performed.

First, the operation at the time of normal shooting in the digital still camera 100 according to the present embodiment will be described with reference to FIG.
When the user half-presses a shutter button (not shown) to photograph the subject, the control unit 60 detects this, and the exposure adjustment unit 12 determines the exposure time and the aperture value of the photographing lens using the output of the imaging unit 11. To do. When the shutter button is fully pressed, shooting is performed based on the determined exposure condition. Digital image data is output from the imaging unit 11 and stored in the image memory 30 (S101).

  The white balance detection unit 21 performs a statistical analysis on the pixel value of a predetermined reference luminance level included in the image data output from the imaging unit 11 and stored in the image memory 30, and estimates the color temperature of the light source. . Then, based on the estimated color temperature of the light source, a coefficient for adjusting the white balance is obtained so that the predetermined statistic calculated from the image data becomes a predetermined value (gray) (S102). At this time, when there are a plurality of types of white balances that can be set in the digital still camera, there is a target value of a statistic corresponding to each white balance, and an optimum coefficient is obtained for each mode. These coefficients are obtained for each RGB channel.

  The pixel interpolation unit 22 interpolates RGB 3-channel color image data from the image data stored in the image memory 30 using the coefficient obtained by the white balance detection unit 21 (S103).

  In the present embodiment, an example of the arrangement of color filters used by the image sensor included in the imaging unit 11 is shown in FIG. As shown in FIG. 2, at each pixel position, only the pixel value of one channel of RGB is obtained, so that the pixel value of a channel different from the filter color needs to be obtained by interpolation. At this time, when obtaining the value of the G channel at the pixel position where the filter color is G, a value corresponding to the G channel is added to the luminance value obtained from that position. Further, when the value of the R or B channel at the position is obtained, the coefficient corresponding to each channel is added to the luminance value at the position corresponding to two adjacent R or B filters, and an average value is obtained.

  Further, in the pixel position where the color of the filter is R or B, when the channel value of the same color as the filter is obtained, the value corresponding to each channel is added to the luminance value at that position. Further, when obtaining pixel values of channels having a color different from that of the filter, the coefficient corresponding to the filter color (channel) of the pixel is added to the luminance values of the four adjacent pixels, and the average value is used as the output value. .

  The above-described pixel interpolation method is an example. For example, when the R or B channel value is obtained at the pixel position where the filter color is G, each R or B pixel value separated by a predetermined distance is obtained. The coefficients corresponding to the channels may be integrated, and a weighted average value based on the distance may be used as the output value.

The image correction unit 23 performs color correction, edge enhancement, and γ correction on the color image data output from the pixel interpolation unit 22 (S104).
First, color correction processing is performed by matrix calculation. The color correction process mainly corrects the hue and saturation of image data.

  Next, the edge portion of the image is emphasized by edge enhancement processing. First, a contour process in a predetermined direction is extracted from an image by filtering the image data. Then, edge enhancement is performed on the extracted contour portion by applying an optimum edge enhancement filter for each direction. At this time, a filter process having a low-pass effect is performed on an isolated noise component with no contour directionality.

  Further, the gradation characteristics of the image data are adjusted so as to obtain a desired gradation characteristic for each of the RGB channels, and γ conversion for matching with the voltage-luminance characteristics (γ characteristics) of the display is performed by referring to the table.

  The image data output from the image correction unit 23 is temporarily stored in the image memory 30. The image compression unit 51 performs compression encoding processing on the image data in accordance with a preset image quality or compression rate setting, and outputs the image data to the recording / playback unit 52. The recording / playback unit 52 adds information at the time of shooting together with the compressed image data in accordance with a predetermined recording format, and records the image data file on the recording medium 200 (S105).

  In such normal shooting, when the subject brightness is low as described above, the shutter speed set by the exposure adjustment unit 12 becomes slow, so that there is a risk that the image quality may deteriorate due to blurring. Therefore, for example, when the subject brightness is extremely dark as in night scene shooting, it is preferable to perform shooting processing using image composition described below.

  Hereinafter, photographing processing using image composition in the digital still camera of the present embodiment will be described with reference to the flowchart of FIG. Note that a process for performing the same operation as the normal photographing described with reference to FIG. 6 will be briefly described. Further, in the present embodiment, for simplicity, a case will be described in which two images having overlapping portions are photographed under the same photographing condition, and one image is generated by combining these two images. Such composite shooting can be set, for example, with a shooting mode setting dial (not shown), and the control unit 60 performs the following shooting control when the composite shooting mode is set.

  When the user half-presses a shutter button (not shown) to photograph the subject, the control unit 60 detects this, and the exposure adjustment unit 12 determines the exposure time and the aperture value of the photographing lens using the output of the imaging unit 11. To do. When the shutter button is fully pressed, two images are taken continuously based on the determined exposure condition. Two pieces of digital image data are output from the imaging unit 11 and stored in the image memory 30 (S201). At this time, since the image is synthesized in the subsequent processing, the exposure time is set to ½ of that in normal shooting.

  Then, the white balance detection unit 21 applies a method similar to that during normal shooting to each image data output from the imaging unit 11 and stored in the image memory 30 to obtain a coefficient for adjusting the white balance. (S202).

  However, each image data is taken with a normal exposure time of 1/2 assuming that two images are to be combined in the subsequent stage. If the coefficient is calculated from the result of analyzing and estimating the color temperature of the light source, the appropriate white balance cannot be adjusted. This is because a pixel value having the same luminance level as that in normal photographing actually corresponds to a pixel value having a luminance level twice that in normal photographing.

  Further, in the subsequent image correction processing (color correction and gradation correction), the processing parameters are optimized on the assumption that an image whose exposure is optimally adjusted and the white balance is corrected based on a predetermined luminance level is input. . Therefore, when the reference luminance level for calculating the coefficient for white balance adjustment is shifted, there is a concern that the color reproduction of the resulting image becomes unnatural.

  This is the reason why the color reproducibility of an image obtained by a conventional composite photographing process that simply combines image data becomes unnatural. Also, if the brightness level that is the standard for detecting white balance differs between normal shooting and composite shooting, even if the same subject is shot, color reproduction differs between normal shooting and composite shooting. .

  In view of such a problem, in the present embodiment, considering that two images are combined in advance in the subsequent stage, at the time of composite shooting in which two images are combined, a reference for white balance in normal shooting is used. A coefficient for adjusting the white balance is obtained with reference to a luminance level ½ of the luminance level.

  Next, the pixel interpolation unit 22 adjusts the white balance of the two pieces of image data stored in the image memory 30 using the coefficient obtained by the white balance detection unit 21, and then generates RGB 3-channel color image data by interpolation. (S203). Then, the generated two pieces of color image data are temporarily stored in the image memory 30.

  First, the motion detection unit 41 detects a positional deviation amount as a corresponding point vector for each partial region from the two pieces of image data stored in the image memory 30. Based on the detected positional deviation amount, the camera movement parameter is estimated and output (S204). The configuration and operation of the motion detection unit 41 will be described in detail later.

  Next, the image synthesis unit 42 synthesizes the two image data stored in the image memory 30 using the camera movement parameters estimated by the motion detection unit 41 (S205).

  In the present embodiment, the pixel position in the second image data corresponding to each pixel of the first image data is obtained, and the corresponding two pixels are synthesized to obtain the pixel value of the synthesized image.

  First, the pixel position in the second image data for each pixel of the first image data is calculated using the camera movement parameter estimated by the motion detector 41. That is, when the coordinate of each pixel of the first image data is (x, y) in (Expression 1), the corresponding pixel position in the second image data is obtained as (x ′, y ′). Then, the pixel value at the pixel position (x, y) of the first image data and the pixel value at the pixel position (x ′, y ′) of the second image data are read from the image memory 30 and the two pixel values are added. Thus, the pixel value in the composite image is stored in the image memory 30 as the pixel value of the composite image. The above processing is repeated for all pixel positions of the first image data to obtain composite image data.

  In the above processing, the pixel value represents an RGB value. Further, for the pixel of the first image data in which the pixel position (x ′, y ′) calculated by (Expression 1) is outside the image area range, a value obtained by doubling the pixel value is used as the pixel value of the composite image. .

  Next, the image correction unit 23 performs color correction, edge enhancement processing, and γ conversion on the combined image data combined by the image combining unit 42 and stored in the image memory 30 in the same manner as during normal shooting ( S206).

  The image data output from the image correction unit 23 is temporarily stored in the image memory 30. Then, the image is compressed by the image compression unit 51 based on the setting and output to the recording / reproducing unit 52. The recording / playback unit 52 adds information at the time of shooting together with the compressed image data in accordance with a predetermined image format, and records it on the recording medium 200 (S207).

Next, the operation of the motion detection means 41 will be described in detail with reference to FIGS. FIG. 9 is a block diagram showing a configuration example of the motion detection means 41, and FIG. 10 is a flowchart for explaining the motion detection processing.
First, the corresponding point detection unit 411 detects a positional deviation amount as a corresponding point vector for each partial region from the two pieces of image data stored in the image memory 30 (S401).

3A and 3B show examples of the first image data and the second image data stored in the image memory 30. FIG.
Hereinafter, in the present embodiment, an operation in a case where corresponding points of two images having overlapping portions are detected using template matching will be described. Template matching is basically a method of associating portions having the highest correlation among a plurality of images. In the present embodiment, first, the RGB values of the first image data and the second image data are converted into luminance values. For the conversion of RGB values into luminance values, for example, a method of outputting an average value of RGB values constituting each pixel as a pseudo luminance value is used.

  Next, a predetermined region is cut out from the first image converted into the luminance value. For example, as shown in FIG. 4, a small area t obtained by dividing the first image data is cut out as a template. Next, corresponding points are extracted from the second image for all cut out templates.

  First, an area for searching for a point corresponding to the cut out template is set in the second image. This search area is set according to the focal length and the exposure time because the blur amount depends on the focal length and the exposure time which are the photographing conditions of the camera. FIG. 5 shows an example of the search area S when the area t of the first image is used as a template. The template is shifted in parallel within the search region S, and a difference in luminance value is calculated for each pixel in the first image (template) and the corresponding second image region. The position where the sum of the absolute values of the differences is the minimum is taken as the corresponding point position.

  When the corresponding point position is obtained, the reliability is determined. The determination of reliability can be performed based on a value representing a correlation between a template and a region of the second image corresponding to the template at the corresponding point. For example, the minimum value of the sum total of the luminance difference absolute values is less than or equal to the first predetermined threshold value, and the difference between the minimum value of the sum total of the brightness difference absolute values and the second smallest value is greater than or equal to the second predetermined threshold value. In this case, it is determined that the corresponding point is reliable. Then, the coordinates in the first and second images of the corresponding points determined to be reliable are stored in the corresponding point storage unit 412. At the same time, the minimum value of the sum of the absolute values of the differences is stored in the corresponding point storage unit 412 as an index of the reliability of the corresponding points.

  Here, the position where the sum of the absolute values of the luminance value differences is the minimum is set as the corresponding point position. However, for example, the position where the correlation value is maximized by performing the correlation calculation may be set as the corresponding point position. Further, the corresponding points are extracted based on the correlation with the luminance value, but a luminance differential value may be used. Although the corresponding points are extracted using template matching, feature points may be extracted from each image, and the corresponding points may be extracted based on the correlation between the feature amounts.

  Next, the movement parameter of the camera is estimated from the corresponding point vector that is the positional deviation amount of each partial region. The camera movement parameter is used for coordinate conversion during synthesis.

  The geometric positional relationship of the second image with respect to the first image is formulated as (Equation 1) below. This relationship is based on the assumption that the two images are in a relationship of translation, in-plane rotation, and deformation. (Expression 1) represents that the point (x, y) in the first image moves to the point (x ′, y ′) in the second image by the coordinate transformation matrix.

  The camera movement parameters a11, a12,... Which are the coordinate transformation matrix elements of (Equation 1). . . , A32 are obtained from the set of extracted corresponding points. The camera movement parameter calculation based on (Equation 1) requires at least three sets of corresponding points.

  Therefore, first, in the corresponding point selection unit 413, the sum of the absolute values of the differences is smaller than a predetermined threshold from the set of the plurality of corresponding points stored in the corresponding point storage unit 412, that is, a highly reliable response. Three sets of points are selected (S402).

  Next, the parameter calculation unit 414 calculates camera movement parameters from the coordinates of the three selected corresponding points. That is, the camera movement parameters a11, a12,. . . , A32 are calculated from the point (x, y) in the three sets of first images and the point (x ′, y ′) in the second image (S403).

  Next, in the pass determination unit 415, an error between the corresponding point estimated using the camera movement parameter obtained by the parameter calculation unit 414 and the actually detected corresponding point stored in the corresponding point storage unit 412 is obtained. A pass judgment is made based on the error amount. This process is performed for all pairs of corresponding points stored in the corresponding point storage unit 412, and the number of corresponding points (passing corresponding points) that pass is counted (S404).

  Details of the error calculation process for each corresponding point in the pass determination unit 415 will be described below.

  First, for all the corresponding points stored, the estimated value (x ″, y) of the point in the second image from the point (x, y) in the first image and the parameter obtained by the parameter calculation unit 414. ") Is obtained as shown in (Formula 2).

  Then, the square of the distance between the obtained estimated value (x ″, y ″) and the actually detected point (x ′, y ′) in the second image is evaluated as an error amount.

  Then, the corresponding points including the points in the second image in which the obtained error amount is equal to or less than the predetermined threshold value are counted as acceptable corresponding points. It should be noted that the pass corresponding points counted at this time and all pointers indicating the pass corresponding points in the corresponding point storage unit 412 are stored in the pass corresponding point storage unit 416.

  The processes from S402 to S404 are repeated for all pairs of three corresponding points with high reliability. That is, another three pairs of corresponding points that are smaller than a predetermined threshold are selected from a plurality of corresponding point groups stored in the corresponding point storage unit 412 in S402, parameters are calculated in S403, and a pass response is performed in S404. Count the number of points. When the number of accepted corresponding points is greater than the number of accepted corresponding points already stored in the accepted corresponding point storage unit 416, the newly accepted number of accepted corresponding points and the accepted corresponding points in the corresponding point storage unit 412 are indicated. All pointers are updated so as to be stored in the pass corresponding point storage unit 416.

  When processing is performed for all combinations of three corresponding points with high reliability, a large number of corresponding points with high reliability results in a huge amount of combinations. It is preferable to control a threshold value for determining sex.

  As described above, when the processing from S402 to S404 is performed for all combinations of three corresponding points with high reliability, the number of acceptable points indicated by the pointer stored in the acceptable corresponding point storage unit 416 is the largest at the end of the process. Become.

  Then, the final camera movement parameter is estimated by the parameter calculation unit 414 based on the most pairs of acceptable correspondence points (S405). That is, with reference to the pointer indicating the corresponding point corresponding to the stored corresponding point storage unit 416, the data of the corresponding corresponding point among the corresponding points stored in the corresponding point storage unit 412 is read. Then, the relationship of (Equation 1) is established with respect to a set of points (x ′, y ′) in the second image corresponding to the points (x, y) in the first image of all the acceptable corresponding points. , The camera movement parameters a11, a12,. . . , A32 is estimated. Note that the LMedS method can be used in place of the least square method for camera movement parameter estimation from all acceptable points.

  As described above, in this embodiment, a change in luminance level due to composition is predicted in advance for image data before image composition, and white balance correction is performed in substantially the same manner as under optimal exposure conditions. . As a result, an image having color reproducibility equivalent to that during normal photographing can be acquired without generating an unnatural color reproduction image even during composite photographing.

  In the present embodiment, when white balance detection is performed, a luminance level that is ½ of a luminance level that is a reference for white balance in normal shooting is used as a reference, but it is a reference for performing white balance. It goes without saying that the same effect can be obtained even if the white balance detection is performed by correcting the luminance level of the input image data by a factor of two while maintaining the luminance level.

  In the present embodiment, the case where one image is generated by combining two images has been described. However, one image may be combined from a larger number of images. For example, when combining n (n ≧ 3) images, shooting is performed with an exposure time of 1 / n in normal shooting, and a normal brightness level is used as a reference when obtaining a white balance correction coefficient. It may be 1 / n of the hour.

  Further, in the present embodiment, after estimating a plurality of camera movement parameter candidates based on a plurality of combinations of highly reliable corresponding points among the corresponding points between a plurality of images, each of the plurality of camera movement parameter candidates is actually A search is made for the one that corresponds to the maximum number of corresponding points detected in (1). Then, the optimum camera movement parameter is estimated again from the corresponding point (passed corresponding point) that matches the searched camera movement parameter candidate. Thereby, it is possible to estimate the camera movement parameter with high stability and accuracy, and as a result, the alignment accuracy of a plurality of images is improved.

  Therefore, blurring of images obtained by composite shooting is suppressed and quality is improved, and dynamic range expansion is achieved by positioning in image composition processing such as panoramic image generation, noise removal, and composition of images shot under different exposure conditions Needless to say, it can also be applied to processing.

  In this embodiment, the case of estimating the camera movement parameter expressed by (Equation 1) has been described, but the case of estimating the F matrix (Fundamental Matrix) representing the three-dimensional movement of the camera is also described. The invention is applicable.

(Second Embodiment)
Hereinafter, a second embodiment of the present invention will be described. Since the configuration of the present embodiment is substantially the same as the configuration of the first embodiment, only the operation at the time of composite shooting that combines and records captured images will be described.

  FIG. 8 is a flowchart for explaining the operation at the time of composite shooting in the digital still camera of the present embodiment. In the present embodiment, detailed operations in each unit are the same as those described in the first embodiment unless otherwise specified. Also in this embodiment, a case where two images are combined will be described.

  First, in S301, two images are taken with an exposure time ½ that of normal shooting, as in S201 of the first embodiment.

  Next, the motion detection unit 41 first detects a positional deviation amount as a corresponding point vector for each partial region from the two pieces of image data stored in the image memory 30. Based on the detected positional deviation amount, the camera movement parameter is estimated and output (S302). In this embodiment, since motion detection is performed before pixel interpolation processing, the image data used by the motion detection unit 41 is not color image data in which each pixel has an RGB value, and each pixel follows a color filter array. This is image data having a luminance value of one color channel. However, in this embodiment, such pixel values are used as they are, and corresponding point vectors are obtained by template matching in accordance with the corresponding point extraction method shown in the first embodiment.

  Next, the image composition unit 42 synthesizes the two image data stored in the image memory 30 using the camera movement parameters estimated by the motion detection unit 41 (S303). Regarding the image composition processing, as in the motion detection processing, the input image is not color image data. However, in accordance with the image composition method shown in the first embodiment, two corresponding pixels are synthesized to generate a composite image. The pixel value.

  The white balance detection unit 21 estimates the color temperature of the light source by performing statistical analysis on the pixel value of the predetermined luminance level of the composite image data stored in the image memory 30. Then, a coefficient for adjusting the white balance is obtained so that the predetermined statistic calculated from the image data becomes a predetermined value (gray) (S304). At this time, the synthesized image data has already been brightened by the synthesizing process by the image synthesizing unit 42, so that it is the same brightness as that in the case of normal shooting. With this analysis, the color temperature of the light source can be estimated.

  Next, the pixel interpolation unit 22 performs interpolation processing on the composite image data stored in the image memory 30 using the coefficient obtained by the white balance detection unit 21, and color composite image data in which each pixel has RGB 3 channel values. Is generated (S305). The generated color composite image data is temporarily stored in the image memory 30.

  Next, the image correction unit 23 performs color correction, edge enhancement processing, and γ conversion on the color composite image data stored in the image memory 30 (S306).

  The color composite image data output from the image correction unit 23 is temporarily stored in the image memory 30. Then, the image is compressed by the image compression unit 51 with the image quality or the compression rate according to the setting, and is output to the recording / reproducing unit 52. The recording / playback unit 52 adds information at the time of shooting together with the compressed image data in accordance with a predetermined image format, and records it on the recording medium 200 (S307).

  As described above, according to the present embodiment, since the white balance correction coefficient is calculated from the image data after the image synthesis, in addition to the effects of the first embodiment, the same processing as in normal shooting is performed. Can be applied to calculate the coefficient.

  Note that in this embodiment as well, as in the first embodiment, three or more images can be combined.

(Third embodiment)
The composite shooting in the first and second embodiments is to combine a plurality of images shot under the same exposure conditions. However, the present invention can also be applied to a composite photographing process that combines a plurality of images photographed under different exposure conditions.

  That is, an imaging device such as a CMOS sensor or a CCD sensor has a narrow dynamic range as compared with a silver salt film, and overexposure and underexposure occur during backlighting. In order to solve such a problem, a method has been proposed in which a plurality of images continuously captured under different exposure conditions are combined to realize an image having a substantially wide dynamic range (Japanese Patent No. 3110797).

  Hereinafter, as an example in which the present invention is applied to such a composite photographing method, a case where two images having exposure amounts of 1/2 and 2 times the optimum exposure conditions are combined will be described.

  Since the basic configuration of the digital still camera of this embodiment and the flow of operations during composite shooting may be substantially the same as those in the first embodiment, the processing during composite shooting in the digital still camera of this embodiment will be described below. Will be described using the flowchart of FIG. 7 again.

  First, in S201, two images are continuously captured. However, in the present embodiment, the exposure time of each image is set to 1/2 (short-time exposure image) and twice (long-time exposure image) compared to the case of normal shooting.

  Then, the white balance detection unit 21 obtains a coefficient for adjusting the white balance from each image data stored in the image memory 30 in the same manner as in the first embodiment (S202). However, in this embodiment, the reference luminance level of the pixel used for estimating the color temperature of the light source is set to a luminance level that is twice that of normal shooting for a short-time exposure image and 1/2 times that of normal shooting for a long-time exposure image. As a level, a coefficient for adjusting the white balance is obtained.

  Next, the pixel interpolation unit 22 interpolates the image data stored in the image memory 30 using the coefficient obtained by the white balance detection unit 21, and generates RGB three-channel color image data (S203). Then, the two interpolated color image data are temporarily stored in the image memory 30.

  Then, the movement detection unit 41 estimates and outputs the movement parameter of the camera in the same manner as in the first embodiment (S204). Note that since the two input images in the present embodiment have different exposure amounts, the luminance values are different even if the subject luminance is the same. Therefore, the luminance value is corrected before the difference calculation is performed. Specifically, the brightness value is corrected to twice for a short exposure image, and the brightness value is corrected to ½ for a long exposure image. In addition, in a long-exposure image, a region where the pixel value is saturated is likely to be erroneously extracted, so that corresponding points are not extracted.

  Next, as in the first embodiment, the image composition unit 42 synthesizes two pieces of image data stored in the image memory 30 using the camera movement parameters estimated by the motion detection unit 41 ( S205).

  Thereafter, similarly to the first embodiment, color correction by the image correction unit 23, edge enhancement processing, γ conversion (S206), compression by the image compression unit 51, and recording by the recording / reproducing unit 52 (S207) are performed.

  As described above, according to the present embodiment, the same color reproducibility as that in the normal shooting can be realized even in the composite shooting process using a plurality of images shot under different exposure conditions.

  Also in this embodiment, it can be configured to synthesize three or more images. In this case, the reference luminance level of the pixel used for estimating the color temperature of the light source and the luminance correction amount at the time of motion detection may be set according to the relationship between the exposure time of each image and the exposure time during normal shooting.

(Other embodiments)
In the above-described embodiment, the configuration in which a plurality of images are combined in the digital still camera has been described. For example, the image data acquired by the imaging unit 11 is recorded on the recording medium 200 as it is, and the plurality of image data is recorded from the recording medium 200. Needless to say, the object of the present invention can be achieved even if the above-described motion detection and image synthesis are performed using a computer program for realizing the functions of the above-described embodiments. In that case, a recording medium (or storage medium) in which a computer program for realizing the functions of the above-described embodiments is recorded is supplied to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus records the recording medium. The computer program stored in is read and executed. In this case, the computer program itself read from the recording medium realizes the functions of the above-described embodiments, and the recording medium recording the computer program constitutes the present invention.

  Further, by executing the computer program read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) operating on the computer based on the instructions of the computer program. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Further, after the computer program read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instructions of the computer program. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, a computer program corresponding to the flowchart described above is stored in the recording medium.

It is a block diagram which shows the structure of the digital still camera which concerns on embodiment of this invention. It is a figure which shows the example of an arrangement | sequence of the color filter which the image pick-up element of the digital still camera concerning embodiment of this invention uses. It is a figure which shows an example of an input image. It is a figure which shows the example of cutting out the template in a corresponding point detection. It is a figure which shows the example of a setting of the search area | region in a corresponding point detection. 6 is a flowchart illustrating an operation at the time of normal shooting in the digital still camera according to the embodiment of the present invention. 6 is a flowchart illustrating an operation at the time of composite shooting in the digital still camera according to the first embodiment of the present invention. It is a flowchart explaining the operation | movement at the time of synthetic | combination imaging | photography in the digital still camera which concerns on the 2nd Embodiment of this invention. It is a figure which shows the structural example of the motion detection part in the digital still camera which concerns on the 1st Embodiment of this invention. It is a flowchart explaining operation | movement of the motion detection part of FIG.

Claims (11)

Based on the detected motion, the plurality of images are aligned and synthesized based on motion detection means for detecting motion between a plurality of images having overlapping portions, which are photographed under an exposure condition different from the proper exposure condition. An image processing apparatus having a combining means,
Using each of the plurality of images as a target image, a white balance adjustment coefficient is calculated based on a color temperature of a light source estimated using a pixel having a predetermined luminance value, and the white of the target image is calculated using the coefficient. It has white balance adjustment means to adjust the balance,
The white balance adjustment unit estimates the color temperature of the light source using a value corrected according to the relationship between the different exposure condition and the appropriate exposure condition as the predetermined luminance value,
The image processing apparatus, wherein the motion detection unit and the synthesis unit perform the motion detection and the synthesis on the image after the white balance adjustment. Based on the detected motion, the plurality of images are aligned and synthesized based on motion detection means for detecting motion between a plurality of images having overlapping portions, which are photographed under an exposure condition different from the proper exposure condition. An image processing apparatus having a combining means,
Using the image synthesized by the synthesizing unit as a target image, a white balance adjustment coefficient is calculated based on the color temperature of a light source estimated using a pixel having a predetermined luminance value, and the target image is calculated using the coefficient. White balance adjusting means for adjusting the white balance of
The image processing apparatus, wherein the white balance adjustment unit estimates a color temperature of the light source using a luminance value used for an image photographed under the proper exposure condition as the predetermined luminance value.   The image processing apparatus according to claim 1, wherein the plurality of images are images taken under an exposure time shorter than the appropriate exposure condition and under the same exposure condition.   The said several image consists of the image image | photographed with the exposure time shorter than the said appropriate exposure condition, and the image image | photographed with the exposure time longer than the said appropriate exposure condition, The Claim 1 or Claim characterized by the above-mentioned. 2. The image processing apparatus according to 2. A motion detection device for determining a movement parameter for aligning first and second images having overlapping portions,
Corresponding point detecting means for detecting corresponding points between the first image and the second image;
Determining means for determining the reliability of the corresponding points;
First estimation means for estimating a movement parameter based on a combination of corresponding points determined to be highly reliable by the determination means;
For each of the movement parameters estimated by the first estimating means, a pass corresponding point whose error from the corresponding point estimated using the estimated movement parameter is not more than a predetermined value is detected from the detected corresponding point. Corresponding point detecting means;
A second estimation unit that estimates the movement parameter using the number of the corresponding points corresponding to the maximum number of the corresponding points, and uses the estimation result as a final movement parameter. Detection device. The corresponding point detection means detects a corresponding point in the second image by template matching in a search area of the second image, using a partial image of the first image as a template;
The motion according to claim 5, wherein the determination unit determines the reliability based on information representing a correlation between the partial image and the partial image of the second image corresponding to the partial image at the corresponding point. Detection device. The image processing device according to any one of claims 1 to 4, comprising the motion detection device according to claim 5 or 6 as the motion detection means,
An imaging apparatus comprising: an imaging unit that captures the plurality of images. A motion detection step for detecting motion between a plurality of images having overlapping portions, which is photographed under an exposure condition different from the proper exposure condition, and aligning and combining the plurality of images based on the detected motion An image processing method including a synthesis step,
Using each of the plurality of images as a target image, a white balance adjustment coefficient is calculated based on a color temperature of a light source estimated using a pixel having a predetermined luminance value, and the white of the target image is calculated using the coefficient. Has a white balance adjustment process to adjust the balance,
The white balance adjustment step estimates the color temperature of the light source using a value corrected according to the relationship between the different exposure condition and the appropriate exposure condition as the predetermined luminance value,
The image processing method, wherein the motion detection step and the synthesis step perform the motion detection and the synthesis on the image after the white balance adjustment. A motion detection step for detecting motion between a plurality of images having overlapping portions, which is photographed under an exposure condition different from the proper exposure condition, and aligning and combining the plurality of images based on the detected motion An image processing method including a synthesis step,
A coefficient for white balance adjustment is calculated based on a color temperature of a light source estimated using a pixel having a predetermined luminance value, using the image synthesized by the synthesis step as a target image, and the target image is calculated using the coefficient. A white balance adjustment process for adjusting the white balance of
The image processing method, wherein the white balance adjustment step estimates a color temperature of the light source using a luminance value used for an image photographed under the appropriate exposure condition as the predetermined luminance value. A motion detection method for obtaining a movement parameter for aligning first and second images having overlapping portions, comprising:
A corresponding point detecting step of detecting corresponding points between the first image and the second image;
A determination step of determining the reliability of the corresponding points;
A first estimation step for estimating a movement parameter based on a combination of corresponding points determined to be highly reliable by the determination step;
For each of the movement parameters estimated by the first estimation step, a pass that detects an acceptable corresponding point whose error from a corresponding point estimated using the estimated movement parameter is equal to or less than a predetermined value is detected from the detected corresponding point. Corresponding point detection process;
A second estimation step of estimating the movement parameter using the number of acceptable corresponding points when the number of acceptable corresponding points is the maximum, and using the estimation result as a final movement parameter. Detection method.   The program which makes a computer perform each process of at least 1 of the image processing method of Claim 7 or Claim 8, and the motion detection method of Claim 10.

Images