JP4781233B2 - Image processing apparatus, imaging apparatus, and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method, and more particularly to an apparatus suitable for combining a plurality of image data and expanding a dynamic range of the image data.

  In recent years, with the development of digital image processing technology, it has become possible to perform various processes that have been difficult until now by processing a plurality of captured images. One example is a process of creating a composite image with an expanded dynamic range from a plurality of images with different exposures. This process generates a low-brightness image and a high-brightness image for the same subject, and uses the low-brightness image in the low-brightness region and the high-brightness image in the high-brightness region, respectively. Thus, an image with good gradation is obtained.

  In Patent Document 1, after aligning the positional relationship of a plurality of pieces of image data of the same subject picked up with different exposure amounts, the brightness level of each image data is set based on the ratio of the average values of the brightness of each image data. In addition, a technique for synthesizing image data with matching brightness levels is disclosed. In this patent document 1, image data having a large dynamic range can be obtained by a combination of a technique for aligning image data and a technique for combining image data.

  By the way, when shooting the same subject a plurality of times with different exposure amounts, there is a slight time difference between the shooting. For this reason, there is a possibility that a relative position shift may occur in a plurality of obtained image data due to camera shake or the like. Therefore, this alignment technique is extremely important in order to synthesize a plurality of image data with high accuracy.

Japanese Patent No. 3553999

However, the technique described in Patent Document 1 described above does not consider processing for ensuring alignment accuracy. Therefore, the conventional technique has a problem in that there is a possibility that the alignment accuracy between two image data having different photographing conditions may be lowered.
The present invention has been made in view of such problems, and an object of the present invention is to accurately align a plurality of image data when generating image data having a large dynamic range.

The image processing apparatus of the present invention is an area that retains the gradation that is common to the plurality of images from an area that retains the gradation among a plurality of images captured under different exposure conditions. Extraction means for extracting a common area; luminance correction means for correcting the luminance of the common area extracted by the extraction means for each of the plurality of images based on an average value of pixel values of the common area; A position correcting unit that corrects the positions of the plurality of images based on a positional shift between the plurality of images in the common area whose luminance is corrected by the correcting unit, and a plurality of positions whose positions are corrected by the position correcting unit. Generating means for generating a composite image based on the image of the image , wherein the extracting means retains the gradation when pixels at the same position of the plurality of images retain gradation. Painting Extracting a common pixel, and extracts the common area based on the common pixel.
The image pickup apparatus of the present invention includes an image pickup device, storage means for temporarily storing a plurality of images of the same subject imaged under different exposure conditions by the image pickup device, and the plurality of images stored in the storage means. Based on an average value of pixel values of the common area, and extraction means for extracting a common area that is the area holding the gradation common to the plurality of images from the area holding the gradation A luminance correction unit that corrects the luminance of the common area extracted by the extraction unit for each of the plurality of images, and a position between the plurality of images in the common region whose luminance is corrected by the luminance correction unit. based on the deviation, including a position correction means for correcting the positions of the plurality of images, on the basis of the position correction means a plurality of the positions are corrected by the image, and generating means for generating a composite image, the extraction When the pixels at the same position of the plurality of images have gradation, the means extracts the pixels having the gradation as common pixels, and the common pixel based on the common pixels is extracted. A region is extracted .

The image processing method of the present invention is a region that retains the gradation property common to the plurality of images from a region that retains the gradation property among a plurality of images captured under different exposure conditions. An extraction step for extracting a common region; a luminance correction step for correcting the luminance of the common region extracted by the extraction step for each of the plurality of images based on an average value of pixel values of the common region; A position correction step for correcting the positions of the plurality of images based on a positional shift between the plurality of images in the common area whose luminance has been corrected by the correction step, and a plurality of positions whose positions have been corrected by the position correction step. based on the image, have a generation step of generating a composite image, in the extraction step, the pixel holds together gradation at the same positions of the plurality of images When it is extracts pixels that holds the gradation of a common pixel, and extracts the common area based on the common pixel.
According to another aspect of the image processing method of the present invention, a plurality of images of the same subject imaged under different exposure conditions by the image sensor are temporarily stored in a storage medium, and stored by the storage step. An extraction step of extracting a common area that is an area holding the gradation property common to the plurality of images from an area holding the gradation property of the plurality of images; and pixels of the common area Based on the average value, the luminance of the common area extracted by the extraction step is corrected for each of the plurality of images, and the luminance of the common area is corrected by the luminance correction step. A position correction step for correcting the positions of the plurality of images based on positional deviation between the plurality of images, and a plurality of images whose positions are corrected by the position correction step Based on, have a generation step of generating a composite image, in the extraction step, when the pixels in the same positions of the plurality of images are both holding the gradation, to hold the floor tonality Pixels are extracted as common pixels, and the common region is extracted based on the common pixels .

A computer program according to the present invention causes a computer to execute each step of the image processing method.
A computer-readable storage medium according to the present invention stores the computer program.

  According to the present invention, alignment of a plurality of image data performed when generating image data having a large dynamic range can be performed with higher accuracy than in the past. Therefore, an image with an expanded dynamic range can be generated more reliably than before.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram illustrating an example of a configuration of a digital camera that is an example of an imaging apparatus according to the present embodiment.
In FIG. 1, the imaging unit 101 includes an aperture, a shutter, an imaging lens group, and a semiconductor imaging element such as a C-MOS or CCD. The imaging control unit 102 controls shutter drive, aperture drive, focus drive, zoom drive, and the like.

  The A / D conversion unit 103 converts the video signal output from the imaging unit 101 into digital image data, and outputs the digital image data to the image processing unit 104. The image processing unit 104 generates color image data by performing processing such as forming a luminance signal and a color difference signal based on the input digital image data.

  The image display unit 111 displays the color image data generated by the image processing unit 104 on, for example, a liquid crystal monitor included in the digital camera body. Note that the color image data generated by the image processing unit 104 is recorded by, for example, a recording medium provided inside the digital camera by the image recording unit 112.

Hereinafter, an example of processing in the image processing unit 104 when the dynamic range of image data is expanded will be described.
In order to generate image data with an expanded dynamic range, the shooting control unit 102 changes the exposure (exposure conditions such as exposure amount and exposure time) for each shooting, and performs shooting a plurality of times at short intervals. Image data is obtained. As a method for performing such shooting, a method called bracket shooting is generally used.

  The video signal obtained by such shooting is converted into digital image data by the A / D conversion unit 103. This digital image data is temporarily stored in the memory unit 105. The plurality of digital image data stored in the memory unit 105 is accompanied by a relative positional shift due to camera shake or the like. Therefore, the position correction unit 109 corrects a relative position shift with respect to the plurality of digital image data stored in the memory unit 105.

The image composition unit 110 selects pixels in a plurality of digital image data in which the relative positional deviation is corrected, and performs a composition process of the plurality of digital image data.
The positional deviation detection unit 108 detects a relative positional deviation amount in a plurality of image data as information necessary for correcting the positional deviation performed by the position correction unit 109. When detecting the relative positional deviation amounts in a plurality of digital image data, two pre-processing performed to increase the accuracy of the detection are processing performed by the region extraction unit 106 and the level correction unit 107.

The area extraction unit 106 obtains a common area having an appropriate range from a plurality of digital image data. As a result, it is possible to prevent erroneous determination when detecting a relative positional shift amount in a plurality of digital image data.
The level correction unit 107 matches the luminance level in the common area between the plurality of digital image data. Thereby, it is possible to improve the detection accuracy of the relative positional deviation amount in the plurality of digital image data.

FIG. 2 is a flowchart for explaining an example of an outline of processing in the image processing unit 104.
In step S201 in FIG. 2, the memory unit 105 inputs and stores a plurality of digital image data obtained by the A / D conversion unit 103 and having different exposure conditions.
Next, in step S <b> 202, the region extraction unit 106 detects an appropriate region from each digital image data stored in the memory unit 105. As will be described later, the appropriate area refers to an area of the image area of the digital image data excluding an area where gradation is lost due to blackout or whiteout.

Next, in step S203, the region extraction unit 106 detects an appropriate region common to each digital image data. In the following description, an appropriate area common to each digital image data is referred to as a common area.
Next, in step S204, the level correction unit 107 performs luminance level correction processing in the common area of each digital image data.
Next, in step S <b> 205, the positional deviation detection unit 108 detects a relative positional deviation amount between the common areas whose luminance levels are corrected.
Details of steps S202 to S205 will be described later.

  When a relative positional shift amount between the common areas is detected, the process proceeds to step S206. Then, the position correction unit 109 corrects the position of the digital image data (original digital image data) temporarily stored in the memory unit 105 according to the deviation amount detected in step S205. For example, when two pieces of digital image data are stored in the memory unit 105, the position of the second digital image data is corrected and the position of the second digital image data is changed to the first digital image data. It may be adjusted to the data position or vice versa. In this manner, when the relative alignment of the plurality of digital image data stored in the memory unit 105 is performed, in step S207, the image composition unit 110 performs a composition process of the plurality of digital image data. .

Below, an example of the detailed process of each step shown in FIG. 2 is demonstrated.
FIG. 3 is a flowchart for explaining an example of the processing (processing for extracting an appropriate region) in step S202 of FIG. This process is performed for each digital image data.
In an image shot with underexposure (a state where the amount of exposure is small), there may be a region (dark region) in which a gradation called blackout cannot be reproduced. In this region, image features are lost. Therefore, if a relative positional shift in a plurality of digital image data is detected including this region, there is a possibility of causing an erroneous determination. Therefore, in order to exclude this region, in step S301, the region extraction unit 106 determines whether or not the pixel value of the input digital image data is greater than or equal to a predetermined threshold value Tl.

If the value of the pixel of the input digital image data is smaller than the predetermined threshold value Tl as a result of this determination, the pixel is determined to be a pixel in a blackened area, and the process proceeds to step S302. Then, the region extraction unit 106 excludes the pixel from pixels used when detecting a relative positional shift in the plurality of digital image data. That is, the pixels in the area that is blacked out are excluded from the detection targets of the misalignment detection unit 108.
On the other hand, if the pixel value of the input digital image data is greater than or equal to a predetermined threshold value Tl, the process proceeds to step S303.

  In an image captured in an overexposed state (a state where the amount of exposure is large), there may be a region (bright region) in which gradation cannot be reproduced, which is called overexposure. In this region, image features are lost. Therefore, if a relative positional shift in a plurality of digital image data is detected including this region, there is a possibility of causing an erroneous determination. Therefore, in order to exclude this region, in step S303, the region extraction unit 106 determines whether or not the pixel value of the input digital image data is equal to or less than a predetermined threshold Th. If the pixel value of the input digital image data is larger than a predetermined threshold value Th as a result of this determination, the pixel is determined to be a pixel in an overexposed region, and the process proceeds to step S302 described above. In step S <b> 302, the region extraction unit 106 excludes pixels in the overexposed region from detection targets of the misalignment detection unit 108.

  On the other hand, if the pixel value of the input digital image data is greater than or equal to a predetermined threshold Th, the process proceeds to step S304. Then, the region extraction unit 106 sets the pixel whose value is determined to be greater than or equal to the threshold Th in step S303 as a pixel candidate used when detecting a relative positional shift in the plurality of digital image data. That is, the region extraction unit 106 sets only pixels within the range of the threshold value Tl or more and the threshold value Th or less among the pixels of the input digital image data as pixels in an appropriate region that retains gradation, The pixel is set as a candidate for detection by the misalignment detection unit 108.

  Next, in step S305, the region extraction unit 106 determines whether or not processing for all the pixels of the input digital image data has been completed. As a result of this determination, if the processing for all the pixels of the input digital image data has not been completed, the processing of steps S301 to S305 is repeated until the processing for all the pixels of the input digital image data is completed. . When the processing is completed for all the pixels of the input digital image data, an appropriate area for one screen is extracted. And it returns to the flowchart of FIG.

  FIG. 4 is a flowchart for explaining an example of the processing in step S203 in FIG. This process is performed between images for which an appropriate area has been obtained. Here, in order to simplify the description, a case where processing between two digital image data is performed will be described as an example. In the following description, these two digital image data are referred to as a first input image and a second input image, respectively.

  First, in step S401, the region extraction unit 106 determines whether or not the pixel of the first input image is a pixel in the appropriate region. If the result of this determination is that the pixel of the first input image is not within the appropriate region, processing proceeds to step S402. Then, the region extraction unit 106 excludes the pixel from the detection target of the positional deviation detection unit 108. In the following description, pixels excluded from detection targets in the misalignment detection unit 108 are referred to as excluded pixels as necessary.

  On the other hand, if the pixel of the first input image is a pixel in the appropriate region, the process proceeds to step S403. Then, the region extraction unit 106 determines whether or not the pixel of the second input image is a pixel in the appropriate region. Here, the region extraction unit 106 extracts, from the second input image, a pixel at the same position as the pixel of the first input image determined in step S401, and whether or not the extracted pixel is a pixel in the appropriate region. Determine. As a result of this determination, if the pixel of the second input image is not a pixel in the appropriate region, the process proceeds to step S <b> 402, and the region extraction unit 106 excludes the pixel from the detection target by the misalignment detection unit 108.

  On the other hand, if the pixel of the second input image is a pixel in the appropriate region, the process proceeds to step S404. Then, the region extraction unit 106 sets the two pixels determined to be within the appropriate region in steps S401 and S403 as pixels in the common region, and sets the two pixels as detection targets in the misalignment detection unit 108. That is, the region extraction unit 106 determines that the pixels in the common region only when both the pixels of the first input image and the pixels of the second input image are in the appropriate range. These two pixels are set as detection targets in the misregistration detection unit 108. In the following description, a pixel to be detected by the misregistration detection unit 108 is referred to as a common pixel as necessary.

  Next, in step S405, the region extraction unit 106 determines whether or not the processing has been completed for all the pixels of the first input image and the second input image. As a result of the determination, if the processing has not been completed for all the pixels of the first input image and the second input image, the processing for all the pixels of the first input image and the second input image is performed. Steps S401 to S405 are repeated until the process ends. When the processing is completed for all the pixels of the first input image and the second input image, a set of common areas is extracted.

  For example, when the dynamic range is expanded using three images, the following may be performed. First, the processing of FIG. 4 is performed between the first image and the second image, and then the processing of FIG. 4 is performed between the second image and the third image. If all the pixels of the three images are in the appropriate area, those pixels are set as the pixels in the common area.

  FIG. 5 is a flowchart for explaining an example of the process (extraction method of a common area in units of blocks) following FIG. The process shown in FIG. 5 is a process for improving the detection accuracy of the position shift when detecting the position shift of a plurality of digital image data in units of blocks as will be described later. This is a process of narrowing the detection area further than the common area.

First, in step S501, the region extraction unit 106 divides the entire screen into blocks having a certain pixel size (a block composed of one or more vertical and horizontal pixels). The following processing is processing in units of blocks.
Next, in step S502, the region extraction unit 106 determines whether or not there is an excluded pixel determined in step S402 in the block divided in step S501. If the result of this determination is that there is an excluded pixel determined in step S402 in the block divided in step S501, the process proceeds to step S503. Then, the region extraction unit 106 excludes the block from the detection target of the misalignment detection unit 108. In the following description, the pixels excluded from the detection target by the misalignment detection unit 108 are referred to as excluded blocks as necessary.

  On the other hand, if there is no excluded pixel determined in step S402 in the block divided in step S501, the process proceeds to step S504. Then, the region extraction unit 106 determines whether or not there is an excluded pixel in a block (reference image block described later) within the motion vector search range detected for the block determined to have no excluded pixel in step S502. Determine whether. If the result of this determination is that there is an excluded pixel within the search range, the process proceeds to step S503, and the region extracting unit 106 excludes the plurality of blocks from the detection target of the misalignment detecting unit 108.

  On the other hand, when there is no excluded pixel in the block determined to have no excluded pixel in step S502 and the block within the motion vector search range detected for the block (No in step S504), step S505 is performed. Proceed to Then, the region extraction unit 106 sets the plurality of blocks as detection targets in the positional deviation detection unit 108. In the following description, blocks used for detection by the misalignment detection unit 108 are referred to as common blocks as necessary.

  Next, in step S506, the region extraction unit 106 determines whether or not processing has been completed for all the blocks divided in step S501. If the result of this determination is that processing has not been completed for all the blocks divided in step S501, steps S502 to S506 are repeated until the processing is completed. Then, when the processing of steps S502 to S506 is completed for all the blocks divided in step S501, the process returns to the flowchart of FIG.

  FIG. 6 is a flowchart illustrating an example of the process (common area level correction process) in step S204 of FIG. In addition, here, in order to simplify the description, a case where processing between two digital image data (first and second input images) is performed will be described as an example.

  In general, the detection of the position shift of a plurality of digital image data is performed based on the correlation value between the plurality of digital image data that is the detection target of the position shift. However, if the luminance levels do not match between the plurality of digital image data that are the position shift detection targets, the accuracy of calculating the correlation value between the plurality of digital image data is lowered. When the dynamic range of the image data is expanded, a positional shift is detected between a plurality of digital image data photographed at different exposures, so that the luminance levels between corresponding images are greatly different. Therefore, the correction of the luminance level is an important process for improving the detection accuracy of the position shift. One of the features of this embodiment is that, after matching the luminance levels between the common blocks, a positional shift occurring in the common blocks is detected.

First, in step S601, the level correction unit 107 calculates the average value a1 of the pixels of the first input image in the common block.
Similarly, in step S602, the level correction unit 107 calculates the average value a2 of the pixels of the second input image in the common block.

Next, in step S603, the level correction unit 107 calculates the gain r1 of the first input image. The relationship between the gain r1 of the first input image and the average values a1 and a2 can be expressed by Expression (1).
r1 = (a1 + a2) / (2 × a1) (1)

Similarly, in step S604, the level correction unit 107 calculates the gain r2 of the second input image. The relationship between the gain r2 of the second input image and the average values a1 and a2 can be expressed by Expression (2).
r2 = (a1 + a2) / (2 × a2) (2)

Next, in step S605, the level correction unit 107 calculates a correction value for the luminance level by multiplying the pixel value of each pixel of the first input image by the gain r1. That is, a value obtained by multiplying the pixel value of each pixel of the first input image by the gain r1 is the luminance level correction value.
Similarly, in step S606, the level correction unit 107 calculates a luminance level correction value by multiplying the pixel value of each pixel of the second input image by the gain r2. That is, a value obtained by multiplying the pixel value of each pixel of the second input image by the gain r2 is the correction value of the luminance level. When the series of level correction processing is thus completed, the process returns to the flowchart of FIG.

  FIG. 7 is a flowchart for explaining an example of processing (position shift detection processing) in step S205 of FIG. Here, a method of obtaining a motion vector for each block and obtaining the motion amount of the entire screen as an affine parameter using the obtained motion vector will be described as an example.

In step S <b> 801, the positional deviation detection unit 108 determines an effective block as preprocessing for obtaining a motion vector for each block. This is a process of excluding blocks that may not find a correct motion vector. Details will be described later.
Next, in step S802, the misregistration detection unit 108 calculates a motion vector of the block. Here, a case where a motion vector of a block is calculated using a general block matching method will be described as an example.

  In the block matching method, a sum of squared differences or a sum of absolute differences between pixels in a block is used as a matching evaluation value. If the blocks in the reference image in the target block for which the motion vector is to be calculated are sequentially moved within the search range of the reference image, the evaluation value is obtained. The position (block of reference image) having the smallest evaluation value among all the evaluation values obtained within the search range is the position (block) having the highest correlation with the target block. The amount of movement between the target block and the block having the highest correlation with the target block is the motion vector. Here, the method of obtaining the evaluation value for each pixel within the search range is called full search. On the other hand, a method of obtaining a minimum evaluation value from pixels thinned out in the search range and performing a fine search for pixels near the pixel having the obtained minimum evaluation value is called a step search. Step search is well known as a method for obtaining a motion vector at high speed.

Next, in step S <b> 803, the positional deviation detection unit 108 determines an effective motion vector. This is a process of excluding those motion vectors that are determined to have incorrect calculation results. Details will be described later.
Next, in step S804, the misregistration detection unit 108 determines whether or not processing has been completed for all blocks. If the result of this determination is that processing has not been completed for all blocks, the processing of steps S801 to S804 is repeated until processing is completed. Then, when the processing is completed for all blocks, the process proceeds to step S805. Then, the positional deviation detection unit 108 detects an affine parameter using the effective motion vector obtained in step S803.

FIG. 8 is a diagram for explaining the detection of affine parameters.
As shown in FIG. 8, it is assumed that the center coordinates of the target block are (x, y). If the center coordinate (x, y) of the target block is moved to the center coordinate (x ′, y ′) of the block in the reference image from the calculation result of the effective motion vector, these relationships are expressed by the formula ( It can be expressed as 3).

  Here, the 3 × 3 matrix on the right side of Equation (3) is the affine transformation matrix. Each element of this affine transformation matrix is an affine parameter. When the affine parameters a, b, d, and e are 1, 0, 0, 1 (a = 1, b = 0, d = 0, e = 1), this conversion is a parallel movement. At this time, the affine parameter c is the amount of movement in the horizontal direction, and the affine parameter f is the amount of movement in the vertical direction. Further, when the rotation angle is θ, the rotational movement can be expressed as in equations (4) to (7).

a = cos θ (4)
b = −sin θ (5)
d = sin θ (6)
e = cos θ (7)

Moreover, Formula (3) can be expressed as Formula (8) in the form of a generalized matrix.
x ′ = A · x (8)
Here, x and x ′ are 1 × 3 matrices, and A is a 3 × 3 matrix. When there are n effective motion vectors, the coordinate value of the target image can be expressed by an n × 3 matrix as shown in Equation (9).
X = (x ₁ x ₂ ... X _n ) (9)

Similarly, the coordinate values after movement can be expressed by an n × 3 matrix as shown in Equation (10).
_{X '= (x 1' x} 2 '··· x n') ··· (10)
Therefore, for n motion vectors, the expression is as shown in Expression (11).
X ′ = A · X (11)

That is, if the affine transformation matrix A in Equation (11) is obtained, it becomes the amount of displacement of the entire screen. By transforming equation (11), the affine transformation matrix A is obtained as in equation (12).
A = X ′ · X ^T · (X · X ^T ) ⁻¹ (12)

  As described above, when the positional deviation of a plurality of digital image data is obtained, the amount of motion of the plurality of digital image data can be expressed using the affine parameters. For this reason, in addition to the amount of movement due to shift blur that occurs when holding the digital camera, it is possible to determine the amount of movement due to roll blur that occurs in the in-plane direction, zoom blur that occurs in the front-rear direction, and the like.

Here, an example of processing (valid block determination processing) in step S801 in FIG. 7 will be described with reference to the flowchart in FIG.
When trying to obtain the correlation between blocks by the block matching method, the image in the block needs to have some feature amount. A correct motion vector cannot be obtained for a flat block containing almost only a DC component. On the other hand, if the edge is included in the horizontal direction or the vertical direction, it is considered that matching can be easily performed. The process shown in FIG. 9 is an example of a process for excluding such a flat portion block. Here, in order to simplify the description, a case where processing is performed on one block will be described as an example.

  First, in step S901, the misregistration detection unit 108 calculates a difference value between the maximum value and the minimum value of pixel values in one horizontal line in the block. For example, if the block size is composed of 50 × 50 pixels, the maximum value and the minimum value are obtained from 50 pixels in the horizontal direction in the block, and the difference value between the maximum value and the minimum value is obtained. calculate.

  Next, in step S902, the positional deviation detection unit 108 determines whether or not the calculation in step S901 has been performed for all horizontal lines of the block. In the example described above, it is determined whether or not the calculation in step S901 has been repeated for the number of horizontal lines, that is, 50 times. As a result of this determination, if the calculation in step S901 has not been performed for all horizontal lines in the block, steps S901 and S902 are repeated until the calculation in step S901 is performed for all horizontal lines in the block. Do.

  When the calculation in step S901 is performed for all horizontal lines of the block, the process proceeds to step S903. Then, the misalignment detection unit 108 obtains the maximum difference value from the difference values calculated in step S901. In the example described above, the maximum difference value is obtained from the 50 difference values.

  Next, in step S904, the misalignment detection unit 108 determines whether or not the maximum difference value obtained in step S903 is larger than a preset threshold value Tx. As a result of this determination, if the maximum difference value obtained in step S903 is not larger than the preset threshold value Tx, the misalignment detection unit 108 does not have a feature amount in the horizontal direction of the block to be processed. The process proceeds to step S905 assuming that the block is a block. Then, the misregistration detection unit 108 determines that the processing target block is an invalid block.

On the other hand, if the maximum difference value obtained in step S903 is larger than the preset threshold value Tx, the misregistration detection unit 108 regards the processing target block as a block having a feature amount in the horizontal direction. The same verification is performed in the vertical direction as in the horizontal direction.
First, in step S906, the misregistration detection unit 108 calculates a difference value between the maximum value and the minimum value of pixel values in one line in the vertical direction in the block. In the above-described example, the maximum value and the minimum value are obtained from 50 pixels in the vertical direction in the block, and the difference value between the maximum value and the minimum value is calculated.

  Next, in step S907, the positional deviation detection unit 108 determines whether or not the calculation in step S906 has been performed for all the vertical lines of the block. In the example described above, it is determined whether or not the calculation in step S906 has been repeated for the number of vertical lines, that is, 50 times. As a result of the determination, if the calculation in step S906 is not performed for all the vertical lines of the block, steps S906 and S907 are repeated until the calculation of step S906 is performed for all the vertical lines of the block. Do.

  When the calculation in step S906 is performed for all the vertical lines of the block, the process proceeds to step S908. Then, the misalignment detection unit 108 obtains the maximum difference value from the difference values calculated in step S906. In the example described above, the maximum difference value is obtained from the 50 difference values.

  Next, in step S909, the positional deviation detection unit 108 determines whether or not the maximum difference value obtained in step S908 is larger than a preset threshold value Ty. As a result of this determination, if the maximum difference value obtained in step S908 is not greater than the preset threshold value Ty, the misalignment detection unit 108 does not have a feature amount in the vertical direction for the block to be processed. The process proceeds to step S905 assuming that the block is a block. As described above, the misalignment detection unit 108 determines that the block to be processed is an invalid block.

  On the other hand, if the maximum difference value obtained in step S908 is larger than the preset threshold value Ty, the misregistration detection unit 108 determines that the block to be processed has a feature amount in both the vertical direction and the horizontal direction. It is considered. In this case, it can be expected that accurate block matching can be performed. In step S910, the misalignment detection unit 108 determines that the processing target block is a valid block.

Next, an example of the process (valid motion vector determination process) in step S803 in FIG. 7 will be described with reference to the flowchart in FIG.
First, in step S1001, the positional deviation detection unit 108 inputs the motion vector detected in step S802 in FIG. Then, in step S1002, the misregistration detection unit 108 calculates the frequency of occurrence of the motion vector.

  Next, in step S1003, the misregistration detection unit 108 determines whether or not the generation frequency of all the motion vectors detected in step S802 in FIG. As a result of this determination, if the occurrence frequencies of all the motion vectors detected in step S802 of FIG. 7 have not been obtained, steps S1001 to S1003 are repeated until the occurrence frequency has been obtained. When the generation frequencies of all the motion vectors detected in step S802 in FIG. 7 are obtained, the process proceeds to step S1004. The misregistration detection unit 108 obtains a motion vector having the maximum occurrence frequency based on the occurrence frequency of the motion vector obtained in step S1002.

  Next, in step S1005, the positional deviation detection unit 108 inputs again the motion vector detected in step S802 of FIG. In step S <b> 1006, the positional deviation detection unit 108 determines whether the input motion vector is a motion vector having the highest occurrence frequency or a motion vector in the vicinity of the motion vector. When the blur of the entire screen is only the shift blur described above, it is considered that the motion vector of each block substantially matches the motion vector of the maximum occurrence frequency. In addition, when the above-mentioned roll shake is involved, it is considered that many motion vectors are generated in the vicinity of the motion vector having the maximum occurrence frequency.

Therefore, if the result of determination in step S1006 is that the input motion vector is a motion vector with the maximum occurrence frequency or a motion vector in the vicinity of the motion vector, the process proceeds to step S1007. Then, the positional deviation detection unit 108 determines that the motion vector is an effective motion vector.
On the other hand, if the input motion vector is not the motion vector having the maximum occurrence frequency or a motion vector in the vicinity of the motion vector, the process proceeds to step S1008. Then, the displacement detection unit 108 determines that the motion vector is an invalid motion vector.

  Next, in step S1009, the positional deviation detection unit 108 determines whether or not the processing has been completed for all the motion vectors detected in step S802 of FIG. If the result of this determination is that processing has not been completed for all motion vectors detected in step S802 of FIG. 7, steps S1005 to S1009 are repeated until the processing is completed.

  FIG. 11 is a diagram illustrating an example of the image data synthesis process performed in step S207 of FIG. When synthesizing image data, the digital image data on the low luminance side is used when the pixel value (pixel level) is below a certain threshold value, and the digital image data on the high luminance side is used when the pixel value is larger than the threshold value. Basic. However, in practice, a process for avoiding that the set threshold value becomes a discontinuous point between the low luminance side and the high luminance side is performed. Here, two threshold values Ta and Tb are set within a range of values that the low luminance side and the high luminance side have in common. The image composition unit 110 of the present embodiment determines the pixel value (pixel level) using these two threshold values Ta and Tb, so that the mixing ratio of the digital image data from the low luminance side to the high luminance side is 1 Image data composition processing is performed so as to sequentially (gradually) change to zero.

  As described above, in the present embodiment, the area extraction unit 106 selects pixels from a plurality of pixels of digital image data, excluding pixels in areas where gradation is lost due to blackout or whiteout, as appropriate pixels. And Then, the region extraction unit 106 sets pixels that are pixels at the same position in a plurality of digital image data and are both appropriate pixels as common pixels, and sets other pixels as excluded pixels. Furthermore, the area extraction unit 106 blocks the entire screen of a plurality of digital image data. If the excluded pixel is not included in both the motion vector detection target block and the reference image block corresponding to the block, the region extraction unit 106 sets these blocks as common blocks. The level correction unit 107 obtains the average values a1 and a2 of the pixel values (luminance levels) in the common block, and matches the luminance levels of the common block using the obtained average values a1 and a2. The position shift detection unit 108 detects a position shift using a common block with a matching luminance level, and the position correction unit 109 corrects the positions of a plurality of digital image data according to the position shift amount.

  As described above, by detecting a motion vector using a common block whose luminance levels are matched to each other, a position shift of a plurality of digital image data is detected. Then, after aligning the positions of the plurality of digital image data in accordance with the detected amount of deviation, the plurality of digital image data is synthesized. Therefore, it is possible to perform alignment of a plurality of digital image data performed when generating image data having a large dynamic range with higher accuracy than in the past.

  Further, in the present embodiment, blocks that do not have a feature amount in the horizontal direction or the vertical direction are excluded from processing targets. In addition, it is possible to detect a shift in the position of a plurality of digital image data using only a motion vector estimated to be caused by an amount of movement due to shift blur, roll blur occurring in the in-plane direction, or zoom blur occurring in the front-rear direction. did. Therefore, it is possible to further improve the accuracy of alignment of a plurality of digital image data.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment described above, the average values a1 and a2 of the pixel values in the common block are obtained, and the luminance levels of the common block are matched using the obtained average values a1 and a2. On the other hand, in the present embodiment, the luminance level of the common block is adjusted using the imaging information obtained by the imaging control unit 102. As described above, this embodiment is different from the first embodiment described above mainly in part of the processing for matching the luminance level of the common block. Therefore, in the following description, the same parts as those in the first embodiment described above are denoted by the same reference numerals as those in FIGS.

FIG. 12 is a block diagram illustrating an example of a configuration of a digital camera that is an example of an imaging apparatus according to the present embodiment.
The difference between the digital camera shown in FIG. 12 and the digital camera of the first embodiment shown in FIG. 1 is that the imaging information obtained by the imaging control unit 102 in the processing of the level correction unit 1202 in the image processing unit 1201 is different. There is to use. An example of the process of the level correction unit 1202 of this embodiment (the process in step S204 of FIG. 2 (common area level correction process)) will be described with reference to the flowchart of FIG. Here, in order to simplify the description, a case where processing between two digital image data (first and second input images) is performed will be described as an example.

Specifically, the shooting information obtained by the shooting control unit 102 is information indicating how much underexposure or overexposure is taken from the appropriate exposure. By using this information, it is possible to correct the luminance level without performing the calculation of the average value described in FIG. 6 of the first embodiment.
First, in step S701, the level correction unit 1202 acquires shooting information of the first input image.
Similarly, in step S702, the level correction unit 1202 acquires shooting information of the second input image.

Next, in step S703, the level correction unit 1202 calculates the gain r1 of the first input image using the shooting information of the first input image. For example, if the first input image is taken one step below, the exposure time when the first input image is obtained is half of the original. Accordingly, the gain r1 for obtaining an appropriate brightness is “2”.
Similarly, in step S704, the level correction unit 1202 calculates the gain r2 of the second input image using the shooting information of the second input image. For example, if the second input image is taken with one step over, the exposure time when the second input image is obtained is twice the original exposure time. Accordingly, the gain r2 for obtaining an appropriate brightness is “½”.

Next, in step S705, the level correction unit 1202 calculates a correction value for the luminance level by multiplying the pixel value of each pixel of the first input image by the gain r1. That is, a value obtained by multiplying each pixel of the first input image by the gain r1 is a correction value of the luminance level.
Similarly, in step S706, the level correction unit 1202 calculates a correction value for the luminance level by multiplying the pixel value of each pixel of the second input image by the gain r2. That is, a value obtained by multiplying each pixel of the second input image by the gain r2 is a correction value of the luminance level. Thus, a series of level correction processing is completed, and the process returns to the flowchart of FIG.

  As described above, in this embodiment, the luminance level of the common block is adjusted using the imaging information regarding the deviation from the appropriate exposure. Therefore, the process of calculating the average values a1 and a2 can be omitted as in the first embodiment. Therefore, in this embodiment, in addition to the effect described in the first embodiment, there is an effect that a plurality of digital image data can be easily aligned.

  Note that when the above-described processing is performed in the imaging apparatus (digital camera), imaging information is obtained from the imaging control unit 102. However, when the later processing is performed by PC application software, imaging information is obtained from the additional information recorded in the image file, and the same processing as described above is performed by the PC.

  FIG. 14 is a diagram illustrating an example of a hardware configuration of a computer capable of executing the processing described in the first and second embodiments. Note that this computer can be connected to the imaging apparatus via an interface, for example, and inputs the video signal captured by the imaging apparatus and performs the processing described in the first and second embodiments. Execute.

  14, a central processing unit (CPU) 2002, a ROM 2003, a RAM 2004, a network interface 2005, an input device 2006, an output device 2007, and an external storage device 2008 are connected to a bus 2001. The CPU 2002 performs data processing or calculation and controls various components connected via the bus 2001. The ROM 2003 stores a control procedure (computer program) of the CPU 2002 in advance. The computer is activated by the CPU 2002 executing the computer program.

  A computer program is stored in the external storage device 2008, and the computer program is copied to the RAM 2004 and executed. The RAM 2004 is used as a work memory for data input / output and transmission / reception, and is used as a temporary storage area for controlling each component.

  The external storage device 2008 is, for example, a hard disk storage device or a CD-ROM, and stores image data and the like. Even if the power is turned off, the stored contents of the external storage device 2008 are not erased. The CPU 2002 executes the processing of the above-described embodiment by executing a computer program in the RAM 2004. A network interface 2005 is an interface for connecting to a network. The input device 2006 is, for example, a keyboard and a mouse, and can perform various designations or inputs. The output device 2007 is a display, a printer, or the like.

(Other embodiments of the present invention)
In order to operate various devices to realize the functions of the above-described embodiments, program codes of software for realizing the functions of the above-described embodiments are provided to an apparatus or a computer in the system connected to the various devices. You may supply. What was implemented by operating said various devices according to the program stored in the computer (CPU or MPU) of the system or apparatus is also included in the category of the present invention.

  In this case, the program code of the software itself realizes the functions of the above-described embodiment. The program code itself and means for supplying the program code to a computer, for example, a recording medium storing the program code constitute the present invention. As a recording medium for storing the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, the functions of the above-described embodiments are not only realized by executing the program code supplied by the computer. It goes without saying that the program code is also included in the embodiment of the present invention even when the function of the above-described embodiment is realized in cooperation with an operating system or other application software running on the computer. Yes.

Further, after the supplied program code is stored in the memory provided in the function expansion board of the computer, the CPU provided in the function expansion board performs part or all of the actual processing based on the instruction of the program code. Needless to say, the present invention includes the case where the functions of the above-described embodiments are realized by the processing.
Further, after the supplied program code is stored in the memory provided in the function expansion unit connected to the computer, the CPU or the like provided in the function expansion unit performs part or all of the actual processing based on the instruction of the program code. Do. Needless to say, the present invention includes the case where the functions of the above-described embodiments are realized by the processing.

  When the contents of the processing in each of the embodiments described above are applied to the recording medium, the program code corresponding to the flowchart described above is stored in the recording medium.

  Note that each of the above-described embodiments can be applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, and a printer). For example, an apparatus).

  In addition, each of the above-described embodiments is merely a specific example for carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. . That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

1 is a block diagram illustrating an example of a configuration of a digital camera according to a first embodiment of the present invention. 5 is a flowchart illustrating an example of an outline of processing in an image processing unit according to the first embodiment of this invention. FIG. 3 is a flowchart illustrating an example of processing (processing for extracting an appropriate region) in step S202 of FIG. 2 according to the first embodiment of this invention. 5 is a flowchart illustrating an example of processing (extraction processing of a common area in units of pixels) in step S203 in FIG. 2 according to the first embodiment of this invention. FIG. 5 is a flowchart illustrating an example of processing (a method of extracting a common area in units of blocks) following FIG. 4 according to the first embodiment of this invention. 5 is a flowchart illustrating an example of processing (common region level correction processing) in step S204 of FIG. 2 according to the first embodiment of this invention. 3 is a flowchart illustrating an example of processing (position shift detection processing) in step S205 in FIG. 2 according to the first embodiment of this invention. It is a figure for showing the 1st embodiment of the present invention and explaining detection of an affine parameter. FIG. 8 is a flowchart illustrating an example of processing (valid block determination processing) in step S801 in FIG. 7 according to the first embodiment of this invention. FIG. 8 is a flowchart illustrating an example of processing (valid motion vector determination processing) in step S803 in FIG. 7 according to the first embodiment of this invention. FIG. 4 is a diagram illustrating an example of image data combining processing performed in step S207 of FIG. 2 according to the first embodiment of this invention. It is the block diagram which showed the 2nd Embodiment of this invention and showed an example of the structure of a digital camera. FIG. 9 is a flowchart illustrating an example of processing of a level correction unit (processing in step S204 of FIG. 2 (common region level correction processing)) according to the second embodiment of this invention. 1 is a diagram illustrating an example of a hardware configuration of a computer according to an embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Image pick-up part 102 Image pick-up control part 103 A / D conversion part 104, 1201 Image processing part 105 Memory part 106 Area extraction part 107, 1202 Level correction part 108 Position shift detection part 109 Position correction part 110 Image composition part 111 Image display part 112 Image recording unit

Claims (12)

Extracting means for extracting, from among a plurality of images picked up under different exposure conditions, a common region that is a region having the gradation property common to the plurality of images from a region having the gradation property; ,
Luminance correction means for correcting the luminance of the common area extracted by the extraction means for each of the plurality of images based on an average value of pixel values of the common area;
Position correcting means for correcting the positions of the plurality of images based on positional deviations between the plurality of images in the common area whose luminance is corrected by the luminance correcting means;
Generating means for generating a composite image based on a plurality of images whose positions have been corrected by the position correcting means ;
The extraction means extracts pixels having the gradation property as common pixels when pixels at the same position of the plurality of images have gradation properties, and based on the common pixels An image processing apparatus that extracts the common area . The luminance correcting means, based on the average value of pixel values in the common region of each front Symbol plurality of images, wherein multiplying the correction factor for each of the plurality of images in pixel values of the plurality of images according Item 8. The image processing apparatus according to Item 1.   The brightness correction means multiplies a pixel value of the plurality of images by a correction coefficient for each of the plurality of images based on information relating to an exposure condition set when the plurality of images are captured. The image processing apparatus according to 1. The extracting means divides the entire screen of the plurality of images into blocks each composed of one or more vertical and horizontal pixels, and extracts a block in which all the pixels in the block are the common pixels as the common region. The image processing apparatus according to any one of claims 1 to 3 . Motion vector detection means for detecting a motion vector using a block composed of one or more vertical and horizontal pixels in the entire screen of the plurality of images,
The extraction means, when all the previous SL detecting subject to detection subject to be within the search range of the motion vector for the block block of pixels and the motion vector of the block pixel motion vector is the common pixel the image processing apparatus according to any one of claims 1 to 3, characterized in that pixels in the block to extract a block are all the common pixels as the common region. The detecting means is effective when each block that is a detection target of the motion vector is valid and each motion vector detected from each block that is the detection target of the motion vector is valid. The image processing apparatus according to claim 5 , wherein motion amounts of the plurality of images are calculated based on the effective motion vector detected in a block. The detection means calculates an affine parameter indicating a motion amount of the plurality of images,
The image processing apparatus according to claim 6 , wherein the position correction unit corrects a relative shift amount generated in the plurality of images by conversion using the affine parameter. An image sensor;
Storage means for temporarily storing a plurality of images of the same subject imaged under different exposure conditions by the image sensor;
Extraction means for extracting a common area, which is an area holding the gradation, common to the plurality of images, from an area holding the gradation of the plurality of images stored in the storage means; ,
Luminance correction means for correcting the luminance of the common area extracted by the extraction means for each of the plurality of images based on an average value of pixel values of the common area;
Position correcting means for correcting the positions of the plurality of images based on positional deviations between the plurality of images in the common area whose luminance is corrected by the luminance correcting means;
Generating means for generating a composite image based on a plurality of images whose positions have been corrected by the position correcting means ;
The extraction means extracts pixels having the gradation property as common pixels when pixels at the same position of the plurality of images have gradation properties, and based on the common pixels An imaging apparatus, wherein the common area is extracted . An extraction step of extracting a common area, which is an area holding the gradation, common to the plurality of images, from an area holding the gradation, from a plurality of images captured under different exposure conditions; ,
A luminance correction step of correcting the luminance of the common region extracted by the extraction step for each of the plurality of images based on an average value of pixel values of the common region;
A position correction step of correcting the positions of the plurality of images based on a shift in position between the plurality of images in the common region whose luminance is corrected by the luminance correction step;
The position correction on the basis of a plurality of images whose position is corrected by step, have a generation step of generating a composite image,
In the extraction step, when the pixels at the same position of the plurality of images have gradation, the pixels having the gradation are extracted as common pixels, and based on the common pixels And extracting the common area . A storage step of temporarily storing in a storage medium a plurality of images of the same subject imaged under different exposure conditions by the image sensor;
An extraction step of extracting a common region that is a region that retains the gradation property common to the plurality of images from a region that retains the gradation property of the plurality of images stored in the storage step; ,
A luminance correction step of correcting the luminance of the common region extracted by the extraction step for each of the plurality of images based on an average value of pixel values of the common region;
A position correction step of correcting the positions of the plurality of images based on a shift in position between the plurality of images in the common region whose luminance is corrected by the luminance correction step;
The position correction on the basis of a plurality of images whose position is corrected by step, have a generation step of generating a composite image,
In the extraction step, when the pixels at the same position of the plurality of images have gradation, the pixels having the gradation are extracted as common pixels, and based on the common pixels And extracting the common area . Computer programs for causing to execute the steps of the image processing method according to claim 9 or 1 0 to the computer. Computer-readable storage medium characterized by storing a computer program according to claim 1 1.

Images