JP5195973B2 - Image processing apparatus, electronic camera, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an electronic camera, and an image processing program.

2. Description of the Related Art Conventionally, a technique is known in which the same subject is photographed a plurality of times while changing exposure settings, and a composite image with a wide dynamic range is generated by combining a plurality of obtained image groups (for example, Patent Document 1). .
Further, Patent Document 2 below is known as a technique for detecting a positional deviation of a pattern between images output from different imaging elements having different resolutions. In this prior art, first, a global search is performed between a high-resolution image and a low-resolution image to find a candidate for a matching region. Next, both images are subjected to pixel interpolation within the range of the matching region, and a region in which the pattern matches in detail is found by comparison between the interpolated pixels.
JP 2002-305684 A International Publication WO95 / 04329 Pamphlet

By the way, in the prior art of Patent Document 1, it is necessary to read a high-resolution still image a plurality of times. Usually, since a high-resolution image reading time is long, the shooting time between these images is greatly opened. For this reason, when the subject or the camera moves, the degree of coincidence of the patterns between the plurality of images becomes low, and it becomes difficult to obtain a good gradation expansion effect.
On the other hand, on the electronic camera side, the camera angle must be firmly fixed so that the camera angle does not move during the plurality of still image shooting periods, and it has been difficult to easily perform gradation enlargement shooting.
In view of the above-described problems, an object of the present invention is to provide a technique for easily obtaining a composite image having a wide dynamic range.

<< 1 >> The image processing apparatus of the present invention includes an image input unit, a shift detection unit, and a gradation enlargement unit.
The image input unit captures a low-resolution image and a high-resolution image obtained by photographing the same subject with different exposure conditions and resolution.
The correction unit corrects the luminance level of the low resolution image based on the luminance level of the high resolution image with respect to the low resolution image having different exposure.
The deviation detection unit detects a positional deviation of the pattern between the low resolution image and the high resolution image.
The gradation enlarging unit performs alignment between the low resolution image and the high resolution image based on the positional deviation. The gradation enlargement unit extracts gradation information from the low-resolution image and combines it with the high-resolution image, thereby generating a composite image with an expanded gradation reproduction range.
<< 2 >> Preferably, the low-resolution image is a plurality of images. The gradation enlarging unit synthesizes the low resolution gradation information of each of the plurality of low resolution images in a multiplexed manner with the high resolution image. << 3 >> Preferably, the deviation detection unit includes a phase division unit and a precision detection unit.
The phase division unit extracts an edge component of the pattern from the high resolution image. The phase division unit performs sub-sampling while shifting the sample position of the edge component, thereby generating a plurality of sampling information with shifted phases.
The precision detection unit detects a positional shift with a finer accuracy than the pixel interval of the low-resolution image by detecting a positional shift in which the pattern most closely matches between the low-resolution image and the plurality of sampling information.
<< 4 >> Preferably, the gradation enlarging unit determines a high luminance region and a low luminance region for a high resolution image. The gradation enlargement unit increases the composition ratio of the under-exposed low resolution image in the high luminance area of the high resolution image. In addition, the gradation enlargement unit increases the composition ratio of the over-exposed low resolution image in the low luminance region of the high resolution image.
<< 5 >> An electronic camera of the present invention includes the image processing apparatus according to any one of << 1 >> to << 4 >> and an imaging unit that captures an object with at least two types of resolutions. In this case, the high-resolution image processed by the image processing apparatus is a high-resolution still image captured by the imaging unit. The low-resolution image processed by the image processing apparatus is a low-resolution image captured by the imaging unit under exposure conditions different from those of the high-resolution image before and / or after capturing the still image.
<< 6 >> Preferably, the electronic camera of the present invention includes a monitor unit that displays an image. The imaging unit sequentially captures low-resolution through images and displays moving images on the monitor unit. Further, the imaging unit captures a low-resolution image at a timing that does not overlap with the capturing of the through image and under an exposure condition different from that of the high-resolution image.
<< 7 >> The image processing program of the present invention is a program for causing a computer to function as the image processing apparatus described in any one of << 1 >> to << 4 >>.

  The image processing apparatus of the present invention synthesizes a low resolution image and a high resolution image with different exposure conditions to expand the gradation reproduction range. In this case, since the number of pixels of the low resolution image is small, the readout time at the time of shooting can be shortened. For this reason, the photographing times of the high-resolution image and the low-resolution image can be brought close to each other, and the degree of coincidence between the images can be increased. As a result, the pattern matches well during the image composition, so that a good gradation expansion effect can be obtained.

  The electronic camera of the present invention captures a high resolution image and a low resolution image by changing the exposure conditions. In this case, since the readout time of the low-resolution image is short, the photographing of the low-resolution image can be completed in a short time. Therefore, the period during which the user should fix the camera angle is shortened, and it is possible to perform gradation enlargement shooting that is easier than before.

[Description of electronic camera configuration]
FIG. 1 is a block diagram showing an electronic camera 10 (including an image processing device 25) according to the present embodiment.
In FIG. 1, a photographing lens 12 is attached to the electronic camera 10. In the image space of the photographic lens 12, the imaging surface of the imaging element 11 is arranged. The image pickup device 11 has a mode for reading a high-resolution image and a mode for reading a low-resolution image by performing pixel thinning and pixel addition inside the device. These readout modes are controlled by the imaging control unit 14. The image signal generated by the image sensor 11 is processed through the signal processing unit 15 and the A / D conversion unit 16 and then temporarily stored in the memory 17.
This memory 17 is connected to a bus 18. The bus 18 is also connected with an imaging control unit 14, a microprocessor 19, a recording unit 22, an image compression unit 24, a monitor display unit 30, an image processing device 25, and the like. The microprocessor 19 is connected to an operation unit 19a such as a release button. A recording medium 22a is detachably attached to the recording unit 22.

[Description of Image Processing Device 25]
FIG. 2 is a block diagram schematically showing the configuration of the image processing device 25.
The gain correction unit 31 performs gain correction and gradation correction on the image in the memory 17. The high resolution image read from the memory 17 is supplied to the reduced image generation unit 32, the feature amount extraction unit 33, and the gradation synthesis unit 34b. The output data of the reduced image generation unit 32 is supplied to the rough detection unit 36 via the feature amount extraction unit 35. The output data of the feature quantity extraction unit 33 is supplied to the precision detection unit 38 via the phase division unit 37. In addition, information about the edge is supplied from the feature amount extraction unit 33 to the gain correction unit 31.

On the other hand, the plurality of low resolution images read from the memory 17 are supplied to the feature amount extraction unit 39 and the alignment unit 34a, respectively. The output data of the feature quantity extraction unit 39 is supplied to the rough detection unit 36 and the precise detection unit 38, respectively.
The positional deviation roughly detected by the rough detection unit 36 is supplied to the precision detection unit 38. The positional deviation detected with high accuracy by the precision detection unit 38 is supplied to the alignment unit 34a. The alignment unit 34a adjusts the pixel position of the low-resolution image based on this positional shift, and outputs it to the gradation composition unit 34b. The tone synthesizing unit 34b acquires information about the gain correction amount from the gain correction unit 31, and combines a plurality of low resolution images with the high resolution image based on the acquired information.
Note that the gradation enlargement unit described in the claims corresponds to the alignment unit 34a and the gradation synthesis unit 34b.

[Description of operation]
3 and 4 are flowcharts for explaining the operation of the electronic camera 10. Hereinafter, this operation will be described along the step numbers shown in these drawings.

Step S1: When the main power supply of the electronic camera 10 is turned on, the microprocessor 19 instructs the imaging control unit 14 to read through images. The imaging control unit 14 drives the imaging device 11 in a low-resolution readout mode, and sequentially reads through images at, for example, 30 frames / second as shown in FIG.

Step S2: The through image read from the image sensor 11 is processed through the signal processing unit 15 and the A / D conversion unit 16. Of the through images, those with little change in brightness due to exposure adjustment described later are displayed as moving images on the monitor display unit 30. The user can determine the shooting composition of the electronic camera 10 by viewing the moving image display of the through image.
On the other hand, the microprocessor 19 performs exposure calculation based on the photometry result of the photometry unit (not shown) and the brightness of the through image, and determines the exposure condition of the high resolution image.

Step S3: The imaging control unit 14 drives the imaging element 11 in the low resolution readout mode during the through image capturing period. As a result, as shown in FIG. 5, one or more through images (low-resolution images) are generated.
At this time, the imaging control unit 14 adjusts the exposure condition of the low-resolution image so that the gradation reproduction range is different from that of the high-resolution image. For example, the exposure time of the low resolution image is changed to −2 steps, −1 step, +1 step, and +2 steps with respect to the exposure time of the high resolution image. The low resolution image generated in this way is temporarily recorded in the memory 17.
If the number of low-resolution images recorded exceeds a predetermined upper limit, the imaging control unit 14 deletes the old low-resolution images in order. The upper limit number is preferably determined according to the usable storage capacity of the memory 17.

Step S4: Here, the microprocessor 19 determines whether or not the release button has been fully pressed by the user.
When the release button is fully pressed, the microprocessor 19 proceeds to step S5.
On the other hand, when the release button is not fully pressed, the microprocessor 19 returns the operation to step S1.

Step S5: Here, the microprocessor 19 determines whether or not the exposure time of the high-resolution image determined in Step S2 is equal to or less than an allowable upper limit where blur is not noticeable. For example, this allowable upper limit is set to about 1 / (35 mm equivalent focal length of the photographing lens 12) seconds.
If the exposure time setting is less than or equal to the allowable upper limit, the microprocessor 19 proceeds to step S6. On the other hand, when the exposure time setting exceeds the allowable upper limit, the microprocessor 19 shifts the operation to step S7.

Step S6: The imaging control unit 14 performs shutter control on the imaging device 11 according to the set exposure time. Subsequently, the imaging control unit 14 drives the imaging device 11 in a high-resolution readout mode and reads out a high-resolution image. This high resolution image (still image) is temporarily recorded in the memory 17 via the signal processing unit 15 and the A / D conversion unit 16.
After this operation, the microprocessor 19 moves the operation to step S9.

Step S7: On the other hand, if it is determined that the exposure time setting exceeds the allowable upper limit of blurring, the microprocessor 19 limits the exposure time to an allowable upper limit that does not cause blurring.
The imaging control unit 14 performs shutter control on the imaging element 11 according to the exposure time limited to be short. In this state, the imaging control unit 14 drives the imaging device 11 in a high resolution readout mode and reads out a high resolution image. This high-resolution image is an image with a low signal level due to insufficient exposure but with little image blur. This high resolution image is temporarily recorded in the memory 17.

Step S8: The gain correction unit 31 corrects the gain of the under-exposed high resolution image.

Step S9: The gain correction unit 31 determines whether or not the high resolution image and the low resolution image in the memory 17 have been gamma corrected. The gain correction unit 31 performs inverse gamma correction on the gamma-corrected image (in this case, it is preferable to increase the number of quantization bits of the gradation component so that the gradation width of the image is not substantially limited).
With this processing, it is possible to perform image composition processing, which will be described later, on a substantially linear gradation axis.

Step S10: The feature amount extraction unit 33 takes in a high-resolution image and extracts a vertical edge component gv and a horizontal edge component gh using an edge extraction filter.
Note that the edge extraction filter here is preferably switched as follows according to the low-resolution image readout method.
・ When a low-resolution image is created by pixel addition or pixel averaging
gv (x, y) = [-f (x, y-4) -f (x, y-3) -f (x, y-2) -f (x, y-1) + f (x, y +4) + f (x, y + 5) + f (x, y + 6) + f (x, y + 7)] / 4
gh (x, y) = [-f (x-4, y) -f (x-3, y) -f (x-2, y) -f (x-1, y) + f (x + 4 , y) + f (x + 5, y) + f (x + 6, y) + f (x + 7, y)] / 4
・ When a low-resolution image is created by pixel decimation
gv (x, y) =-f (x, y-4) + f (x, y + 4)
gh (x, y) =-f (x-4, y) + f (x + 4, y)
In order to reduce the influence of noise, the feature amount extraction unit 33 preferably replaces the vertical edge component gv and the horizontal edge component gh that fall within a predetermined minute amplitude with zero.
Based on the edge components gv and gh, the feature amount extraction unit 33 selects a region having a large amount of edge components in the image and sets the region as a region of interest.

Step S11: Since the low-resolution image has a different exposure condition from the high-resolution image, the luminance level is different. Therefore, the gain correction unit 31 performs gain correction on the low resolution image in the memory 17 to match the luminance level of the high resolution image.
For example, a low-resolution image that has been subjected to n-stage exposure correction can be adjusted to the luminance level of the high-resolution image by multiplying the linear luminance level by 1 / (2 ⁿ ).
Note that the gain of the low-resolution image may be corrected so that the luminance levels of the high-resolution image and the low-resolution image coincide with each other in the attention area obtained in step S10.

Step S12: The reduced image generation unit 32 converts the resolution of the gain-adjusted high resolution image to match the number of pixels of the low resolution image.
For example, by extracting an average value of 4 × 4 pixels, the number of vertical and horizontal pixels of a high-resolution image can be converted to 1/4 each.
The image with the reduced resolution (hereinafter referred to as a reduced image) is transmitted to the feature amount extraction unit 35.

Step S13: Subsequently, the coarse detection unit 36 detects a positional shift between the reduced image and the low resolution image.

FIG. 6 is a diagram showing a procedure for obtaining this positional deviation by comparing projected edges. Hereinafter, a process for detecting a positional shift at high speed will be described with reference to FIG.
First, the feature quantity extraction unit 35 extracts a vertical edge component gv ′ from the reduced image f (x, y) using a vertical edge extraction filter (see FIG. 6A) represented by the following equation.
gv ′ (x, y) =-f (x, y-1) + f (x, y + 1)
Furthermore, the feature quantity extraction unit 35 extracts a horizontal edge component gh ′ from the reduced image f (x, y) using a filter for horizontal edge extraction (see FIG. 6B) expressed by the following equation.
gh ′ (x, y) =-f (x-1, y) + f (x + 1, y)
In order to reduce the influence of noise, the feature amount extraction unit 35 preferably replaces the vertical edge component gv ′ and the horizontal edge component gh ′ that fall within a predetermined minute amplitude with zero.
Next, as shown in FIG. 6A, the feature quantity extraction unit 35 calculates the vertical projection waveform by cumulatively adding the vertical edge component gv ′ to the horizontal unit.
Furthermore, as shown in FIG. 6B, the feature quantity extraction unit 35 calculates a horizontal projection waveform by accumulating the horizontal edge component gh ′ in units of vertical columns.
On the other hand, the feature amount extraction unit 39 captures a plurality of low resolution images from the memory 17. The feature amount extraction unit 39 performs the same processing as the feature amount extraction unit 35 on each low-resolution image, and obtains a vertical projection waveform and a horizontal projection waveform, respectively.
Here, as shown in FIG. 6A, the coarse detection unit 36 takes a difference while shifting the vertical projection waveform in the central area of the reduced image and the vertical projection waveform in the central area of the low-resolution image, and the difference The waveform shift that minimizes the absolute value sum of is detected. This waveform shift corresponds to the vertical position shift between the reduced image and the low resolution image.
Further, as shown in FIG. 6B, the coarse detection unit 36 calculates a difference while shifting the horizontal projection waveform in the central area of the reduced image and the horizontal projection waveform in the central area of the low-resolution image, and calculates the absolute difference. Detects the waveform shift that minimizes the sum of values. This waveform shift corresponds to a horizontal position shift between the reduced image and the low resolution image.
In this way, the coarse detection unit 36 obtains positional deviations (coarse detection results) of the plurality of low resolution images using the reduced image as a position reference, and outputs the obtained positional deviations to the precise detection unit 38.

Step S14: Next, a positional shift between the high resolution image and the low resolution image is detected precisely.
First, the feature amount extraction unit 33 calculates the vertical projection waveform of the high-resolution image by cumulatively adding the vertical edge component gv obtained in step S10 to the horizontal unit. In addition, the feature amount extraction unit 33 calculates the horizontal projection waveform of the high-resolution image by accumulating and adding the horizontal edge component gh in units of vertical columns in step S10.
The phase division unit 37 subsamples the vertical projection waveform of the high resolution image every four pixels. At this time, the phase splitting unit 37 generates four types of sampling information whose phases are shifted from each other as shown in FIG. 7 by shifting the sub-sampling phase.
Similarly, the phase division unit 37 subsamples the horizontal projection waveform of the high resolution image every four pixels. At this time, by shifting the sub-sampling phase, four types of sampling information whose phases are shifted from each other are generated.

Step S15: The precision detection unit 38 shifts the sampling information of the vertical projection waveform obtained from the high resolution image and the vertical projection waveform of the low resolution image from the coarse detection result of the positional deviation by the coarse detection unit 36 as a starting point. The difference is taken, and the waveform shift that minimizes the sum of the absolute values of the differences is detected.
The precision detection unit 38 performs the detection of the waveform deviation for each of the four types of sampling information, thereby obtaining the waveform deviation that most closely matches the feature of the pattern (in this case, the waveform). This waveform shift corresponds to a positional shift in the horizontal direction. Further, the precision detection unit 38 similarly detects a positional deviation in the vertical direction.
In this way, the precision detection unit 38 obtains positional deviations (precision detection results) of a plurality of low resolution images with a high resolution image as a position reference with a finer accuracy than the pixel interval of the low resolution image, and the alignment unit. To 34a.

Step S16: The alignment unit 34a enlarges (4 × 4 times) the low-resolution image. At this time, the alignment unit 34a does not perform pixel interpolation, and obtains an enlarged image with a large pixel interval.
Next, the alignment unit 34a displaces the pixel position of the enlarged image of the low resolution image based on the precise detection result of the positional deviation obtained by the precise detection unit 38, and performs mapping (rearrangement) as shown in FIG. )I do. In this way, a rearranged image having the same number of vertical and horizontal pixels as the high-resolution image can be obtained.

Step S17: The rearranged image that has been subjected to the mapping process includes pixels that are not filled with gaps, pixels that deviate from normal pixel positions, and overlapping pixels.
Therefore, the alignment unit 34a picks up neighboring pixels for each normal pixel position of the rearranged image. The alignment unit 34a applies a Gaussian filter of the following expression to the color difference components of these neighboring pixels.
f (x, y) is a color difference component at the pixel position (x, y) of the rearranged image. m is the range of neighboring pixels. σ is a numerical value for adjusting the weight ratio. For example, m = 5 and σ = 3 are preferable as specific numerical values.
The alignment unit 34a determines the color difference component of the rearranged image by using the Gaussian filter calculation result as the color difference component at the normal pixel position.

Step S18: The gradation composition unit 34b performs the following filter processing on the luminance component of the high resolution image. First, the tone synthesizing unit 34b extracts a luminance component from the high-resolution image after gain correction, and performs filter processing that combines median processing and Gaussian filtering. For example, the Gaussian filter is implemented by setting the filter size to 3 × 3 pixels and extracting the central three values from nine pixels within the filter size. By this processing, it is possible to reduce noise generated in the luminance component due to insufficient exposure.

Step S19: The gradation composition unit 34b extracts an image area having a high possibility of gradation collapse from the high resolution image as map information. For example, an image region whose luminance level reaches the saturation level of the image sensor 11 is extracted as a high luminance region. Further, for example, an image area whose luminance level does not exceed the blacked-out range of the image sensor 11 is extracted as a low luminance area.

Step S20: The tone synthesizing unit 34b acquires information about the gain correction amount of the low-resolution image performed in Step S11 from the gain correcting unit 31. In accordance with the gain correction amount, the tone composition unit 34b determines the composition ratio of the high resolution image and the low resolution image as follows.
(1) Low-luminance region of high-resolution image: The composition ratio of a low-resolution image with a gain correction amount smaller than 1 is set to a high value, for example, about 30 to 50%. On the other hand, for a low resolution image with a gain correction amount greater than 1, the composition ratio is set to a low value of about 0%. With this setting, it is possible to reflect the tone information of the dark part of the low-resolution image subjected to plus exposure correction in the high-resolution image.
(2) High-luminance region of high-resolution image: The composition ratio of a low-resolution image with a gain correction amount larger than 1 is set to a high value, for example, about 30 to 50%. On the other hand, for a low resolution image with a gain correction amount smaller than 1, the composition ratio is set low to about 0%. With this setting, it is possible to reflect the gradation information of the bright part of the low-resolution image subjected to the minus exposure correction in the high-resolution image. (3) Edge of high-resolution image: A portion (edge) where the luminance component changes locally exceeding a predetermined ratio is extracted. The composition ratio of the low resolution image with respect to the edge is lowered to about 0%. With this setting, the edge structure of the high resolution image can be maintained.

Step S21: The alignment unit 34a corrects the positional deviation of the low resolution image according to the precise detection result of the positional deviation detected in Step S15. The tone synthesizing unit 34b extracts a luminance component from the low-resolution image after the positional deviation correction, and weights and adds the luminance component to the luminance component of the high-resolution image according to the synthesis ratio determined in step S20. By this weighted addition, the gradation reproduction range of the composite image is expanded substantially linearly.

Step S22: The tone synthesizing unit 34b performs tone correction on the luminance component of the synthesized image using tone conversion characteristics as shown in FIG. In this gradation conversion characteristic, gradation information restored in the high luminance region is within a practical signal value range by compressing the gradation range of medium to high luminance. Furthermore, for the low-brightness gradation range, the gradation information restored in the low-brightness area falls within a practical signal value range by suppressing black crushing and expanding the gradation to such an extent that dark noise does not increase. .

Step S23: The color difference component (rearranged image) generated in step S17 and the luminance component (composite image) generated in step S22 are combined to complete a color image with an expanded gradation reproduction range.
This color image is recorded and stored in the recording medium 22a via the image compression unit 24, the recording unit 22, and the like.

[Effects of the embodiment, etc.]
In this embodiment, a low resolution image generated in the low resolution readout mode is used for gradation expansion of a still image (high resolution image). This low resolution image is read out at a high speed of, for example, 30 to 60 frames / second. Therefore, when synthesizing a high resolution image and a low resolution image, it is difficult for a failure due to a difference in the pattern to occur, and a good gradation expansion effect can be obtained.

  Further, in the present embodiment, a plurality of low resolution images are generated while correcting exposure in the plus direction and the minus direction. Therefore, gradation information can be obtained for various gradation areas. Therefore, as shown in FIG. 10, a good gradation expansion effect can be obtained in the high luminance and low luminance gradation regions.

  In addition, even if a plurality of low-resolution images are captured in this manner, continuous capture can be performed at a high frame rate, so that capturing is completed in an instant. Therefore, the time for the user to fix the camera angle is much shorter than before. Therefore, the user can easily perform gradation enlargement photography.

  Further, in the present embodiment, a plurality of sampling information whose sampling phases are shifted from each other is generated from the high resolution image. By detecting the positional deviation between the sampling information and the low resolution image, the positional deviation can be detected with a finer accuracy than the pixel interval of the low resolution image. As a result, it is possible to further improve the alignment accuracy of the pattern of the high resolution image and the low resolution image, and to obtain a further excellent gradation expansion effect.

  In the present embodiment, the synthesis ratio of the low-resolution image is adaptively lowered for the region where the gradation is less crushed. Therefore, it is possible to leave the original gradation information of the high resolution image faithfully.

  Furthermore, in the present embodiment, the synthesis ratio of the low resolution image is locally lowered at the edge of the high resolution image. For this reason, it is possible to avoid problems such as edges becoming multi-line after synthesis.

<< Additional items of embodiment >>
In addition, this inventor has disclosed the procedure which further speeds up position shift detection in Japanese Patent Application No. 2005-345715. According to this procedure, the position shift detection of this embodiment may be speeded up.

  In step S13, absolute positional deviation is roughly detected between the reduced image of the high resolution image and the low resolution image. However, the present invention is not limited to this. A relative positional shift may be roughly detected between a plurality of low-resolution images. The precise detection unit 38 can roughly know the remaining absolute positional deviation based on the relative coarse detection result and the precise detection result of at least one positional deviation. The precision detection unit 38 can quickly detect a precise positional deviation by performing a positional deviation search using the absolute rough detection result as a starting point.

  In the above-described embodiment, the positional deviation of the image is detected by comparing the projected waveforms. However, the present invention is not limited to this. For example, the positional deviation may be detected by spatial comparison of the pixel arrangement of both images.

  In the above-described embodiment, the case where the image processing apparatus 25 is mounted on the electronic camera 10 has been described. However, the present invention is not limited to this. An image processing program in which the above-described image processing is program-coded may be created. By executing this image processing program on a computer, it is possible to effectively utilize the gradation information of the low-resolution image and expand the gradation of the high-resolution image.

  In the above-described embodiment, the low resolution image is acquired before the high resolution image is captured. However, the present invention is not limited to this.

  For example, the low-resolution image may be acquired after the high-resolution image is captured. Further, a plurality of low resolution images may be acquired before and after the high resolution image is captured.

  In the above-described embodiment, the gradation information of the low resolution image is supplemented for both the high luminance region and the low luminance region. However, the present invention is not limited to this. The gradation information of the low resolution image may be supplemented with respect to one of the high luminance region and the low luminance region. For example, the gradation reproduction area of the high luminance area can be expanded by supplementing the gradation information of the low resolution image subjected to the minus exposure correction to the high luminance area of the high resolution image. Further, for example, the gradation reproduction area of the low luminance area can be expanded by supplementing the gradation information of the low resolution image subjected to plus exposure correction to the low luminance area of the high resolution image.

  In the above-described embodiment, the case where the image signal of the luminance color difference is handled has been described. However, the present invention is not limited to this. In general, the present invention may be applied to cases where RGB, Lab and other image signals are handled.

  In the above-described embodiment, the positional deviation of the pattern is detected by image processing. However, the present invention is not limited to this. For example, an acceleration sensor or the like may be mounted on the camera side to determine the movement (vibration) of the shooting area of the camera, and the positional deviation of the pattern of the multiple images may be detected from the movement (vibration) of the shooting area.

  In the above-described embodiment, only the low resolution image with different exposure conditions is combined with the high resolution image. However, the present invention is not limited to this. For example, a high resolution image and a low resolution image having the same exposure conditions may be added to the synthesis. For a low-resolution image having the same exposure condition, for example, there is an effect of increasing the S / N of the composite image.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiments are merely examples in all respects and should not be interpreted in a limited manner. The scope of the present invention is shown by the scope of claims, and is not restricted by the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

  As described above, the present invention is a technique that can be used for an image processing apparatus or the like.

It is a block diagram showing electronic camera 10 (including image processing device 25) of this embodiment. 3 is a block diagram schematically showing a configuration of an image processing device 25. FIG. 5 is a flowchart (1/2) illustrating the operation of the electronic camera 10. 3 is a flowchart (2/2) illustrating an operation of the electronic camera 10. 3 is a flowchart illustrating a shooting sequence of the electronic camera 10. It is a figure explaining the detection procedure of position shift. It is a figure explaining the subsampling of a high resolution image. It is a figure explaining the production | generation of a rearranged image. It is a figure explaining the gradation conversion of a synthesized image. It is a figure explaining the gradation expansion effect.

DESCRIPTION OF SYMBOLS 10 ... Electronic camera, 11 ... Imaging device, 12 ... Shooting lens, 14 ... Imaging control part, 15 ... Signal processing part, 16 ... A / D conversion part, 17 ... Memory, 18 ... Bus, 19 ... Microprocessor, 19a ... Operation unit, 22 ... recording unit, 22a ... recording medium, 24 ... image compression unit, 25 ... image processing device, 30 ... monitor display unit, 31 ... gain correction unit, 32 ... reduced image generation unit, 33 ... feature amount extraction unit , 34a ... alignment unit, 34b ... gradation synthesis unit, 35 ... feature amount extraction unit, 36 ... coarse detection unit, 37 ... phase division unit, 38 ... precision detection unit, 39 ... feature amount extraction unit

Claims (4)

An image input unit that captures a low-resolution image and a high-resolution image obtained by photographing the same subject while changing exposure conditions and resolution;
A correction unit that corrects the luminance level of the low-resolution image based on the luminance level of the high-resolution image for a low-resolution image with different exposure;
A misalignment detection unit that detects misalignment of a pattern between the low resolution image and the high resolution image;
By aligning the low-resolution image and the high-resolution image based on the positional deviation, extracting gradation information of the low-resolution image and synthesizing the high-resolution image, a gradation reproduction range A gradation enlargement unit that generates a composite image in which
The gradation enlargement part
Determining a brightness level for the high-resolution image;
For a relatively high brightness region of the high resolution image, the composition ratio of the relatively underexposed low resolution image is set larger than the composition ratio of the relatively overexposed low resolution image,
For the relatively low brightness area of the high resolution image, the composition ratio of the low resolution image that is relatively overexposed is set larger than the composition ratio of the low resolution image that is relatively underexposed,
Characterized in that based on the synthesis ratio, the gradation information of each pixel included in the corresponding range of the high resolution image, weighted addition of the gradation information of each pixel included in the corresponding range of the low-resolution image An image processing apparatus. An image processing apparatus according to claim 1;
An imaging unit that captures a subject with at least two resolutions;
The high-resolution image processed by the image processing device is a high-resolution still image captured by the imaging unit,
The low-resolution image processed by the image processing device is a low-resolution image captured by the imaging unit under exposure conditions different from those of the high-resolution image before and / or after capturing of the still image. A featured electronic camera. The electronic camera according to claim 2,
It has a monitor that displays images,
The imaging unit
Shoot low-through images sequentially, and display a video on the monitor.
An electronic camera, wherein a low-resolution image is shot at a timing that does not overlap with the shooting of the through image and under an exposure condition different from that of the high-resolution image. An image processing program for causing a computer to function as the image processing apparatus according to claim 1.