JP2007049227A - Image processing apparatus and method thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus for forming an image having reduced noise from low luminance to high luminance when images having different exposure times are synthesized and output. <P>SOLUTION: The image processing apparatus has an input means 1 for inputting a long-time exposure image picked up at a first exposure time, and a plurality of short-time exposure images picked up at different times and an exposure time shorter than the first exposure time; image forming means 5-12 each for forming a corrected image having reduced noise compared to the plurality of short-time exposure images by using the plurality of short-time exposure images; and image synthesizing means 13-18 each for forming a composite image on the basis of the corrected image having the reduced noise and the long-time exposure image. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an apparatus and method for synthesizing a plurality of images obtained by imaging a subject with different exposure amounts to form a single image having excellent gradation and a wide dynamic range.

  In recent years, solid-state imaging devices such as CCD imaging devices and CMOS imaging devices have been used as imaging devices for image input devices such as television cameras, video cameras, and digital cameras. However, the dynamic range of the solid-state imaging device is narrower than the dynamic range of the film of a silver salt camera, and it has been desired to expand the dynamic range.

  Therefore, as a method for expanding the dynamic range of the solid-state imaging device, there is a method of capturing a plurality of images with different exposure amounts in the same scene and combining the plurality of images to obtain an image with an expanded dynamic range. .

  Patent Document 1 describes a control method for adjusting the dynamic range width of image data output to an output device to the dynamic range width or output characteristics of the output device. As a method for synthesizing the images, a difference in luminance level between the same pixels is calculated using a non-collapsed pixel as a reference point in both the long exposure image and the short exposure image. Then, by combining the brightness levels of both images and creating a composite image by replacing the long-exposure image signal with a short-exposure image signal that has been differentially corrected when the brightness level of the long-exposure image signal exceeds a set threshold value. A method is disclosed.

  Patent Document 2 discloses a configuration in which a long exposure image and a short exposure image are alternately output. When outputting a short-exposure image, a pixel whose luminance value is a predetermined value or less is determined to be blacked out, and is switched to a long-exposure image signal stored in the frame memory. A configuration is disclosed in which when a long exposure image is output, a pixel whose luminance value is equal to or greater than a predetermined value is determined to be whitened and switched to a short exposure image signal stored in a frame memory.

  Patent Document 3 discloses a configuration for determining an addition ratio for synthesizing a high luminance video signal and a standard luminance video signal from a peak value of the standard luminance video signal.

  Further, in Patent Document 4, when the exposure ratio is large, the signal level of the short-time exposure image signal decreases, and the middle luminance portion of the composite image (near the switching point between the long-time exposure image signal and the short-time exposure image signal). In response to a problem that S / N deteriorates, a short time exposure image signal is multiplexed and added for a plurality of fields to generate a medium time exposure image signal. There are three types of coefficients that vary depending on the level of the input image signal so that the long exposure image signal is assigned to the low luminance portion, the medium exposure image signal is assigned to the medium luminance portion, and the short exposure image signal is assigned to the high luminance portion. A configuration is disclosed in which an exposure image signal is multiplied and combined.

Japanese Patent Application Laid-Open No. 07-131704 Japanese Patent Laid-Open No. 02-174470 JP 2001-094870 A JP 2004-056573 A

  However, in Patent Document 1, a luminance level offset value is simply added to a short-time exposure signal to obtain a composite image. Since the low S / N area of the low-brightness area of the short-time exposure signal is assigned as it is from the middle brightness to the high brightness area of the synthesized image, the S / N of the middle brightness area of the synthesized image is deteriorated.

  Similarly, in Patent Document 2, the S / N of the medium luminance region corresponding to the low-level portion of the short-time exposure image is deteriorated.

  In Patent Document 3, the ratio of addition at the time of synthesis is determined according to the peak value of the short-time exposure signal. In particular, when the peak value is low, the low luminance region of the short-time exposure image is assigned from the medium luminance to the high luminance region of the composite image, and thus the S / N of the medium luminance region of the composite image is extremely deteriorated.

  Further, in Patent Document 4, a plurality of short-time exposure images are simply added to obtain a medium-time exposure image having a large level, and the S / N can be improved by assigning this to the middle luminance portion of the composite image. However, with simple addition, since the noise level increases with the signal level, it is difficult to dramatically improve the S / N of the middle luminance portion of the composite image. In addition, since a short-time exposure image with a low level that does not take any noise countermeasures is assigned to the high-brightness portion of the composite image, the composite image having a low S / N ratio from the middle brightness to the high brightness, particularly in the high-brightness portion. Become.

  An object of the present invention is to provide an image processing apparatus and method for forming an image in which noise is reduced from low luminance to high luminance when images with different exposure times are synthesized and output.

  The image processing apparatus of the present invention includes a long-time exposure image captured at a first exposure time, and a plurality of short-time exposure images captured at an exposure time shorter than the first exposure time at different times. Input means for inputting the image, image forming means for forming a corrected image with reduced noise compared to the plurality of short-time exposure images using the plurality of short-time exposure images, and a corrected image with reduced noise And an image composition means for forming a composite image based on the long-time exposure image.

  The image processing method of the present invention includes a long exposure image captured at a first exposure time, and a plurality of short exposure images captured at an exposure time shorter than the first exposure time at different times. An input step for inputting the image, an image forming step for forming a corrected image with reduced noise compared to the plurality of short-time exposure images using the plurality of short-time exposure images, and a corrected image with reduced noise And an image composition step for forming a composite image based on the long-time exposure image.

  When a composite image is formed based on a long-time exposure image and a short-time exposure image with different exposure times, noise is removed while maintaining high resolution even when the signal level of the short-time exposure image is low. A short-exposure image signal with a high N can be generated, and an image with excellent gradation expression can be obtained.

(First embodiment)
The first embodiment of the present invention focuses on the fact that a signal captured with a short exposure time has a higher ratio of noise to the signal compared to a signal captured with a long exposure time. The image is synthesized after removing noise from the generated signal.

An image processing method in this embodiment will be described.
FIG. 2 shows an example of output waveforms from the image sensor of the short-time exposure image and the long-time exposure image before synthesis in the present embodiment. This is a signal when a subject that is brighter (brightness increases) from left to right. Sig1-a is a first exposure image signal imaged at an exposure time Ta starting from time (time) t1. Sig1-b is a second exposure image signal imaged at an exposure time Ta starting from time (time) t2. Sig1-c is a third exposure image signal imaged with an exposure time Ta starting from time (time) t3. These images are taken with the same exposure time although the start time is different. Next, Sig1-d is an exposure image signal captured at an exposure time Tb that is an exposure time longer than Ta, starting from time (time) t4.

  In the present embodiment, digital filter processing in the time axis direction is performed as a noise removal method. Noise removal by digitally filtering the exposure image signals (pixel data that is different in the time axis direction but the same in spatial coordinates) captured with the first, second, and third exposure times relatively short. To realize. The image data Sig1-a, Sig1-b, and Sig1-c of the same spatial position of the exposure image signal sampled in the time axis direction (that is, at different times) are multiplied by an optimum weighting factor at each sampling time, and convolution is performed. By performing the integration, it is possible to perform a digital filter process in the time axis direction to generate an exposure image signal in which noise is removed and S / N is improved. Convolution integration will be described in detail later.

  FIG. 1 is a configuration diagram for explaining an image processing apparatus. Reference numeral 1 denotes an image signal input terminal, to which a plurality of short-time exposure image signals Sig1-a, Sig1-b, and Sig1-c shown in FIG. 2 and a time-division signal Sig1 of the long-time exposure image signal Sig1-d are input. Reference numeral 2 denotes a first switch, 3 denotes a second switch, which is turned on / off by a select signal 4 (Sel). When the first exposure image signal Sig1-a is input to the input terminal 1, the select signal 4 becomes a high level (H), the second switch 3 is turned on, and the first switch 2 is turned off. Thereafter, the select signal 4 becomes low level (L), the second switch 3 is turned off, and the first switch 2 is turned on. When the select signal 4 is at a high level (H), the second switch 3 is turned on, and the first exposure image signal Sig1-a is input to the three-dimensional FIR type digital filter section via the first frame delay 5. Is done. The first frame delay 5 delays the first exposure image signal Sig1-a by one frame. Thereafter, similarly, second and third exposure image signals corresponding to Sig1-b and Sig1-c in FIG. 2 are input.

  In the three-dimensional FIR digital filter unit, the first exposure image signal Sig1-a is delayed by two frames by the second frame delay 6 and the third frame delay 7, and Sig1-a and Sig1-b in FIG. , Sig1-c are arranged on the same time so that the same pixels can be calculated spatially.

  That is, the first exposure image signal Sig1-a is delayed by three frames by the three frame delays 5, 6, and 7 and input to the multiplier 11. The second exposure image signal Sig1-b is delayed by two frames by two frame delays 5 and 6, and is input to the multiplier 10. The third exposure image signal Sig1-c is delayed by one frame by one frame delay 5 and input to the multiplier 9.

  Sig1-a is multiplied by the gain coefficient K1 of the multiplier 9, Sig1-b is multiplied by the gain coefficient K0 of the multiplier 10, Sig1-c is multiplied by the gain coefficient K2 of the multiplier 11, and the adder 12 multiplies the multiplier 9, The sum of the outputs of 10 and 11 is taken. With this process, the short-time exposure image signals Sig1-a, Sig1-b, and Sig1-c are subjected to a convolution operation between signals at the same pixel position with different time axes, and noise components are reduced by filtering. The output of the adder 12 is input to the multiplier 13, multiplied by G 2 set by the gain coefficient 15, and input to the adder 18. Next, the long-time exposure image signal Sig1-d is input to the input terminal 1, and at this time, the select signal 4 becomes low level (L), the first switch 2 is turned on, and the second switch 3 turns off. The long-exposure image signal Sig1-d that has passed through the first switch 2 is input to the multiplier 14 and multiplied by G1 set by the gain coefficient 17. The long-exposure image signal Sig1-d multiplied by the coefficient G1 is input to the adder 18, and is added to the short-exposure image signal after the digital filter processing to generate a composite image signal 19 having a wide dynamic range. . The gain control unit 16 inputs the output signal of the adder 12 and the long exposure image signal Sig1-d and controls the gain coefficients 15 and 17.

  The first luminance region of the composite image signal 19 is formed mainly of the long-time exposure image Sig1-d, and the second luminance region on the higher luminance side than the first luminance region is output from the adder 12. It is formed mainly of short-time exposure images with reduced noise.

  In the present embodiment, the noise component of the short-time exposure image signal is removed by digital filter processing by convolution operation in the time axis direction for each pixel in the same spatial position in the time axis direction. It is possible to suppress deterioration in image quality due to noise in the short-time exposure image signal at the joint of the composite image signal, which has been a problem in the past. In addition, since noise removal is performed with a digital filter in the time axis direction, the digital filter processing that has been a problem in digital filter processing by convolution of each pixel at different spatial positions used in normal noise removal image processing An epoch-making noise removal process without lowering the spatial resolution by the filter process can be realized.

(Second Embodiment)
An image processing apparatus according to the second embodiment of the present invention will be described with reference to FIG. 1 having the same functions as those in FIG. 1 are denoted by the same reference numerals and detailed description thereof is omitted. The difference from FIG. 1 is that a level detection means 20 and a gain control means 21 are newly provided. 20 detects the signal level of the long exposure image signal Sig1-d, and transmits the signal level of the long exposure image signal Sig1-d to the gain control unit 21. The gain control unit 21 dynamically changes the multiplication coefficient K1 of the multiplier 9, the multiplication coefficient K0 of the multiplier 10, and the multiplication coefficient K2 of the multiplier 11 according to the level of the long-time exposure image signal Sig1-d. Variable control. Hereinafter, the setting method of the multiplication coefficients K0, K1, and K2 will be described. The multiplication coefficients K0, K1, and K2 of the three-dimensional FIR digital filter of the short-time exposure image signals Sig1-a, Sig1-b, and Sig1-c in the time axis direction are set according to the amplitude level of the long-time exposure image signal Sig1-d. Thus, by dynamically changing as in the following (1) to (3), a filtering process and a synthesis process corresponding to the signal level can be performed to generate a synthesized image having an optimum S / N. It becomes possible.

  (1) When the long-exposure image signal level is a small amplitude less than or equal to the first threshold VL, each coefficient of the digital filter is set to, for example, zero (K0 = K1 = K2 = 0), and the short-exposure image signal is Cut and synthesize.

  (2) When the level of the long-exposure image signal is a medium amplitude greater than or equal to the first threshold VL and less than or equal to the second threshold VH, the digital filter coefficients are, for example, K1 = K2 = 0.25 and K0 = 0.5. And a three-dimensional FIR filter having a steep high-frequency cutoff characteristic, outputting a short-time exposure image signal from which a noise component of a high-frequency component is removed, and synthesizing it with a long-time exposure image signal. The center multiplication coefficient K0 is set to be double the side multiplication coefficients K1 and K2. The sum of the center multiplication coefficient K0 and the side multiplication coefficients K1 and K2 is 1.

  (3) In a large amplitude region where the long-exposure image signal level is greater than or equal to the second threshold value VH, the coefficient of the digital filter is set to, for example, K1 = K2 = K0 = 0.333, and the wideband range from low to high Thus, a filter having a constant attenuation characteristic is output, and a short-time exposure image signal from which constant noise removal has been performed is output, particularly in a wide band range, and is combined with the long-time exposure image signal. All of the multiplication coefficients K0, K1, and K2 of the FIR type digital filter are set equal, and the sum of the multiplication coefficients K0, K1, and K2 is 1.

  As described above, by dynamically changing the coefficient of the digital filter for short-time exposure image signal processing in accordance with the level of the long-time exposure image signal, the following effects can be obtained.

  In the case of (1), the output of the short-time exposure image signal is only a noise component, and it is possible to prevent the deterioration of the S / N of the composite image in the low luminance region by stopping this with the digital filter unit. It becomes.

  FIG. 4 shows the frequency characteristics of the three-dimensional digital filter when the multiplication coefficient (2) is set. In the case of (2), since the output of the short-time exposure image signal is subjected to a three-dimensional digital filter processing in the time axis direction having a characteristic (see FIG. 4) that particularly attenuates the high frequency component, it is a conspicuous short time at a small amplitude. Since the high-frequency noise component of the exposure image signal is removed and the short-time exposure signal with improved S / N is sent to the synthesis unit, the S / N of the composite image in the medium luminance region is improved.

  FIG. 5 shows the frequency characteristics of the three-dimensional digital filter when setting the multiplication coefficient in (3). In the case of (3), since the signal level of the short-time exposure image signal is large and the signal is good in S / N, it is not necessary to perform the three-dimensional digital filter processing having a steep attenuation characteristic in the high frequency area. The S / N of the composite image can be secured. Instead, a short-time exposure image signal that has a characteristic of attenuating a low-frequency noise component (1 / f noise) that is noticeable when the short-time exposure image signal has a large amplitude (see FIG. 5) and that has been subjected to noise removal in a wide band range. Since it is sent to the combining unit, the S / N of the combined image in the high luminance region is improved.

  As described above, according to the first and second embodiments, noise that does not deteriorate in resolution at all can be removed by digitally filtering a plurality of short-time exposure images captured in the same exposure time in the time axis direction. Since the generated short-time exposure image is generated and then assigned to the composite image, it is possible to improve the S / N deterioration in the medium luminance region of the composite image, which has been a problem in Patent Documents 1 to 4. In addition, it is possible to dramatically improve the S / N deterioration in the high luminance region of the composite image, which has been a problem in Patent Document 4.

  In the method of synthesizing images from image signals with different exposure times, even when the level of the short-time exposure image signal is low, a high-S / N short-time exposure image signal with high noise is generated while maintaining high resolution. It is possible to obtain an image with excellent gradation expression.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

1 is a diagram illustrating an image processing apparatus for wide dynamic range image synthesis according to a first embodiment of the present invention. FIG. It is a figure which shows the short exposure image signal and long exposure image signal of a pre-combination process. It is a figure which shows the image processing apparatus for the wide dynamic range image composition by the 2nd Embodiment of this invention. It is a figure which shows the frequency characteristic of the three-dimensional digital fill when a long-time exposure image signal is medium brightness | luminance below VL and VH. It is a figure which shows the frequency characteristic of a three-dimensional digital filter when a long-time exposure image signal is the high brightness | luminance more than VH.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image signal input terminal 2 from an image sensor 1st switch 3 2nd switch 4 Select signal 5 1st frame delay 6 2nd frame delay 7 3rd frame delay 9 Multiplier 10 Multiplier 11 Multiplier 12 Adder 13 Multiplier 14 Multiplier 15 Gain coefficient G1
16 Gain control unit 17 Gain coefficient G1
18 Adder 19 Composite Image Signal

Claims (11)

An input means for inputting a long exposure image captured at a first exposure time and a plurality of short exposure images captured at an exposure time shorter than the first exposure time at different times;
An image forming unit that forms a corrected image with reduced noise compared to the plurality of short-time exposure images using the plurality of short-time exposure images;
An image processing apparatus comprising: an image composition unit configured to form a composite image based on the corrected image with reduced noise and the long-time exposure image.   The first luminance area of the composite image is mainly formed of the long-time exposure image, and the second luminance area on the higher luminance side than the first luminance area is mainly formed of the correction image. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the image forming unit includes an FIR type digital filter.   4. The image processing apparatus according to claim 3, wherein the image forming unit controls a multiplication coefficient of the FIR digital filter in accordance with a signal level of the long-time exposure image.   The image processing apparatus according to claim 4, wherein the image forming unit sets a multiplication coefficient of the FIR digital filter to zero when a signal level of the long-time exposure image is equal to or less than a first threshold value.   The image forming unit sets the center multiplication coefficient of the FIR digital filter to be double the side multiplication coefficient when the signal level of the long-exposure image is not less than a first threshold value and not more than a second threshold value. 6. An image processing apparatus according to claim 4, wherein the image processing apparatus is characterized in that:   The image processing apparatus according to claim 6, wherein a sum of the center multiplication coefficient and the side multiplication coefficient is one.   8. The image forming unit according to claim 4, wherein when the signal level of the long-exposure image is equal to or greater than a second threshold value, all of the multiplication coefficients of the FIR digital filter are set equal. The image processing apparatus according to item 1.   9. The image processing apparatus according to claim 8, wherein the sum of multiplication coefficients of the FIR type digital filter is 1. An input step of inputting a long exposure image captured at a first exposure time and a plurality of short exposure images captured at an exposure time shorter than the first exposure time at different times;
An image forming step of forming a corrected image with reduced noise compared to the plurality of short-time exposure images using the plurality of short-time exposure images;
An image processing method comprising: an image composition step of forming a composite image based on the corrected image with reduced noise and the long-time exposure image.   The first luminance area of the composite image is mainly formed of the long-time exposure image, and the second luminance area on the higher luminance side than the first luminance area is mainly formed of the correction image. The image processing method according to claim 10.

Images