JP3974634B2 - Imaging apparatus and imaging method - Google Patents

Description

  The present invention relates to an imaging apparatus and an imaging method capable of reducing noise by synthesizing captured images.

In recent years, camera miniaturization has progressed as an imaging apparatus, and camera shake correction technology has attracted attention.
With the downsizing of the camera, the holdability is reduced in photography, and the degree of freedom is widened as compared with conventional photography forms such as photography with one hand as a photography method such as a camera equipped with a mobile phone.

As described above, when shooting is performed in various methods or in an unstable state, the camera shakes and shakes when the shutter button is pressed.
For example, when shooting under dark conditions with a long exposure time, there are many cases where the shutter speed becomes slow and blurring occurs.
In addition, when the sensitivity is increased under the exposure conditions and the exposure time is shortened, noise is added and the image becomes rough.

In order to solve this problem, an apparatus for correcting camera shake is mounted on a single-lens reflex camera or recently a compact camera.
For example, an optical camera shake correction device that inclines and corrects a lens in accordance with a shake has been commercialized, but it is very difficult in terms of space in consideration of recent downsizing and mounting in a mobile phone camera.

As other methods, various imaging devices that can obtain a blur-free image from a plurality of images by image processing have been proposed (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 10-108079

However, the conventionally proposed imaging apparatus has a disadvantage that a memory for holding a plurality of images is necessary and expensive.
Further, as the number of shots increases, there is a disadvantage that the processing time becomes longer because a positional shift between images occurring during shooting is corrected.

  An object of the present invention is to provide an imaging apparatus and an imaging method capable of obtaining a blur-free image with a small number of images under different exposure conditions and reducing the processing time.

  According to a first aspect of the present invention, there is provided an imaging apparatus having a function of combining a plurality of captured images taken under different exposure conditions, and detecting means for detecting a positional deviation and a blur amount between the plurality of images. Taking a pixel difference from the first image and the second image with the positional deviation corrected based on the detection result of the detection means, and if the difference value exceeds a preset threshold value, the pixel is regarded as the edge of the image. The first image synthesis ratio is determined to be higher, and when the difference value is equal to or smaller than the threshold value, the first image synthesis ratio is decreased to synthesize the image.

  Preferably, at least one of the plurality of images to be combined includes an image with noise and high resolution.

  Preferably, at least one of the plurality of images to be combined includes an image with low resolution but less noise.

  Preferably, the threshold value set in the synthesizing unit can be changed by a gain magnification with respect to an image signal determined from an exposure condition when an image is captured.

  Preferably, the synthesizing unit performs image synthesis based on at least a G component of data which is an output of the image sensor.

  Preferably, the synthesizing unit performs image synthesis based on at least the luminance signal among the luminance signal Y and the color difference signals U and V.

  Preferably, the detection means extracts a positional shift and a blur amount between images from a difference image of a plurality of images.

  Preferably, as the distance from the edge increases, the composition unit increases the ratio of images with little long-time exposure noise, and the edge portion increases the ratio of images with high resolution of short-time exposure. Set as follows.

  Preferably, when the distance from the edge is closer than a preset distance, the synthesizing unit synthesizes the image by increasing the ratio of the short-time exposure image.

  Preferably, when the distance from the edge is longer than a preset distance, the synthesizing unit synthesizes the image by increasing the ratio of the long-time exposure image.

  According to a second aspect of the present invention, there is provided an imaging method for combining a plurality of captured images taken under different exposure conditions, the first step of detecting a positional deviation and a blur amount between the plurality of images, and detection. When the difference between the first step for correcting the positional deviation based on the result and the second step for obtaining a pixel difference from the second image is compared with a preset threshold, and the difference exceeds the threshold Determines that the pixel is an edge of the image, increases the first image composition ratio, and sets the first image composition ratio to be lower when the difference value is equal to or less than the threshold value; And a fourth step of combining the images with the ratio set in the step.

  According to the present invention, a blur-free image can be obtained with a small number of images with different exposure conditions, and the processing time can be shortened.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing an embodiment of an imaging apparatus according to the present invention.

  The imaging apparatus 10 includes an optical system 11, an imaging element 12, a signal processing unit 13, a CPU 14 as a control and arithmetic processing unit, and a memory 15.

The imaging apparatus 10 according to the present embodiment has a function of capturing a plurality of images including an image with a short exposure time and high resolution but a lot of noise, and an image with a long exposure time but low noise and a low resolution.
Then, the imaging device 10 detects a positional shift and a blur amount between the captured images, and corrects the positional shift based on the detection result. If this difference value exceeds a preset threshold value, this pixel is judged to be an edge of the image and the first image composition ratio is increased, and if the difference value is less than the threshold value, the first image The image forming ratio is lowered (the second image is increased), and the image is combined to form an image that reduces noise without blurring.

  The composition processing of this embodiment can be performed on the output (RAW data) of the RGB Bayer array from the sensor, but the YUV format (Y is a luminance signal and U and V are color difference signals) after the signal processing of the captured image. This can also be applied to the case where the luminance signal Y in is used for image synthesis.

  The imaging device 12 is formed by a CCD or CMOS sensor or the like, receives an optical image of a subject passing through the optical system 11, photoelectrically converts it, and supplies it to the signal processing unit 13. In the following description, it is assumed that the image sensor 12 is formed by a CCD.

  The optical system 11 and the image sensor 12 of the present embodiment have a low CCD sensitivity under the control of the CPU 14, for example, a long exposure shooting with a low noise but a low shutter speed, and a high noise by changing the CCD sensitivity. It is possible to perform shooting with a short exposure time.

The signal processing unit 13 performs processing such as color interpolation, white balance, YUV conversion processing, compression, and filtering, and performs storage in the memory 15 and the like.
The signal processing unit 13 according to the present embodiment has a function of extracting a luminance signal portion from each imaged data in the optical system 11 and the image sensor 12 and a filtering processing function of removing a noise component of the luminance signal Y. The filtering process is performed with a filter coefficient for reducing the noise component without reducing the resolution. A median filter or an edge preserving filter can be applied to general ones.
The signal processing unit 13 stores an image after filtering processing corresponding to each shooting data in the memory 15.

  The CPU 14 performs exposure control and has operation inputs such as an operation unit (not shown), determines the operation of the entire system according to those inputs, and controls the optical system 11, the image sensor 12, the signal processing unit 13, and the like. It governs mediation control for the entire system.

Further, the CPU 14 detects a positional shift and a blur amount between the captured images based on the data after the signal processing of the signal processing unit 13 stored in the memory 13, and the difference between the first image and the second image. It has a function of forming an image in which noise is reduced without blurring by combining the images by changing the ratio of the images in accordance with the comparison result between the threshold value and the threshold value.
More specifically, a detection function for detecting a positional shift and a blur amount between a plurality of images with different exposure conditions, and a first image with a short exposure time in which the positional shift is corrected based on a detection result of the detection function And if the difference value exceeds a preset threshold value, the pixel is determined to be an edge of the image, the composition ratio of the first image is increased, and the difference value Is equal to or less than the threshold value, the composition ratio of the second image is increased to synthesize the image.

The CPU 14 extracts a Gr component from the RAW pixel data of a plurality of captured images, obtains a difference, and separates an edge, a blur component, and a noise component from the difference image by a threshold value.
For example, in the correction function, the CPU 14 calculates an absolute value of a difference image (short-time exposure image S1−long-time image exposure S2) between the images that have been subjected to the positional shift correction processing.
Since this image is obtained by subtracting a long-time exposure blur image from a short-time exposure image, an edge image including noise and blur is generated.
As the distance from the edge increases, the CPU 14 increases the ratio of images with less noise in long-time exposure, and sets the edge portion so that the ratio of images with high resolution of short-time exposure increases.
For example, the ratio of the short-exposure image may be set to 100% for the edge portion, and the ratio of the long-exposure image may be increased stepwise as the distance from the edge increases, and the ratio may be set to 50% for the noise portion.
Then, when the distance from the edge is closer than the preset distance, the CPU 14 synthesizes the image by increasing the ratio of the short-time exposure image. When the distance from the edge is longer than the preset distance, the CPU 14 The ratio is increased to synthesize an image.
The CPU 14 stores the processed image data in the memory 15.

  Hereinafter, the reason for using at least the G (r) component for image synthesis among a plurality of image examples photographed under different exposure conditions and RAW data as the output of the image sensor will be described in more detail.

2A to 2C are diagrams showing two images taken under different exposure conditions and an example of a composite image thereof.
FIG. 2A shows an image with a lot of noise because the exposure time is short and there is no blur but the sensitivity is high. For example, the image was taken with a short shutter speed and high sensitivity, and the resolution was high, but the sensitivity was increased and noise was added to the entire image.
FIG. 2B shows an image with little noise because the exposure time is long and blurring but the sensitivity is low. For example, the image was taken with a long shutter speed and low sensitivity, and the resolution is low, and there is a possibility that camera shake has occurred, but the image is low in noise. FIG. 2 (c) is an image obtained by combining FIGS. 2 (a) and (b).
Even if a plurality of images are continuously photographed, as shown in FIGS. 2A to 2C, positional deviation occurs between the photographing.

FIGS. 3A to 3C show images obtained by extracting a Gr component from a plurality of RAW image signals captured as shown in FIG. In the RAW image signal, since the white balance gain is already applied to R · B at the analog stage, the R · B component has more noise than the G component. Therefore, in this embodiment, the Gr component is extracted.
It is also possible to use other R, B, and Gb. In the present embodiment, the difference is obtained after correcting the positional deviation between a plurality of images, and the image includes the edge and noise of short-time exposure and the blur of long-time exposure. The edge / blur component and the noise component are separated from the difference image by the threshold Th.
In the present embodiment, a plurality of images are synthesized at a ratio according to the distance from the edge with reference to an image having a high resolution but a lot of noise from the difference image.

  FIGS. 4A to 4C are diagrams showing the relationship between the difference image and the ratio in composition, in which FIG. 4A shows an image before noise reduction processing, and FIG. 4B shows a subject portion. FIG. 4C shows an enlarged image and an output image.

The edge portion of the difference image is as shown in the figure. Since pixels in the vicinity of the image do not change greatly, an image with relatively little change is obtained when the distance from the edge exceeds a predetermined distance.
Therefore, in the vicinity of the edge, as shown in FIG. 3B, the difference image is an image obtained by extracting the short-time exposure edge and noise and the long-time exposure blur.
This is divided by a preset threshold value Th, and when the signal is larger than the threshold value Vth, it is determined as edge blurring, and the composition ratio of short-time exposure is increased.
If the signal is smaller than the threshold value Th, it is determined as noise and the long exposure composition ratio is set large. Then, for example, the composition ratio is set for each pixel, and two images are composed.
For example, an image with noise, such as a wall behind the subject or a uniform surface, is an image with noticeable unevenness in noise. By performing this processing, noise can be reduced.

In this embodiment, a plurality of images including an image with a short exposure time and high resolution but a lot of noise and an image with a long exposure time but low noise and a low resolution are taken. When detecting and synthesizing positional deviation and blur between photographing, a difference image between two images to be synthesized is generated. The edge component, blur component, and noise component of the two images are extracted from this image. The edge / blur component and the noise component are separated from the difference image using the threshold Th, and the ratio of the short-exposure image without blur is increased in the portion judged to be the edge or blur, and the other portion is a length with less noise. Noise reduction without blurring is achieved by increasing the ratio of time exposure images.
For example, the ratio of the short-exposure image may be set to 100% for the edge portion, and the ratio of the long-exposure image may be increased stepwise as the distance from the edge increases, and the ratio may be set to 50% for the noise portion.

  Next, the operation according to the above configuration will be described with reference to the flowchart of FIG.

First, under the control of the CPU 14, the optical system 11 and the image sensor 12 are driven to photograph, and in step ST1, a short-time exposure photographing with a high CCD sensitivity, that is, a lot of noise but a high shutter speed is performed.
Next, in step ST2, photographing with a long exposure time is performed while changing the CCD sensitivity to reduce noise.
In step ST3, a Gr component is extracted from the photographed data, and in step ST4, a filter process for removing a noise component of the signal is performed. Filter processing is performed with filter coefficients for reducing noise components without reducing resolution. Common examples include median filters and edge preserving filters.
Next, in step ST5, a positional shift of each processed image is detected and corrected. Here, the method for detecting and correcting the deviation is not limited.
In step ST6, a difference image (short-time exposure image S1−long-time image exposure S2) between the images subjected to the positional deviation correction processing is calculated. Since this image is obtained by subtracting a long-exposure blurred image from a short-exposure image, an edge image including noise and blur is generated.
In step ST7, the value ΔS of each pixel of the difference image is compared with the set threshold value Th. In step ST8, if ΔS is larger than Th, the short exposure combination ratio is set high. On the other hand, if ΔS is smaller than Th, in step ST9, the long exposure composition ratio is set high.
In step ST10, this ratio setting is performed for all pixels, and in step ST11, two images are synthesized based on this ratio.

Note that the threshold Th can be changed by a gain magnification with respect to an image signal determined from an exposure condition when an image is captured.
This is to change the value of the threshold Th according to the gain magnification for the sensor signal set at the time of shooting.
For a dark subject or the like, the ISO sensitivity at the time of shooting is set high, and the gain magnification with respect to the output of the sensor is high. Since the noise increases as the gain increases, the threshold for separating the edge and noise based on the difference image is changed in accordance with the gain magnification.

  In the present embodiment, the case where two images are combined has been described, but three or more images may be combined. For example, in the case of three-sheet composition, the ratio in composition may be set by the distance from the edge according to each exposure condition.

  As described above, according to the present embodiment, the noise reduction effect can be achieved by changing the combination ratio by dividing edge blur and noise due to the difference image from the plurality of images photographed under different exposure conditions by the threshold value. In addition, it is possible to shoot corresponding to positional deviation and blur between the shot images. Compared with the prior art, the memory capacity, the processing time, the component mounting space, and the cost can be greatly reduced.

  In the above description, the composition process is performed on the output (RAW data) of the RGB Bayer array from the image sensor (sensor), but the luminance signal Y in the YUV format after the signal processing of the captured image is used for the image composition. It can also be applied to cases.

  In this case, for example, in the correction function, the CPU 14 calculates the difference (short-time exposure image YS−long-time image exposure YL) between the luminance signals Y of the respective shootings with different exposure conditions processed by the signal processing unit 13, and the positional deviation. And configured to detect the amount of blur. Since this image is obtained by subtracting a long-time exposure blur image from a short-time exposure image, an edge image including a blur is generated.

The YUV format of the captured image will be described.
Signals handled in this format are a luminance signal Y, a color difference signal U, and a color difference signal V. Of these signals YUV, the Y signal component greatly affects the resolution.
The human eye is relatively insensitive to color information, such as reducing the color difference signal, such as JPEG, as the principle of image compression. Therefore, by synthesizing the image of the luminance signal Y, a large noise reduction effect can be achieved. it can.
In this case, the processing time is shortened by removing noise only with the Y component of YUV.

  In this embodiment, positional deviation and blurring are extracted from a plurality of difference images. However, it is also possible to detect an accurate deviation amount by pattern matching or the like to change the composition ratio.

It is a block diagram showing one embodiment of an imaging device concerning the present invention. It is a figure which shows the example of several images image | photographed on different exposure conditions. It is a figure which shows the image of the Y signal component image | photographed with short exposure, the image of the Y signal component image | photographed with long exposure, and the difference image of the Y signal component of several images. It is a figure which shows the relationship between a difference image and the ratio in a synthesis | combination. It is a flowchart for demonstrating operation | movement of this embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Optical system, 12 ... Image sensor, 13 ... Signal processing part, 14 ... CPU, 15 ... Memory.

Claims (11)

An imaging device having a function of combining a plurality of captured images taken under different exposure conditions,
Detecting means for detecting a positional deviation and a blur amount between the plurality of images;
A pixel difference is calculated from the first image and the second image whose positional deviation is corrected based on the detection result of the detection means, and when the difference value exceeds a preset threshold value, the pixel is determined to be an edge of the image. Then, the image pickup apparatus includes: a combining unit that increases the combining ratio of the first image and combines the image by decreasing the combining ratio of the first image when the difference value is equal to or less than the threshold value. The imaging apparatus according to claim 1, wherein at least one of the plurality of images to be synthesized includes an image having noise but high resolution. The imaging apparatus according to claim 1, wherein at least one of the plurality of images to be synthesized includes an image with low resolution but low noise. The imaging apparatus according to any one of claims 1 to 3, wherein the threshold value set in the combining unit can be changed by a gain magnification with respect to an image signal determined from an exposure condition when an image is captured. The imaging device according to any one of claims 1 to 4, wherein the synthesizing unit performs image synthesis based on at least a G component of data that is an output of the imaging element. The imaging apparatus according to claim 1, wherein the synthesizing unit performs image synthesis based on at least a luminance signal among the luminance signal Y and the color difference signals U and V. 6. The imaging device according to any one of claims 1 to 6, wherein the detection unit extracts a positional shift and a blur amount between images from a difference image of a plurality of images. As the distance from the edge increases, the compositing means increases the ratio of images with less noise for long-time exposure, and the edge portion is set so that the ratio of images with high resolution for short-time exposure increases. The imaging device according to claim 1. The imaging device according to claim 8, wherein when the distance from the edge is closer than a preset distance, the synthesizing unit synthesizes the image by increasing the ratio of the short-time exposure image. 10. The imaging apparatus according to claim 8, wherein the synthesizing unit synthesizes the image by increasing the ratio of the long-time exposure image when the distance from the edge is longer than a preset distance. An imaging method for combining a plurality of captured images taken under different exposure conditions,
A first step of detecting a positional shift and a blur amount between the plurality of images;
A second step of taking a pixel difference from the first image and the second image in which the positional deviation is corrected based on the detection result;
The difference value is compared with a preset threshold value. If the difference value exceeds the threshold value, the pixel is determined to be an edge of the image, and the composition ratio of the first image is increased. In this case, a third step of setting the first image composition ratio low;
And a fourth step of synthesizing images with the ratio set in the third step.

Images