JP2013258444A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device, an image processing method, and a program, capable of reducing noise by lowering a ratio of noise mixed after gain processing, and acquiring an image keeping a dynamic range.SOLUTION: An image processing device 1 comprises: image acquisition means 10 for acquiring a first image group including at least one image at a prescribed exposure time according to a prescribed ISO sensitivity and a second image group including at least one image according to a higher ISO sensitivity than the prescribed ISO sensitivity; level matching means 11 for matching a pixel level of the image included in the second image group with a pixel level of the image included in the first image group; and image synthesizing means 12 for generating a synthetic image by synthesizing the image included in the first image group with the image included in the second image group whose pixel level matched with the pixel level of the first image group.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  In order to reduce camera shake and noise in an imaging apparatus such as a digital camera, various so-called electronic shake correction methods have been proposed. For example, in Patent Document 1, a plurality of continuous shootings with a reduced exposure time per sheet are performed, and alignment processing is performed using motion vectors so that the motion between the obtained images is canceled. A technique for obtaining an image with reduced blur and noise by combining the plurality of images after being performed is disclosed.

JP 2006-157568 A

  In general, there are a plurality of noise sources mixed in an image signal in an image sensor. A method for suppressing the amplitude of noise mixed in the image signal after gain processing applied according to ISO sensitivity will be described below. First, the signal value is amplified and read so that the brightness of the image is higher than the appropriate brightness by gain processing. Thereafter, the signal value once amplified is reduced so that an image of appropriate brightness can be obtained by digital processing. By performing such processing, the amplitude of noise mixed in the image signal after the gain processing is also reduced, so that the ratio of the noise to the signal value in the image with appropriate brightness after the digital processing can be reduced. . However, this method has a potential problem that if the gain of the signal value in the gain processing is excessively increased, the signal value is saturated, resulting in a narrow dynamic range of the image signal.

  The present invention has been made in view of the above problems, and reduces an amount of noise mixed after gain processing to reduce noise and acquire an image with a dynamic range maintained, and image processing An object is to provide a method and a program.

In order to achieve the above object, the present invention provides the following means.
The present invention acquires a first image group including at least one image at a predetermined exposure time according to a predetermined ISO sensitivity, and includes at least one image according to an ISO sensitivity higher than the ISO sensitivity. Image acquiring means for acquiring the second image group, level adjusting means for matching the pixel level of the image included in the second image group with the pixel level of the image included in the first image group, and the first An image processing apparatus is provided that includes an image composition unit that generates a composite image by combining an image included in one image group and an image included in the second image group in which pixel levels are matched.

  According to the present invention, when the ISO sensitivity when an image included in the first image group is acquired is a sensitivity at which the image has a preferable brightness, the image included in the second image group is the first image group. Since it is acquired at a higher ISO sensitivity than the images included in the image group, there are many saturated pixels and the image has a narrow dynamic range, but the ratio of noise mixed in the image sensor after gain processing decreases. ing. For this reason, after matching the pixel level of the image included in the second image group with the pixel level of the image included in the first image group, a composite image obtained by combining these is generated in the image sensor. It is possible to obtain an image maintaining the dynamic range while suppressing the influence of noise mixed after the gain processing.

In the above invention, it is preferable that the image synthesizing unit includes an excluded pixel extracting unit that extracts a pixel to be excluded from a synthesis target when generating a synthesized image.
In this way, even when the image included in the second image group includes a pixel whose signal value is saturated by increasing the ISO sensitivity, such a pixel can be excluded from the synthesis target. In addition, for pixels in which saturation does not occur, noise can be further reduced by adding an image with a reduced ratio of noise mixed in the image sensor after gain processing to the synthesis target.

In the above invention, according to the predetermined ISO sensitivity, the number of images included in the first image group and the second image group based on a distribution of luminance values of images acquired at the predetermined exposure time. It is preferable that the apparatus includes an imaging parameter determination unit that determines at least one of the number of images to be included and the ISO sensitivity when acquiring images included in the second image group.
By doing in this way, the ISO sensitivity which can change a synthetic | combination number of sheets, for example, and can anticipate a noise reduction effect can be determined with the luminance value of the image acquired in the predetermined exposure time.

In the above invention, based on the scene determination means for determining the shooting scene when acquiring the image included in the first image group or the second image group, and the shooting scene determined by the scene determination means, At least one of the number of images to be included in the first image group, the number of images to be included in the second image group, and ISO sensitivity when acquiring images included in the second image group is set. It is preferable that a photographing parameter determining unit for determining is provided.
By doing so, it is possible to determine, for example, the ISO sensitivity at which the number of combined images can be changed or a noise reduction effect can be expected in accordance with the shooting scene.

In the above invention, the predetermined exposure time is an exposure time that does not cause camera shake or subject shake, and the first image group and the second image group are acquired with the same exposure time. It is preferable.
In this way, it is possible to suppress the blur of the composite image by suppressing the exposure time of each image used for composition to an exposure time that does not cause camera shake or subject blur, and after gain processing in the image sensor. It is possible to obtain an image that maintains the dynamic range while suppressing the influence of mixed noise.

  According to another aspect of the present invention, there is provided a first image group including at least one image at a predetermined exposure time according to a predetermined ISO sensitivity, and at least one image according to an ISO sensitivity higher than the ISO sensitivity. Obtaining a second image group including: matching a pixel level of an image included in the second image group to a pixel level of an image included in the first image group; and And a step of generating a composite image obtained by combining the image included in the image group and the image included in the second image group in which the pixel levels are matched.

  The present invention also includes a step of obtaining a first image group including at least one image at a predetermined exposure time according to a predetermined ISO sensitivity, and at least one image according to an ISO sensitivity higher than the ISO sensitivity. Acquiring a second image group including: matching a pixel level of an image included in the second image group to a pixel level of an image included in the first image group; An image processing program for causing a computer to execute a step of generating a composite image obtained by combining an image included in the image group and an image included in the second image group with matching pixel levels is provided.

  According to the present invention, it is possible to reduce noise by reducing the ratio of noise mixed in after gain processing and to obtain an image that maintains a dynamic range.

1 is a block diagram illustrating a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. It is explanatory drawing in the image processing apparatus which concerns on the 1st Embodiment of this invention at the time of producing | generating a synthesized image. It is explanatory drawing in the image processing apparatus which concerns on the 2nd Embodiment of this invention in operation | movement at the time of producing | generating a synthesized image. It is a flowchart which shows the flow of a process at the time of producing | generating a synthesized image in the image processing apparatus which concerns on the 3rd Embodiment of this invention. It is an example of the histogram calculation result about the image acquired in the normal scene in the image processing apparatus which concerns on the 3rd Embodiment of this invention. It is an example of the cumulative histogram calculation result about the image acquired in the normal scene in the image processing apparatus which concerns on the 3rd Embodiment of this invention. It is an example of the histogram calculation result about the image acquired in the night view scene in the image processing apparatus which concerns on the 3rd Embodiment of this invention. It is an example of the cumulative histogram calculation result about the image acquired in the night view scene in the image processing apparatus which concerns on the 3rd Embodiment of this invention.

  An image processing apparatus according to an embodiment of the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 shows a schematic configuration of the image processing apparatus 1. As shown in FIG. 1, the image processing apparatus 1 includes an image acquisition unit 10, a level adjustment processing unit 11, and an image composition unit 12.
The image acquisition unit 10 acquires a first image group including at least one image at a predetermined exposure time according to a predetermined ISO sensitivity, and at least one image according to an ISO sensitivity higher than the ISO sensitivity. A second group of images is acquired.
In this embodiment, the image acquisition unit 10 includes an optical system 100 including a lens, an image sensor 101 that captures an image via the optical system 100, and a frame that temporarily stores an image captured by the image sensor 101. A memory 102 is provided and functions as an imaging device. Note that the image acquisition unit 10 does not necessarily include the optical system 100, the imaging device 101, and the frame memory 102, and may be configured to acquire a predetermined image from an external imaging device.

  The level adjustment processing unit 11 is included in the second image group so that the pixel level of the image included in the second image group for the same subject matches the pixel level of the image included in the first image group. Correction is performed on the displayed image. Further, the image composition unit 12 generates a composite image obtained by compositing the images included in the first image group and the images included in the second image group in which the pixel levels are matched by the level matching processing unit 11. Here, the image synthesizing unit 12 includes an excluded pixel extracting unit (not shown), and when the synthesized image is generated, the excluded pixel extracting unit extracts pixels to be excluded from the synthesis target. Then, the image composition unit 12 generates a composite image while excluding the extracted pixels.

Hereinafter, in the image processing apparatus 1 described above, an operation when a composite image is generated will be described with a specific example.
First, the image acquisition unit 10 acquires a first image group and a second image group. That is, in the image acquisition unit 10, a plurality of images are captured by the image sensor 101 through the optical system 100 including a lens and the captured images are temporarily stored in the frame memory 102. At this time, a plurality of high-sensitivity images are taken as a first image group at an ISO sensitivity with an exposure time in which camera shake does not occur and appropriate brightness, and are stored in the frame memory 102. Next, a plurality of high-sensitivity images whose ISO sensitivity is increased with the same exposure time are taken as a second image group and stored in the frame memory 102.

A specific example is shown in FIG. That is, the exposure time (shutter speed) at which camera shake does not occur is 1/10 second, and the ISO sensitivity needs to be 3200 in order to obtain an appropriate brightness when shooting an arbitrary shooting scene with this exposure time. Shall.
Under this circumstance, first, four images are continuously shot as the first image group with a shutter speed of 1/10 second and ISO sensitivity of 3200, and stored in the frame memory. Next, the same shutter speed is set to 1/10 second, and four images are continuously shot as a second image group with ISO sensitivity 6400 higher than the previous ISO sensitivity, and stored in the frame memory.

  The image included in the second image group photographed with ISO sensitivity 6400 has a gain twice as large as appropriate, and there are many pixels that are saturated, and the dynamic range of the image is narrowed. However, since the sensitivity is read out from the image sensor, the ratio of noise mixed in the image sensor after gain is reduced is reduced.

  In this way, the images continuously shot and stored in the frame memory 102 are corrected by the level adjustment processing unit 11 so that the same subject has the same pixel level. In the example shown in FIG. 2, since the second image group is taken with ISO sensitivity 6400, the pixel value is twice the ISO sensitivity 3200 of the first image group with appropriate brightness. For this reason, the pixel level of the image included in the second image group is matched with the pixel level of the image included in the first image group by setting the pixel level of the second image group to ½.

  Note that saturated pixels are generated when the ISO sensitivity is doubled to 6400. However, even when the saturated pixels are halved by the level adjustment processing, the pixel values do not coincide with the pixel values obtained with ISO 3200. On the other hand, the image whose level has been adjusted by the level adjustment processing unit 11 is subjected to a synthesis process by the image synthesis unit 12 to generate a synthesized image. This synthesis process can be performed by a known method, for example, the method described in Patent Document 1 described above.

  When generating a composite image, for pixels that were saturated before level matching processing, the pixel values after level matching are also affected by saturation and are not appropriate pixel values. Need to be added. For this reason, the pixel to be excluded from the synthesis target is extracted by the excluded pixel extraction unit. Specifically, when the pixel value is from 0 to 255, when an image taken with ISO sensitivity 6400 is halved by level adjustment processing, the pixel value near 127 is affected by saturation. Therefore, the pixel is excluded from the synthesis target.

  In this way, in a dark part where saturation does not occur, noise can be further reduced by adding an image with a reduced ratio of noise mixed in the image sensor after gain is applied to the compositing target image. In the example of FIG. 2, four of the eight composite images are halved after being photographed by ISO 6400, so that the ratio of noise mixed in the image sensor after gain is reduced is reduced. It has become. By synthesizing these images, it is possible to obtain a strong noise reduction effect even compared to the normal 8-image synthesis.

In addition, since an image photographed with an appropriate ISO sensitivity is used for synthesis even in a bright portion, and a pixel in which saturation occurs when photographing with an ISO sensitivity larger than the appropriate sensitivity is not used for composition, deterioration such as saturation in a bright portion. Rather, the dynamic range is maintained as usual.
Note that, in a bright part where saturation occurs, the effect of combining a plurality of images is reduced because the combining is not performed. For example, in the example of FIG. 2, in a pixel where saturation occurs, the number of images used for composition is reduced from the normal 8 to 4, and the noise reduction effect by combining multiple images is reduced accordingly. However, since the S / N ratio is usually poor in the dark part and good in the bright part, and the noise reduction effect is expected to be exerted strongly in the dark part, a strong noise reduction effect is obtained in the dark part as in the present invention. A scheme that can be considered effective.

  As described above, the gain processing in the image sensor is performed by combining the image shot with the normal exposure time and ISO sensitivity and the image shot with the same exposure time and ISO sensitivity higher than normal. Thus, it is possible to obtain an image that maintains the dynamic range while suppressing the influence of noise mixed in after.

(Second Embodiment)
In the first embodiment described above, an image included in the first image group captured with an ISO sensitivity with appropriate brightness and an image included in the second image group captured with an ISO sensitivity higher than this are included. By synthesizing, a synthesized image with improved noise reduction effect and maintained dynamic range was obtained. In the present embodiment, a mode in which images taken with three or more ISO sensitivities are combined will be described.

  FIG. 3 shows an example in which images taken with three types of ISO sensitivity are combined. Similar to the example of FIG. 2 described in the first embodiment, the exposure time (shutter speed) at which camera shake does not occur is set to 1/10 second, and an appropriate image is captured when an arbitrary shooting scene is shot with this exposure time. It is assumed that the ISO sensitivity needs to be 3200 for brightness. First, the image acquisition unit 10 captures one image as a first image group with a shutter speed of 1/10 second and an ISO sensitivity of 3200 and stores the image in the frame memory 102.

  Next, the image acquisition unit 10 continuously shoots two images as the second image group with the same shutter speed 1/10 second and ISO sensitivity 6400, and stores them in the frame memory 102. These images taken with ISO sensitivity 6400 have a gain that is twice that of the appropriate value, and there are many pixels that are saturated, and the dynamic range of the image is narrow. However, since the sensitivity is read out from the image sensor, the ratio of noise mixed in the image sensor after gain is reduced is reduced.

  Further, the image acquisition unit 10 continuously shoots five images as another second image group with the same shutter speed 1/10 second and ISO sensitivity 12800, and stores them in the frame memory 102. These images taken with ISO sensitivity 12800 have a gain that is four times higher than appropriate, so that there are more pixels saturated and the dynamic range of the image is narrower. However, since the sensitivity is read out from the image sensor, the ratio of noise mixed in the image sensor after gain is applied is further reduced as compared with the image photographed by ISO 6400.

  Next, the level of the image stored in the frame memory 102 is adjusted to correct the same subject to the same pixel level. In the example shown in FIG. 3, two images are continuously shot with ISO sensitivity 6400. This is a pixel value that is twice that of ISO sensitivity 3200, which is appropriate brightness, and the level is adjusted by setting this to 1/2.

Also, the level of 5 images taken with ISO sensitivity 12800 is adjusted to 1/4. Similar to the first embodiment, the image subjected to level adjustment processing is subjected to synthesis processing except for pixels affected by saturation.
By doing in this way, similarly to the first embodiment, it is possible to obtain a composite image in which noise is further reduced in a dark part where saturation does not occur while maintaining a dynamic range as usual. In the first embodiment, images taken with two types of ISO sensitivities are used for synthesis. As a result, the noise reduction characteristics may change suddenly at the luminance level at which the number of synthesized images is switched. For example, in the example of FIG. 2, when the pixel value is set to 0 to 255, the pixel value larger than 127 is excluded from the synthesis target by the excluded pixel extraction unit. For the pixels whose pixel values are smaller than 127, four images captured with ISO sensitivity 3200 and four images captured with ISO sensitivity 6400 are combined. For this reason, there is a risk that the noise reduction characteristics may change suddenly near the pixel value at which the number of combined images is switched.

On the other hand, in the present embodiment, images captured with three or more ISO sensitivities are used for synthesis, and switching can be made smoother, so that sudden changes in noise reduction characteristics can be mitigated, A high-quality composite image can be generated.
As described above, as in the present embodiment, by combining an image shot with a normal exposure time and ISO sensitivity and an image shot with a plurality of ISO sensitivities that are the same exposure time and stronger than normal, It is possible to obtain an image maintaining the dynamic range while suppressing the influence of noise mixed after the gain processing.
In addition, by using images taken with three or more ISO sensitivities for synthesis, changes in noise reduction characteristics according to the brightness level can be smoothed, and a composite image with natural noise characteristics can be obtained. It becomes.

(Third embodiment)
In the first embodiment and the second embodiment described above, the number of shots at each ISO sensitivity is determined in advance. In the present embodiment, a mode in which the number of shots at each ISO sensitivity is variable according to the shooting scene will be described.
That is, in this embodiment, the image processing apparatus includes a scene determination unit (not shown) and a shooting parameter determination unit, and the scene determination unit acquires images included in the first image group or the second image group. The number of images included in the first image group, the number of images included in the second image group, and the second image are determined based on the shooting scene determined by the scene determination unit. At least one of the ISO sensitivities when acquiring images included in the group can be determined.

FIG. 4 is a flowchart showing the flow of processing when a composite image is generated in the image processing apparatus of this embodiment.
In step S100, one image is taken with an exposure time (blur limit shutter speed) at which camera shake does not occur and an ISO sensitivity with appropriate brightness. As in the first and second embodiments, the exposure time (shutter speed) at which camera shake does not occur is 1/10 second, and an image is captured at an appropriate brightness when an arbitrary shooting scene is shot with this exposure time. To achieve this, the ISO sensitivity needs to be 3200.

  In step S100, one image is taken with a shutter speed of 1/10 second and an ISO sensitivity of 3200. In step S101, a histogram of the image photographed in step S100 is calculated. In step S102, the number of shots at each ISO sensitivity is determined based on the calculated histogram.

  Hereinafter, the method for determining the number of shots at each ISO sensitivity in step S102 will be described in detail. In the present embodiment, two patterns are prepared for normal shooting scenes and night scenes for the number of shots at each ISO sensitivity, and scene discrimination is made based on the histogram to determine which of the two patterns is to be adopted. For example, in a normal shooting scene, as in the first embodiment, four ISO sensitivities 3200 and four ISO sensitivities 6400 are taken, for a total of eight images.

  On the other hand, in the night scene, the proportion of the dark portion in the image is large, and it is desirable to improve the image quality of the dark portion. Therefore, two ISO sensitivities 3200 and six ISO sensitivities 6400 are taken. . A cumulative histogram is calculated from the calculated histogram, and a night scene or a normal scene is determined from the ratio of pixels occupying a dark part.

  FIG. 5 shows an example of a histogram calculation result in a normal scene. In FIG. 5, a histogram is plotted with pixel values on the horizontal axis (0 to 255) and the number of pixels on the vertical axis. For all the pixels of the input image, the number of pixels indicating a certain pixel value is calculated, and this is used as a histogram of pixel values. As the pixel value, a value converted into a luminance value may be used. For example, only the G pixel may be used as the representative pixel value.

  FIG. 6 shows an example of a cumulative histogram calculation result. A cumulative histogram is plotted as pixel values (0 to 255) on the horizontal axis and frequency (%) on the vertical axis. The cumulative histogram is obtained by integrating the histogram calculation result from the pixel value 0 to a certain pixel value, and plotting the ratio of the integrated pixel value to the total number of pixels for each pixel value. For example, in FIG. 6, the pixel value 64 on the horizontal axis has a frequency of about 40%, which means that the number of pixels from pixel value 0 to pixel value 64 is 40% of the total number of pixels. Yes.

  FIG. 7 shows an example of a histogram calculation result in a night scene, and FIG. 8 shows this cumulative histogram. In a typical night scene, since there are many pixels in the dark part, the cumulative histogram shows a high value with a smaller pixel value. For example, paying attention to the cumulative histogram value at the pixel value 64, if it is above a certain threshold value, it is determined as a night scene, otherwise it is determined as a normal scene.

Thereafter, according to the determined ISO sensitivity and the number of images to be photographed, a plurality of images are photographed in step S103, a level adjustment process is performed in step S104, and a composite image is generated based on the plurality of images in step S105. .
As described above, as in the present embodiment, by combining an image shot with a normal exposure time and ISO sensitivity and an image shot with a plurality of ISO sensitivities that are the same exposure time and stronger than normal, It is possible to obtain an image maintaining the dynamic range while suppressing the influence of noise mixed after the gain processing.
Further, by making the number of images of each ISO sensitivity variable according to the scene, an image quality improvement effect suitable for the scene can be obtained.

  In the above-described embodiment, the scene determination unit determines a shooting scene for acquiring images included in the first image group or the second image group, and the shooting scene determined by the scene determination unit. Based on the above, at least one of the number of images included in the first image group, the number of images included in the second image group, and the ISO sensitivity when acquiring images included in the second image group is obtained. However, the present invention is not limited to such a configuration. For example, the shooting parameter determination unit determines the number of images to be included in the first image group and the second image group based on the luminance value distribution of images acquired at a predetermined exposure time according to a predetermined ISO sensitivity. It may be configured to determine at least one of the number of images to be included and the ISO sensitivity at the time of acquiring images included in the second image group.

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 10 Image acquisition part 11 Level adjustment process part 12 Image synthetic | combination part 100 Optical system 101 Image pick-up element 102 Frame memory

Claims (7)

A first image group including at least one image at a predetermined exposure time according to a predetermined ISO sensitivity, and a second image group including at least one image according to an ISO sensitivity higher than the ISO sensitivity. Image acquisition means for acquiring
Level matching means for matching the pixel level of the image included in the second image group to the pixel level of the image included in the first image group;
Image synthesizing means for generating a synthesized image obtained by synthesizing the image included in the first image group and the image included in the second image group in which pixel levels are matched,
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the image synthesizing unit includes an excluded pixel extracting unit configured to extract a pixel to be excluded from a synthesis target when generating a synthesized image.   The number of images to be included in the first image group and the number of images to be included in the second image group based on the luminance value distribution of the images acquired at the predetermined exposure time according to the predetermined ISO sensitivity. 3. The image processing apparatus according to claim 1, further comprising an imaging parameter determination unit that determines at least one of ISO sensitivity when acquiring an image included in the second image group. Scene discriminating means for discriminating a shooting scene when acquiring an image included in the first image group or the second image group;
Based on the shooting scene determined by the scene determination means, the number of images included in the first image group, the number of images included in the second image group, and the images included in the second image group The image processing apparatus according to claim 1, further comprising an imaging parameter determining unit that determines at least one of ISO sensitivities when acquiring the image.   The predetermined exposure time is an exposure time that does not cause camera shake or subject shake, and the first image group and the second image group are acquired with the same exposure time. The image processing apparatus according to claim 4. Obtaining a first image group including at least one image at a predetermined exposure time according to a predetermined ISO sensitivity;
Obtaining a second image group including at least one image according to an ISO sensitivity higher than the ISO sensitivity;
Matching a pixel level of an image included in the second image group to a pixel level of an image included in the first image group;
Generating a composite image obtained by combining the image included in the first image group and the image included in the second image group in which pixel levels are matched, and
An image processing method comprising: Obtaining a first image group including at least one image at a predetermined exposure time according to a predetermined ISO sensitivity;
Obtaining a second image group including at least one image according to an ISO sensitivity higher than the ISO sensitivity;
Matching a pixel level of an image included in the second image group to a pixel level of an image included in the first image group;
Generating a synthesized image obtained by synthesizing an image included in the first image group and an image included in the second image group with matching pixel levels;
An image processing program for causing a computer to execute.

Images