JP2024024478A - Image processing device and method, program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of reducing the difference in noise characteristics between a synthesized plurality of images and a single image.

SOLUTION: An image processing device includes synthesizing means for generating a first image by synthesizing a plurality of images captured at a sensitivity higher than a target sensitivity, generation means for generating a second image by performing noise reduction processing on the first image, and control means for adjusting at least one of the number of images to be synthesized by the synthesizing means and the characteristics of the noise reduction processing so that the noise characteristics of the second image approaches the noise characteristics of the image captured at the target sensitivity.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to an image processing device that combines a plurality of images and reduces noise.

When capturing still images with an imaging device such as a digital camera, it is effective to ensure a sufficient exposure time in order to obtain images with less noise. However, when the exposure time is increased, there is a problem that image blur occurs due to camera movement due to camera shake or movement of the subject, and the photographed image becomes unclear.

Electronic image blur correction has been proposed as a method for dealing with such image blur. For example, Patent Document 1 discloses a technique in which images are taken multiple times in succession with a short exposure time with little image blurring, and after alignment is performed to cancel the deviation between the multiple images obtained, compositing processing is performed. discloses a method for obtaining images with less image blur and noise.

Patent Application No. 2010-210959

However, RAW images captured by image sensors contain multiple types of noise with different characteristics, such as dark noise that does not depend on the amount of light from the subject, that is, the brightness of each pixel, and light shot noise that depends on the brightness of each pixel. It is. Therefore, the number of composite images required to obtain a noise improvement effect equivalent to one step of ISO sensitivity, for example, differs depending on the type of noise.

More specifically, if the ISO sensitivity changes by one step, that is, if the gain changes by a factor of two, the dark noise will double and the light shot noise will increase by a factor of √2. Therefore, the number of composite images required to improve noise by one step of ISO sensitivity is four for dark noise and two for optical shot noise.

In other words, if the same number of images are combined regardless of the brightness of the subject, the noise reduction effect will differ depending on the brightness of the subject, and a single uncombined image taken at a low ISO sensitivity will differ from a composite image with different noise characteristics. It will be generated.

In this way, if the characteristics of the composite image differ from those of a single image taken at a low ISO sensitivity, it may be difficult to combine the composite images when developing the composite RAW image (composite RAW image) with application software, etc. Processing must be performed using characteristics different from those for a single image. As a result, the usability for the user deteriorates. Further, when performing development processing within a digital camera, it is necessary to maintain optimal setting values for each single image and composite image.

The present invention has been made in view of the above-mentioned problems, and its purpose is to provide an image processing device that can reduce the difference in noise characteristics between a composite image obtained by combining a plurality of images and a single image. That's true.

The image processing device according to the present invention includes a composition means for generating a first image by combining a plurality of images taken at a sensitivity higher than a target sensitivity, and a noise reduction process for the first image. and a generation means for generating a second image, and a number of images to be synthesized by the synthesis means so that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity. , and a control means for adjusting at least one of the characteristics of the noise reduction process.

According to the present invention, it is possible to reduce the difference in noise characteristics between a composite image obtained by combining a plurality of images and a single image.

FIG. 1 is a block diagram showing the configuration of a digital camera according to a first embodiment. 5 is a flowchart showing the operation of the first embodiment. 5 is a flowchart showing the operation of the first embodiment. 5 is a flowchart showing the operation of the first embodiment. FIG. 3 is a block diagram showing a configuration example of a composition processing section. 7 is a flowchart showing the operation of the second embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of these features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar components are designated by the same reference numerals, and redundant description will be omitted.

(First embodiment)
In this embodiment, an example will be described in which a noise-reduced image is generated by combining a plurality of captured images within a digital camera.

FIG. 1 is a block diagram showing the configuration of a digital camera 100 according to a first embodiment of the invention.

In FIG. 1, an optical system 101 forms a subject image on an imaging unit 102. The imaging unit 102 includes an imaging element 102a made of, for example, a CCD or a CMOS sensor, and the imaging element 102a photoelectrically converts an optical image formed by the optical system 101 to generate an analog image signal. Further, the imaging unit 102 performs A/D conversion on the analog image signal output from the imaging element 102a to generate digital image data.

The image processing unit 103 applies various image processing such as white balance adjustment processing, color interpolation processing, reduction/enlargement processing, and filtering processing to the image data generated by the imaging unit 102.

The RAM 104 is a rewritable volatile memory, and is used as a temporary storage area for data output during the operation of each block included in the digital camera 100, and as an image buffer.

The ROM 105 is a rewritable nonvolatile memory, and stores operation programs for each block included in the digital camera 100 as well as parameters necessary for the operation of each block. It also stores information on noise characteristics of images taken at the target ISO sensitivity.

The control unit 106 is, for example, a CPU, and controls the operation of each block included in the digital camera 100 by reading out an operation program for each block included in the digital camera 100 from the ROM 105, expanding it to the RAM 104, and executing it. Further, by the control unit 106 instructing the optical system 101 to drive the lens, it is possible to take pictures at different focus positions.

The composition processing unit 107 performs image composition processing, which will be described later, on the image data stored in the RAM 104 to generate a noise-reduced image.

The recording unit 108 records the image processed by the image processing unit 103 and the composite image processed by the composition processing unit 107, which are stored in the RAM 104, as recorded images.

The display unit 109 includes a display device such as a PC monitor, a television, or a smartphone, and displays images recorded in the RAM 104.

The user input unit 110 inputs instructions from the user to the control unit 106. The internal bus 111 transmits data between blocks according to instructions from the control unit 106.

Next, the operation of the digital camera 100 of the first embodiment will be described using FIGS. 2A to 2C and 3.

FIG. 2A is a flowchart showing a series of processes for synthesizing a plurality of captured images to generate a noise-reduced image.

The control unit 106 captures a plurality of images in step S201, synthesizes each image in step S202, and performs noise reduction processing (NR processing) on the synthesized image in step S203. Each of these steps will be explained in detail below.

FIG. 2B is a flowchart showing in detail the image capturing operation in step S201 of FIG. 2A.

In step S211, the control unit 106 determines photographing conditions regarding the optical system 101 and the imaging unit 102. The shooting conditions include shutter speed, aperture value, and ISO sensitivity, and these can be automatically determined by the control unit 106 based on the photometry results of the digital camera 100, or can be set arbitrarily by the photographer according to the subject to be photographed. It is. Further, the shutter speed and aperture value related to the expression of motion and depth can be determined by the photographer, and the ISO sensitivity can be automatically determined by the control unit 106 based on the photometry results of the digital camera 100.

In step S212, the control unit 106 determines the number of shots (N). This value is set by the control unit 106 depending on how many stages of noise reduction effect is targeted. In other words, it is set how many steps lower than the ISO sensitivity of the image captured by the imaging unit 102 and the same noise characteristics as the target are to be aimed for. For example, if the target is a noise reduction effect of one stage, the number of images (N=4) that can reduce dark noise by one stage is set. This is because, compared to optical shot noise, dark noise requires a larger number of images to be synthesized in order to reduce the noise by one step of ISO sensitivity. It is assumed that all N images are taken under the same photographing conditions.

In step S213, the control unit 106 operates the imaging unit 102 based on the imaging conditions set in step S211 and the number of images determined in step S212, and captures a plurality of images.

In step S214, the control unit 106 determines whether the number of captured images has reached the number of captured images (N) set in step S212. If the number of captured images (N) has not been reached, the control unit 106 repeats steps S213 and S214 until the number of captured images (N) is reached. Moreover, if the number of photographed images (N) has been reached, photographing ends.

Next, the image synthesis process in step S202 will be explained using FIG. 2C and FIG. 3.

FIG. 3 is a block diagram showing an example of the configuration of the composition processing unit 107 in this embodiment. The composition processing section 107 includes a block reading section 301, an alignment processing section 302, a noise characteristic setting section 303, an image composition section 304, and an NR (noise reduction) processing section 305.

FIG. 2C is a flowchart showing in detail the image synthesis process (processing by the synthesis processing unit 107) in step S202 of FIG. 2A.

In FIG. 2C, in step S221, the control unit 106 uses the block reading unit 103 to select a reference image to be used as a reference for synthesis from among the captured images recorded in the RAM 104, and reads it as a block image of a predetermined size. (divide into blocks). The block size is, for example, 16 pixels vertically and 16 pixels horizontally. Hereinafter, in the compositing process of this embodiment, it is assumed that processing is performed in units of block images. When the processing of one block is completed, the next adjacent block is processed, and when the processing of all blocks is completed, the compositing processing is also completed. Further, although the selection of the reference image is arbitrary, it is preferable to select an image photographed at the timing when the shutter is pressed.

In step S222, the control unit 106 first inputs to the alignment processing unit 302 a reference image to be a synthesis target image from among the captured images stored in the RAM 104. Then, the block image of the reference image output from the block reading unit 301 (reference block image) is aligned with the block image of the same size in the reference image, and the block image (reference block image). Generally, there is a difference between the standard image and the reference image due to camera shake or a local moving object, so if they are combined as is, the subject will become a multiple image. Therefore, the images are combined by aligning the subject positions between the images. The alignment process may use a known technique such as template matching.

In step S223, the control unit 106 uses the noise characteristic calculation unit 303 to calculate the ISO sensitivity of the photographed image (photographing ISO sensitivity) and the brightness of the ISO sensitivity corresponding to the target noise reduction effect (target ISO sensitivity) from the ROM 105. Read out the variance value of each noise. Then, the number of images to be combined by the image combining unit 304 and the characteristics when performing NR processing by the NR processing unit 305 are calculated. Specifically, first, the number of composite images is calculated by referring to the noise variance value of the shooting ISO sensitivity and the noise variance value of the target ISO sensitivity for the brightness value (brightness information) of the highest brightness pixel in the reference block image. calculate.

Here, Ymax is the brightness value of the brightest pixel in the reference block image, Var_O (Ymax) is the noise variance of the shooting ISO sensitivity at the brightness value Ymax, and Var_T (Ymax) is the noise variance of the target ISO sensitivity. , the composite number N_COMP is calculated using equation (1).

N_COMP=Var_O(Ymax)/Var_T(Ymax)...(1)
In equation (1), if a decimal number appears, it is rounded down.

Note that in this embodiment, the number of composite images N_COMP is calculated using the pixel with the highest brightness in the reference block image, but if only the value of one pixel is referenced, it may be affected by isolated point noise. . Therefore, the number of composite images may be calculated from the brightness values of high brightness areas instead of pixels. For example, it is possible to reduce the influence of isolated point noise by performing low-pass filter processing on the reference block image in advance and calculating Ymax from the image after the filter processing.

By calculating the number of composite images in this manner, it is possible to calculate the number of composite images necessary to obtain a noise reduction effect equivalent to the target ISO sensitivity in a high-brightness region within a block.

Next, when performing NR processing in the NR processing unit 305, a parameter indicating the strength of the noise reduction effect based on the luminance value of each pixel is calculated.

Here, the brightness value of the corresponding pixel of the reference block image is Y, the variance value of the noise of the shooting ISO sensitivity at the brightness value Y is Var_O(Y), the variance value of the noise of the target ISO sensitivity is Var_T(Y), and the number of composite images When N_COMP is used, the noise reduction parameter α is calculated using equation (2).

α=Var_O(Y)/(Var_T(Y)×N_COMP)…(2)
In this way, by calculating the noise reduction parameter α, it is possible to obtain a noise reduction effect equivalent to the target ISO sensitivity for the luminance value of each pixel in each pixel of the image synthesized by the image synthesis unit 304. It is possible to calculate the required filter characteristics (how many pixels should be added to the filter strength).

In step S224, the control unit 106 uses the image synthesis unit 304 to synthesize the reference block image output from the block reading unit 301 and the reference block image output from the alignment processing unit 302. Specifically, N_COMP-1 reference block images are averaged for each pixel with respect to the standard block image. The combined block image (combined block image) is output to the NR processing unit 305.

Note that in this embodiment, the averaging was simply performed for each pixel, but each of the standard block image and reference block image was converted to a frequency image by performing transformation processing such as fast Fourier transformation or discrete cosine transformation. Above, averaging processing may be performed for each frequency component. Furthermore, when performing the averaging process for each frequency component, the difference between the reference block image and the reference block image may be calculated for each frequency component, and the averaging process may be performed only for the frequency components for which the difference is smaller than a predetermined threshold. By adding only frequency components with small differences in this way, noise is suppressed by averaging for frequencies where noise components are dominant, and for frequencies where differences in frequency components occur due to positional shift or changes in the shape of the subject. , not averaged. Therefore, it is possible to reduce the phenomenon of multiple images of the subject.

In step S225, the control unit 106 uses the NR processing unit 305 to perform NR processing on the composite block image output from the image composition unit 304. Specifically, when the pixel to be processed in a composite block image is the pixel of interest, pixels around the pixel of interest (for example, two pixels on the top, bottom, left and right) are sorted in descending order of difference from the pixel of interest, and then noise An additive average is calculated for the α parameters calculated by the characteristic calculation unit 303.

In this way, by performing filter processing to calculate the additive average of the pixel of interest and surrounding pixels based on the parameter α, it is possible to obtain a noise reduction effect equivalent to the target ISO sensitivity.

In step S226, the control unit 106 determines whether processing has been completed for all block images. If all block images have been processed, the control unit 106 ends the image composition process. If all block pixels have not been processed, the process advances to step S227, new block pixels are set, and the processes from step S222 to step S226 are repeated. The above is the detailed operation of the composition processing unit 107. When all operations are completed, an output image is generated and stored in RAM 104.

In this embodiment, by performing image synthesis processing in this manner, it is possible to generate a synthesized image that has little difference in noise characteristics from a single uncomposed image shot under conditions of low ISO sensitivity.

Note that in this embodiment, the compositing process is performed on newly captured images, but the user selects an image from among the images recorded in the recording unit 108 using the user input unit 110 and performs the compositing process. may be executed. For example, for an image specified by the user, images continuously shot under the same conditions are read into the RAM 104 and combined according to the flowchart in FIG. 2C.

At this time, the noise characteristic calculation unit 303 calculates the number of ISO sensitivity steps that can reduce dark noise according to the number of images read out to the RAM 104, and sets the target ISO sensitivity. For example, if the number of images read into the RAM 104 is in the range of 4 to 15, the target ISO sensitivity is set to one level lower than the shooting ISO sensitivity.

By performing the compositing process in this manner, it becomes possible to perform the compositing process not only on newly captured images but also on previously captured images.

(Second embodiment)
In the first embodiment, when images were captured in step S213 of FIG. 2B, all images were captured under the same conditions. In contrast, in the second embodiment, instead of capturing images under the same conditions, some images are captured differently depending on the difference in the number of steps between the shooting ISO sensitivity and the target ISO sensitivity and the number of images that can be stored in the RAM 104. Shoot under the conditions.

FIG. 4 is a flowchart showing the flow of image capturing processing in step S201 in this embodiment.

In step S401, the control unit 106 determines imaging conditions similarly to step S211.

In step S402, the control unit 106 determines the number of shots (N) similarly to step S212. At this stage, similarly to the first embodiment, the number of shots (N) is determined assuming that all N images are taken under the same shooting conditions.

In step S403, the control unit 106 determines whether N images can be stored in the RAM 104. If the control unit 106 can store the images, the process proceeds to step S405, and if there is no free space in the RAM 104 (image storage unit) and the N images cannot be stored, the control unit 106 proceeds to step S404.

In step S404, the control unit 106 reviews the number of shots and the shooting conditions. Specifically, the photographing conditions are reviewed based on the difference in the number of steps between the photographing ISO sensitivity determined in step S401 and the target ISO sensitivity, and the number of images that can be stored in the RAM 104. Then, the number of shots to be taken without changing the shooting conditions (N) and the number of shots to be taken with changing the shooting conditions (M) are determined.

For example, if the difference in the number of steps between the photographing ISO sensitivity and the target ISO sensitivity is two steps, it will be necessary to combine 16 images in order to reduce dark noise by two steps. However, if the number of images that can be stored in the RAM 104 is, for example, 12, the shooting conditions are changed so that some of the captured images are shot with twice the brightness, such as by doubling the shutter speed. do. By multiplying the images (high exposure images) with changed shooting conditions in this way by a gain of 1/2 when compositing, the number of shots (M) required to reduce dark noise can be reduced to 4. becomes possible. Further, the number of images (N) to be photographed under the photographing conditions determined in advance in step S401 is set to the remaining images that can be stored in the RAM 104, in this case eight images.

In step S405, the control unit 106 causes the imaging unit 102 to operate under the imaging conditions set in step S401 or the imaging conditions reviewed in step S404 to capture an image.

In step S406, the control unit 106 determines whether the number of captured images has reached the number of captured images set in step S402 or the number of captured images reviewed in step S404. If the distance has been reached, the control unit 106 ends the image capturing. If the number of images is less than the number of captured images, the process returns to step S405, and the processes of step S405 and step S406 are repeated.

The image synthesis process in step S202 in the second embodiment follows the flowchart in FIG. 2C as in the first embodiment, but when performing the alignment process in step S222, for high exposure images, Positioning processing is performed by multiplying by a gain of 1/2.

In addition, in step S223, when calculating the number of images to be combined in the image combining unit 304 and the characteristics for performing NR processing in the NR processing unit 305, the brightness value Ymax of the highest brightness pixel in the reference block image is If the value is less than 1/2 of the maximum value, a high exposure image is selected.

If the brightness value Ymax of the highest brightness pixel in the reference block image is 1/2 or more of the maximum brightness value, the same processing as in the first embodiment is performed, but N_COMP is greater than the number of captured images (N). Therefore, in that case, processing is performed with N_COMP=N.

By performing the compositing process as described above, even if the difference between the photographing ISO sensitivity and the target ISO sensitivity is large, the noise reduction process can be performed within the range of the number of images that can be stored in the RAM 104.

The disclosure of this specification includes the following image processing apparatus, method, program, and storage medium.

(Item 1)
a synthesizing means for synthesizing a plurality of images taken at a sensitivity higher than a target sensitivity to generate a first image;
generating means for performing noise reduction processing on the first image to generate a second image;
Adjusting at least one of the number of images to be combined by the combining means and the characteristics of the noise reduction processing so that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity. control means for;
An image processing device comprising:

(Item 2)
The image processing device according to item 1, wherein the plurality of images are images captured by an imaging means.

(Item 3
The image processing device according to item 1, wherein the plurality of images are images stored in a storage means.

(Item 4)
The image processing device according to any one of items 1 to 3, further comprising a storage unit that stores noise characteristics of an image captured at the target sensitivity.

(Item 5)
5. The image processing apparatus according to any one of items 1 to 4, wherein the compositing means aligns and composites the plurality of images.

(Item 6)
Items 1 to 5, wherein the combining means divides the image into blocks each consisting of a plurality of pixels, and determines the number of images to be combined based on brightness information of a high brightness area within the block. The image processing device according to any one of the above.

(Item 7)
7. The image processing apparatus according to item 6, wherein the compositing means changes the number of composite images and noise reduction characteristics for each block based on the target sensitivity.

(Item 8)
The image according to any one of items 1 to 5, wherein the generating means changes characteristics of noise reduction processing for the first image based on luminance information of each pixel. Processing equipment.

(Item 9)
9. The image processing device according to any one of items 1 to 8, wherein the compositing means changes the exposure of the captured images when the number of images to be composited exceeds the capacity of the image storage unit.

(Item 10)
The image processing device according to any one of items 1 to 9, wherein the sensitivity is an ISO sensitivity.

(Item 11)
a compositing step of composing a plurality of images captured at a sensitivity higher than a target sensitivity to generate a first image;
a generation step of performing noise reduction processing on the first image to generate a second image;
Adjusting at least one of the number of images to be combined in the combining step and the characteristics of the noise reduction processing so that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity. a control process to
An image processing method comprising:

(Item 12)
A program for causing a computer to function as each means of the image processing apparatus according to any one of items 1 to 10.

(Item 13)
A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing apparatus according to any one of items 1 to 10.

(Other embodiments)
The present invention also provides a system or device with a program that implements one or more functions of the above-described embodiments via a network or a storage medium, and one or more processors in a computer of the system or device reads the program. This can also be achieved by executing a process. It can also be implemented by a circuit (eg, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, the following claims are hereby appended to disclose the scope of the invention.

100: Digital camera, 101: Optical system, 102: Imaging unit, 102a: Imaging element, 103: Image processing unit, 104: RAM, 105: ROM, 106: Control unit, 107: Combination processing unit, 108: Recording unit, 109: Display section, 110: User input section, 111: Internal bus, 301: Block reading section, 302: Alignment processing section, 303: Noise characteristic calculation section, 304: Image composition section, 305: NR processing section

Claims (13)

a synthesizing means for synthesizing a plurality of images taken at a sensitivity higher than a target sensitivity to generate a first image;
generating means for performing noise reduction processing on the first image to generate a second image;
Adjusting at least one of the number of images to be combined by the combining means and the characteristics of the noise reduction processing so that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity. control means for;
An image processing device comprising: The image processing apparatus according to claim 1, wherein the plurality of images are images captured by an imaging means. The image processing apparatus according to claim 1, wherein the plurality of images are images stored in a storage means. The image processing apparatus according to claim 1, further comprising storage means for storing noise characteristics of an image captured at the target sensitivity. The image processing apparatus according to claim 1, wherein the compositing means aligns and composites the plurality of images. 2. The combining means divides the image into blocks each consisting of a plurality of pixels, and determines the number of images to be combined based on brightness information of a high brightness area within the block. The image processing device described. 7. The image processing apparatus according to claim 6, wherein the compositing means changes the number of composite images and noise reduction characteristics for each block based on the target sensitivity. 2. The image processing apparatus according to claim 1, wherein the generating means changes characteristics of noise reduction processing for the first image based on luminance information of each pixel. The image processing apparatus according to claim 1, wherein the compositing means changes the exposure of the captured images when the number of images to be composited exceeds the capacity of the image storage unit. The image processing apparatus according to claim 1, wherein the sensitivity is an ISO sensitivity. a compositing step of composing a plurality of images captured at a sensitivity higher than a target sensitivity to generate a first image;
a generation step of performing noise reduction processing on the first image to generate a second image;
Adjusting at least one of the number of images to be combined in the combining step and the characteristics of the noise reduction processing so that the noise characteristics of the second image approach the noise characteristics of the image captured at the target sensitivity. a control process to
An image processing method comprising: A program for causing a computer to function as each means of the image processing apparatus according to claim 1. A computer-readable storage medium storing a program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 10.

Images