JP4645921B2 - Image signal processing apparatus and method, program, and imaging apparatus - Google Patents

Abstract

An image signal processing apparatus includes the following elements. A control unit determines a gradation correction characteristic representing a conversion characteristic for correcting brightness of an input frame. A separation unit separates the determined gradation correction characteristic into a representative gradation correction value representing an amount of correction for a representative value of a main subject and a remaining gradation correction characteristic obtained by removing the representative gradation correction value from the gradation correction characteristic. A gain processing unit uniformly applies the separated representative gradation correction value to the frame as a gain. A noise reduction processing unit performs noise reduction processing on the frame to which the representative gradation correction value is applied. A gradation correction processing unit performs gradation correction processing on the frame for which the noise reduction processing has been performed using the separated remaining gradation correction characteristic.

Description

  The present invention relates to an image signal processing apparatus and method, a program, and an imaging apparatus, and in particular, an image in which noise is appropriately suppressed can be obtained even when tone correction processing and noise reduction processing are used in combination. The present invention relates to an image signal processing device and method, a program, and an imaging device.

  2. Description of the Related Art In recent years, image data having a wide dynamic range can be taken even with digital cameras due to advances such as noise reduction of solid-state imaging devices and advances in signal processing technology. At this time, in order to prevent saturation in the image sensor, control is performed to suppress the exposure amount more than normal shooting (see, for example, Patent Document 1).

  Hereinafter, imaging performed by this control is referred to as wide DR imaging, and an image captured by this control is referred to as a wide DR image.

  Wide DR image data has a uniformly dark screen compared to the normal image. Therefore, at the time of output, gradation correction processing is performed by digital signal processing inside the camera so that the brightness of the main subject is suitable. For example, tone curve processing or dynamic range compression processing is often used as the gradation correction processing.

  Here, the gradation correction process works to brighten the appearance of the main subject. This operation has the effect of widening the width of the signal assigned to the main subject. Therefore, in the digital signal processing, the continuity of the signal at that portion (the smoothness of the change in the level direction) is deteriorated.

  On the other hand, in a digital camera, noise reduction processing (hereinafter referred to as NR processing) is often performed in order to suppress noise in image data. In the NR process, a noise model indicating the relationship between the input level and the noise component (see, for example, Patent Document 2) and human visual characteristics (for example, see Patent Document 3) are removed from the image data. .

  The NR process is often performed, for example, by nonlinear smoothing represented by a bilateral filter. Digital signal smoothing has an effect of improving level direction continuity (resolution) by averaging a plurality of data including quantization error and noise. Therefore, the NR process has the property of improving the continuity of the input image signal in addition to noise removal.

JP 2008-148180 A JP 2008-131529 A JP 2008-113222 A

  Now, when the above-described gradation correction processing and NR processing are used together for a wide DR image, even if the gradation correction processing is performed first or the NR processing is performed first, As described above, the processing performance is degraded.

  When gradation correction processing is performed first, and then NR processing is performed, noise is amplified or the original signal is suppressed due to the influence of the gradation correction processing, so it becomes difficult to distinguish the noise from the original signal, There is a risk that the performance of the NR processing will deteriorate. In addition, in the image after the gradation correction process, the noise model is not established due to the influence thereof, and it is difficult to appropriately perform the NR process.

  On the other hand, when performing the NR process first and then performing the gradation correction process, because the brightness of the main subject is dark because the gradation correction is not performed, the human visual characteristics are not correctly reflected, There is a risk that the performance of the NR processing will deteriorate. Further, since the gradation correction process corrects the main subject brightly, the continuity of the level of the subject may be deteriorated.

  The present invention has been made in view of such a situation, and is intended to suppress deterioration in processing performance when tone correction processing and noise reduction processing are used in combination.

An image signal processing apparatus according to an aspect of the present invention includes: a control unit that determines a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame based on an exposure adjustment amount at the time of shooting the frame; and the control unit The gradation correction characteristic determined by the gradation correction representative value representing the correction amount with respect to the representative value of the main subject, and the remaining gradation correction characteristics obtained by removing the gradation correction representative value from the gradation correction characteristic. , A gradation processing unit that applies the gradation correction representative value distributed by the distribution unit as a gain uniformly to the frame, and the gradation correction representative value is applied by the gain processing unit. Noise reduction processing means for performing noise reduction processing on the frame, and the remaining tone correction characteristics distributed by the distribution means. There are, and a gradation correcting means for performing gradation correction processing to the frame in which the noise reduction processing is performed by the noise reduction processing means.

  The function is a normalized gradation correction function obtained by normalizing the gradation correction function with respect to the gradation correction representative value.

The gradation correction processing means can perform tone curve processing as the gradation correction processing.

The gradation correction processing means can perform dynamic range compression processing as the gradation correction processing.

The gradation correction processing means can perform a shading correction process as the gradation correction process.

  The image processing apparatus may further include parameter setting means for setting a noise model parameter for the noise reduction processing using the gradation correction representative value distributed by the distribution means.

In the image signal processing method according to one aspect of the present invention, the image signal processing apparatus determines a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame based on an exposure adjustment amount at the time of shooting the frame , The determined gradation correction characteristics are converted into a gradation correction representative value representing a correction amount with respect to a representative value of a main subject, and a remaining gradation correction characteristic obtained by removing the gradation correction representative value from the gradation correction characteristic. The assigned gradation correction representative value is uniformly applied as a gain to the frame, and noise reduction processing is performed on the frame to which the gradation correction representative value is applied. A step of performing gradation correction processing on the frame on which the noise reduction processing has been performed using the remaining gradation correction characteristics;

The program according to one aspect of the present invention determines a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame based on an exposure adjustment amount at the time of shooting the frame, and the determined gradation correction characteristic is determined. And the gradation correction representative value representing the correction amount for the representative value of the main subject and the remaining gradation correction characteristics obtained by removing the gradation correction representative value from the gradation correction characteristics, and the assigned gradation correction A representative value is uniformly applied as a gain to the frame, a noise reduction process is performed on the frame to which the gradation correction representative value is applied, and the remaining gradation correction characteristics thus distributed are used. , Causing the computer to execute processing including a step of performing gradation correction processing on the frame on which the noise reduction processing has been performed.

An imaging apparatus according to another aspect of the present invention corrects the brightness of a frame based on an imaging unit that images a subject and an exposure adjustment amount at the time of shooting a frame corresponding to the subject imaged by the imaging unit. A gradation correction representative value representing a correction amount with respect to a representative value of a main subject among the objects, a control means for determining a gradation correction characteristic representing a conversion characteristic to be performed, and the gradation correction characteristic determined by the control means. And a distribution means for distributing the gradation correction characteristics to the remaining gradation correction characteristics obtained by removing the gradation correction representative values, and the gradation correction representative values distributed by the distribution means for the frame. Gain processing means for uniformly applying the gain, and noise reduction processing for the frame to which the gradation correction representative value is applied by the gain processing means. Using the reduction processing means and the remaining gradation correction characteristics distributed by the distribution means, a gradation correction process is performed on the frame on which the noise reduction processing has been performed by the noise reduction processing means. Tone correction processing means.

In one aspect of the present invention, a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame is determined based on an exposure adjustment amount at the time of shooting of the frame, and the determined gradation correction characteristic is The gradation correction representative value indicating the correction amount with respect to the representative value of the main subject and the remaining gradation correction characteristics obtained by removing the gradation correction representative value from the gradation correction characteristics are sorted. Then, the distributed gradation correction representative value is uniformly applied as a gain to the frame, and noise reduction processing is performed on the frame to which the gradation correction representative value is applied. Using the remaining tone correction characteristics, tone correction processing is performed on the frame on which the noise reduction processing has been performed.

In another aspect of the present invention, a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame based on an exposure adjustment amount at the time of shooting of the frame corresponding to the imaged subject is captured. Is determined, and the determined gradation correction characteristic is a gradation correction representative value indicating a correction amount with respect to a representative value of a main subject, and the remaining gradations obtained by removing the gradation correction representative value from the gradation correction characteristic. Sorted into correction characteristics. Then, the distributed gradation correction representative value is uniformly applied as a gain to the frame, and noise reduction processing is performed on the frame to which the gradation correction representative value is applied. Using the remaining tone correction characteristics, tone correction processing is performed on the frame on which the noise reduction processing has been performed.

  According to the present invention, the gradation correction process and the noise reduction process can be used together. Further, according to the present invention, it is possible to suppress deterioration in processing performance when the tone correction processing and the noise reduction processing are used in combination.

Hereinafter, a first embodiment and a second embodiment will be described as embodiments of an image signal processing apparatus to which the present invention is applied. The description will be given in the following order.

1. 1st Embodiment (example comprised with a digital camera)
2. Second embodiment (example configured with a personal computer)

<1. First Embodiment>
[Example of digital camera configuration]
FIG. 1 is a block diagram showing a configuration example of an embodiment of a digital camera as an image signal processing apparatus to which the present invention is applied.

  In the example of FIG. 1, the digital camera 1 includes an exposure controller 11, an exposure amount adjustment unit 12, an image sensor 13, an A / D (Analog to Digital) conversion unit 14, an image signal processing unit 15, a signal processing unit 16, and a display. The unit 17 and the recording unit 18 are included.

  The exposure control controller 11 and the exposure amount adjustment unit 12 perform exposure control. The exposure controller 11 sets the exposure adjustment amount (that is, the control value of the exposure amount adjustment unit 12) so that the exposure amount of the image sensor 13 is appropriate according to the luminance distribution of the subject and the shooting mode of the digital camera 1. Set. Further, the exposure controller 11 outputs the exposure adjustment amount at this time to the DET processing unit 21. The exposure adjustment unit 12 adjusts the exposure of the image sensor 13 using the control value from the exposure controller 11, and performs exposure. As this control value, for example, aperture stop, exposure time, and density of a neutral density (ND) filter are often used.

  The image sensor 13 is composed of a solid-state image sensor such as a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor), for example, and receives light incident from an object through an optical block (not shown) such as a lens. Convert to signal.

  The A / D converter 14 quantizes the image signal converted into an analog signal by the image sensor 13 and converts it into digital image data. Image data from the A / D conversion unit 14 is output to the DET processing unit 21 and the gain processing unit 23 of the image signal processing unit 15.

  The image signal processing unit 15 performs image signal processing of the image data from the A / D conversion unit 14. That is, the image signal processing unit 15 performs gradation correction so that the brightness of the main subject in the image data (frame) from the A / D conversion unit 14 is suitable, and suppresses noise in the image data. Perform noise reduction processing. At that time, the image signal processing unit 15 substantially performs the brightness gradation correction process in two stages before and after the noise reduction process. For example, a gradation correction process for increasing gradation is performed as the first stage, and a gradation correction process for suppressing high-intensity gradation is performed as the second stage. Hereinafter, the noise reduction processing is appropriately referred to as NR processing or noise removal processing.

  The image signal processing unit 15 includes a DET (detection) processing unit 21, a distribution processing unit 22, a gain processing unit 23 that performs first-stage gradation correction processing, an NR (noise reduction) adjustment processing unit 24, and an NR processing unit 25. , And a TM (tone map) processing unit 26 that performs second-stage gradation correction processing.

  The DET processing unit 21 analyzes the image data from the A / D conversion unit 14, determines the gradation correction characteristic to be applied to the image data using the exposure adjustment amount from the exposure controller 11, and determines The gradation correction characteristics thus obtained are output to the distribution processing unit 22. For example, the DET processing unit 21 determines a gradation correction function as an example of the gradation correction characteristic. The tone correction function is a function that defines conversion characteristics that make the brightness of image data suitable as the input / output relationship of brightness. For example, the overexposure of high-luminance portions is suppressed and the main subject becomes brighter. To be determined.

  The distribution processing unit 22 uses the gradation correction characteristics from the DET processing unit 21 as components necessary for the gain processing unit 23 that performs gradation correction processing for the first-stage brightness and the second-stage brightness level. The TM processing unit 26 that performs the tone correction processing is decomposed into components necessary for distribution. For example, when a gradation correction function as a gradation correction characteristic is input from the DET processing unit 21, the distribution processing unit 22 converts the gradation correction function from the gradation correction representative value and the gradation correction function to the gradation correction function. It decomposes into a normalized gradation correction function which is the remaining gradation correction function excluding the tone correction representative value. Then, the distribution processing unit 22 distributes the gradation correction representative value to the gain processing unit 23, and distributes the normalized gradation correction function to the TM processing unit 26. The gradation correction representative value is also output to the NR adjustment processing unit 24.

  Here, the gradation correction representative value is a gain amount when the correction processing performed on the representative value (representative level value) of the main subject is represented by a gain. The normalized gradation correction function is a gradation correction function normalized so that the input / output of the gradation correction function is the same as the representative value of the main subject.

  The gain processing unit 23 uses the gradation correction representative value from the distribution processing unit 22 to perform gradation correction processing for brightness at the first stage. That is, the gain processing unit 23 uniformly corrects (adjusts) the brightness of the image data from the A / D conversion unit 14 using the gradation correction representative value from the distribution processing unit 22 to obtain the brightness. Is output to the NR processing unit 25.

  The NR adjustment processing unit 24 uses the gradation correction representative value from the distribution processing unit 22 to calculate a noise model parameter indicating the relationship between the input level and the noise component, and sets it as the noise model of the NR processing unit 25. .

  In the NR processing unit 25, for example, noise removal processing described in Patent Document 3 is performed. The NR processing unit 25 removes the noise of the image data from the gain processing unit 23 using the noise model set by the NR adjustment processing unit 24 and outputs it to the TM processing unit 26. Note that noise removal in the NR processing unit 25 not only means that noise is completely removed, but also includes noise reduction.

  The TM processing unit 26 uses the normalized gradation correction function from the distribution processing unit 22 to perform the second-level brightness gradation correction process. That is, the TM processing unit 26 performs tone correction of the image data from the NR adjustment processing unit 24 using the normalized tone correction function from the distribution processing unit 22, and the tone-corrected image data is The signal is output to the signal processing unit 16. As the tone correction, tone curve processing, dynamic range compression processing, shading processing, or the like is used. In the following description, tone curve processing will be described as an example of tone correction.

  The signal processing unit 16 performs signal processing for converting the image data from the TM processing unit 26 into image data suitable for display on the subsequent display unit 17 and recording in the recording unit 18, and corresponding display units 17 and the recording unit 18. The signal processing unit 16 performs signal processing that combines signal format conversion such as gamma conversion and YCbCr conversion, and color conversion processing using an LUT.

  The display unit 17 is composed of, for example, an LCD (Liquid Crystal Display) panel and displays an image corresponding to the image data from the signal processing unit 16. The recording unit 18 records the image data from the signal processing unit 16 on a recording medium such as an optical disk or a magnetic disk (not shown).

[Configuration example of NR processing unit]
FIG. 2 is a diagram illustrating a configuration example of the NR processing unit.

  The NR processing unit 25 illustrated in FIG. 2 includes a physical characteristic value calculation unit 31, a visual characteristic value calculation unit 32, a threshold value determination unit 33, and a noise removal unit 34.

  Image data from the gain processing unit 23 is input to the physical characteristic value calculating unit 31, the visual characteristic value calculating unit 32, and the noise removing unit 34, respectively.

The physical characteristic value calculation unit 31 calculates a noise amount (physical characteristic value) σ ² that is a value depending on the luminance of the input image data, using the noise model in which the parameter is set by the NR adjustment processing unit 24. And output to the threshold value determination unit 33.

  The visual characteristic value calculation unit 32 determines the colors R, G, and B of the image indicated by the input image data, and uses a visual model to correct a correction parameter (visual characteristic value) k that is a coefficient for correcting the physical characteristic value. Is calculated and output to the threshold value determination unit 33.

  The visual model includes, for example, a color and correction value conversion table created in consideration of human visual characteristics, which will be described later with reference to FIG.

The threshold value determination unit 33 uses the noise amount σ ² from the physical characteristic value calculation unit 31 and the correction parameter k from the visual characteristic value calculation unit 32 to use a filter threshold value used by the noise removal unit 34 to perform noise removal processing. ε is calculated, and the calculated filter threshold ε is output to the noise removing unit 34. For example, the threshold value determination unit 33 is configured by a multiplier, and multiplies the noise amount σ ² from the physical characteristic value calculation unit 31 by the correction parameter k from the visual characteristic value calculation unit 32 and is a multiplied value kσ. ² is output to the noise removing unit 34 as the filter threshold ε. As described above, by determining the filter threshold ε, it is possible to perform noise removal processing using an appropriate threshold considering the physical characteristics of noise and the human visual characteristics.

  The noise removal unit 34 performs noise removal using, for example, an ε filter on the input image data using the filter threshold value ε from the threshold value determination unit 33, and the image data subjected to the noise removal process is processed later. Is output to the TM processing unit 26. Note that a method other than the ε filter may be used as the noise removal method in the noise removal unit 34.

[Conversion table configuration example]
FIG. 3 is a diagram illustrating an example of a conversion table used as a visual model.

  The conversion table is a conversion table in which colors, values of (r, g, b), and correction parameters k are associated with each other, and values of image colors R, G, and B are used as correction parameters k. To convert.

  In the conversion table, green, (r, g, b) = (100,170,100), correction parameter k = 2.0 are associated, yellow, (r, g, b) = (240,200,60), correction parameter k = 2.0 is associated. Also, blue, (r, g, b) = (90,90,200), correction parameter k = 3.0 are associated, red, (r, g, b) = (200,50,70), correction parameter k = 3.0 is associated. Further, skin color, (r, g, b) = (220, 170, 170), correction parameter k = 3.0 are associated, and in addition, correction parameter k = 2.5 is associated.

  The visual characteristic value calculation unit 32 discriminates the colors R, G, and B of the image indicated by the input image data, and calculates a value closest to the discriminated values of R, G, and B in the conversion table (r, g , b). Then, the visual characteristic value calculation unit 32 selects the correction parameter k corresponding to the selected value of (r, g, b) from the conversion table, and outputs the selected correction parameter k to the threshold value determination unit 33.

  When selecting the closest value to each value of R, G, B from the values of (r, g, b) in the conversion table, it is determined based on the color space distance (vector) of each value. Is done.

[Explanation of wide DR shooting mode]
Next, the shooting mode of the digital camera 1 will be described with reference to FIG.

  The digital camera 1 is provided with at least a wide DR shooting mode for shooting an image having a wide dynamic range and a standard shooting mode for shooting an image having a standard dynamic range instead of a wide dynamic range. It has been. In the case of the example of FIG. 4, two types of wide DR shooting A mode and wide DR shooting B mode using different exposure adjustment methods are provided as the wide DR shooting mode.

  When the shooting mode of the digital camera 1 is the wide DR shooting A mode, the exposure controller 11, for example, described in Patent Document 1 prevents the overexposure on the image sensor 13 and the exposure amount of the main subject is extremely high. Set the amount of exposure adjustment using a method that adjusts the exposure so that it is not too low.

  Further, when the shooting mode of the digital camera 1 is the wide DR shooting B mode, the exposure controller 11 uses, for example, a method of adjusting the exposure so as to eliminate all overexposure on the image sensor 13 and adjust the exposure. Set the amount.

  Note that the specific method of adjusting the exposure is not limited to the above-described wide DR shooting A mode and wide DR shooting B mode, but is arbitrary. However, when compared with shooting in the standard mode, shooting in the DR shooting mode is possible. The exposure will be reduced when the exposure is adjusted.

  In the example of FIG. 4, images 51 to 53 of image data photographed in each of the photographing modes described above and their histograms 54 to 56 are shown. That is, the upper part shows an image 51 of image data shot in the standard shooting mode and a histogram 54 thereof. The middle row shows an image 52 of image data shot in the wide DR shooting A mode and a histogram 55 thereof, and the lower row shows an image 53 of image data shot in the wide DR shooting B mode. The histogram 56 is shown.

  In the histograms 54 to 56, the vertical axis represents the frequency, the horizontal axis represents the brightness (luminance), and it is shown that the brighter the color goes to the right. The main subject in the images 51 to 53 is a person at the center of the screen, and the range E in each of the histograms 54 to 56 indicates the brightness distribution of the main subject. Hereinafter, although not particularly mentioned, the other histograms are similarly expressed.

  In shooting in the standard shooting mode, the brightness of the main subject has sufficient brightness as shown in the range E of the histogram 54, which is preferable, but in the high-luminance sky portion above the image 51, A jump occurs and the subject (empty part) cannot be reflected in the signal.

  On the other hand, in wide DR shooting A mode shooting, shooting is performed with a lower exposure than in standard shooting mode shooting. In the wide DR shooting A mode shooting, since the exposure is lower than in the standard shooting mode shooting, the empty portion of the image 52 is reflected in the signal as compared with the image 51 in the standard shooting mode. .

  In wide DR shooting B mode shooting, exposure is further reduced compared to wide DR shooting A mode shooting. In the case of wide DR shooting B mode shooting, the sky portion in the image 53 is completely reflected in the signal by lowering the exposure further than shooting in the wide DR shooting A mode. However, the brightness of the main subject is dark as shown in the range E of the histograms 55 and 56 by reducing the exposure in both wide DR shooting modes.

[Description of gradation correction characteristics]
FIG. 5 is a diagram showing an example of gradation correction characteristics of each image data in FIG. In the example of FIG. 5, the graph corresponds to the tone curves 61 to 63 representing the tone correction characteristics of the image data shot in each shooting mode of FIG. 4, and the image data after tone correction using the tone curves 61 to 63. Images 64 to 66 to be displayed and histograms 67 to 69 of the images 64 to 66 are shown.

  In each of the graphs of the tone curves 61 to 63, the horizontal axis represents the input level, and indicates that the input level increases as it goes to the right. The vertical axis represents the output level, and the higher the level, the higher the output level. Hereinafter, although not particularly mentioned, other tone curve graphs are similarly expressed.

  The main subject in these images 64 to 66 is the person at the center of the screen, as in the images 51 to 53. A range F in the graphs of the tone curves 62 and 63 indicates the brightness range of the main subject, and a range E in each of the histograms 67 to 69 indicates the brightness distribution of the main subject.

  In the DET processing unit 21, gradation correction characteristics for correcting the brightness of image data obtained by imaging are determined. This gradation correction characteristic is a processing characteristic for correcting the brightness of a main subject that is dark due to shooting in the wide DR shooting mode while retaining a signal of a high-luminance portion. Or a tone correction function represented by 63.

  Although the method for determining the gradation correction characteristic is arbitrary, the DET processing unit 21 determines the gradation correction characteristic so as to suppress, for example, overexposure in a high-luminance portion and to brighten the main subject. More specifically, the DET processing unit 21 is lowered in the brightness range of the main subject indicated by the range F of the tone curve 62 by the exposure adjustment amount of the exposure controller 11 as compared with the case of the standard shooting mode. The gradation correction characteristic is determined so as to cancel (cancel) the exposure. Then, as the brightness range F is exceeded, the DET processing unit 21 determines the gradation correction characteristics so as to gradually suppress the correction amount.

  Similarly, the DET processing unit 21 is further lowered in the brightness range of the main subject indicated by the range F of the tone curve 63 by the exposure adjustment amount of the exposure controller 11 than in the wide DR shooting A mode. Tone correction characteristics are set so as to cancel the exposure. Then, as the brightness range F is exceeded, the DET processing unit 21 determines the gradation correction characteristics so as to gradually suppress the correction amount. Therefore, the slope of the range F of the tone curve 63 is steeper than the slope of the range F of the tone curve 62. Note that “x (1 / exposure adjustment amount)” shown in each graph of the tone curves 62 and 63 represents that the exposure adjustment amount is canceled out.

  By this tone curve (gradation correction characteristic) 62, the low luminance side of the image 52 in FIG. 4 is corrected brightly. That is, the main subject of the image 52 in FIG. 4 has a suitable brightness (for example, the brightness equivalent to the brightness shown in the range E of the image 64 and the histogram 67 in the standard shooting mode) as shown in the image 65 and the histogram 68. Can be corrected.

  Similarly, the tone curve (tone correction characteristic) 63 corrects the low brightness side of the image 53 in FIG. That is, the main subject of the image 53 in FIG. 4 has a suitable brightness (for example, the brightness shown in the range E of the image 64 and the histogram 67 in the standard shooting mode) as shown in the range E of the image 66 and the histogram 69. Equivalent brightness).

  Note that image data shot in the standard shooting mode is not subject to gradation correction in the image signal processing unit 15, but in the example of FIG. 5, for comparison with image data shot in the wide DR shooting mode. A tone curve 61, an image 64, and a histogram 67 are shown. Accordingly, the tone curve 61 shown in FIG. 5 is a linear tone curve, the image 64 after gradation correction matches the image 51 in FIG. 4, and the histogram 67 after gradation correction is shown in FIG. 4 is consistent with the histogram 54 of FIG.

[Description of image signal processing]
Next, details of the processing of the image signal processing unit 15 will be described with reference to FIG. In the example of FIG. 6, for convenience of explanation, the NR adjustment processing unit 24 is omitted, and the distribution processing unit 22 includes a representative value detection unit 71 and a gradation correction normalization unit 72. .

  Further, in the example of FIG. 6, graphs of tone curves 81 to 83 and histograms 91 to 93 are shown. The tone curve 81 represents the tone correction characteristic determined by the DET processing unit 21, and the tone curve 82 is input of processing for uniformly applying the tone correction representative value G detected by the representative value detecting unit 71. The output characteristics are expressed as tone curves for explanation. A tone curve 83 represents the normalized gradation correction characteristic obtained by the gradation correction normalization unit 72. Histograms 91 to 93 are a histogram of image data input to the gain processing unit 23, a histogram of image data after gain processing, and a histogram of image data after gradation correction, respectively.

  For example, data of gradation correction characteristics represented by a tone curve 81 is input from the DET processing unit 21 to the representative value detection unit 71 and the gradation correction normalization unit 72.

  The representative value detection unit 71 detects a gradation correction representative value G that is a value representing a correction amount to be performed on the representative value of the main subject in the gradation correction characteristic determined by the DET processing unit 21. The gradation correction representative value G is a numerical value represented by a gain. As an example, when the gradation correction characteristic is represented by a function y = f (x), the input / output ratio f (x) / x (that is, The slope of the function y = f (x)) is obtained from the following equation (1).

G = argmax (f (x) / x) (1)

However, if the brightness range of the main subject is already known, x is the brightness range of the main subject. For example, when the brightness of the main subject is known by detection and is a certain value x0 (that is, when the representative value of the main subject is x0), G = f (x0) / x0 It becomes. Further, as the gradation correction representative value G when the brightness of the main subject has a range, in addition to the maximum value of f (x) / x in the range, for example, an average value of f (x) / x May be used. Further, when the brightness of the main subject is unknown, the maximum value of f (x) / x is used.

  By this gradation correction representative value G detection process, an element for correcting the brightness of the main subject is detected.

The gradation correction normalization unit 72 obtains the remaining components excluding the gradation correction representative value G detected by the representative value detection unit 71 from the gradation correction characteristics determined by the DET processing unit 21. That is, the tone correction normalization unit 72 normalizes the tone correction characteristics determined by the DET processing unit 21 with respect to the main subject, and obtains new tone correction characteristics. For example, the function f ′ (x) representing the normalized gradation correction characteristic which is a new gradation correction characteristic is detected by the representative value detection unit 71 when the gradation correction characteristic is represented by the function y = f (x). Using the gradation correction representative value G, the following equation (2) is obtained.

f '(x) = f (x × G) / G (2)

  In equation (2), the x-axis is multiplied by G in order to cancel out the extension of the level width due to the gradation correction representative value G in the gain processing unit 23 to be described later. The y-axis has been multiplied by 1 / G to restore the original amount. That is, the equation (2) for obtaining f ′ (x) is an equation that makes the tone correction characteristic a linear straight line around the main subject x0 as shown by the tone curve 83, in other words, This is an expression for normalizing the tone correction characteristic f (x). Therefore, the detection of the gradation correction representative value G agrees that a normalization constant is detected.

  As described above, the distribution processing unit 22 represents the tone correction characteristic represented by the tone curve 81 with the uniform gain G represented by the tone curve 82 and the tone curve 83 with respect to the frame. They are decomposed into normalized gradation correction characteristics, and are distributed to the gain processing unit 23 and the TM processing unit 26, respectively.

  That is, the gradation correction representative value G detected by the representative value detection unit 71 is output to the gain processing unit 23, and the normalized gradation correction characteristic obtained by the gradation correction normalization unit 72 is the TM processing unit 26. Is output. The gradation correction representative value G is also output to the NR adjustment processing unit 24 (not shown) in the example of FIG. Details of the processing in the NR adjustment processing unit 24 will be described later with reference to FIG.

  The gain processing unit 23 uniformly applies a gain (gradation correction representative value) G to the input image data as indicated by the tone curve 82. As a result, as shown in the histogram 92, the brightness axis (horizontal axis) is extended by the gain, so that the level width of the image signal is widened. Therefore, before applying the gradation correction representative value G, the brightness distribution of the main subject that is dark as shown in the range E of the histogram 91 matches the main subject as shown in the range E of the histogram 92. Brightly corrected.

  The NR processing unit 25 performs NR processing on the image data after the gain processing unit 23. At this time, since the brightness of the image data is adjusted by the gain processing unit 23 and a foundation for the normal operation of the visual model described above with reference to FIG. 2 is formed, the NR processing unit 25 uses the main subject. Therefore, a suitable NR process can be performed.

  The TM processing unit 26 performs tone correction on the image data after NR processing using the normalized tone correction characteristic represented by the tone curve 83. Since the normalized gradation correction characteristic is normalized with respect to the representative value (for example, x0) of the main subject, it is linear up to the representative value x0, but in general, from the representative value x0, It has the property of reducing the brightness of high brightness parts. As a result, over-exposure in the signal processing unit 16 at the subsequent stage is suppressed.

  In gradation correction, gradation correction characteristics normalized with respect to the representative value of the main subject are used. Thereby, as shown in the histogram 92 and the histogram 93, the brightness axis (level width of the image signal) spread by the NR process can be narrowed without changing the width of the brightness range E of the main subject. .

  As described above, in the image signal processing unit 15, the image data is processed according to the gradation correction characteristics determined by the DET processing unit 21 by the two-step brightness gradation correction processing of the gain processing unit 23 and the TM processing unit 26. The brightness of is corrected.

  That is, the TM processing unit 26 operates in cooperation with the gain processing unit 23, whereby the brightness correction of the image data according to the determination (tone correction characteristics) of the DET processing unit 21 is achieved.

  For reference, when the TM processing unit 26 is not provided, the entire image is input to the subsequent signal processing unit 16 while being brightly corrected by the brightness adjustment by the gain processing unit 23. Wide DR shooting may not be effective.

[Operation of digital camera]
Next, the wide DR shooting process of the digital camera 1 will be described with reference to the flowchart of FIG. For example, when the digital camera 1 is turned on by a user operation and the operation mode of the digital camera 1 is switched to the wide DR shooting mode, the process of FIG. 7 is started.

  In step S11, the exposure controller 11 and the exposure amount adjustment unit 12 perform exposure control. That is, the exposure controller 11 confirms that the shooting mode of the digital camera 1 is the wide DR shooting mode. Then, for example, as described above with reference to FIG. 4, the exposure controller 11 adjusts the exposure so that the exposure amount of the main subject is not extremely reduced while preventing overexposure on the image sensor 13. Is used to set the exposure adjustment amount (control value of the exposure amount adjustment unit 12). The exposure adjustment unit 12 adjusts the exposure of the image sensor 13 using the control value from the exposure controller 11, and performs exposure.

  In step S <b> 12, the image sensor 13 images the subject with the exposure adjusted by the exposure amount adjustment unit 12. That is, the image sensor 13 converts light incident from a subject or the like through an optical block (not shown) including a lens or the like into an analog signal.

  The exposed light is converted into an analog signal by the image sensor 13 and further quantized by the A / D converter 14 into digital image data. Image data from the A / D conversion unit 14 is output to the DET processing unit 21 and the gain processing unit 23 of the image signal processing unit 15.

  In step S13, the DET processing unit 21 analyzes the image data from the A / D conversion unit 14, and uses the exposure adjustment amount from the exposure controller 11 to correct the brightness of the image data. That is, a gradation correction function representing gradation correction characteristics is determined. As described above with reference to FIG. 5, the gradation correction function is determined, for example, so as to suppress overexposure in a high-luminance portion and to brighten the main subject. The gradation correction function determined by the DET processing unit 21 is output to the representative value detection unit 71 and the gradation correction normalization unit 72 of the distribution processing unit 22.

  In step S <b> 14, the representative value detection unit 71 detects the gradation correction representative value G from the gradation correction function determined by the DET processing unit 21. As described above with reference to FIG. 6, for example, when the gradation correction characteristic is represented by the function y = f (x), the gradation correction representative value G is expressed by the input / output ratio f (x) / x. And obtained from equation (1). The gradation correction representative value G detected by the representative value detection unit 71 is output to the gradation correction normalization unit 72, the gain processing unit 23, and the NR adjustment processing unit 24.

  In step S15, the gradation correction normalization unit 72 normalizes the remaining components obtained by removing the gradation correction representative value G detected by the representative value detection unit 71 from the gradation correction function determined by the DET processing unit 21. A normalized gradation correction function f ′ (x) representing gradation correction characteristics is obtained. As described above with reference to FIG. 6, the normalized gradation correction function f ′ (x) is obtained by the representative value detection unit 71 when the gradation correction characteristic is represented by a function y = f (x), for example. Using the detected gradation correction representative value G, it is obtained by Expression (2).

  The normalized gradation correction function f ′ (x) obtained by the gradation correction normalization unit 72 is output to the TM processing unit 26.

  In step S <b> 16, the gain processing unit 23 corrects the brightness of the entire image of the image data from the A / D conversion unit 14 using the gradation correction representative value G determined by the representative value detection unit 71. That is, the gain processing unit 23 uniformly applies a gain (gradation correction representative value) G to the image data. Thereby, the brightness of the entire image of the image data is corrected according to the main subject. The image data whose brightness has been corrected by the gain processing unit 23 is output to the NR processing unit 25.

  In step S <b> 17, the NR adjustment processing unit 24 sets a noise model used by the NR processing unit 25 using the gradation correction representative value G detected by the representative value detection unit 71. For example, the noise model used by the NR processing unit 25 is defined by the following equation (3) described in Patent Document 1.

式（３）において、パラメータaは、入力レベルに略比例するノイズ成分を定義するものであり、パラメータbは、入力レベルの平方根に略比例するノイズ成分を定義するものであり、パラメータcは、入力レベルに依存しないノイズ成分を定義するものである。なお、σ(x)は、ノイズ信号の振幅の分散を表しており、物理的に計算される値である。

In equation (3), parameter a defines a noise component that is approximately proportional to the input level, parameter b defines a noise component that is approximately proportional to the square root of the input level, and parameter c is It defines noise components that do not depend on the input level. Note that σ (x) represents the variance of the amplitude of the noise signal and is a physically calculated value.

  The parameters a, b, and c used in the noise model σ (x) represented by the equation (3) can be defined by the following equations (4) to (6), respectively.

a ≡ V _a ² (4)
b ≡ g _FD ²・ g _sig ² (5)
c ≡ n _ri ²・ g _sig ²・ ・ ・ （６）

Parameter V _a of formula (4) is a value indicating the characteristics of the opening (sensitivity) unevenness in the image pickup device 13, the average value of the output of each pixel when is incident uniform light to the light receiving portion of the image pickup device 13 Is shown as a percentage. More specifically, it is calculated as the square root of the value obtained by subtracting the dispersion of the optical shot noise and the floor noise from the dispersion of the output value when the uniform light of the light receiving unit is incident.

The parameter b corresponds to optical shot noise generated by the image sensor 13. A parameter g _{FD in} Expression (5) indicates a gain value generated when the charge amount of the image sensor 13 is converted into a voltage value.

The parameter c corresponds to the floor noise that wants to depend on the input level. The parameter n _{ri in} Expression (6) indicates a converted noise value obtained by converting floor noise such as thermal noise, dark shot noise, dark unevenness, etc. into a signal level in the analog processing system input unit. Specifically, it is calculated as the square root of the variance of the output value for each pixel obtained when the incident light to the light receiving portion of the image sensor 13 is blocked.

The parameter g _sig in the equations (5) and (6) indicates the total gain in the analog image signal transmission system and the A / D conversion system. This parameter g _sig is expressed by the following equation (7).

Here, the parameter g _CDS indicates a gain generated in a CDS (not shown). The parameter g _a is the gain generated in the analog amplifier (not shown) provided in the previous stage of the A / D converter 14. The parameter n _bits indicates the maximum number of bits of output data of the A / D conversion unit 14. The parameter V _fs indicates an allowable maximum value of the input voltage of the A / D conversion unit 14.

The parameter g _d indicates a gain (gradation correction representative value) G applied to the image data after A / D conversion by the gain processing unit 23 provided in the subsequent image signal processing unit 15.

That is, the NR adjustment processing unit 24 multiplies Expression (7) by a gain (gradation correction representative value) G, sets parameters, obtains the shape of the function of Expression (3), and performs NR processing on it. The noise model used by the unit 25 is set. Thereby, the influence which the gain (gradation correction representative value) G applied by the gain processing unit 23 has on the parameter g _d of the equation (7) can be taken into consideration.

In step S <b> 18, the NR processing unit 25 performs NR processing on the image data after the gain processing unit 23. The processing in step S18 will be specifically described. The physical characteristic value calculation unit 31 uses a noise model in which parameters are set in step S16, and is a noise that is a value depending on the luminance of the image data from the gain processing unit 23. The amount σ ² is calculated and output to the threshold value determination unit 33.

  The visual characteristic value calculation unit 32 determines the colors R, G, and B of the image indicated by the image data from the gain processing unit 23, and uses the visual model including the conversion table described above with reference to FIG. A correction parameter k, which is a coefficient for correcting the value, is calculated and output to the threshold value determination unit 33.

The threshold value determination unit 33 multiplies the noise amount σ ² from the physical characteristic value calculation unit 31 by the correction parameter k from the visual characteristic value calculation unit 32, and uses the multiplied value kσ ² as the filter threshold value ε to generate noise. The data is output to the removal unit 34.

  The noise removing unit 34 performs, for example, noise removal using an ε filter on the image data from the gain processing unit 23 using the filter threshold value ε from the threshold value determining unit 33, and is subjected to the noise removal processing. The data is output to the TM processing unit 26 at the subsequent stage.

As described above, the NR processing unit 25 determines the filter threshold ε obtained by multiplying the noise amount σ ² from the noise model and the correction parameter k from the visual model. Therefore, the brightness of the input image data is adjusted by the gain processing unit 23, so that the visual model operates correctly and the main subject is appropriately considered in consideration of physical characteristics of noise and human visual characteristics. It is possible to perform noise removal processing using a simple threshold.

  In step S19, the TM processing unit 26 performs tone correction on the image data after the NR processing using, for example, the normalized tone correction function represented by the tone curve 83 described above with reference to FIG. As processing, tone curve processing is performed. The tone-corrected image data is output to the signal processing unit 16.

  In step S <b> 20, the signal processing unit 16 performs signal processing for converting the image data from the TM processing unit 26 into image data suitable for display on the subsequent display unit 17 or recording in the recording unit 18, The data is output to the corresponding display unit 17 and recording unit 18.

  For example, image data that has undergone camera signal processing suitable for display on the display unit 17 is displayed on the display unit 17. Image data that has undergone camera signal processing suitable for recording in the recording unit 18 is recorded by a recording unit 18 on a recording medium such as an optical disk or a magnetic disk (not shown).

  As described above, the representative value detection unit 71 detects the gradation correction representative value G from the gradation correction function determined for the image data, and the gain processing unit 23 detects the detected gradation correction representative value. G is used to correct the brightness of the entire image data. Thereby, NR processing can be suitably performed on the main subject. That is, even when tone correction processing and noise reduction processing are used in combination, an image in which noise is appropriately suppressed can be obtained.

  Further, the TM processing unit 26 operates in cooperation with the gain processing unit 23, whereby the brightness correction of the image data according to the determination (tone correction function) of the DET processing unit 21 is achieved. That is, in the TM processing unit 26, gradation correction is performed using a normalized gradation correction characteristic that generally has a property of reducing the brightness of a portion with high luminance. Thereby, overexposure in the signal processing unit 16 at the subsequent stage can be suppressed, and the effect of wide DR photography can be made effective.

  Further, the gradation correction characteristic is normalized by the gradation correction normalization unit 72, and the gradation correction is performed by the TM processing unit 26 using the normalized gradation correction function. As a result, it is possible to improve the deterioration of signal continuity (smooth change in the level direction) during wide DR shooting while ensuring the performance of NR processing.

  As described above, it is possible to suppress deterioration in processing performance when the tone correction processing and the noise reduction processing are used in combination.

[Description of effects]
With reference to FIGS. 8 to 10, a detailed description will be given of an improvement effect on the deterioration of the continuity of the signal during the wide DR shooting described above. FIG. 8 shows graphs 111 to 114 representing signal continuity in each process of the conventional configuration during standard imaging. FIG. 9 shows graphs 121 to 124 representing signal continuity in each process of the conventional configuration during wide DR shooting. FIG. 10 shows graphs 131 to 135 representing signal continuity in each process of the image signal processing unit 15 during wide DR shooting.

  As a conventional configuration shown in FIGS. 8 and 9, a configuration in which NR processing is performed first and tone correction processing is performed next is considered. This is because the reverse configuration (the configuration in which gradation correction processing is performed first) is not generally used because it reduces the discrimination between the noise of the NR processing and the original signal. Further, in FIGS. 8 to 10, as shown in the 0.5 / 0.5 NR processing LPF (Low Pass Filter) coefficient, a 2-tap ε filter is used as the NR processing for convenience of explanation.

  A graph 111 in FIG. 8 represents the level at the pixel position of the image data of the analog signal. A graph 112 in FIG. 8 represents the continuity of the signal of the image data after A / D conversion. A graph 113 in FIG. 8 represents the continuity of the image data signal after the NR processing. A graph 114 in FIG. 8 represents the continuity of the signal of the image data after gradation correction.

  A graph 121 in FIG. 9 represents the level at the pixel position of the image data of the analog signal. A graph 122 in FIG. 9 represents the continuity of the signal of the image data after A / D conversion. A graph 123 in FIG. 9 represents the continuity of the image data signal after the NR processing. A graph 124 in FIG. 9 represents the continuity of the signal of the image data after the gradation correction.

  A graph 131 in FIG. 10 represents the level at the pixel position of the image data of the analog signal. A graph 132 in FIG. 10 represents the continuity of the signal of the image data after A / D conversion. A graph 133 in FIG. 10 represents the continuity of the image data signal after the gain processing. A graph 134 in FIG. 10 represents the continuity of the signal of the image data after the NR processing. A graph 135 in FIG. 10 represents the continuity of the image data signal after gradation correction.

  In the graph 111 in FIG. 8, the graph 121 in FIG. 9, and the graph 131 in FIG. 10, the horizontal axis represents the pixel position, and the vertical axis represents the level. That is, the levels of the graph 121 of FIG. 9 and the graph 131 of FIG. 10 are lower than the level of the graph 111 of FIG.

  Further, the scales indicated by the dotted lines in the graphs 112 to 114 in FIG. 8, the graphs 122 to 124 in FIG. 9, and the graphs 132 to 135 in FIG. 10 represent the quantization accuracy. The gentler the step, the finer the quantization accuracy and the more continuous the signal. That is, one step of the scales of the graphs 112 to 114 of FIG. 8, the graphs 122 and 123 of FIG. 9, and the graphs 132, 134, and 135 of FIG. 10 is the scale of the graph 124 of FIG. 9 and the graph 133 of FIG. It is gentler than the two steps, the quantization accuracy is fine, and the signal is continuous.

  With reference to these graphs, first, the case of wide DR imaging in the conventional configuration will be described. As shown in the graph 111 in FIG. 8 and the graph 121 in FIG. 9, in the case of wide DR shooting in the conventional configuration, A / D conversion is performed on an analog signal having a lower level than in the case of standard shooting. Yes.

  In the case of wide DR shooting in the conventional configuration, after A / D conversion, after NR processing, tone correction is performed, and the brightness of the main subject is corrected. At this time, the signal range assigned to the main subject is expanded by the gradation correction. Therefore, in the conventional wide DR shooting, as shown in the graph 124 of FIG. 9, the continuity of the signal is deteriorated after the gradation correction as compared with the image data at the time of the standard shooting shown in the graph 114 of FIG. Image data is output.

  On the other hand, as shown in the graph 111 in FIG. 8 and the graph 131 in FIG. 10, the image signal processing unit 15 has a lower level than in the case of standard imaging, as in the case of wide DR imaging in the conventional configuration. A / D conversion is performed on the analog signal.

  Then, after the A / D conversion, the image signal processing unit 15 extends the brightness axis (horizontal axis) by the gain in the gain processing as shown in the graph 133 of FIG. Image data with degraded continuity is generated. That is, by increasing the gain of the digital signal, the influence of the quantization error increases, and the signal has only a jump value (the amount of change increases and the continuity deteriorates).

  However, the image data becomes image data in which the continuity of the signal is improved by the smoothing action of the subsequent NR process, as shown in the graph 134 of FIG. That is, since the threshold ε of the ε filter is usually larger than the accuracy of the signal (= resolution in the level direction), the effect of improving the continuity of the signal can be obtained by the smoothing action of the NR process.

  Further, in the gradation correction, by using the normalized gradation correction characteristics, as described above using the histogram 92 and the histogram 93 in FIG. 6, the level width of the signal of the image stretched by the NR process Works to narrow. Therefore, unlike the conventional image data after gradation correction shown by the graph 124 in FIG. 9, the continuity of the signal does not deteriorate due to the gradation correction. As a result, as shown in the graph 135 of FIG. 10, the image data after the gradation correction of the image signal processing unit 15 is compared with the image data after the gradation correction shown in the graph 124 of FIG. Only the smoothing effect of NR processing is excellent in signal continuity.

  As described above, by performing NR processing with the brightness adjusted for the main subject, the parameters of NR processing are optimized, and even when NR processing and gradation correction processing are used together, An image in which noise is appropriately suppressed can be obtained.

  In addition, the deterioration of signal continuity due to gain processing is improved, and furthermore, since tone correction processing acts in a direction to improve signal continuity, it is possible to obtain an image with excellent signal continuity as a whole. it can.

  In the above description, tone curve processing is used as an example of tone correction. However, not only tone curve processing but also dynamic range compression processing can be used.

  Dynamic range compression processing is processing that compresses the range of a signal while ensuring the appearance of details in the image. Technically, the image signal is decomposed or extracted based on spatial characteristics. This is a process of dividing the intensity of each signal into two or more components.

  For example, Japanese Patent Laid-Open No. 2001-275015 (hereinafter referred to as Patent Document 4) extracts a low frequency band component of an image, compresses a range with an LUT, extracts a high frequency band component, and enhances it. A dynamic range compression process is described. Japanese Unexamined Patent Application Publication No. 2007-049540 (hereinafter referred to as Patent Document 5) extracts a low-frequency component of an image, compresses the range, and selectively enhances the high-frequency component by contrast correction. Processing is described.

  That is, when dynamic range compression processing as shown in Patent Documents 4 and 5 is used as gradation correction in the TM processing unit 26, for example, normalization is performed on the low-frequency component during range compression of the low-frequency component. Tone curve processing is performed with the tone correction function.

  Further, as an example of gradation correction, a shading process for increasing the brightness of the peripheral portion of the image can be used. Next, with reference to FIG. 11, a description will be given of a case where a shading process, which is a shading correction process for a lens having a function of reducing the exposure amount in the peripheral part of the image, is used as the gradation correction in the TM processing unit 26. .

[Description of image signal processing]
FIG. 11 is a diagram illustrating details of the processing of the image signal processing unit 15 when the shading processing is used as the gradation correction. 11 is different from the example of FIG. 6 in that the DET processing unit 21 and the TM processing unit 26 are replaced with the DET processing unit 151 and the TM processing unit 152, respectively. The gain processing unit 23 and the NR processing unit 25 are common.

  In the example of FIG. 11, graphs of gain curves 171 to 173 for shading correction and images 181 to 183 are shown. In addition, in the images 181 to 183, the change in the brightness of the person who is the main subject is not shown, but in reality, the brightness of the person changes in the same way as the background of the respective images 181 to 183. .

  In each graph of the gain curves 171 to 173, the horizontal axis represents the image height (pixel position with the position of the optical axis as the origin) in the images 181 to 183, and the image height 0 (the left end in the figure) is The center position (position of the optical axis) of the screen 181 and the image height x0 indicate the image height of the main subject (the person in the images 181 to 183). The vertical axis represents the gain.

  The gain curve 171 represents the gradation correction characteristic determined by the DET processing unit 151, and the gain curve 172 is a gain curve (1) that uniformly applies the gradation correction representative value G detected by the representative value detection unit 71. Straight line). The gain curve 173 represents the normalized gradation correction characteristic obtained by the gradation correction normalization unit 72. Images 181 to 183 are images corresponding to the image data input to the gain processing unit 23, the image data after gain processing, and the image data after gradation correction, respectively.

  The image 181 of the image data obtained by imaging becomes darker as it goes to the outside of the screen due to the characteristics of the lens. This is called shading, and the DET processing unit 151 determines gradation correction characteristics for shading correction that makes the brightness of the main subject c suitable.

  In this case, a shading correction gain corresponding to the pixel position of the image height x0 of the main subject as represented by the gain curve 171 is used as the gradation correction characteristic. For example, data of gradation correction characteristics represented by a gain curve 171 is input from the DET processing unit 151 to the representative value detection unit 71 and the gradation correction normalization unit 72.

  Similar to the example of FIG. 6, the representative value detection unit 71 uses the expression (1) to represent the gradation correction representative value that is a value representing the correction amount of the gradation correction characteristic determined by the DET processing unit 151. G is detected. When the shading correction gain according to the pixel position is used as the gradation correction, the gradation correction representative value G is, for example, an average value or a maximum value of the correction gain in the range of the pixel position of the main subject, or It is also possible to simply use the maximum value of the correction gain. By this gradation correction representative value G detection process, an element for correcting the brightness of the main subject is detected.

  Similar to the example of FIG. 6, the gradation correction normalization unit 72 uses the equation (2) to determine the gradation detected by the representative value detection unit 71 from the gradation correction characteristics determined by the DET processing unit 151. The remaining components excluding the corrected representative value G are obtained. That is, the tone correction normalization unit 72 normalizes the tone correction characteristics determined by the DET processing unit 151 with respect to the main subject, and obtains new tone correction characteristics.

  As described above, the distribution processing unit 22 represents the gradation correction characteristic represented by the gain curve 171 with the uniform gain G represented by the gain curve 172 and the gain curve 173 with respect to the frame. Decompose into normalized tone correction characteristics. The gradation correction representative value G detected by the representative value detection unit 71 is output to the gain processing unit 23, and the normalized gradation correction characteristic obtained by the gradation correction normalization unit 72 is output to the TM processing unit 152. Is done.

  The gain processing unit 23 uniformly applies a gain (gradation correction representative value) G to the input image data as indicated by the gain curve 172. Thus, before the application of the gradation correction representative value G, the brightness of the dark background and the main subject as shown in the image 181 matches the main subject as shown in the background of the image 182. Brightly corrected.

  The NR processing unit 25 performs NR processing on the image data after the gain processing unit 23. At this time, since the brightness of the image data is adjusted by the gain processing unit 23, the NR processing unit 25 can perform suitable NR processing on the main subject.

  The TM processing unit 152 performs gradation correction using the normalized gradation correction characteristic represented by the gain curve 173 normalized with respect to the image height x0 of the main subject with respect to the image data after NR processing. Perform shading correction. Thereby, an image 183 subjected to shading correction can be obtained. In other words, as shown in the background of the image 183, the image 183 is corrected so that it goes brighter toward the outside of the screen, and the image 183 is obtained by uniformly correcting the peripheral portion of the image and the central portion of the image. Can do.

  As described above, the image signal processing unit 15 performs shading of image data according to the tone correction characteristics determined by the DET processing unit 151 by the two-stage brightness correction processing of the gain processing unit 23 and the TM processing unit 152. Can also be corrected.

<2. Second Embodiment>
[Configuration example of personal computer]
FIG. 12 is a block diagram showing a configuration example of an embodiment of an image signal processing system including a personal computer as an image signal processing apparatus to which the present invention is applied.

  In the example of FIG. 12, the image signal processing system 201 includes an imaging device 211 having each part of the preceding stage up to the A / D conversion unit 14 of the digital camera 1 of FIG. 1, and the image signal processing unit 15 and later of the digital camera 1. The personal computer 212 having the subsequent stages is configured. In the example of FIG. 12, portions corresponding to those of the example of FIG. 1 are denoted by the corresponding reference numerals, and the description thereof will be repeated, and will be omitted as appropriate.

  That is, the imaging device 211 includes the exposure controller 11, the exposure amount adjustment unit 12, the imaging element 13, and the A / D conversion unit 14 in FIG. The imaging device 211 records undeveloped image data (RAW image data) captured by the imaging device 13 and digitally converted by the A / D conversion unit 14 on a recording medium (not shown). At this time, the imaging device 211 also records the exposure adjustment amount during imaging controlled by the exposure controller 11 as metadata in the RAM image file. This exposure adjustment amount is added to, for example, a tag ExposureBiasValue (ID: 37380) of Exif (Exchangeable image file format) information. Note that the use tag is an example, and other tags such as a manufacturer-specific tag can also be used.

  The imaging device 211 transfers RAW image data with metadata added to the personal computer 212 via, for example, a USB (Universal Serial Bus) cable, the recording medium itself, or a network. Deliver.

  The personal computer 212 includes an image signal processing unit 15 including a DET processing unit 21, an allocation processing unit 22, a gain processing unit 23, an NR adjustment processing unit 24, an NR processing unit 25, and a TM processing unit 26 in FIG. The unit 16, the display unit 17, and the recording unit 18 are configured.

  The personal computer 212 acquires the RAW image data to which the metadata is added from the imaging device 211, inputs the acquired RAW image data to the image signal processing unit 15, and refers to FIG. The above-described NR process and gradation correction process are executed.

  That is, the RAW image data and the exposure adjustment amount, which is metadata added to the RAW image data, are input to the DET processing unit 21 of the image signal processing unit 15 and are used for determining gradation correction characteristics. The RAW image data is also input to the gain processing unit 23 of the image signal processing unit 15 and is output to the NR processing unit 25 after the brightness is adjusted, and is output to the TM processing unit 26 after the NR processing is performed. Tone correction processing is performed.

  Next, image signal processing of the personal computer 212 will be described with reference to the flowchart of FIG. Note that the processing in steps S112 to S119 in FIG. 13 is basically the same as the processing in steps S13 to S20 in FIG.

For example, the power of the imaging device 211 is turned on by a user operation, and the operation mode of the imaging device 211 is switched to the wide DR shooting mode. Since the exposure controller 11 of the image pickup apparatus 211 is in the wide DR shooting mode, the exposure adjustment amount (exposure) is set so that the exposure amount of the main subject is not extremely reduced while preventing overexposure on the image pickup device 13. Control value of the amount adjusting unit 12). The exposure adjustment unit 12 adjusts the exposure of the image sensor 13 using the control value from the exposure controller 11, and performs exposure. The imaging element 13 images the subject with the exposure adjusted by the exposure adjustment unit 12.

  The exposed light is converted into an analog signal by the image sensor 13 and further quantized by the A / D converter 14 to become digital image data. Undeveloped image data (RAW image data) from the A / D converter 14 is recorded on a recording medium (not shown) with the exposure adjustment amount from the exposure controller 11 added as metadata.

  The RAW image data recorded on the recording medium of the imaging device 211 is transferred to the personal computer 212 via a USB cable or the like, for example, according to a user operation.

  In step S <b> 111, the personal computer 212 inputs the RAW image data to which the metadata from the imaging device 211 is added to the image signal processing unit 15. The RAW image data is input to the DET processing unit 21 and the gain processing unit 23. The exposure adjustment amount added as metadata to the RAW image data is input to the DET processing unit 21.

  In step S112, the DET processing unit 21 analyzes the image data from the imaging device 211 and uses the exposure adjustment amount from the exposure controller 11 to correct the brightness of the image data, that is, the gradation correction characteristic. A gradation correction function representing a tone correction characteristic is determined. As described above with reference to FIG. 5, the gradation correction function is determined, for example, so as to suppress overexposure in a high-luminance portion and to brighten the main subject. The gradation correction function determined by the DET processing unit 21 is output to the representative value detection unit 71 and the gradation correction normalization unit 72 of the distribution processing unit 22.

  In step S <b> 113, the representative value detection unit 71 detects the gradation correction representative value G from the gradation correction function determined by the DET processing unit 21. As described above with reference to FIG. 6, for example, when the gradation correction characteristic is represented by the function y = f (x), the gradation correction representative value G is expressed by the input / output ratio f (x) / x. And obtained from equation (1). The gradation correction representative value G detected by the representative value detection unit 71 is output to the gain processing unit 23 and the NR adjustment processing unit 24.

  In step S114, the gradation correction normalization unit 72 normalizes the remaining components obtained by removing the gradation correction representative value G detected by the representative value detection unit 71 from the gradation correction function determined by the DET processing unit 21. A normalized gradation correction function f ′ (x) representing gradation correction is obtained. As described above with reference to FIG. 6, the normalized gradation correction function f ′ (x) is obtained by the representative value detection unit 71 when the gradation correction characteristic is represented by a function y = f (x), for example. Using the detected gradation correction representative value G, it is obtained by Expression (2).

  The normalized gradation correction function f ′ (x) obtained by the gradation correction normalization unit 72 is output to the TM processing unit 26.

  In step S <b> 115, the gain processing unit 23 corrects the brightness of the entire image of the image data from the imaging device 211 using the gradation correction representative value G determined by the representative value detection unit 71. That is, the gain processing unit 23 uniformly applies a gain (gradation correction representative value) G to the image data. Thereby, the brightness of the entire image of the image data is corrected according to the main subject. The image data whose brightness has been corrected by the gain processing unit 23 is output to the NR processing unit 25.

  In step S <b> 116, the NR adjustment processing unit 24 sets a noise model used by the NR processing unit 25 using the gradation correction representative value G detected by the representative value detection unit 71. That is, the NR adjustment processing unit 24 multiplies the above-described equation (7) by a gain (gradation correction representative value) G, sets parameters, and obtains the shape of the function of the above-described equation (3). It is set as a noise model used by the NR processing unit 25.

  In step S117, the NR processing unit 25 performs NR processing on the image data after the gain processing unit 23. At this time, since the brightness of the image data is adjusted by the gain processing unit 23, the NR processing unit 25 can perform suitable NR processing on the main subject. The image data that has been subjected to noise removal processing is output to the TM processing unit 26 in the subsequent stage.

  In step S118, the TM processing unit 26 performs tone curve processing as gradation correction on the image data after NR processing using the normalized gradation correction characteristic. The tone-corrected image data is output to the signal processing unit 16.

  In step S119, the signal processing unit 16 performs signal processing for converting the image data from the TM processing unit 26 into image data suitable for display on the subsequent display unit 17 and recording in the recording unit 18, The data is output to the corresponding display unit 17 and recording unit 18.

  For example, image data that has undergone camera signal processing suitable for display on the display unit 17 is displayed on the display unit 17. Image data that has undergone camera signal processing suitable for recording in the recording unit 18 is recorded by a recording unit 18 on a recording medium such as an optical disk or a magnetic disk (not shown).

  As described above, the NR process is performed with the brightness adjusted for the main subject. Accordingly, in the personal computer 212 of FIG. 12, even when the NR processing parameters are optimized and the NR processing and the gradation correction processing are used together, an image in which noise is appropriately suppressed can be obtained.

  In the above description, the example of the NR processing unit 25 having the block (visual characteristic value calculating unit 32) using the visual model has been described. However, the present invention explicitly includes the NR processing unit having no block. It can also be applied to.

  In other words, even NR processing that does not have a block that explicitly uses a visual model has, for example, a framework that modulates a parameter for each pixel level, and there is a difference between the parameter and the noise model. In this case, since it is considered that the parameter includes an element corresponding to the visual model, by applying the present invention, the performance of the NR processing is improved as in the case of the NR processing having a block using the visual model. Can be improved.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

  FIG. 14 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processes using a program.

  A CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, and a RAM (Random Access Memory) 303 are connected to each other via a bus 304.

  An input / output interface 305 is further connected to the bus 304. The input / output interface 305 is connected to an input unit 306 including a keyboard and a mouse, and an output unit 307 including a display and a speaker. In addition, a storage unit 308 including a hard disk and a nonvolatile memory, a communication unit 309 including a network interface, and a drive 310 that drives the removable medium 311 are connected to the bus 304.

  In the computer configured as described above, the CPU 301 loads the program stored in the storage unit 308 to the RAM 303 via the input / output interface 305 and the bus 304 and executes the program, for example. Is done.

  The program executed by the CPU 301 is recorded in the removable medium 311 or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and is installed in the storage unit 308.

  Note that the program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a block diagram which shows the structural example of embodiment of the digital camera to which this invention is applied. FIG. 2 is a block diagram illustrating a configuration example of an NR processing unit in FIG. 1. It is a figure which shows the example of a conversion table. It is a figure which shows the image of the image data image | photographed in imaging | photography mode. FIG. 5 is a diagram illustrating an example of gradation correction characteristics of the image data in FIG. 4. It is a figure explaining the process of an image signal process part. 3 is a flowchart for explaining a wide DR shooting process of the digital camera of FIG. 1. It is a figure which shows the graph of the continuity of the signal in each process of the conventional structure at the time of standard imaging | photography. It is a figure which shows the graph of the continuity of the signal in each process of the conventional structure at the time of wide DR imaging | photography. It is a figure which shows the graph of the continuity of the signal in each process of the image signal process part at the time of wide DR imaging | photography. It is a figure explaining the shading process as gradation correction. It is a block diagram which shows the structural example of embodiment of the image signal processing system to which this invention is applied. 13 is a flowchart for describing image signal processing of the personal computer in FIG. 12. It is a block diagram which shows the structural example of the hardware of a computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Digital camera, 11 Exposure control controller, 15 Image signal processing part, 21 DET processing part, 22 Distribution processing part, 23 Gain processing part, 24 NR adjustment processing part, 25 NR processing part, 26 TM processing part, 31 Physical characteristic Value calculating unit, 32 visual characteristic value calculating unit, 33 threshold value determining unit, 34 noise removing unit, 71 representative value detecting unit, 72 gradation correction normalizing unit, 151 DET processing unit, 152 TM processing unit, 201 image signal processing system , 211 imaging device, 212 personal computer

Claims (9)

Control means for determining a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame based on an exposure adjustment amount at the time of shooting the frame ;
The gradation correction characteristic determined by the control means includes a gradation correction representative value representing a correction amount with respect to a representative value of a main subject, and the remaining gradations obtained by removing the gradation correction representative value from the gradation correction characteristic. A distribution means for distributing the correction characteristics;
Gain processing means for applying the gradation correction representative value distributed by the distribution means uniformly as a gain to the frame;
Noise reduction processing means for performing noise reduction processing on the frame to which the gradation correction representative value is applied by the gain processing means;
Gradation correction processing means for performing gradation correction processing on the frame on which the noise reduction processing has been performed by the noise reduction processing means using the remaining gradation correction characteristics distributed by the distribution means; An image signal processing apparatus. The image signal processing apparatus according to claim 1, wherein the remaining gradation correction characteristics are normalized gradation correction functions obtained by normalizing the gradation correction characteristics with respect to representative values of the main subject. The image signal processing apparatus according to claim 2, wherein the tone correction processing unit performs tone curve processing as the tone correction processing. The image signal processing apparatus according to claim 2, wherein the gradation correction processing unit performs dynamic range compression processing as the gradation correction processing. The image signal processing apparatus according to claim 2, wherein the gradation correction processing unit performs a shading correction process as the gradation correction process. The image signal processing apparatus according to claim 2, further comprising: a parameter setting unit that sets a noise model parameter for the noise reduction process using the gradation correction representative value distributed by the distribution unit. The image signal processing device
Based on the exposure adjustment amount at the time of shooting the frame, to determine a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame,
The determined gradation correction characteristics are converted into a gradation correction representative value representing a correction amount with respect to a representative value of a main subject, and a remaining gradation correction characteristic obtained by removing the gradation correction representative value from the gradation correction characteristic. Sorting,
The assigned gradation correction representative value is uniformly applied to the frame as a gain,
A noise reduction process is performed on the frame to which the gradation correction representative value is applied,
An image signal processing method including a step of performing gradation correction processing on the frame on which the noise reduction processing has been performed using the remaining gradation correction characteristics that have been distributed. Based on the exposure adjustment amount at the time of shooting the frame, to determine a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame,
The determined gradation correction characteristics are converted into a gradation correction representative value representing a correction amount with respect to a representative value of a main subject, and a remaining gradation correction characteristic obtained by removing the gradation correction representative value from the gradation correction characteristic. Sorting,
The assigned gradation correction representative value is uniformly applied to the frame as a gain,
A noise reduction process is performed on the frame to which the gradation correction representative value is applied,
A program for causing a computer to execute a process including a step of performing a gradation correction process on the frame subjected to the noise reduction process using the remaining gradation correction characteristics that have been distributed. Imaging means for imaging a subject;
Control means for determining a gradation correction characteristic representing a conversion characteristic for correcting the brightness of the frame based on an exposure adjustment amount at the time of shooting a frame corresponding to the subject imaged by the imaging means;
The gradation correction characteristic determined by the control means is a gradation correction representative value representing a correction amount for a representative value of a main subject of the subject, and the gradation correction representative value is excluded from the gradation correction characteristic. Distribution means for distributing the remaining gradation correction characteristics;
Gain processing means for applying the gradation correction representative value distributed by the distribution means uniformly as a gain to the frame;
Noise reduction processing means for performing noise reduction processing on the frame to which the gradation correction representative value is applied by the gain processing means;
Gradation correction processing means for performing gradation correction processing on the frame on which the noise reduction processing has been performed by the noise reduction processing means using the remaining gradation correction characteristics distributed by the distribution means; An imaging apparatus provided.

Images