JP2020136890A - Image processing device, image processing method, imaging apparatus, program, and storage medium - Google Patents

Abstract

To provide an image processing device capable of extending a dynamic range on a high luminance side, while suppressing color curving, in an input image.SOLUTION: The image processing device includes an input part for inputting the input image which is an image signal having a plurality of color signals, a white balance processing part for performing white balance adjustment to the input image, a first extension processing part for performing first extension processing for extending the dynamic range in a first color space to a signal after the white balance processing, a conversion part for converting the signal after the first extension processing is performed into the signal of a second color space, and a second extension processing part for performing second extension processing for further extending the dynamic range in the second color space to the signal converted into the second color space by the conversion part.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for gradation conversion of an image.

Conventionally, a white balance gain process is performed on an image taken by an image pickup device such as a digital camera to adjust the color balance by applying different gains according to the color in consideration of the characteristics of the image pickup device and the shooting conditions. There is. However, among the signals of the black and white subject at the time of the input image, when the G signal has reached the saturation level, the signal is not input at the correct color ratio. Therefore, even if the white balance gain processing is performed, the values of the R, G, and B signals do not match, and coloring occurs. In order to suppress this coloring, conventionally, the signal values of other color signals have been dealt with by uniformly clipping at the saturation level of the G signal, but there is a problem that the gradation on the high luminance side is lost. there were.

To solve this problem, a technique is known in which the input signal value is expanded and the saturation level of each color signal is adjusted to maintain the gradation on the high luminance side without causing coloring. For example, in Patent Document 1, after determining the upper limit value (extension upper limit value) of the dynamic range expansion of the input image and adjusting the signal level in the high luminance region of each color signal, the signal having a high saturation level has a low saturation level. A technique has been proposed in which the signal is replaced so that the extended upper limit becomes white.

Japanese Unexamined Patent Publication No. 2015-156616

However, in the prior art disclosed in Patent Document 1 described above, the gradation characteristics in the high luminance region of each color are converted so that the extended upper limit value becomes white. Therefore, in a subject whose brightness changes continuously with the same hue, the RGB composition ratio of the color signal in the middle to high luminance range fluctuates, causing image quality deterioration (color bending) in which the original hue of the subject is different. May be done.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image processing apparatus capable of expanding the dynamic range on the high-luminance side of an input image while suppressing color bending. Is.

The image processing apparatus according to the present invention includes an input means for inputting an input image which is an image signal having a plurality of color signals, a white balance processing means for adjusting the white balance with respect to the input image, and a white balance processing. The first expansion processing means for performing the first expansion processing for expanding the dynamic range in the first color space with respect to the later signal, and the signal after the first expansion processing is performed for the second color. A second expansion process for further expanding the dynamic range in the second color space is performed on the conversion means for converting into a spatial signal and the signal converted into the second color space by the conversion means. It is characterized by comprising the extended processing means of the above.

According to the present invention, it is possible to extend the dynamic range on the high-luminance side of an input image while suppressing color bending.

The block diagram which shows the structure of the image pickup apparatus which is one Embodiment of the image processing apparatus of this invention. The block diagram which shows the structure of the image processing part. The figure which showed the relationship between the amount of light input to an image sensor, and the RGB signal value of an input image. The figure which shows an example of the signal level after applying the white balance gain. The flowchart which shows the operation of the 1st extended processing part. The figure which shows the weight characteristic of the 1st signal expansion amount. The figure which shows an example of the characteristic of the output signal of the 1st expansion processing part. The figure which shows the expansion characteristic of the 2nd expansion processing part.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiment, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the attached drawings, the same or similar configurations are designated by the same reference numbers, and duplicate description is omitted.

FIG. 1 is a block diagram showing a configuration of an image pickup apparatus according to an embodiment of the image processing apparatus of the present invention. In FIG. 1, the image pickup device 100 includes an image pickup element 111, a mechanical shutter 113, a camera control unit 114, an image processing unit 115, an operation unit 117, a display unit 118, and a recording medium 119. Further, as a photographing optical system for forming an image of a subject image, it has a condensing lens 121, an aperture 122, a focus lens 123, and a lens control unit 124.

The user can specify the control contents of the camera control unit 114 and the lens control unit 124 by operating the AF start button, the shooting start button, etc. (not shown) of the operation unit 117, and perform the AF operation, the shooting operation, and the like. it can. In addition, a menu button and a control button are also arranged on the operation unit 117, and the menu screen can be displayed on the display unit to set the operation contents of the camera such as still image shooting and moving image shooting.

FIG. 2 is a block diagram showing the configuration of the image processing unit 115 shown in FIG. The image processing unit 115 of the present embodiment expands the dynamic range on the high-luminance side of the input image based on the user's instruction or the feature analysis result of the input image.

In FIG. 2, an image signal having signal values of all three colors of R, G, and B is input to the image input terminal 200 of the image processing unit 115 per pixel. The system control unit 114 controls the entire image pickup apparatus 100 and also controls the image processing unit 115. Further, the system control unit 114 determines the extension upper limit value at the time of expanding the dynamic range on the high brightness side of the input image based on the user's operation or the analysis result of the brightness of the input image and the distribution of the color signal in the screen. It is sent to the second expansion processing unit 205. Further, the system control unit 114 divides the input image into M × N rectangular areas, calculates the integrated value of the RGB signals for each divided area, and calculates the white balance gain to be applied to the input image from those ratios. Then, it is sent to the white balance gain processing unit 202.

The white balance gain processing unit 202 applies the white balance gain determined by the system control unit 114 to each of the R, G, and B signals to the image input from the image input terminal 200 to adjust the white balance. Do.

The first expansion processing unit 203 refers to the R and B signals after the white balance gain processing, and extends the G signal in the high brightness region so that the color ratios of R, G, and B do not fluctuate significantly in the RGB color space. Perform the process of The color space conversion processing unit 204 converts the expanded RGB color space signal output from the first expansion processing unit 203 into a YUV color space signal.

The second expansion processing unit 205 performs processing in the YUV color space to expand the amplitude of the YUV signal in the high-luminance range up to a preset development range. The image output terminal 206 outputs the YUV signal processed by the second expansion processing unit 205.

Next, the operation of each processing block of the image processing unit 115 in the present embodiment will be described with reference to FIGS. 3 to 8. In FIG. 2, the input image input from the image input terminal 200 is generated by the image sensor 111. A color filter is arranged on the image sensor 111 to disperse the light from the subject and output each signal value of R, G, and B. The signal levels of R, G, and B are determined according to the amount of light input to the image sensor 111, and the upper limit of the signal is clipped at a predetermined saturation level TH0.

FIG. 3 is a diagram showing an example of the relationship between the amount of light input to the image sensor 111 and the RGB signal value of the input image input from the image input terminal 200. Pin_R, Pin_G, and Pin_B correspond to R, G, and B signals, respectively. The horizontal axis shows the amount of light input to the image sensor 111, and the vertical axis shows an example of each RGB signal level of the input image. The RGB value of the input image is predetermined according to the amount of light input to the image sensor 111. It changes at the ratio of. When the amount of light input to the image sensor 111 of the G signal becomes BV0 or more, the output signal level from the image sensor 111 is saturated at TH0. On the other hand, since the spectral sensitivity of the color filter of the R and B signals is lower than that of G, the output signal level from the image sensor 111 is not saturated even if the amount of light input to the image sensor 111 is BV0 or more.

Next, the operation of the white balance gain processing unit 202 will be described.

Assuming that the signal values of R, G, and B of the input image are Pin_R, Pin_G, and Pin_B, respectively, the signals P1_R, P1_G, and P1_B after the white balance gain processing are represented by the equations (1) to (3). WB_R, WB_G, and WB_B are coefficients of white balance gain.

P1_R = WB_R x Pin_R ... (1)
P1_G = WB_G x Pin_G ... (2)
P1_B = WB_B x Pin_B ... (3)
In the present embodiment, the white balance gain is applied to the R and B signals with reference to the G signal to adjust the white balance. Therefore, in the equations (1) to (3), the value of the gain WB_G of the G signal, which is the reference color of the white balance gain, is 1 (equal magnification), and the gains WB_R of R and B, which are other non-reference colors, The value of WB_B is greater than 1.

FIG. 4 is a diagram showing an example of the signal level after applying the white balance gain. The horizontal axis is the amount of light input to the image sensor 111, and the vertical axis is the signal level after applying the white balance gain in the equations (1) to (3) to each RGB signal input from the image input terminal 200. Is shown. In FIG. 4, in a subject having a light intensity of BV0 or more, the G signal (P1_G) is saturated by the output saturation of the image sensor 111 and the clip processing after the white balance processing. However, the R signal (P1_R) and the B signal (P1_B) maintain the gradation characteristics of the sensor saturation level TH0 or higher.

Here, it is assumed that the system control unit 114 sets the extension upper limit value of the dynamic range expansion for the input image to TH_max. In that case, if the signals in the high-luminance range of G and B are expanded so as to approach the R signal (P1_R) having the highest saturation level, the RGB ratio in the high-luminance range fluctuates and color bending occurs. Therefore, in the present embodiment, the signal level is expanded by dividing the extension upper limit into two stages of the RGB color space and the YUV color space. As a result, processing is performed so that the brightness of the final image output reaches a desired extended upper limit value while suppressing fluctuations in the color ratio in the high brightness range.

Specifically, first, in the first expansion processing unit 203, the G signal in the high luminance region is expanded while referring to the R signal and the B signal so that the color ratio does not fluctuate as much as possible in the RGB color space. After that, the expanded RGB signal is converted into a YUV color space by the color space conversion processing unit 204, and the second expansion processing unit 205 converts the expanded RGB signal into a YUV color space so that the maximum value of the output luminance becomes the expansion upper limit value. Extend the signal in the luminance range. Hereinafter, a more specific description will be given.

First, the operation of the first expansion processing unit 203 will be described. In the first expansion processing unit 203, the signal values of R and B after the white balance processing are maintained in the RGB space, and only the G signal expands the signal in the high luminance region. As an example of the G signal expansion process, in the present embodiment, the G signal for generating the RG color difference and the BG color difference are expanded independently, and the average value of the two types of G signals is obtained after the expansion. I do.

In the following description, P2_R, P2_G, and P2_B are R signal, G signal, and B signal output from the first expansion processing unit 203, respectively. Further, P2_GR and P2_GB are G signals for RG color difference generation and G signals for BG color difference generation, which are intermediately generated for G signal generation after expansion.

FIG. 5 is a flowchart showing the operation of the first expansion processing unit 203.

In FIG. 5, the processing is started in S700, and the RGB signal after the white balance gain processing is input to the first expansion processing unit 203. In S701, the R signal after the white balance gain processing is output as it is as the R signal of the output of the first expansion processing unit 203. In S711, the B signal after the white balance gain processing is output as it is as the B signal of the output of the first expansion processing unit 203.

S702 to S708 show the flow of the G signal expansion processing for generating the RG color difference. In this processing flow, the R signal is set as the target signal for expansion of the G signal, and the expansion processing is performed when the G signal is equal to or more than a predetermined threshold value and equal to or less than the R signal.

In S702, the R and G signals after the white balance gain processing are referred to. In S703, it is determined whether or not the signal value of the G signal is equal to or greater than the threshold value for starting the expansion process. If it is equal to or more than the threshold value, the process proceeds to S704, and if it is less than the threshold value, the process proceeds to S707. In S707, the G signal before the expansion process is selected as the G signal for generating the RG color difference.

In S704, it is determined whether or not the signal value of the G signal is equal to or less than the R signal, and if it is equal to or less than the R signal, the process proceeds to S705. If the signal value of the G signal is larger than the R signal, the process proceeds to S707, and the G signal before the expansion process is selected as the G signal for generating the RG color difference.

In S705, the processing shown in the equation (4) or the equation (5) is performed with reference to the R signal to expand the G signal level for generating the RG color difference.

P2_GR = (P1_R-P1_G) * f ((P1_G-TH_t1) / (TH_t2-TH_t1)) + P1_G ... (4)
P2_GR = P1_R ... (5)
Equation (4) is applied when the signal level of the G signal to be extended is lower than that of the R signal and the saturation level of the G signal has not been reached. In the formula (4), TH_t1 indicates a signal level at which the expansion process of the G signal is started. Further, TH_t2 indicates the saturation level of the G signal before expansion. That is, the signal level of the G signal after white balance processing is P1_G by the weight determined by f ((P1_G-TH_t1) / (TH_t2-TH_t1)) until the signal level of the G signal changes from TH_t1 to TH_t2. Is expanded so as to approach the signal level P1_R of the expansion target, and is output as P2_GR. The weight f (x) of the expansion process at this time has the characteristics shown in FIG. 6, for example, assuming that the input is x and the output is y.

Here, when the signal level of P2_GR exceeds the signal level of the expansion target by the processing of the equation (4), the equation (5) is applied and the R signal is output. Further, even when the signal level of the G signal is lower than that of the R signal but the saturation level of the G signal is reached, the equation (5) is applied and the R signal is output as the G signal for generating the RG color difference. Will be done.

In S706, P2_GR is selected as the G signal for generating the RG color difference. In S708, either the G signal output from S707 or 706 is selected as the G1 signal which is the G signal for generating the RG color difference, and the process proceeds to S719.

Here, an example of the R signal and the G signal for generating the RG color difference is shown in FIG. 7A. In FIG. 7A, since P2_R is the signal of the expansion target, the signal level does not change before and after the first expansion processing, and is equal to the R signal P1_R after the white balance gain processing. Further, the signal level of the G signal is gradually expanded from the point where it exceeds the signal level TH_t1 at the start of expansion, asymptotically approaches the R signal so as not to exceed the R signal, and becomes equal to the R signal (P2_R) after TH_t3.

Returning to the description of FIG. 5, S712 to S718 show the flow of the G signal expansion processing for BG color difference generation. In this processing flow, the B signal is set as the target signal for expansion of the G signal, and the expansion processing is performed when the G signal is equal to or more than a predetermined threshold value and is equal to or less than the B signal.

In S712, the B and G signals after the white balance gain processing are referred to. In S713, it is determined whether or not the signal value of the G signal is equal to or greater than the threshold value for starting the expansion process. If it is equal to or more than the threshold value, the process proceeds to S714, and if it is less than the threshold value, the process proceeds to S717. In S717, the G signal before the expansion process is selected as the G signal for generating the BG color difference.

In S714, it is determined whether or not the signal value of the G signal is equal to or less than the B signal, and if it is equal to or less than the B signal, the process proceeds to S715. If the signal value of the G signal is larger than the B signal, the process proceeds to S717, and the G signal before the expansion process is selected as the G signal for generating the BG color difference.

In S715, the process shown in the equation (6) or the equation (7) is performed with reference to the B signal to expand the G signal level for generating the BG color difference.

P2_GB = (P1_B-P1_G) * f ((P1_G-TH_t1) / (TH_t2-TH_t1)) + P1_G ... (6)
P2_GB = P1_B ... (7)
Equation (6) is applied when the signal level of the G signal to be extended is lower than that of the B signal and the saturation level of the G signal has not been reached. In equation (6), TH_t1 indicates the signal level at which the expansion process of the G signal is started. Further, TH_t2 indicates the saturation level of the G signal before expansion. That is, the signal level of the G signal after white balance processing is P1_G by the weight determined by f ((P1_G-TH_t1) / (TH_t2-TH_t1)) until the signal level of the G signal changes from TH_t1 to TH_t2. Is expanded so as to approach the signal level P1_B of the expansion target, and is output as P2_GB. The weight f (x) of the expansion process at this time has the characteristics shown in FIG. 6, for example, assuming that the input is x and the output is y.

Here, when the signal level of P2_GB exceeds the signal level of the expansion target by the processing of the equation (6), the equation (7) is applied and the B signal is output. Further, even when the signal level of the G signal is lower than that of the B signal but the saturation level of the G signal is reached, the equation (7) is applied and the B signal is output as the G signal for generating the BG color difference. Will be done.

In S716, P2_GB is selected as the G signal for generating the BG color difference. In S718, either the G signal output from S717 or S716 is selected as the G2 signal, which is the G signal for generating the BG color difference, and the process proceeds to S719.

Here, an example of the B signal and the G signal for generating the BG color difference is shown in FIG. 7 (b). In FIG. 7B, since P2_B is the signal of the expansion target, the signal level does not change before and after the first expansion processing, and is equal to the B signal P1_B after the white balance gain processing. Further, since the B signal of the expansion target exceeds the saturation level of the G signal of the expansion target, the G signal of the expansion target is replaced with the B signal.

In S719, the average value of the G signal G1 for generating the RG color difference and the G signal G2 for generating the BG color difference is calculated and output as the final G signal of the first expansion processing unit 203.

FIG. 7C shows an example of the R, G, and B signals after the first expansion process. In FIG. 7C, P2_G is expanded with a characteristic that is an average value of the expansion result of the G signal referring to R and the expansion result of the G signal referring to B. Therefore, the color ratio of RGB to the brightness does not change significantly as compared with the R, G, and B signals before the first expansion process shown in FIG. 3, and the dynamic range of the signal is expanded from TH_0 to TH_3. Is done.

Next, the processing contents of the color space conversion processing unit 204 will be described. The color space conversion processing unit 204 performs the calculations of equations (8) to (10) using the RGB signals P2_R, P2_G, and P2_B after the first expansion processing to generate the luminance / color difference signal YUV.

Y = a * P2_R ^γ + b * P2_G ^γ + c * P2_B ^γ ... (8)
V = p * (P2_R ^γ −P2_RG ^γ ) + q * (P2_B ^γ −P2_BG ^γ )… (9)
U = r * (P2_B ^γ- P2_BG ^γ ) + s * (P2_R ^γ- P2_RG ^γ ) ... (10)
In equations (8) to (10), γ is a coefficient indicating the characteristics of the gamma conversion process applied to the input expanded signal, and for example, γ = 0.45. a, b, and c are matrix coefficients for generating a luminance signal from the RGB signal after gamma conversion, and a coefficient such that a + b + c = 1 is applied. p, q, r, and s are matrix coefficients for generating a color difference signal from the RGB signal after γ conversion. The matrix coefficients p, q, r, and s shall be appropriately adjusted in consideration of the output video signal standard such as BT.601 and BT.709 and the elements of image quality adjustment.

Next, the processing content of the second expansion processing unit 205 will be described. The second expansion processing unit 205 performs the operations shown in equations (11) to (13) with respect to the YUV signal output from the color space conversion processing unit 204. Then, the YUV signals, Y_out, U_out, and V_out corresponding to the extended upper limit value set by the system control unit 114 are generated.

Y_out = Y * g (Y) ... (11)
U_out = U * g (Y) * k ... (12)
V_out = V * g (Y) * k ... (13)
In equations (11) to (13), g (Y) is an amplitude modulation characteristic of the YUV signal, which is output from the color space conversion processing unit 204 and is determined according to the luminance value Y after the first expansion processing. is there. The characteristics of the gain with respect to the luminance value Y are shown in FIG.

Here, among RGB, the G signal has the largest composition ratio with respect to the luminance signal, so the characteristics of FIG. 8 are determined based on the G signal. In FIG. 8, the start luminance level x1 and the end luminance level x2 of the amplitude modulation set the lower limit expansion level TH_t1 of the first expansion process for the G signal and the upper limit extension level TH_t3 of the first expansion process for the G signal.

Further, in FIG. 8, the lower limit of the amplitude modulation gain is 1.0, and the upper limit of the amplitude modulation gain is determined by the ratio of the parameters y1 and y2. The upper limit expansion level TH_t3 of the first extension for the G signal is set in y1, and the extension upper limit value TH_max sent from the system control unit 114 is set in y2. That is, the amplitude modulation gain g (Y) gradually increases from 1.0 in the range of the luminance Y from x1 to x2, and finally becomes the upper limit value y2 / y1 = TH_max / TH_t3.

The amplitude of the YUV signal output from the color space conversion processing unit 204 is modulated by using the gain g (Y), which is an expansion amount calculated according to the difference from the target value in this way. Further, for the U and V signals, an arbitrary coefficient k is set in the equations (12) and (13) so that the saturation can be finely adjusted independently of the luminance signal Y.

By the above processing, in the present embodiment, the signal level is expanded by dividing it into two stages of RGB color space and YUV color space, so that the input image can be obtained while suppressing the fluctuation of the color ratio in the high luminance range. On the other hand, it is possible to realize a preset upper limit brightness value.

(Other embodiments)
The present invention also supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads the program. It can also be realized by the processing to be executed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The invention is not limited to the above embodiments, and various modifications and modifications can be made without departing from the spirit and scope of the invention. Therefore, a claim is attached to make the scope of the invention public.

100: Imaging device, 114: System control unit, 115: Image processing unit, 202: White balance gain processing unit, 203: First expansion processing unit, 204: Color space conversion processing unit, 205: Second expansion processing unit

Claims (12)

An input means for inputting an input image, which is an image signal having a plurality of color signals,
A white balance processing means for adjusting the white balance of the input image,
A first expansion processing means for performing a first expansion processing for expanding the dynamic range in the first color space for the signal after the white balance processing, and
A conversion means for converting the signal after the first expansion process into a signal in the second color space, and
A second expansion processing means that performs a second expansion processing for further expanding the dynamic range in the second color space with respect to the signal converted into the second color space by the conversion means.
An image processing apparatus comprising. The image processing apparatus according to claim 1, further comprising a setting means for setting an upper limit value for expanding the dynamic range with respect to the input image. In the white balance processing means, the first extended processing means uses a color to which a gain of the same magnification is applied or a color to which a minimum gain is applied as a reference color, and other colors as non-reference colors. 2. The second aspect of the present invention is to extend the dynamic range of the reference color so that the difference between the gradation characteristic of the color signal of the reference color and the gradation characteristic of the color signal of the non-reference color becomes small. The image processing apparatus according to. The second expansion processing means is characterized in that the dynamic range of the signal converted into the second color space is expanded so that the brightness of the second color space reaches the upper limit value. Item 3. The image processing apparatus according to item 3. The second expansion processing means is characterized in that the expansion amount of the dynamic range is determined based on the difference between the signal value after the expansion processing is performed by the first expansion processing means and the upper limit value. The image processing apparatus according to claim 4. The image processing apparatus according to any one of claims 2 to 5, wherein the upper limit value is a brightness level higher than the saturation level of the image signal of the input image. The first color space is an RGB color space, and the second color space is a color space composed of a brightness signal, a first color difference signal, and a second color difference signal. The image processing apparatus according to any one of claims 1 to 6. The image processing apparatus according to claim 7, wherein the second color space is a YUV color space. An imaging means for capturing a subject image and
The image processing apparatus according to any one of claims 1 to 8.
An imaging device characterized by comprising. An input process for inputting an input image, which is an image signal having a plurality of color signals,
A white balance processing step for adjusting the white balance of the input image, and
A first expansion processing step of performing a first expansion processing for expanding the dynamic range in the first color space for the signal after the white balance processing, and
A conversion step of converting the signal after the first expansion process into a signal in the second color space, and
A second expansion processing step of performing a second expansion processing for further expanding the dynamic range in the second color space with respect to the signal converted into the second color space in the conversion step.
An image processing method characterized by having. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8. A computer-readable storage medium that stores a program for causing the computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.

Images