JP6742892B2 - Image processing apparatus and image processing method - Google Patents

Description

The present invention relates to a technique for performing gradation conversion in consideration of saturation of a signal value of an image signal.

Patent Document 1 discloses that an image signal is divided into a Bayer signal for generating a luminance signal and a color difference signal generated from a difference between the Bayer signals, and gradation processing is performed on each signal.

Japanese Patent Laid-Open No. 2015-136014

However, when the same gain value is applied to the Bayer signal and the color difference signal to perform the gradation conversion, as shown in FIG. 1, in the Bayer signal, the R signal is the number of output bits in the output signal value after the gain is applied. The signal value specified in may be exceeded. In this case, the signal value of the R signal is clipped.

On the other hand, although the color difference signal is subjected to the clipping process determination after the gain is applied, since the signal values of both CbCr signals are within the limits of the output signal value, the value is output as it is. Therefore, the value of the Cr signal after the gain application generated from the color difference signal becomes larger than the value of the Cr signal after the gain application generated from the Bayer signal, which causes a problem that coloring occurs in the image after the gain application. is there.

One embodiment of the present invention is an image processing apparatus that performs gradation conversion processing, the first image signal having a first format, which is a format composed of a plurality of color signals, generated based on the image signal. And a second image signal having a second format, which is a format composed of color difference signals, and an input unit for inputting the first image signal input by the input unit, and a gain value is calculated. And a processing unit that uses the second image signal to apply the gain value calculated by the calculating unit to perform a gradation conversion process, and the processing unit includes the first unit. The second image signal is converted into an image signal having the first format by using the color signal of the image signal, and the gain value is applied to the converted image signal, and the gain value is applied. The converted image signal is converted into an image signal having the second format .

When the gradation conversion is performed by applying the gain to the color difference signal, the region clipped in the output signal value is taken into consideration, so that the occurrence of coloring can be suppressed.

It is a figure explaining the subject of this invention. It is a system diagram of the 1st Embodiment of this invention. It is a flow chart of a 1st embodiment. It is a figure explaining the gradation process of 1st Embodiment. It is a flow chart explaining gain application processing of a 1st embodiment. It is a figure which shows conversion of the image format of 1st Embodiment. It is a figure which shows the position of the pixel signal of 1st Embodiment. It is a system diagram of the 2nd Embodiment of this invention. It is a flow chart of a 2nd embodiment. 9 is a graph illustrating a clip process of the second embodiment. It is a flow chart explaining gain application processing of a 2nd embodiment. It is a figure which shows the position of the pixel signal of 2nd Embodiment.

An embodiment of the present invention will be described in detail with reference to the drawings by taking an image processing device used for an image pickup device as an example. The present invention is applicable not only to the image processing apparatus used for the image pickup apparatus but also to other image processing apparatuses.

(First embodiment)
In the first embodiment, a Bayer signal for generating a luminance signal (a first format image signal composed of a plurality of color signals) and a color difference signal (a plurality of color signals) are generated from an image signal input to an image processing apparatus. Image signals of a second format composed of color difference signals calculated by using the color difference signals), and gradation processing is performed on each signal. In this gradation processing, the color difference signal is converted into the Bayer signal format and then the gain processing is performed.

The first embodiment will be described with reference to FIGS. 2 to 7. In this embodiment, the Bayer signal is defined as a first format image signal and the color difference signal is defined as a second format image signal.

2 is a system diagram of the image processing apparatus, FIG. 3 is a flowchart showing processing in the image processing apparatus, FIG. 4 is a diagram illustrating gradation processing, FIG. 5 is a flowchart illustrating gradation processing for color difference signals, and FIG. The figure which showed the 1st image format used when transforming a 2nd image format, and FIG. 7 is a figure explaining the position of the pixel signal used for calculation.

First, the system configuration of the image processing apparatus will be described with reference to FIG.

A captured image acquisition unit 201 acquires a captured image used for image processing.

Reference numeral 202 denotes a gain calculation unit, which calculates a gain to be used for gradation correction processing of a captured image based on image information of an image signal in the first image format with the captured image acquired by the captured image acquisition unit 201 as an input. I do.

Reference numeral 203 denotes a first gain processing unit, which uses the gain input from the gain calculation unit 202 to perform gain processing on the image signal of the first image format.

A second gain processing unit 204 uses the gain input from the gain calculation unit 202 and the image signal in the first image format acquired from the captured image acquisition unit 201 to generate an image signal in the second image format. The gain process is performed on.

Reference numeral 205 denotes a signal processing unit, which performs a signal processing unit such as a developing process on the basis of the signal to which the first gain processing unit 203 and the second gain processing unit 204 have applied the gain to generate an output image.

FIG. 3 is a flowchart showing a process performed by each component of the image processing apparatus described above, and details of each process will be described with reference to FIGS. 4 to 7.

In step S301, the captured image acquisition unit 201 acquires an image that is the target of the gradation correction processing.

In S302, the captured image acquisition unit 201 performs a process of dividing the acquired image signal into a Bayer signal and a color difference signal. Both signals separated from the image signal of the captured image are subjected to signal processing according to the respective signals, and the processed Bayer signal is input to the gain calculation unit 202.

In S303, the gain calculation unit 202 calculates the gain used for gradation processing from the input Bayer signal.

The gain calculation process will be described with reference to FIG. FIG. 4 shows a captured image 401, a graph 402 showing gradation characteristics used for gradation conversion processing, and a calculated gain 403.

First, based on the Bayer signal of the captured image of the input image 401, first, the input luminance Y is calculated using the following formula (1).
Y=0.299×R+0.587×G+0.114×B (1)

Next, a gain is generated using the calculated input luminance Y and the gradation characteristic. The gradation characteristic used in the gain calculation process indicates a gain table in which the horizontal axis represents the input luminance signal and the vertical axis represents the gain signal, as shown in the graph 402. In the gain generation process, a gain value corresponding to the input luminance Y is calculated using the graph 402, and a gain signal corresponding to each pixel position of the image such as 403 is generated. The gain calculation unit 202 outputs the calculation result to the first gain processing unit 203 and the second gain processing unit 204. In this embodiment, the formula (1) is used when calculating the input luminance Y, but the present invention is not limited to this, and another system may be used.

In step S304, the first gain processing unit 203 uses the input Bayer signal and the calculated gain to multiply the gain as gradation processing of the first image format. The gain process is calculated using the following equation (2).
out(x,y)=Gain(x,y)×in(x,y) (2)
out(x, y): output RGB signal value after gain of coordinates (x, y) in(x, y): input RGB signal value of coordinates (x, y) Gain(x, y): gain value

Then, the first gain processing unit 203 outputs the Bayer signal after applying the gain to the signal processing unit 205.

In step S305, the second gain processing unit 204 performs gradation processing on the image signal of the second image format using the color difference signal and the calculated gain.

The gradation process performed on the image signal of the second image format will be described in detail with reference to FIGS. FIG. 5 is a flowchart for explaining gradation processing for color difference signals, FIG. 6 is a diagram showing a first image format used when converting a second image format, and FIG. 7 is a position of a pixel signal used in calculation. It is the figure which showed. Details of the process will be described with reference to a flowchart.

In step S501, the input Bayer signal is used to convert the color difference signal that is the second image format into the RGB signal values that form the first image format. The signal used in this conversion process is the G signal of the Bayer signals of the first image format. First, as shown in FIG. 6, an interpolation process is performed on the G signal in order to obtain a single-plate image having only the G signal. As the interpolation processing used here, a known technique such as linear interpolation processing can be used.

Next, after performing interpolation processing on the image signal of the first image format, the following signals (3) and (4) are calculated for the image signal of the second image format using the G signal, A B signal and an R signal that are in the first image format are generated.
B=Cb+G (3)
R=Cr+G (4)
G: G signal after interpolation processing in the first image format Cb: Cb signal in the second image format Cr: Cr signal in the second image format B: B signal R generated by converting the second image format : R signal generated by converting the second image format

The G signal used in this calculation is a signal at the same position as the pixel position of the Cb signal, and the Cr signal is also calculated using the same G signal used in the calculation of the Cb signal. That is, using the signal values of FIG. 7A, the following expressions (5) and (6) are obtained.
B(x,y)=Cb(x,y)+G(x,y) (5)
R(x+1,y)=Cr(x+1,y)+G(x,y) (6)

In step S502, the B signal and R signal converted into the first image format and the G signal used for conversion in the first image format are subjected to gain processing using the gain input from the gain calculation unit 202. For this gain processing, the calculation is performed using the equation (2) used when the gain is applied to the first image format in S304. The gain value used at that time is the gain value at the same position as the pixel position of the Cb signal. That is, using the signal values of FIG. 7B, the following expressions (7), (8), and (9) are obtained.
outB(x,y)=Gain(x,y)×B(x,y) (7)
outR(x+1,y)=Gain(x,y)×R(x+1,y) (8)
outG(x,y)=Gain(x,y)×G(x,y) (9)
outB(x,y): Output B signal value after gain at coordinates (x, y) outR(x+1, y): Output R signal value after gain at coordinates (x+1, y) outG(x, y): Output G signal value Gain(x, y) after gain of coordinates (x, y) in one image format: gain value

In S503, clipping processing is performed on the signal after the gain processing by using an arbitrary threshold value. The arbitrary threshold value here is set in consideration of the number of bits of the signal value of the output image, for example. The clip processing is, more specifically, a processing of replacing a signal value exceeding a threshold value with a threshold value. Further, the signals resulting from the replacement are defined as a signal B', a signal R', and a signal G', respectively, and the following processing is performed.

In step S<b>504, the clipped B′ signal and R′ signal are converted into the image signal of the second image format using the clipped G′ signal. The conversion of this signal is performed using the following equations (10) and (11).
Cb'=B'-G'... (10)
Cr'=R'-G'... (11)
G′: G signal in the first image format after clipping B′: B signal after clipping R′: R signal after clipping Cb′: Cb signal in the second image format after transformation Cr′: After conversion Cr signal of the second image format

That is, using the signal values of FIG. 7C, the following expressions (12) and (13) are obtained.
Cb'(x,y)=B'(x,y)-G'(x,y)... (12)
Cr'(x+1,y)=R'(x+1,y)-G'(x,y)... (13)

By performing these processes, the Bayer signal and the color difference signal are subjected to the gain process in consideration of the region where the signal is clipped, so that the adverse effect such as coloring of the output image does not occur.

In step S306, the Bayer signal and the color difference signal after the gain processing are input to the signal processing unit 205 that performs signal processing such as development processing, and the processed development result is output.

(Second embodiment)
In the second embodiment, an image signal input to the image processing device is divided into a luminance signal and an edge signal, and gradation processing is performed on each signal. At this time, when the edge enhancement processing is performed, the edge signal is converted using the luminance signal and then the gain processing is performed.

The second embodiment will be described with reference to FIGS. 8 to 12. In this embodiment, the luminance signal is defined as a first format image signal and the edge signal is defined as a second format image signal.

FIG. 8 is a system diagram of the image processing apparatus, FIG. 9 is a flowchart showing processing in the image processing apparatus, FIG. 10 is a graph explaining a clip value corresponding to a luminance value for removing noise, and FIG. 11 is a floor for an edge signal. FIG. 12 is a flowchart illustrating the adjustment process, and FIG. 12 is a diagram showing the positions of pixel signals used in the calculation.

When performing edge enhancement processing, edge extraction processing is performed from the luminance signal of the captured image, and gain processing is performed on the edge signal and the luminance signal. In this gain processing, when the gain is applied as it is without performing the processing of the present embodiment, the signal value is clipped in consideration of the saturation region after the gain is applied in the luminance signal, while the clipping is considered in the edge signal. Gain is applied without it. Therefore, there is a problem that when the luminance signal after gain and the edge signal are combined, the edge signal emphasized in the saturated region is added. In order to solve this problem, in the present embodiment, the edge signal is converted using the luminance signal and then the gain process is performed.

The system configuration of the image processing apparatus of this embodiment will be described with reference to FIG. The difference from the system configuration shown in FIG. 2 is the edge extraction unit 803. Therefore, the description of the common components will be omitted, and the edge extraction unit 803 will be described.

In FIG. 8, an edge extraction unit 803 extracts the edge component in the image by using the luminance signal of the captured image and inputs it to the second gain processing unit 805.

The flowchart of FIG. 9 shows the operation process of each component, and the details of each process will be described with reference to FIGS. 10 to 12. The processing of the present embodiment will be described with reference to FIG. 9. The difference from FIG. 3 is that S902 that performs edge extraction on the image signal and S905 that performs noise component removal processing on the edge signal are calculated. In step S906, the gain process is performed on the edge signal using the gain. Therefore, description of the other processes will be omitted, and S902, S905, and S906 will be described.

In step S902, the edge extraction unit 803 performs edge extraction using the captured image signal input from the captured image acquisition unit 801. Here, the input signal is a luminance signal, and the edge signal is extracted by performing band-pass filter processing on this signal.

In step S905, coring processing that is processing for removing noise components included in the extracted edge signal is performed. The graph of FIG. 10 is used for this processing. In the graph 1001 of FIG. 10, the horizontal axis represents the input luminance signal, and the vertical axis represents the value used in the coring process as the clip value. In order to consider the noise amount according to the brightness value, the edge value considering noise can be obtained by subtracting the clip value corresponding to each brightness from the edge signal. Therefore, the clip value at each pixel position is calculated using the luminance signal of the first image format at the same pixel position used when the edge was extracted. Then, the clip value is subtracted from the edge signal to obtain the noise-free edge signal, and the obtained signal is output to the second gain processing unit 805.

In step S906, the second gain processing unit 204 performs gain processing using the edge signal input from the edge extraction unit 803 to the second gain processing unit 805 and the calculated gain. The gain processing of the second image format will be described with reference to FIGS. 11 and 12. FIG. 11 is a flowchart for explaining gradation processing for edge signals, and FIG. 12 is a diagram showing the positions of pixel signals used in calculation. Details of the process will be described with reference to a flowchart.

In step S1101, the input luminance signal is used to convert an edge signal in the second image format into a luminance signal in the first image format. In this process, the following equation (14) is calculated for the second image format, and using the signal values in FIG. 12A, a signal EY having the same image format as the first image format is generated. ..
EY(x,y)=E(x,y)+Y(x,y) (14)
Y(x,y): luminance signal in the first image format E(x,y): edge signal after noise removal in the second image format

The luminance signal Y used in this calculation is a signal at the same position as the pixel position of the edge signal E. By adding the luminance signal and the edge signal after noise removal using Expression (14), it is possible to perform gain processing with the signal value after edge enhancement, and at that time, obtain a value that considers the saturation region. Then, even when the edge signal which is the second image format is restored, the edge signal value in consideration of the saturated region can be obtained.

In step S1102, the edge signal converted into the first image format and the luminance signal used in the conversion in the first image format are subjected to gain processing using the calculated gain. In this gain processing, the calculation is performed using the equation (2) used when the gain is applied to the first image format in S304 of the first embodiment. The gain value used at that time is expressed by the following equations (15) and (16) by using the gain value at the same position as the pixel position of the edge signal and the luminance signal, which are the signal values of FIG. Become.
outEY(x,y)=Gain(x,y)×EY(x,y) (15)
outY(x,y)=Gain(x,y)×Y(x,y) (16)
outEY(x, y): Output edge signal value after gain of coordinates (x, y) outY(x, y): Output luminance signal value after gain of coordinates (x, y) Gain(x, y): Gain value

In step S1103, clipping processing is performed on the signal after gain processing using an arbitrary threshold value. The arbitrary threshold value here is set in consideration of the saturation value of the output image. A signal that exceeds a threshold that is a set saturation value is replaced with the threshold value. The signals resulting from the replacement are defined as a luminance signal Y'and an edge signal EY', respectively.

In step S1104, the clipped edge signal EY' is converted into the second image format using the clipped luminance signal Y'. This signal conversion is calculated by the following equation (17) using the signal value of FIG. 12(c).
E′(x,y)=EY′(x,y)−Y′(x,y) (17)
EY': edge signal after clipping Y': luminance signal after clipping E': edge signal of second image format after conversion

By performing these processes, the gain processing of the edge signal can be performed in consideration of the saturation area, and when the luminance signal and the edge signal are finally combined, the edge signal emphasized in the saturation area is added. It is possible to prevent such a harmful effect.

201 captured image acquisition unit 202 gain calculation unit 203 first gain processing unit 204 second gain processing unit 205 signal processing unit

Claims (8)

An image processing device that performs gradation conversion processing ,
A first image signal having a first format which is a format composed of a plurality of color signals and a second image signal having a second format which is a format composed of color difference signals, which are generated based on the image signal Input means for inputting and
Calculation means for calculating a gain value using the first image signal input by the input means ;
Processing means for applying the gain value calculated by the calculating means using the second image signal to perform gradation conversion processing,
The processing means, applying the gain value with respect to the first by using a color signal of the image signal into said second image signal to an image signal having the first format, the converted image signal and an image processing apparatus and converting the image signal in which the gain value is applied to the image signals having the second format. The image processing apparatus according to claim 1, wherein the processing unit further applies the gain value to the first image signal to perform gradation conversion processing. The image processing apparatus according to claim 1 or 2, characterized in that it comprises a developing means for developing processing using the first image signal the is gradation conversion processing and the second image signal. Said processing means further claim 1, characterized in that clipping against being converted into an image signal a second image signal which the gain value is applied, the signal value on the basis of the threshold having the first format The image processing apparatus according to any one of items 1 to 3 . The image processing apparatus according to any one of claims 1 to 4, wherein the image signal having the first format is a Bayer signal. The image processing apparatus according to claim 1, wherein the plurality of color signals are R, G, and B signals . The processing means, characterized in that the conversion using the G signal of said first image signal, the second image signal, the image signal having a first format consisting of R and B signals The image processing apparatus according to claim 6 . A method of controlling an image processing device, comprising:
A first image signal having a first format which is a format composed of a plurality of color signals and a second image signal having a second format which is a format composed of color difference signals, which are generated based on the image signal An input step of inputting and
A calculation step of calculating a gain value using the first image signal input in the input step;
A processing step of applying the gain value calculated in the calculation step to perform gradation conversion processing using the second image signal;
In the processing step, the color signal of the first image signal is used to convert the second image signal into an image signal having the first format, and the gain value is applied to the converted image signal. Then, the image signal to which the gain value is applied is converted into an image signal having the second format .

Images