JP7282536B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, a program, and a recording medium.

In the HDR video system proposed in recent years, the gradation compression disclosed in Non-Patent Document 1 is performed in the camera and transmitted to the monitor, so that the gradation with a reflectance of 100% or more, which was compressed in SDR, can be reduced. It is possible to reproduce the contrast close to the appearance of the subject. Here, the gradation compression characteristic (Inverse-EOTF Nonlinear Encoding Equation) of Non-Patent Document 1, which is adopted in the HDR video signal standard, is a dark area conversion characteristic compared to SDR and the HLG method compatible with SDR. is highly nonlinear. Therefore, approximation with a low-order function suitable for hardware implementation is difficult.

In Patent Document 1, by interpolating a plurality of pre-stored basic gamma characteristics according to the brightness difference of the subject of the image, a gamma characteristic corresponding to the brightness difference is generated, and the generated gamma characteristic is used as the luminance of the image. An image processing apparatus for applying levels is disclosed.

JP 2014-121019 A

ST 2084:2014 - SMPTE Standard - High Dynamic Range Electro-Optical Transfer Function of Mastering Reference Displays (p.7 Eauation5.2)

However, in the image processing apparatus disclosed in Patent Document 1, in order to generate a target gamma characteristic by linear interpolation of a plurality of gamma characteristics, if a high-order approximation is required for tone conversion to be realized, There is a problem that it is difficult to reduce the conversion error.
SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems, and it is an object of the present invention to reduce conversion errors when performing gradation conversion on the signal level of an input image.

An image processing apparatus according to the present invention comprises dividing means for dividing a dynamic range of an input image into a plurality of signal ranges, and gradation conversion of the signal level of the input image for each signal range divided by the dividing means. determining means for determining tone conversion characteristics; adjusting means for adjusting the signal level of the input image for each signal range; and determining means for adjusting the signal level of the input image adjusted by the adjusting means for each signal range. and converting means for performing gradation conversion using the gradation conversion characteristics determined by the determination means, wherein the determination means determines that the gradation conversion of signal levels included in a predetermined signal range is applied to the input image. The gradation conversion characteristic is determined for each signal range so as to be constant regardless of the dynamic range of the signal.

According to the present invention, it is possible to reduce conversion errors when converting the signal level of an input image into gradation.

It is a figure which shows an example of a structure of an image processing apparatus. It is a figure which shows an example of a structure of a gradation conversion process part. 6 is a flow chart showing an example of an operation by a gradation conversion processing unit; 7 is a graph showing an example of gamma settings applied to the gradation conversion processing section;

Preferred embodiments of the present invention will be described below with reference to the drawings.
In this embodiment, a case where a camera is applied as an example of an image processing device will be described.
FIG. 1 is a diagram showing an example of the configuration of an image processing apparatus 100. As shown in FIG.
The image processing apparatus 100 includes an optical system 101, an image sensor 102, an A/D conversion unit 103, a color adjustment processing unit 104, a gradation conversion processing unit 105, a luminance/color difference encoding processing unit 106, a recording processing unit 107, a display It has a unit 108 , a user instruction unit 109 and a system control unit 110 .

The optical system 101 includes a focus lens, diaphragm, and shutter.
The imaging device 102 is a device such as a CMOS that converts the amount of light of an object imaged by the optical system 101 into an electrical signal by photoelectric conversion. The focus, aperture, and shutter of the optical system 101, and the imaging sensitivity of the imaging device 102 are controlled by the system control unit 110 based on the dynamic range during imaging.
The A/D conversion unit 103 converts the electrical signal from the image sensor 102 into a digital signal to generate a RAW image.

The color adjustment processing unit 104 performs gain processing for white balance adjustment and matrix processing for adjusting color reproduction characteristics on the RGB values of the generated RAW image.
The gradation conversion processing unit 105 performs gradation conversion on the RGB signals that have undergone color adjustment processing using characteristics in which the absolute luminance of the monitor display is associated with the signal level after gradation conversion.
A luminance/color-difference encoding processing unit 106 performs matrix processing for converting the tone-converted RGB signals into luminance/color-difference signals.
A recording processing unit 107 performs encoding/compression processing for recording the converted luminance/color difference signals in a predetermined file format.

The display unit 108 is, for example, a liquid crystal monitor or the like installed on the back of the camera. A display unit 108 displays the luminance/color difference signals converted by the luminance/color difference encoding processing unit 106 .
A user instruction unit 109 is an operation dial arranged on the camera body, a touch panel type liquid crystal monitor arranged on the rear surface of the camera, or the like. The user inputs a setting value via the user instruction unit 109 when setting the exposure for shooting according to the shooting scene.
A system control unit 110 controls the operation of the entire image processing apparatus 100 .

Next, the gradation conversion processing unit 105 will be described in detail.
An input image with an arbitrary dynamic range is input to the gradation conversion processing unit 105 . In the present embodiment, the dynamic range of the input image is represented by the reflectance of the object saturated when the image is captured. For example, a dynamic range of 400% indicates that exposure control is performed so that the output of the image sensor 102 is saturated when an object with a reflectance of 400% is photographed.

The dynamic range of the input image is determined by the system control unit 110 based on the exposure setting input from the user instruction unit 109 at the time of shooting, or by analyzing the luminance distribution of the image captured before shutter release. .
The gradation conversion processing unit 105 of the present embodiment divides the signal range based on a threshold value TH, which will be described later, regardless of the dynamic range of the input image, and obtains the highest approximation accuracy in a predetermined signal range corresponding to the dark side. To control gradation conversion processing so that gradation conversion with a high value is performed.

FIG. 2 is a diagram showing an example of the configuration of the gradation conversion processing unit 105. As shown in FIG.
The gradation conversion processing section 105 has an image input terminal 200 , a signal range division section 201 , a signal level adjustment section 202 , a gamma conversion section 203 , a gamma setting DB 204 and an image output terminal 205 .

An image of an arbitrary dynamic range is input from the color adjustment processing unit 104 to the image input terminal 200 .
The signal range division unit 201 determines whether the signal level of the input image is equal to or less than the threshold for each pixel, and outputs the determination result to the signal level adjustment unit 202 and the gamma conversion unit 203 . The signal range dividing unit 201 acquires a threshold that serves as a criterion based on the dynamic range of the input image and the gamma setting held in the gamma setting DB 204 .

The signal level adjustment unit 202 adjusts the signal level amplitude for each pixel of the input image based on the determination result of the signal range division unit 201 .
The gamma conversion unit 203 determines the gamma setting corresponding to the signal range of the pixel of interest from the gamma setting DB 204 based on the determination result of the signal range division unit 201, and converts the signal level to the signal level adjusted by the signal level adjustment unit 202. Tone conversion, that is, gamma conversion is performed.

The gamma setting DB 204 holds multiple gamma settings. Each gamma setting is of a different type and corresponds to a different dynamic range. A gamma setting is an example of a gradation conversion characteristic.
The image output terminal 205 receives the signal level that has undergone gradation conversion by the gamma conversion section 203 . The image output terminal 205 outputs the tone-converted signal level to the luminance/color difference encoding processing unit 106 .

Next, gradation conversion processing by the gradation conversion processing unit 105 will be described with reference to the flowchart of FIG. The flowchart in FIG. 3 is started when the image processing apparatus 100 is activated.
In S<b>300 , the signal range division unit 201 acquires the input image that is output from the color adjustment processing unit 104 and input to the image input terminal 200 .
In S<b>301 , the signal range division unit 201 acquires information on the dynamic range of the input image from the system control unit 110 .

In S302, the signal range dividing unit 201 calculates the first reference reflectance and the second reference reflectance for determining the signal level of the input image based on the acquired dynamic range and the gamma setting held in the gamma setting DB 204. Determine the reference reflectance of
Specifically, the signal range dividing unit 201 selects a gamma setting that saturates with a reflectance lower than the dynamic range of the input image from among the plurality of gamma settings held in the gamma setting DB 204, and selects the gamma setting. Let the saturation level be the first reference reflectance. Note that the gamma setting selected at this time is applied when performing gradation conversion on a signal level equal to or lower than the threshold TH in S305, which will be described later.
Further, the signal range dividing unit 201 selects a gamma setting that saturates with a reflectance higher than the dynamic range of the input image from among the plurality of gamma settings held in the gamma setting DB 204, and sets the saturation level of the gamma setting to A second reference reflectance is used. Note that the gamma setting selected at this time is applied when tone conversion is performed on a signal level greater than the threshold TH in S307, which will be described later.

For example, if the dynamic range of the input image is X% and the gamma setting DB 204 holds a gamma setting saturated with reflectance L% and reflectance H%, then the first reference reflectance is L and the first reference reflectance is L%. The reference reflectance of 2 becomes H. Here, the magnitude relationship among X, L, and H is L<X≦H. Note that the reference dynamic range for the gamma setting saturated at the reflectance L% is L%, and the reference dynamic range for the gamma setting saturated at the reflectance H% is H%.

In S303, the signal range dividing unit 201 divides the dynamic range of the input image into a plurality of signal ranges, and calculates a threshold value TH for determining which signal range each signal level of the input image belongs to by formula (1 ). This processing corresponds to an example of processing by the dividing means. The signal range dividing unit 201 determines whether the signal level is equal to or less than the threshold TH for each pixel of the input image.
TH=2 ⁿ ×(first reference reflectance)/(dynamic range of input image) (1)
Here, n is the bit precision of the input signal input to the gradation conversion processing unit 105 .

For example, assuming that the bit precision of the input signal is 14 bits, the dynamic range of the input image is X%, and the first reference reflectance is L%, the threshold TH is given by Equation (2).
TH=16383×L/X Expression (2)
If the signal level of the input image is equal to or less than the threshold TH, the process proceeds to S304, and if greater than the threshold TH, the process proceeds to S306. The signal range dividing unit 201 divides the input image into a signal range with a low signal level or a signal range with a high signal level by making a determination for each pixel of the input image.

In S304, the signal level adjustment unit 202 adjusts the signal level based on the ratio between the dynamic range of the input image and the first reference reflectance when the signal level is equal to or lower than the threshold TH. This process corresponds to an example of adjustment means.
Specifically, when the dynamic range of the input image is X% and the first reference reflectance is L%, the gain L_gain of equation (3) is used to adjust the signal level to be amplified.
L_gain=(dynamic range of input image)/(first reference reflectance)
=X/L Expression (3)
The signal level adjustment section 202 outputs the adjusted signal level to the gamma conversion section 203 .

In S305, the gamma conversion unit 203 tone-converts the first gamma setting corresponding to the first reference reflectance among the gamma settings held in the gamma setting DB 204 when the signal level is equal to or lower than the threshold TH. determine the gamma setting for This processing corresponds to an example of a determining means. Specifically, when the first reference reflectance is L %, the gamma setting for the dynamic range L % is determined as the gamma setting for tone conversion. Subsequently, the gamma conversion unit 203 applies the determined gamma setting to tone-convert the adjusted signal level. This process corresponds to an example of conversion means.

In S306, the signal level adjuster 202 adjusts the signal level based on the ratio between the dynamic range of the input image and the second reference reflectance when the signal level is greater than the threshold TH. This process corresponds to an example of adjustment means.
Specifically, when the dynamic range of the input image is X% and the second reference reflectance is H%, the signal level is adjusted to attenuate using the gain H_gain of equation (4).
H_gain=(dynamic range of input image)/(second reference reflectance)
=X/H Expression (4)
The signal level adjustment section 202 outputs the adjusted signal level to the gamma conversion section 203 .

In S307, if the signal level is greater than the threshold TH, the gamma conversion unit 203 tone-converts the second gamma setting corresponding to the second reference reflectance among the gamma settings held in the gamma setting DB 204. to determine the gamma setting for This processing corresponds to an example of a determining means. Specifically, when the second reference reflectance is H %, the gamma setting for dynamic range H % is determined as the gamma setting for tone conversion. Subsequently, the gamma conversion unit 203 applies the determined gamma setting to tone-convert the adjusted signal level. This processing corresponds to an example of conversion means.
The gamma conversion unit 203 performs the processing of S305 or S307 for all pixels, and outputs the signal level after gradation conversion to the image output terminal 205. FIG.
With the above processing, the operation of the gradation conversion processing unit 105 ends.

Next, an example of gamma setting applied in tone conversion processing will be described with reference to FIG.
In FIG. 4, when the dynamic range of the input image is X%, the gamma settings corresponding to the dynamic ranges L% and H% held in the gamma setting DB 204 are applied to obtain the input image with the dynamic range of X%. gradation conversion is performed. It should be noted that the magnitude relationship among X, L, and H is L<X≦H.

In FIG. 4, the horizontal axis indicates the signal level of the 14-bit quantized input image input to the gradation conversion processing unit 105, and the vertical axis indicates the 10-bit signal level after the gradation conversion. This 10-bit signal level is a value associated with the absolute luminance of the monitor display.
The input image is divided into two with the threshold TH as the boundary, and for the signal range (X%_range_0_L) from 0 to L%, the signal level of the input image is amplified by a gain of ×X/L, and the dynamic range L% is converted with the gamma setting (L%_range_0_L) corresponding to . In addition, for the signal range (X%_range_L_X) of L% to X% of the input image, the signal level of the input image is attenuated by the gain of ×X/H, and the gamma setting (H% _range_L_X).

Here, the gamma curve for the gamma setting corresponding to the dynamic range L% is the lowest signal range gamma curve among the gamma settings held in the gamma setting DB 204 . That is, since it is possible to finely perform gradation conversion with an accuracy of 14 bits for input and 10 bits for output for signals with brightness up to reflectance L%, it is possible to reduce the error from target gradation conversion. can.

Also, the dynamic range X% of the input image may fluctuate depending on the exposure setting at the time of shooting. However, as long as the relationship L<X≦H is satisfied, gradation conversion can be performed with high precision using the gamma curve of the gamma setting for the dynamic range L% for signal levels below the threshold TH in the input image. can. Therefore, in this embodiment, the gamma setting is determined for each signal range so that the gradation conversion of signal levels included in a predetermined signal range is constant regardless of the dynamic range of the input image.

Specifically, it is assumed that the image processing apparatus 100 can capture images with dynamic ranges of 300%, 600%, and 1000%, and acquires input images with dynamic ranges of 300%, 600%, and 1000%. The gamma conversion unit 203 sets a gamma setting based on a dynamic range of less than 300% (for example, 200%) when tone-converting any input image. gamma setting. The gamma conversion unit 203 performs gradation conversion on the signal level of the signal range on the dark side by applying the determined gamma setting. By performing gradation conversion in this manner, the brightness of the signal range on the dark side of any input image can be reproduced with the same brightness when displayed on a monitor, and gradation conversion with less approximation error can be performed. can be done.

Therefore, according to this embodiment, it is possible to reduce the conversion error when converting the gradation by associating the brightness of the shooting scene with the absolute luminance of the monitor display, and to record a good image. Further, since the tone conversion according to the present embodiment requires relatively little processing load, the tone conversion processing unit 105 can be implemented in the image processing apparatus 100 as hardware.

Further, in the present embodiment, the gamma conversion unit 203 sets the gamma setting corresponding to the dynamic range lower than the dynamic range of the input image to the low signal range among the plurality of signal ranges, and sets the gamma setting corresponding to the dynamic range lower than the dynamic range of the input image. Set the gamma setting. Therefore, since fine gradation conversion can be performed for signal levels in a low signal range, it is possible to reduce errors from target gradation conversion.

(Other embodiments)
The present invention is also realized by executing the following processing. That is, it is a process in which a program that implements the functions of the above-described embodiments is supplied to a system or device via a network or various recording media, and a computer (CPU, MPU, etc.) of the system or device reads and executes the program. In this case, the program and the recording medium recording the program constitute the present invention.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.
In the above-described embodiment, the case where the gamma conversion unit 203 determines the gamma setting for gradation conversion has been described. You may

101: Optical system 102: Imaging element 103: A/D conversion unit 104: Color adjustment processing unit 105: Gradation conversion processing unit 106: Luminance/color difference encoding processing unit 107: Recording processing unit 108: Display unit 109: User Instruction unit 110: System control unit 201: Signal range division unit 202: Signal level adjustment unit 203: Gamma conversion unit 204: Gamma setting DB

Claims (11)

a dividing means for dividing the dynamic range of the input image into a plurality of signal ranges;
determining means for determining gradation conversion characteristics when performing gradation conversion of the signal level of the input image for each signal range divided by the dividing means;
adjusting means for adjusting the signal level of the input image for each signal range;
converting means for tone-converting the signal level of the input image adjusted by the adjusting means for each signal range using the tone conversion characteristics determined by the determining means, the image processing apparatus comprising:
The determining means is
An image processing apparatus, wherein tone conversion characteristics are determined for each signal range so that tone conversion of signal levels included in a predetermined signal range is constant regardless of the dynamic range of the input image. The determining means is
In a low signal range among the plurality of signal ranges divided by the dividing means, a gradation conversion characteristic corresponding to a dynamic range lower than the dynamic range of the input image is used for gradation conversion when performing gradation conversion of the signal level. 2. The image processing apparatus according to claim 1, wherein the determination is based on characteristics. The determining means is
In the higher signal range among the plurality of signal ranges divided by the dividing means, the gradation conversion characteristic corresponding to the dynamic range equal to or greater than the dynamic range of the input image is used as the gradation conversion characteristic when performing gradation conversion of the signal level. 3. The image processing apparatus according to claim 1, wherein the image processing apparatus determines that The determining means is
wherein, from a plurality of tone conversion characteristics respectively corresponding to a plurality of dynamic ranges, a tone conversion characteristic for performing tone conversion of a signal level is determined for each signal range divided by the dividing means; 4. The image processing apparatus according to any one of items 1 to 3. The adjustment means is
3. The signal level of the input image is adjusted for each signal range based on the dynamic range of the input image and the dynamic range that is a reference for tone conversion characteristics determined by the determining means. 5. The image processing apparatus according to any one of 1 to 4. The adjustment means is
In the low signal range among the plurality of signal ranges divided by the dividing means, the dynamic range of the input image and the dynamic range serving as a reference of the gradation conversion characteristics corresponding to the low signal range determined by the determining means. adjusting to amplify the signal level of the input image using a ratio;
In the high signal range among the plurality of signal ranges divided by the dividing means, the dynamic range of the input image and the dynamic range serving as a reference of the gradation conversion characteristic corresponding to the high signal range determined by the determining means. 6. The image processing apparatus according to any one of claims 1 to 5, wherein a ratio is used to adjust the signal level of the input image so as to attenuate. The dividing means is
The dynamic range of the input image is divided into a plurality of signal ranges based on the dynamic range of the input image and the dynamic range as a reference of the gradation conversion characteristics determined by the determining means. Item 7. The image processing apparatus according to any one of Items 1 to 6. 8. The method according to any one of claims 1 to 7, wherein the dynamic range of the input image is determined based on at least one of an exposure setting when the input image was captured and a luminance distribution of the input image. The described image processing device. a splitting step of splitting the dynamic range of the input image into a plurality of signal ranges;
a determination step of determining a gradation conversion characteristic when performing gradation conversion of the signal level of the input image for each signal range divided by the division step;
an adjusting step of adjusting the signal level of the input image for each signal range;
a conversion step of tone-converting the signal level of the input image adjusted in the adjustment step for each signal range using the tone conversion characteristic determined in the determination step, wherein
In the determining step,
An image processing method, comprising: determining gradation conversion characteristics for each signal range so that gradation conversion of signal levels included in a predetermined signal range is constant regardless of the dynamic range of the input image. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8. A computer-readable recording medium recording a program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.

Images