JP7446847B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing technique that performs gradation processing or the like on a captured image.

High Dynamic Range (HDR) display can express images with a wider dynamic range than Standard Dynamic Range (SDR) display. In the case of HDR display, it is possible to express high brightness and colors with high brightness, which cannot be fully expressed with SDR display.

Furthermore, as a conventional technique, there is a technique for improving the contrast of an image by generating a low-frequency image from an input image and performing gradation processing on the input image based on the low-frequency image. Patent Document 1 discloses that an emphasis signal is generated based on a gain map generated from a reduced image of an input image, and the contrast is improved by adding the emphasis signal to the gain map of the gain characteristic that is originally desired to be applied to the input image. Techniques for generating highly effective gain maps are disclosed.

JP2019-53590A

By the way, when a RAW image photographed using HDR is subjected to HDR development, the image is developed with the desired contrast, so no reduction in contrast occurs. On the other hand, when a RAW image photographed using HDR is subjected to SDR development, gradation compression occurs due to SDR development, resulting in a decrease in the contrast of the image. Furthermore, the impression of the image will be different when a RAW image shot in HDR is developed using HDR and displayed on a display device capable of displaying HDR, and when developed using SDR and displayed on a display device capable of displaying only SDR.

Therefore, an object of the present invention is to make it possible to generate an image with a contrast similar to that obtained when HDR development is performed even when a RAW image photographed using HDR is subjected to SDR development.

The image processing apparatus of the present invention includes a generation means for generating a gain signal used for gain processing when developing an undeveloped image, and a gain signal generated by the generation means to generate a gain signal for the undeveloped image. processing means for performing the gain processing using the undeveloped image, and the undeveloped image is used to perform development processing corresponding to a second dynamic range that is a narrower dynamic range than the first dynamic range; The generating means is configured to generate a gamma characteristic corresponding to the first dynamic range that has a common characteristic in a region where there is no difference between the gamma characteristic corresponding to the first dynamic range and the gamma characteristic corresponding to the second dynamic range. In a region where there is a difference between the characteristics and the gamma characteristics corresponding to the second dynamic range, the gamma characteristics corresponding to the second dynamic range are changed to correspond to the first dynamic range, The method is characterized in that the gain signal is generated based on gain characteristics .

According to the present invention, even when a RAW image photographed using HDR is subjected to SDR development, it is possible to generate an image with a contrast similar to that obtained when HDR development is performed.

1 is a diagram showing a configuration example of an image processing apparatus according to an embodiment. It is a flowchart of image processing of an embodiment. FIG. 3 is an explanatory diagram of contrast reduction. FIG. 3 is an explanatory diagram of gain characteristics and a gain map. 5 is a flowchart of gain map emphasis signal generation processing. FIG. 3 is a diagram showing an example of gain characteristics. FIG. 3 is an explanatory diagram of contrast reduction due to global gain characteristics. FIG. 2 is an explanatory diagram of a method for generating gain characteristics in this embodiment. FIG. 3 is an explanatory diagram of contrast improvement effect. FIG. 3 is an explanatory diagram of an emphasis signal of a gain map.

Hereinafter, embodiments of the present invention will be described in detail based on the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. Identical configurations or processes will be described using the same reference numerals.
FIG. 1 is a functional block diagram showing a schematic configuration of an image processing apparatus according to this embodiment. Further, FIG. 2 is a flowchart showing the flow of image processing in the image processing apparatus of this embodiment.

Before describing the configuration of the image processing apparatus shown in FIG. 1 and the details of the processing in the flowchart shown in FIG. 2, an overview of the image processing in this embodiment will be described.
The image processing apparatus of this embodiment performs image processing such as performing gain processing on an input image and outputting the processed image. It is assumed that the input image to the image processing device is a RAW image obtained by imaging with an imaging device capable of high dynamic range (HDR) photography, that is, an undeveloped captured image. It is also assumed that the image input/output to the image processing device is a luminance image composed of luminance signals. The configuration and explanation thereof for various signal processing such as acquisition of a captured image, development processing for the captured image signal, and generation processing of a brightness image will be omitted.

Here, a change in image impression that occurs when a RAW image (undeveloped captured image) obtained by HDR photography is subjected to HDR development and SDR development will be described.
FIG. 3 is a diagram showing the difference that occurs in luminance signals when a RAW image obtained by HDR photography is subjected to SDR development and when HDR development is performed. The vertical axis in FIG. 3 indicates the luminance signal, and the horizontal axis indicates the x direction (x coordinate direction) of the image. A solid line portion 302 in FIG. 3 indicates an example of a luminance signal obtained by HDR development, and a dotted line portion 301 indicates an example of a luminance signal obtained by SDR development. Comparing the brightness signals of the solid line part 302 and the dotted line part 301, if the maximum brightness value of the signal by SDR development shown in the dotted line part 301 is Y1, the maximum brightness value of the signal by HDR development shown by the solid line part 302 The value is Y2. In the case of the example in FIG. 3, the difference in maximum luminance value between the signal shown by the dotted line part 301 and the signal shown by the solid line part 302 is Y2 - Y1, and the difference in the maximum luminance value between the signal shown by the dotted line part 301 and the signal shown by the solid line part 302 is Y2 - Y1, and the difference in the maximum luminance value between the signal shown by the dotted line part 301 and the signal shown by the solid line part 302 is Y2 - Y1. It can be seen that the contrast has decreased. The reason why there is a difference in signal values between HDR development and SDR development is that HDR can express a brighter range than SDR, and SDR developed signals are different from HDR. This is because the gradation is compressed compared to the developed signal. For example, when SDR development is performed, the signal value of the white portion of the image becomes lower than when HDR development is performed, resulting in a decrease in contrast. Therefore, when a RAW image obtained by HDR photography is subjected to HDR development and SDR development, the impression of the image differs in terms of contrast (that is, the contrast effect differs).

For this reason, when performing SDR development on a RAW image captured using HDR, the image processing apparatus of the present embodiment uses the gamma characteristics of both HDR development and SDR development for gain processing during development processing. Adjust gain characteristics.
The image processing apparatus of this embodiment has the same gamma characteristics (no difference in gamma characteristics) between HDR development and SDR development as in the case of SDR development of a RAW image shot with SDR (common). Allow characteristic gain processing to be performed. On the other hand, for areas where there is a difference in gamma characteristics between HDR development and SDR development, the image processing device uses a gain map based on a low-frequency image generated from the input image and a gamma map between HDR development and SDR development. A gain map emphasis signal is generated based on the characteristic difference. Then, the image processing device generates a gain map with a high contrast improvement effect by adding the emphasis signal of the gain map generated as described above to the gain map of the gain characteristic originally desired to be applied to the input image, and Apply gain processing to the input image using a map. This makes it possible to obtain an image with a contrast similar to that during HDR development even during SDR development. In the following description, the gamma characteristic during HDR development will be expressed as HDR gamma, and the gamma characteristic during SDR development will be expressed as SDR gamma.

Here, the gain characteristic is a characteristic expressed by a table in which the horizontal axis is the input luminance signal and the vertical axis is the gain signal, as shown in the graph of FIG. 4A, for example. Further, the gain map is a map in which gain values to be applied are determined corresponding to each pixel position of the image, as shown in FIG. 4(b). In the example of FIG. 4(b), the map is a two-dimensional array of gain values Gain(x, y) applied to the position of the coordinates (x, y) of each pixel. The x value of the coordinates (x, y) of each pixel is 0, 1, 2, ..., W-1, and the y value is 0, 1, 2, ..., H-1. It will be.

Hereinafter, the configuration and image processing of the image processing apparatus of this embodiment shown in FIG. 1, which can generate a gain map with a high contrast improvement effect as described above, will be described in detail.
The image processing apparatus of this embodiment is roughly divided into three components: a local gain map generation section 110, a global gain map generation section 111, and a gain processing section 112.

The local gain map generation unit 110 generates a local gain map (local contrast gain map) for locally adjusting contrast from the luminance image input to the image processing device. The local gain map is a map in which gain values are two-dimensionally arranged corresponding to each pixel position, as shown in FIG. 4(b) described above. The local gain map generation unit 110 generates a low frequency image from the input luminance image, and generates a local gain map by applying gain characteristics to the low frequency image. Details of the gain characteristics used in the local gain map generation section 110 will be described later.

The global gain map generation unit 111 generates a global gain map (global contrast gain map) for adjusting the brightness of the image according to the brightness value of the input brightness image. The global gain map is also a map in which gain values are two-dimensionally arranged corresponding to each pixel position, as shown in FIG. 4(b) described above.

Furthermore, the global gain map generation unit 111 generates a final gain map by adding the local gain map generated by the local gain map generation unit 110 to the global gain map. The final gain map is also a map in which gain values are two-dimensionally arranged corresponding to each pixel position, as shown in FIG. 4(b). The final gain map is a gain map used when performing gain processing on the input image.

In this manner, the image processing apparatus of this embodiment is capable of independently controlling brightness adjustment using the global gain map and local contrast adjustment using the local gain map. Furthermore, the image processing device of this embodiment generates a local gain map with a high contrast improvement effect by adding an emphasis signal to the local gain map. Then, the image processing apparatus of this embodiment generates a final gain map from these global gain maps and local gain maps.

Image processing performed in the image processing apparatus of this embodiment will be described in detail below using the flowchart of FIG. Furthermore, the correspondence between the flowchart in FIG. 2 and the functional block diagram in FIG. 1 will be explained.

First, in step S201, the local gain map generation unit 110 generates a local gain map based on a luminance image that is an input image. Details of the local gain map generation process performed in step S201 will be described later.

Next, in step S202, the global gain map generation unit 111 generates a global gain map for adjusting brightness according to the brightness of the input image, which is a brightness image. Furthermore, the global gain map generation unit 111 generates a final gain map based on the generated global gain map and the local gain map generated in step S201. Details of the global gain map generation process and the final gain map generation process performed in step S202 will be described later.

Next, in step S203, the gain processing unit 112 performs a process of applying a gain to the input image (gain process) using the final gain map generated in step S202. Here, the luminance signal value at the coordinates (x, y) of the input image is set as IMin (x, y), and the luminance signal value of the output image after gain processing is set as IMout (x, y). In addition, when performing SDR development on a RAW image obtained by HDR photography, GMlast sets the gain value of the coordinates (x, y) in the final gain map in order to obtain the same contrast as the HDR developed image in the image after SDR development. Let it be (x, y). In this case, the luminance signal value IMout(x,y) of the output image after gain processing is expressed as in equation (1).

IMout(x,y)=GMlast(x,y)×IMin(x,y) Equation (1)

In the following explanation, when performing SDR development on a RAW image obtained by HDR photography, the gain characteristic of the gain map for making the image after SDR development have the same contrast as the HDR developed image will be referred to as the HDR-SDR conversion gain characteristic. It is written as. Details of the HDR-SDR conversion gain characteristics will be described later.

The above is the general flow of image processing in the image processing apparatus of this embodiment, and the main focus in the image processing of this embodiment is the process of step S201.
Hereinafter, the local gain map generation process performed in step S201 will be described in detail using the flowchart of FIG. The local gain map generation process shown in the flowchart of FIG. 5 is the process performed by the local gain map generation unit 110 of FIG. 1.

The local gain map generation section 110 includes a first low frequency image generation section 101, a second low frequency image generation section 102, a correction gain conversion section 103, a gain conversion section 104, a gain map conversion section 105, and a gain map synthesis section 106. , a local gain map calculation unit 107.

In step S501 of FIG. 5, the first low-frequency image generation unit 101 performs a reduction process on the input image to generate a first reduced image, and converts the first reduced image into a second low-frequency image. It is output to the generation unit 102. Furthermore, the first low-frequency image generation unit 101 generates a low-frequency image having a different frequency level from the input image by enlarging the first reduced image to the size of the original input image, and The image is output to the correction gain conversion unit 103.

The second low-frequency image generation unit 102 further performs reduction processing on the first reduced image to generate a second reduced image. In this way, the first reduced image is generated by reducing the input image, and the second reduced image is generated by further reducing the first reduced image. In addition, the second low-frequency image generation unit 102 enlarges the second reduced image to the size of the input image, thereby generating a low-frequency image whose frequency level is different from that of the input image and the low-frequency image obtained by enlarging the first reduced image. An image is generated and the low frequency image is output to the correction gain conversion unit 103.

Hereinafter, the luminance image that is the input image will be referred to as the input luminance image IMin, the luminance image that is the low-frequency image obtained by enlarging the first reduced image will be referred to as the first-layer luminance image IMM, and the low-frequency image obtained by enlarging the second reduced image will be referred to as the first layer luminance image IMM. The brightness image that is the image is referred to as a second layer brightness image IMS.
Note that, as a method of reduction processing and enlargement processing performed in the first low-frequency image generation unit 101 and the second low-frequency image generation unit 102, a known method such as processing using a bilinear method is used. be able to. In addition, a known method such as low-pass filter processing may be used as a method for obtaining a low-frequency image.

Next, in step S502, the correction gain conversion unit 103 generates a correction gain characteristic for compensating for contrast reduction that may occur due to the global gain characteristic used when generating the global gain map.

Here, in explaining the method for generating the corrected gain characteristic, the global gain characteristic used when generating the global gain map and the contrast reduction that may occur due to the global gain characteristic will be explained.
First, global gain characteristics will be explained using FIG. 6.
In the graph of FIG. 6(a), the horizontal axis shows the input luminance signal and the vertical axis shows the output luminance signal. In the graphs of FIGS. 6(b) and 6(c), the horizontal axis shows the input luminance signal and the vertical axis shows the input luminance signal, respectively. The axis shows the output gain signal. FIG. 6A is a diagram showing a gradation correction curve 601 that is the target of brightness adjustment using the global gain map generated by the global gain map generation unit 111 of the image processing apparatus of this embodiment. Further, a solid line portion 604 in FIG. 6(b) is obtained by converting the gradation correction curve 601 in FIG. 6(a) into a gain table, and represents a global gain characteristic for brightness adjustment.

Next, contrast reduction that may occur due to the global gain characteristic will be explained using FIG. 7.
Graph (a1) in FIG. 7A represents global gain characteristics, where the horizontal axis represents the input luminance signal and the vertical axis represents the output gain signal. Graphs (a2) to (a4) in FIG. 7(a) show that in the global gain characteristic shown in graph (a1), the input luminance signal Y71 to Y72 has a characteristic in which the gain almost monotonically decreases. FIG. 3 is an explanatory diagram of the effect of improving local contrast using a frequency image. In graphs (a2) and (a4) in FIG. 7A, the horizontal axis indicates the x direction (x coordinate direction), and the vertical axis indicates the luminance signal. In the graph (a3) of FIG. 7A, the horizontal axis indicates the x direction, and the vertical axis indicates the gain signal.

In the case of an image with a difference in brightness such as the input brightness signals Y71 and Y72 in graph (a1) of FIG. 7A, the brightness signal can be expressed as in graph (a2). A solid line portion 701 in graph (a2) of FIG. 7A indicates a change in the brightness signal when the input image has an edge portion where the brightness value changes rapidly at a certain position in the x direction, for example. represents. A dotted line section 702 in graph (a2) indicates a change in the luminance signal of a low frequency image obtained by performing reduction processing, enlargement processing, or filter processing on the input image of the luminance signal as represented by the solid line section 701. represents. In other words, the luminance signal of the low-frequency image represented by the dotted line portion 702 is a signal in which the luminance signal at the edge portion of the solid line portion 701 gradually monotonically decreases due to reduction processing and enlargement processing of the input image, or filter processing. .

Then, based on the signal shown in the dotted line part 702 of the graph (a2), a gain map generated based on the gradation characteristic where the gain monotonically decreases with respect to the luminance is shown in the dotted line part 703 of the graph (a3). It becomes as expressed. In other words, the gain map represented by the dotted line section 703 in graph (a3) is generated based on the luminance signal that changes slowly as shown in the dotted line section 702 of the low frequency image shown in graph (a2), so the gain map is The gain changes gradually in response to changes in the luminance signal.

Here, the key point in the contrast improvement effect is that in the input image, the luminance signal decreases in the x direction, whereas the gain signal gradually increases. That is, in the examples of graphs (a2) and (a3), the luminance signal of the input image is decreasing in the x direction, while the gain signal is gradually increasing. Further, since the gain signal is determined based on the characteristics of the graph (a1), the portion that is the input luminance signal Y72 in the graph (a2) becomes the gain G72 in the graph (a3). On the other hand, the portion of the graph (a2) that is the input luminance signal Y71 has a gain G71 in the graph (a3).

Then, when gain processing is performed on the luminance signal of the input image shown in the solid line part 701 of the graph (a2) using the gain map shown in the dotted line part 703 of the graph (a3), after the gain processing The luminance signal becomes a signal as shown in the solid line portion 704 of graph (a4). That is, the luminance signal after gain processing shown in the solid line portion 704 of graph (a4) is a signal in which the contrast improvement effect is obtained in the signal portion in the range shown by the arrow 705.

However, when a gradation characteristic is such that the gain monotonically increases with respect to luminance, a gain map generated based on the gradation characteristic cannot produce an effect of improving contrast.
A problem with a gain map generated based on a low-frequency image having gradation characteristics in which the gain monotonically increases with respect to luminance will be explained using FIG. 7(b).
Graph (b1) in FIG. 7(b), like graph (a1) in FIG. 7(a), represents the global gain characteristic, with the horizontal axis representing the input luminance signal and the vertical axis representing the output gain signal. It shows. Graphs (b2) to (b4) in FIG. 7(b) show that in the global gain characteristic shown in graph (b1), the input luminance signal Y73 to Y74 has a characteristic in which the gain increases monotonically. It is a figure explaining the effect of local contrast using a frequency image. In graphs (b2) and (b4) in FIG. 7(b), the horizontal axis indicates the x direction (x coordinate direction), and the vertical axis indicates the luminance signal. In the graph (b3) of FIG. 7(b), the horizontal axis indicates the x direction, and the vertical axis indicates the gain signal.

In the case of an image with a difference in brightness such as the input brightness signals Y73 and Y74 in graph (b1) of FIG. 7(b), the brightness signal can be expressed as in graph (b2). A solid line portion 711 in graph (b2) of FIG. 7(b) indicates a change in the brightness signal when the input image has an edge portion where the brightness value changes suddenly at a certain position in the x direction, for example. represents. A dotted line portion 712 in graph (b2) represents a change in the luminance signal of a low frequency image obtained by performing reduction processing, enlargement processing, or filter processing on the input image of the luminance signal as represented by the solid line portion 711. represents. In other words, the luminance signal of the low-frequency image represented by the dotted line portion 712 is a signal in which the luminance signal at the edge portion of the solid line portion 711 gradually monotonically decreases due to reduction processing and enlargement processing of the input image, or filter processing. .

Then, based on the signal shown in the dotted line part 712 of the graph (b2), a gain map generated by the gradation characteristic in which the gain monotonically decreases with respect to the luminance is represented by the dotted line part 706 of the graph (b3). It will be something like this. In other words, the gain map represented by the dotted line section 706 in graph (b3) is generated based on the luminance signal that changes slowly like the dotted line section 712 of the low frequency image shown in graph (b2). The gain changes gradually in response to changes in the luminance signal.

The key point here is that while the luminance signal in the input image decreases in the x direction, the gain signal also decreases gradually. In other words, in the examples of graphs (b2) and (b3), while the luminance signal of the input image is decreasing in the x direction, the gain signal is also decreasing gradually. Furthermore, since the gain signal is determined based on the characteristics of the graph (b1), the portion that is the input luminance signal Y73 in the graph (b2) becomes the gain G73 in the graph (b3). On the other hand, the portion of the graph (b2) that is the input luminance signal Y74 has a gain G74 in the graph (b3).

Then, when gain processing is performed on the luminance signal of the input image shown in the solid line part 711 of the graph (b2) using the gain map shown in the dotted line part 706 of the graph (b3), after the gain processing The luminance signal becomes a signal as shown in the solid line portion 707 of graph (b4). The luminance signal after the gain processing shown in the solid line portion 707 of graph (b4) becomes a signal in which the gain changes gently in the signal portion within the range shown by the arrow 708. In other words, in this case, the contrast decreases.

Therefore, in the image processing apparatus, a correction gain characteristic is used to compensate for the decrease in contrast due to the global gain map as described above. A method for generating the correction gain characteristic will be explained using FIG. 6(b).
As described above, the solid line portion 604 in FIG. 6(b) represents the global gain characteristic for brightness adjustment. In contrast to this global gain characteristic, the correction gain characteristic monotonically increases as shown in the solid line section 603 in FIG. 6(c), as in the brightness region 602 of the global gain characteristic in FIG. 6(b). It is treated as a characteristic that converts the part into a monotonically decreasing part. The reason why the correction gain characteristic is made to be a monotonically decreasing characteristic is to obtain the effect of improving contrast.

Here, the characteristic to be applied in the global gain characteristic, which is the original gain processing, to the brightness region 602 from input brightness signal Y61 to Y62 shown in FIG. 6(b) is expressed by the following equation (2). The gain characteristic is as shown in equation (3) below. In the equation, Yin represents the input luminance signal value, A and Ad represent the slope, and Gα(Yin) and Gαd(Yin) each represent the gain when the input luminance signal value is Yin. Further, G61 indicates a gain value when the input luminance signal of the gain characteristic is Y61.

Gα(Yin)=A(Yin-Y61)+G61 (A>0) Formula (2)
Gαd(Yin)=Ad(Yin-Y61)+G61 (Ad<0) Formula (3)

The explanation returns to the flowchart of FIG.
In step S503, the gain conversion unit 104 generates a gain characteristic that compensates for the decrease in contrast that occurs when performing SDR development of the RAW image obtained by HDR photography, based on the correction gain characteristic generated in step S502, the SDR gamma, and the HDR gamma. generate. In the present embodiment, the gain conversion unit 104 generates an HDR-SDR conversion gain characteristic as a gain characteristic that compensates for a decrease in contrast that occurs when a RAW image obtained by HDR photography is subjected to SDR development.

FIG. 8 is a diagram used to explain a method of generating HDR-SDR conversion gain characteristics in the image processing apparatus of this embodiment. The vertical axis in FIG. 8(a) indicates the output luminance signal, and the horizontal axis indicates the input luminance signal. The vertical axis of FIG. 8(b) shows the gain signal, and the horizontal axis shows the input luminance signal.
A dotted line section 801 in FIG. 8A indicates HDR gamma, and a solid line section 802 indicates SDR gamma. Further, a solid line portion 803 in FIG. 8(b) indicates a gain characteristic applied when performing SDR development of a RAW image obtained by SDR photography.

When a RAW image obtained by HDR photography is subjected to SDR development, in order to obtain the same contrast as the HDR developed image in the SDR developed image, it is necessary to make the slope of the gain characteristic during SDR development steep. In other words, in order to obtain the same contrast as the HDR developed image in the image after SDR development, the gain characteristic during SDR development is shown in the dotted line section 804, which has a steeper slope than the gain characteristic shown in the solid line section 803. It is necessary to have such HDR-SDR conversion gain characteristics.

The gain conversion unit 104 of this embodiment determines the steepness of the gain characteristic in the HDR-SDR conversion gain characteristic based on the gain characteristic during SDR development and the difference between HDR gamma and SDR gamma.
Here, in a region where there is a difference between the HDR gamma and the SDR gamma, such as near the input luminance signal Y82 in FIG. The gain characteristics during SDR development are converted based on the difference between the output luminance signals. That is, in this embodiment, the difference between the output luminance signals corresponding to the input luminance signals of HDR gamma and SDR gamma represents the gamma difference between HDR gamma and SDR gamma, and based on this gamma difference, HDR- Generate SDR conversion gain characteristics.

For example, let the value of the input luminance signal be Yin, and let Gαd(Yin) be the gain when Yin based on the correction gain characteristic obtained in step S502. Furthermore, it is assumed that the output luminance signal in the SDR gamma corresponding to Yin is Ysdr, the output luminance signal in the HDR gamma corresponding to Yin is Yhdr, and the adjustment coefficient is m. At this time, Gβ(Yin), which is the HDR-SDR conversion gain characteristic, is expressed by equation (4). Note that in equation (4), the degree of contrast enhancement can be controlled by changing the value of the adjustment coefficient m.

Gβ (Yin) = Gαd (Yin) - m x Gαd (Yin) x (Yhdr - Ysdr) Equation (4)

Furthermore, the gain conversion unit 104 of the present embodiment has the same characteristics as shown in FIG. 8(b) in a region where the SDR gamma and HDR gamma have the same characteristics, such as from the origin to the vicinity of the input luminance signal Y81 in FIG. ), the gain characteristics are the same during SDR development and HDR development. That is, when performing SDR development on a RAW image obtained by HDR photography, the HDR-SDR conversion gain characteristic of an area where the SDR gamma and HDR gamma have the same characteristics is made into the gamma characteristic during SDR development.

Next, in step S504, the gain map conversion unit 105 generates a gain map by applying the HDR-SDR conversion gain characteristic generated in step S503 described above to the input luminance image. Hereinafter, a gain map generated by applying HDR-SDR conversion gain characteristics to an input luminance image will be referred to as a same-size hierarchical gain map.

In addition, the gain map conversion unit 105 applies the HDR-SDR conversion gain characteristics to the first layer brightness image IMM and the second layer brightness image IMS generated in step S501, thereby obtaining a gain for each layer. Generate a map. Hereinafter, a gain map generated by applying the HDR-SDR conversion gain characteristic to the first layer luminance image IMM will be referred to as a first layer gain map. Further, a gain map generated by applying the HDR-SDR conversion gain characteristic to the second layer luminance image IMS will be referred to as a second layer gain map.

In Fig. 1, the HDR-SDR conversion gain characteristic is expressed as Gβ, and a same-size hierarchical gain map is created by applying the HDR-SDR conversion gain characteristic Gβ to the coordinates (x, y) of the input luminance image IMin. It is written as GML (x, y). Furthermore, a first layer gain map obtained by applying the HDR-SDR conversion gain characteristic Gβ to the coordinates (x, y) of the first layer luminance image IMM is expressed as GMM (x, y). Similarly, the first layer gain map obtained by applying the HDR-SDR conversion gain characteristic Gβ to the coordinates (x, y) of the second layer luminance image IMS is expressed as GMS (x, y).

Then, the gain map conversion unit 105 outputs the same-size hierarchical gain map GML (x, y) generated by applying the HDR-SDR conversion gain characteristics as described above to the local gain map calculation unit 107. Further, the gain map conversion unit 105 converts the first layer gain map GMM (x, y) and the second layer gain map GMS (x, y) generated by applying the HDR-SDR conversion gain characteristics into a gain map synthesis unit. 106.

Next, in step S505, the gain map synthesis unit 106 synthesizes the first layer gain map GMM (x, y) and the second layer gain map GMS (x, y) generated in step S504. Generate a gain map. Note that in FIG. 1, the composite gain map at coordinates (x, y) is expressed as GMc(x, y). As a method for generating the composite gain map, a known method may be used in which a gain signal of a gain map with a large image size and a gain signal of a gain map with a small image size are weighted and added according to the signal difference in gain. The composite gain map GMc(x,y) generated by the gain map composition section 106 is sent to the local gain map calculation section 107.

FIG. 9A is a diagram showing an example of a brightness signal of an input brightness image (input brightness signal 901), in which the vertical axis represents the brightness signal and the horizontal axis represents the coordinate in the x direction. It is assumed that the input luminance signal 901 shown in FIG. 9A is a signal having an edge portion where the luminance signal Y1 increases to the luminance signal Y2 at a certain position in the x direction. Further, a solid line portion 902 in FIG. 9(b) represents a gain signal generated from the input luminance signal 901 shown in FIG. 9(a), and the gain signal G1d corresponds to the luminance value Y2 of the input luminance signal 901. However, the gain signal G2d corresponds to the luminance value Y1 of the input luminance signal 901. On the other hand, a dotted line portion 903 in FIG. 9(b) represents a composite gain signal based on a composite gain map generated by the gain map synthesis unit 106 for the input luminance signal 901 in FIG. 9(a). The composite gain signal indicated by the dotted line part 903 is obtained by combining the gain maps generated from the reduced image (low frequency image) as described above. It has the characteristic of gradually decreasing in G2d.

Next, in step S506, the local gain map calculation unit 107 calculates an emphasis signal of the local gain map based on the composite gain map generated in step S505 and the same-size hierarchical gain map generated in step S504. . Note that in FIG. 1, the value of the local gain map at coordinates (x, y) is expressed as GMlocal (x, y). If the value of the composite gain map at coordinates (x, y) is GMcomp(x, y), the value of the gain map of the equal-sized layer is GML(x, y), and the adjustment coefficient is k, then the value of the local gain map GMlocal( x, y) is expressed by equation (5).

GMlocal (x, y) = k (GMcomp (x, y) - GML (x, y)) Equation (5)

A solid line portion 904 in FIG. 9(c) is a local gain map calculated based on the input luminance signal 901 in FIG. 9(a) described above and a composite gain signal shown in a dotted line portion 903 in FIG. Shows emphasized signal. By using the gain signal shown in the solid line section 902 generated from the input luminance signal 901 and the composite gain signal shown in the dotted line section 903, the calculation (subtraction) of equation (5) is performed, so that the result as shown in the solid line section 904 is performed. An emphasis signal of a local gain map is generated.

Note that the adjustment coefficient k multiplied by the difference value between the composite gain map value GMcomp(x, y) and the gain map value GML(x, y) of the equal-sized layer in equation (5) can be set arbitrarily. be. That is, the emphasis signal of the gain map can be controlled by setting the adjustment coefficient k. Further, the adjustment coefficient k may be changed according to the value of the composite gain map or the gain map of the equal-sized layer.

FIG. 10(a) shows an example of the emphasis signal of the local gain map during SDR development, and FIG. 10(b) shows an example of the emphasis signal of the local gain map obtained as described above in this embodiment. FIG. By using the emphasis signal shown in FIG. 10(b), when performing SDR development of a RAW image obtained by HDR photography, it is possible to obtain a contrast effect during HDR development. For example, the peak value of the emphasis signal of the local gain map during SDR development becomes the gain signal G101 in FIG. 10(a). On the other hand, according to the emphasis signal of the local gain map according to the present embodiment, as shown in FIG. 10(b), when a RAW image obtained by HDR photography is subjected to SDR development, the peak value is equivalent to that during HDR development. This becomes a gain signal G102. That is, according to this embodiment, when performing SDR development on a RAW image obtained by HDR photography, an emphasis signal with a high contrast effect can be obtained.

The above is the image processing according to the image processing apparatus of this embodiment, and the main point in this embodiment is the process of step S503 in FIG. 5 performed by the gain conversion unit 104 in FIG. 1.
Next, the global gain map and final gain map generation processing performed by the global gain map generation unit 111 of FIG. 1 in step S202 of FIG. 2 will be described with reference to FIGS. 6 and 9.
The global gain map generation section 111 includes a global gain map calculation section 108 and a final gain map generation section 109.

As mentioned above, in FIG. 6(a), the horizontal axis represents the input brightness signal and the vertical axis represents the output brightness signal, and in FIG. 6(b), the horizontal axis represents the input brightness signal and the vertical axis represents the output gain signal. There is. FIG. 6A shows a tone correction curve 601 that is the target of brightness adjustment performed by the global gain map generation unit 111 in the image processing apparatus of this embodiment. The gradation correction curve is calculated using a known method that can calculate the gain for the entire image area by, for example, performing spline interpolation to match the brightness value of the target area in the image to the target brightness value. can do. Furthermore, as described above, the solid line portion 604 in FIG. 6(b) is the gradation correction curve 601 in FIG. 6(a) converted into a gain table, and represents the gain characteristics for brightness adjustment. ing.

A solid line portion 905 in FIG. 9(d) indicates each gain of the global gain map generated based on the input luminance signal 901 illustrated in FIG. 9(a) using the global gain characteristic illustrated in FIG. 6(b). It shows a gain signal with values plotted in the x direction. In FIG. 9(d), the horizontal axis represents the position in the x direction, and the vertical axis represents the gain signal. The gain signal in the solid line portion 905 in FIG. 9(d) has a gain characteristic in which the gain value increases from G1 to G2 in response to the portion where the luminance value increases from Y1 to Y2 in the input luminance signal 901 in FIG. 9(a). It is a signal with Note that the gain value G1 is a gain value corresponding to the luminance value Y1 of the input luminance signal, and the gain value G2 is a gain value corresponding to the luminance value Y2.

In this way, the global gain map calculation unit 108 uses the global gain characteristics to generate a global gain map with the gain characteristics shown in FIG. 9(d) based on the input luminance signal 901 in FIG. 9(a). . Then, the global gain map generated by the global gain map calculation section 108 is output to the final gain map generation section 109. Note that in FIG. 1, the value of the global gain map at coordinates (x, y) is expressed as GMglobal (x, y), and the value of the final gain map is expressed as GMlast (x, y).

The final gain map generation unit 109 generates a final gain map based on the local gain map generated by the local gain map calculation unit 107 and the global gain map generated by the global gain map calculation unit 108. Then, the final gain map generation section 109 outputs the global gain map to the gain processing section 112.

Here, in the example of FIG. 9(d), the coordinates are (x, y), the gain signal of the global gain map is GMglobal (x, y), and the gain signal of the local gain map is GMlocal (x, y). . In this case, the final gain map generation unit 109 calculates the gain signal GMlast(x,y) of the final gain map using equation (6).

GMlast (x, y) = GMlocal (x, y) + GMglobal (x, y) Equation (6)

Next, in step S203, the gain processing unit 112 performs gain processing to apply a final gain map to the input image. As mentioned above, if the input luminance signal value at coordinates (x, y) is IMin (x, y) and the final gain map value is GMlast (x, y), then the output luminance signal value after gain processing IMout ( x, y) are determined by the above-mentioned equation (1).

By performing the image processing as described above, the image processing apparatus of this embodiment can obtain a RAW image that is an output image after gain processing to which the HDR-SDR conversion gain characteristic is applied.
Note that although the input image and the output image are described as RAW images in this embodiment, the input image and the output image may be images in other formats such as so-called YUV.
As described above, according to the image processing apparatus of this embodiment, when SDR development is performed from a RAW image shot in HDR, it is possible to obtain a contrast effect close to the HDR development result.

The present invention provides a system or device with a program that implements one or more functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
The above-described embodiments are merely examples of implementation of the present invention, and the technical scope of the present invention should not be construed as limited by these embodiments. That is, the present invention can be implemented in various forms without departing from its technical idea or main features.

101: First low frequency image generation section, 102: Second low frequency image generation section, 103: Correction gain conversion section, 104: Gain conversion section, 105: Gain map conversion section, 106: Gain map synthesis section, 107 : Local gain map calculation unit, 108: Global gain map calculation unit, 109: Final gain map generation unit, 110: Local gain map generation unit, 111: Global gain map generation unit, 112: Gain processing unit

Claims (8)

generating means for generating a gain signal used in gain processing when developing an undeveloped image;
processing means for performing the gain processing on the undeveloped image using the gain signal generated by the generation means;
When a development process corresponding to a second dynamic range that is a narrower dynamic range than the first dynamic range is performed using the undeveloped image, the generating means In a region where there is no difference between the gamma characteristics and the gamma characteristics corresponding to the second dynamic range, the gamma characteristics are common, and the gamma characteristics corresponding to the first dynamic range and the gamma characteristics corresponding to the second dynamic range are the same. In a region where there is a difference between the two, the gain signal is generated based on a gain characteristic that is a characteristic obtained by changing the gamma characteristic corresponding to the second dynamic range to correspond to the first dynamic range. Characteristic image processing device. The generating means generates a gain map consisting of gain values corresponding to each pixel of the image,
The image processing apparatus according to claim 1 , wherein the processing means performs the gain processing using a gain value corresponding to each pixel of the gain map. The generating means is
a global gain map that performs brightness adjustment on the image;
generating a local gain map for locally performing contrast adjustment on the image;
3. The image processing apparatus according to claim 2 , wherein the processing means generates a final gain map used in the gain processing by adding the global gain map and the local gain map. The generating means is
generating a correction gain characteristic from the image to compensate for contrast reduction caused by the gain processing according to the global gain map;
In a region where there is no difference between the correction gain characteristic, the gamma characteristic corresponding to the first dynamic range, and the gamma characteristic corresponding to the second dynamic range, the characteristic is common, and the gamma characteristic corresponding to the first dynamic range is a common characteristic. In a region where there is a difference between the gamma characteristics and the gamma characteristics corresponding to the second dynamic range, the gamma characteristics corresponding to the second dynamic range are changed to correspond to the first dynamic range. The image processing apparatus according to claim 3 , wherein the local gain map is generated based on the gain characteristics . The generating means generates the local gain map based on the image and an image obtained by reducing the image and then enlarging it to the size of the image, or an image obtained by filtering the image. The image processing apparatus according to claim 3 or 4 . The generating means is configured to generate at least two images having different frequencies, the image and the images obtained by reducing the image and then enlarging it to the size of the image, or by filtering the image and having different frequencies. The image processing apparatus according to claim 5 , wherein the local gain map is generated based on at least two images . An image processing method executed by an image processing device, the method comprising:
a generation step of generating a gain signal used for gain processing when developing an undeveloped image;
a processing step of performing the gain processing on the undeveloped image using the gain signal generated in the generation step,
When the undeveloped image is subjected to development processing corresponding to a second dynamic range, which is a narrower dynamic range than the first dynamic range, in the generation step, processing corresponding to the first dynamic range is performed. In a region where there is no difference between the gamma characteristics and the gamma characteristics corresponding to the second dynamic range, the gamma characteristics are common, and the gamma characteristics corresponding to the first dynamic range and the gamma characteristics corresponding to the second dynamic range are the same. In a region where there is a difference between the two, the gain signal is generated based on a gain characteristic that is a characteristic obtained by changing the gamma characteristic corresponding to the second dynamic range to correspond to the first dynamic range. Featured image processing method. A program for causing a computer to function as each means included in the image processing apparatus according to any one of claims 1 to 6 .

Images