JP2020190711A - Image processing device and image processing method - Google Patents

Abstract

To provide a technique to make it possible to easily identify a pixel exceeding a dynamic range of an output device from a pixel within the dynamic range while performing output such that a change in a gradation is visible.SOLUTION: An image processing device comprises a setting part for setting a threshold of a luminance level and an image processing unit for generating a second image by converting the luminance level of at least a part of a region of a first image. The image processing unit generates the second image such that the luminance level of a pixel of the second image corresponding to a pixel having a first luminance level greater than the threshold in the first image is higher than the luminance level of a pixel of the second image corresponding to a pixel having a second luminance level greater than the first luminance level in the first image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image processing apparatus and an image processing method.

In recent years, in display devices such as televisions and monitors, products having a wider dynamic range (displayable brightness range) than before have appeared. Along with this, high dynamic range (HDR; High Dynamic Range) images corresponding to a wide dynamic range are becoming widespread. In contrast to HDR images, conventional images are referred to as SDR (Standard Dynamic Range) images in the sense that they have a standard dynamic range.

On the other hand, with the spread of such HDR images, there is an increasing use for checking HDR images on a monitor having a limited dynamic range as compared with the HDR standard (that is, a monitor having a narrower dynamic range than the HDR image). When displaying an HDR image that corresponds to a dynamic range wider than the dynamic range of the monitor, the pixels corresponding to the range (overrange) that the monitor cannot express are displayed with the maximum brightness (white) that is expressed as so-called overexposure. Processing is performed. For example, an image is confirmed on the rear monitor of a camera or an electronic viewfinder at the time of shooting, but many of these types of monitors have a limited dynamic range.

Patent Document 1 describes a method of displaying an overranged region by superimposing a zebra pattern on a region having a brightness level equal to or higher than a threshold value.

JP-A-2014-167609

However, although the method of Patent Document 1 can determine the presence or absence of an overranged pixel region and its range, it is not suitable for visually recognizing a continuous change in gradation or an image included in the overrange. For example, in the method of Patent Document 1, it is not possible to determine whether or not an object is included in the overranged pixel region of the image.

The present invention has been made in view of the above circumstances, and it is possible to output pixels that exceed the dynamic range of the output device so that they can be easily distinguished from the pixels within the dynamic range and the change in gradation can be visually recognized. The purpose is to provide the technology to do so.

The first aspect of the present invention includes a setting unit for setting a luminance level threshold and an image processing unit for converting a luminance level in at least a part of a region of the first image to generate a second image. In the image processing unit, the brightness level of the pixel of the second image corresponding to the pixel having the first luminance level larger than the threshold of the first image is higher than the first luminance level of the first image. Provided is an image processing apparatus characterized in that the second image is generated so as to be higher than the brightness level of the pixel of the second image corresponding to the pixel having a high second brightness level.

The second aspect of the present invention includes a step of setting a luminance level threshold and a generation step of converting the luminance level of at least a part of the first image to generate a second image. In the step, the brightness level of the pixel of the second image corresponding to the pixel having the first luminance level larger than the threshold of the first image is higher than the first luminance level of the first image. Provided is an image processing method characterized in that the second image is generated so as to be higher than the brightness level of the pixel of the second image corresponding to the pixel having the two luminance levels.

A third aspect of the present invention provides a program for causing a computer to function as each part of an image processing apparatus according to the first aspect of the present invention.

According to the present invention, it is possible to output pixels that exceed the dynamic range of the output device so that they can be easily distinguished from the pixels within the dynamic range and the change in gradation can be visually recognized.

A block diagram showing the configuration of an image display system. The figure which shows the flow of the image conversion processing by an image processing apparatus. The figure which shows an example of the relationship between an input dynamic range, an output dynamic range, and an overrange. The figure which shows an example of the relationship between the luminance level before conversion and the luminance level after conversion. The figure which shows an example of the relationship between the luminance level before conversion and the luminance level after conversion. The figure which shows an example of the relationship between the luminance level before conversion and the luminance level after conversion. The figure which shows an example of the relationship between the luminance level before conversion and the luminance level after conversion. The figure which shows the example of the output image generated by the conversion process.

The present invention converts an input image to an output image by a process of converting the dynamic range of the input image (hereinafter, "input dynamic range") into a dynamic range consisting of brightness levels equal to or less than a set threshold value (hereinafter, "output dynamic range"). Regarding the technology to generate. Such a technique can be used to generate an output image suitable for output by the output device from the input image when the dynamic range of the output device is narrower than the input dynamic range. For the purpose of confirming the brightness and gradation of the input image, it is preferable that the pixels (areas) whose brightness level is included in the output dynamic range of the input image are reproduced as faithfully as possible to the input image. On the other hand, among the input images, it is desired that the pixels (areas) whose luminance level is not included in the output dynamic range can be easily identified as being overranged. Here, among the input images, pixels (areas) whose luminance level is not included in the output dynamic range are referred to as overrange pixels (overrange areas). However, if the overrange area is displayed with a uniform brightness or the same warning color is displayed, the image information (gradation information) included in the overrange area is lost. Specifically, since it is not possible to confirm the gradation change included in the overrange area, the user cannot confirm the objects and patterns included in the overrange area. Therefore, the purpose of confirming the input image cannot be achieved. In addition, in the method of performing gradation compression so that all or part of the input dynamic range (for example, the part to be confirmed) fits in the output dynamic range, the brightness level of the input image is included in the output dynamic range. Is output with a brightness different from the original. Therefore, such gradation compression is not suitable for checking an input image.

Therefore, in the present invention, a process of converting the brightness level of the input image having a brightness level exceeding the output dynamic range is performed so that the converted brightness level is included in the output dynamic range. Further, the conversion process uses a correspondence relationship in which the brightness level after conversion decreases with respect to the increase in the brightness level before conversion for at least a part of the overrange. In other words, for at least a part of the overrange, the conversion process is performed so that the mapping (function) from the luminance level before conversion to the luminance level after conversion is monotonically reduced. That is, for pixels whose luminance level is included in the output dynamic range of the input image, the larger the luminance level (gradation value) in the input image, the higher the luminance level of the converted output image. For pixels whose luminance level is not included in the output dynamic range of the input image, the larger the luminance level (gradation value) in the input image, the lower the luminance level of the converted output image. As a result, it is possible to faithfully reproduce the input image for the pixels whose luminance level is included in the output dynamic range. On the other hand, for the pixels of the input image whose luminance level is not included in the output dynamic range, it is easy to distinguish the pixels of the input image whose luminance level is included in the output dynamic range, and the change in gradation is visually recognized. The output that can be done is possible.
In this way, the process of converting so that the higher the brightness level of the image before conversion is, the lower the brightness level of the image after conversion is also called a positive / negative process (positive / negative inversion process). It is also simply called inversion processing.

In addition, in this specification, "luminance level" is used in the meaning of the index indicating the brightness, specifically, the value proportional to the brightness. For example, it may be an absolute brightness value, a standardized value of brightness, or a relative brightness value. "The mapping from the brightness level before conversion to the brightness level after conversion is monotonically reduced" means that the larger the brightness level before conversion, the smaller the brightness level after conversion, that is, the mapping is f, and the brightness level before conversion is Is Li, and the brightness level after conversion is f (Li).
If L1 <L2, then f (L1) ≥ f (L2)
Means that holds true. This mapping f is also called monotonic non-increasing. The domain of the map f, that is, the range of the luminance level Li before conversion may be the entire range exceeding the output dynamic range or a part of the range exceeding the output dynamic range. For example, since pixels are rarely distributed over the entire range exceeding the output dynamic range, even if the conversion by the above mapping f is performed only in a part of the range where a relatively large number of overrange pixels are concentrated. Good. The mapping can be said to be the correspondence between the luminance level before conversion and the luminance level after conversion, or a function (conversion function) for converting the luminance level before conversion into the luminance level after conversion.

In the following embodiments, an example in which the present invention is applied to an application for confirming an HDR image on a display device having a limited dynamic range will be described.

FIG. 1 is a block diagram showing a configuration of an image display system according to an embodiment of the present invention. The image display system is a system used for confirming the input image 12, and is composed of an image processing device 10 and an image display device 11. The image processing device 10 receives the input image 12 and the output dynamic range information 13 as inputs, performs necessary processing on the input image 12, generates an output image 14, and outputs the output image 14 to the image display device 11. The user confirms the brightness and gradation of the input image 12 by looking at the output image 14 displayed on the image display device 11. The image processing device 10 and the image display device 11 may be separate devices or may be single devices.

The input image 12 is, for example, an HDR image conforming to a standard such as a PQ method or an HLG method. However, as long as the image has a signal gradation corresponding to a brightness level exceeding the reference white of the SDR image, an image other than these standards (for example, a method unique to the camera such as the Log method) may be used as the input image 12. For example, in an HDR image of the HLG system, a signal gradation up to a brightness level corresponding to 12 times the brightness of the reference white of the SDR image is defined. Further, in the HDR image of the PQ system, assuming that the reference white of the SDR image is 100 nits, the signal gradation up to the brightness level corresponding to 100 times the brightness (10000 nits) is defined. In addition, 1 nit is equal to 1 cd / m ² , and the area of 1 square meter is the brightness that shines uniformly with the brightness of 1 candela.

The correspondence between the gradation value and the brightness (brightness) is called EOTF (Electro-Optical Transfer Function). The EOTF of the input image 12 may be associated with the input image 12 and input to the image processing device 10 (however, if the EOTF of the input image 12 is known, the input of the EOTF is not always necessary).

The output dynamic range information 13 is information for determining the dynamic range (output dynamic range) of the output image 14 output from the image processing device 10. When the output image 14 is used for display on the image display device 11 as in the present embodiment, information indicating the dynamic range of the image display device 11 may be given as the output dynamic range information 13. For example, when the gradation value of the input image 12 is defined by the absolute brightness as in the HDR image of the PQ method, the value of the maximum brightness (absolute brightness) that can be displayed by the image display device 11 is the output dynamic range information 13. It is preferable to be given as. On the other hand, when the gradation value of the input image 12 is defined by the relative brightness as in the HDR image of the HLG method or the Log method, the value of the relative brightness corresponding to the maximum brightness that can be displayed by the image display device 11 is set. It is preferable that the output dynamic range information 13 is given. Alternatively, it is not a value of absolute brightness or relative brightness, but a gradation value corresponding to the maximum displayable brightness (that is, information indicating to which gradation value in the gradation range of the input image 12 can be displayed). May be given as the output dynamic range information 13. The image processing device 10 can also read the EDID information of the image display device 11 and acquire information indicating the value of the maximum luminance (absolute luminance) that the image display device 11 can display.

The image display device 11 is an output device for displaying the output image 14 output from the image processing device 10. In the image display device 11, the brightness (brightness) corresponding to the gradation value is displayed for each pixel of the output image 14. The image display device 11 is, for example, a liquid crystal display device, an organic EL (Electro Luminescence) display device, a plasma display device, a MEMS (Micro Electro Mechanical System) shutter type display device, and the like.

The image processing device 10 has an image acquisition unit 101, a setting unit 102, an image processing unit 103, and a communication unit 104 as main functional units. The image acquisition unit 101 is a function of acquiring the data of the input image 12.

The setting unit 102 is a function for setting the output dynamic range. The setting unit 102 sets at least an upper limit value (threshold value) of the output dynamic range. The setting unit 102 may set an upper limit value of the output dynamic range according to a user's instruction received from an input device such as a mouse or a touch panel, for example. The setting unit 102 may set an upper limit value of the output dynamic range based on the output dynamic range information 13 acquired from the image display device 11 via the communication unit 104. When the device to which the output image 14 is output is undecided, the former method (input or designation by the user) is preferable. The upper limit of the input image 12 and the output dynamic range is given to the image processing unit 103.

When the input image 12 includes pixels having a brightness level larger than the upper limit of the output dynamic range (overrange pixels), the image processing unit 103 converts the brightness level of the input image 12 and based on the converted brightness level. It is a function to generate an output image 14. The process performed by the image processing unit 103 on the overrange pixel may include processing other than the luminance level conversion, for example, color conversion, superimposition of an image pattern (warning pattern), and the like. For pixels whose luminance level is equal to or less than the upper limit of the output dynamic range (that is, pixels that are not overranged), the value of the input image 12 may be output to the output image 14 as it is. This is to faithfully reproduce the brightness and color of the input image 12.

The communication unit 104 is a communication interface for transmitting and receiving information indicating the output image 14 and the output dynamic range of the image display device 11 to and from the image display device 11.

The image processing device 10 may be composed of, for example, a computer including a processor (CPU), a memory, and a non-volatile storage medium. In the case of this configuration, the functional units 101 to 103 shown in FIG. 1 are realized by loading the program stored in the non-volatile storage medium into the memory and executing the program by the processor. However, the present invention is not limited to this configuration, and all or part of the functional parts of the image processing apparatus 10 may be configured by a circuit such as an ASIC or FPGA. Alternatively, all or part of the functional parts of the image processing device 10 may be realized by cloud computing or distributed computing.

FIG. 2 shows the flow of image conversion processing by the image processing device 10. In step S201, the image acquisition unit 101 acquires the input image 12. At this time, the image acquisition unit 101 also acquires the EOTF associated with the input image 12.

In step S202, the setting unit 102 acquires the output dynamic range information 13 and sets the upper limit value (threshold value) of the output dynamic range based on the output dynamic range information 13. The upper limit of the input image 12 and the output dynamic range is given to the image processing unit 103.

In step S203, the image processing unit 103 converts the gradation value of each pixel of the input image 12 into a luminance level according to the EOTF of the input image 12.

In step S204, the image processing unit 103 performs brightness level conversion processing on the pixels whose brightness level exceeds the threshold value in the input image 12. At this time, at least a part of the range exceeding the upper limit of the output dynamic range is converted by the monotonically decreasing mapping (specific examples will be described later). For pixels whose brightness level is within the output dynamic range, the brightness level is not converted and is output as it is.

In step S205, the image processing unit 103 converts the brightness level of each pixel after conversion into a gradation value according to the EOTF of the image display device 11, and generates an output image 14. The EOTF of the image display device 11 may be preset in the image processing unit 103, may be included in the output dynamic range information 13, or may be acquired directly from the image display device 11.

The image processing unit 103 may perform the conversion processing of steps S203 to S205 by a function, or uses a data table (hereinafter referred to as a look-up table (LUT)) to which the values before and after the conversion are associated. You may go. Since the method using the LUT requires less calculation, the processing time can be shortened. An individual function or LUT may be used for each of the three conversion processes of steps S203 to S205, or a function or LUT that performs the three conversion processes collectively may be used. In the conversion processing function or LUT of step S203, the input is the gradation value and the output is the luminance level. The function or LUT of the conversion process in step S204 has both an input and an output of the luminance level. In the conversion processing function or LUT of step S205, the input is the luminance level and the output is the gradation value. Further, the function or LUT that collectively performs the three conversion processes of steps S203 to S205 has gradation values for both input and output. Once the combination of "EOTF of the input image", "EOTF of the image display device" and "output dynamic range" is determined, the gradation value (input gradation) of the input image and the gradation value (output gradation) of the output image are determined. Correspondence can be calculated in advance. By using the calculated value, it is possible to generate a function or LUT that collectively performs the three conversion processes. By performing the three conversion processes together, the processing time can be further shortened. It is assumed that the function or LUT used by the image processing unit 103 is stored in the non-volatile storage medium in advance. Further, although the conversion processing is performed on the overrange pixels in step S204, for example, using the input / output correspondence as shown in FIG. 4 described later, the input image is collectively converted regardless of the brightness level. May be applied.

If the correspondence between the input value and the output value (hereinafter referred to as "input / output relationship") is defined for all the input values, the data size of the LUT becomes enormous. Therefore, it is preferable to use a LUT that defines an input / output relationship only for discrete input values (a limited number of input values; hereinafter referred to as "discrete input values"). As a result, the data size of the LUT can be reduced, and the memory size required for storing the LUT can be reduced. When an input value not defined by the LUT is given, one or more discrete input values having similar values are selected, and the output value is calculated by interpolation calculation using the input / output relationship of the discrete input values. For example, two output values corresponding to two discrete input values with a given input value in between are obtained from the LUT, and the ratio according to the distance between the given input value and each discrete input value is two. By linearly interpolating one output value, the output value corresponding to the input value is calculated.

Care must be taken when applying the above-mentioned discrete LUT and interpolation calculation to the LUT for the conversion process in step S204 and the LUT for performing the three conversion processes of steps S203 to S205 together. That is, in the present embodiment, the correspondence between the input value and the output value changes discontinuously depending on whether the input value is within the output dynamic range or exceeds the upper limit value thereof. For example, in the case of being within the output dynamic range, the relationship between the input value and the output value is monotonically increasing, but when the upper limit value of the output dynamic range is exceeded, the relationship is switched to the monotonically decreasing relationship. Therefore, if a discrete input value smaller than the upper limit value of the output dynamic range and a discrete input value larger than the upper limit value are selected when performing the interpolation calculation, the result of the interpolation calculation deviates greatly from the desired input / output relationship. , Inappropriate converted images may be obtained. In order to prevent such an unintended conversion, it is effective to include the input / output relationship regarding the input value corresponding to the upper limit value of the output dynamic range in the LUT. In other words, one of the plurality of input / output relationships (correspondence relationship between input value and output value) defined in the LUT is the input / output relationship related to the input value corresponding to the upper limit value (luminance level threshold value) of the output dynamic range. It is good to be. By using such a LUT, the interpolation calculation is not performed between the discrete input value smaller than the upper limit value of the output dynamic range and the discrete input value larger than the upper limit value, and it is possible to eliminate the invalid interpolation calculation. it can. The input value and output value of the LUT may be a gradation value or a brightness level.

FIG. 3 shows an example of the relationship between the dynamic range of the input image, the output dynamic range, and the overrange. In this example, the input image is of the PQ method and has gradations up to 10000 nits, and the upper limit of the output dynamic range is 100 nits, and the gradation range exceeding 100 nits is the overrange. In each pixel included in the input image, the pixel whose gradation value is within the output dynamic range (that is, the pixel having the gradation value or less corresponding to 100 nits) can be displayed according to the EOTF information of the input image. However, the pixel whose gradation value corresponds to the overrange cannot be displayed.

By the way, as is generally known as Weber-Fechner's law, the amount of sensation of gradation in image display is perceived in proportion to the change ratio of brightness. Therefore, even if the change ratio of the brightness is changed from an increase to a decrease of the same degree, the relative relationship between the gradations can be perceived. Therefore, for the pixels in the gradation range that cannot be displayed by the image display device 11, the relationship of the brightness change with respect to the gradation increase of the input image 12 is replaced from the increase to the same degree of decrease. By doing so, it becomes possible to display the gradation exceeding the dynamic range of the image display device 11 on the image display device 11 while reproducing the relative relationship between the gradations.

FIG. 4 shows an example of the relationship between the luminance level before conversion and the luminance level after conversion in the luminance level conversion process of step S204. The horizontal axis of FIG. 4 is the normalized input luminance, which is a value obtained by normalizing the luminance level before conversion (luminance level of the input image 12) so that the upper limit of the output dynamic range is 1.0. The vertical axis is the normalized output luminance, which is a value obtained by normalizing the luminance level after conversion (luminance level of the output image 14) so that the upper limit of the output dynamic range is 1.0. Both the horizontal axis and the vertical axis are logarithmic axes.

The thick line in FIG. 4 represents the mapping (function) from the luminance level before the conversion to the luminance level after the conversion in the luminance level conversion process of step S204. The brightness level within the output dynamic range (range in which the normalized brightness is 1.0 or less) indicates that the output is output as it is without being converted (rather than not being converted, it is represented by a linear function of slope 1. It can also be considered that the transformation was performed so as to be a mapping.) On the other hand, for the luminance level exceeding the output dynamic range (the range in which the normalized luminance is larger than 1.0), the mapping is expressed or approximated by a linear function whose slope is -1 with respect to the logarithm of the luminance level. Conversion is performed. As a result, for example, the standardized input luminance 100 (corresponding to 10000 nits in the input image of the PQ method) is converted into the normalized output luminance 0.01 (corresponding to 1 nit).

By such a luminance level conversion, it is possible to faithfully reproduce the luminance of the input image 12 for the pixels in the output dynamic range. On the other hand, for pixels exceeding the output dynamic range, an output image that can be easily distinguished from the pixels within the output dynamic range and that the change in gradation can be visually recognized can be obtained. Therefore, the user can preferably use the image display device 11 having a limited dynamic range to check the brightness and gradation of the HDR image, check the pixel region that is overranged, and the like.

As shown in FIG. 4, the luminance level conversion for the overrange is preferably a mapping represented or approximated by a linear function of slope -1. However, this mapping is only a preferable example, and the same effect can be obtained by using other mappings. Examples of other mappings are shown below.

Reference numeral 50 in FIG. 5 is an example of a mapping represented or approximated by a linear function having a slope smaller than -1. By performing the conversion shown in the map 50, the effect of increasing the contrast of the gradation of the overrange can be obtained. Therefore, if you want to highlight the overranged area in the image, the mapping 50 is suitable.

Reference numeral 51 in FIG. 5 is an example of a mapping represented or approximated by a linear function having a gradient greater than -1 (since it is assumed to be a monotonic decreasing function, the gradient is smaller than 0). By performing the conversion shown in the mapping 51, the effect of weakening the contrast of the gradation of the overrange can be obtained. Therefore, when it is desired to display the overrange region in the image in a soft tone, the mapping 51 is suitable.

FIG. 6 is an example of using a mapping other than a linear function. Reference numeral 60 in FIG. 6 is an example of a mapping represented or approximated by a piecewise linear function. In the piecewise linear function, the domain is divided into a plurality of divisions, and each division is represented by a linear function. In FIG. 6, the domain (the range in which the standardized input luminance is greater than 1.0) is divided into two categories, and the slope of the section with the lower brightness level is made smaller than the slope of the section with the higher brightness level. This is an example. By performing the conversion shown by the mapping 60, for example, an effect of highlighting a range in which the normalized input luminance is 1 to 10 and softly displaying a range exceeding 10 can be obtained. In general, overrange pixels tend to be distributed in a large range of 1000 nits or less, and very few pixels exceed 1000 nits. Therefore, by allocating a large number of output gradations to a range in which the brightness level is low even in the overrange, the visibility of the overrange can be further improved.

Reference numeral 61 in FIG. 6 is an example of a mapping represented or approximated by a curve (for example, an nth-order function (n is an integer of 2 or more)) (since it is assumed to be a monotonic decreasing function, it is convex upward. It is a curve). The same effect can be obtained by using such a mapping 61. Further, in the example of FIG. 6, an n-th order function is used such that when the normalized input luminance is 1.0, the normalized output luminance is 1.0 and the slope (derivative) is 0. .. As a result, the change in which the output brightness changes from an increase to a decrease gradually becomes so that a gradation display that gives priority to a natural impression can be realized before and after the upper limit of the output level.

In the examples of FIGS. 4 to 6, the luminance level conversion is performed so that the output luminance level is continuous between the range where the input luminance level is equal to or less than the upper limit value and the range where the input luminance level is larger than the upper limit value. That is, the mapping in the range where the standardized input luminance is 1.0 or less (linear function of inclination 1) and the mapping in the range where the normalized input luminance is greater than 1.0 are continuous. As a result, the display brightness does not become discontinuous before and after the upper limit of the output level, so that the unnaturalness of the display can be alleviated.

On the other hand, as shown in FIG. 7, the output luminance level may be intentionally discontinuous between the range where the input luminance level is equal to or less than the upper limit value and the range where the input luminance level is larger than the upper limit value. As a result, the display brightness changes significantly before and after the upper limit of the output level, so that the user can easily distinguish between the pixels in the output level and the pixels in the overrange.

As shown in FIGS. 4 to 7, in the brightness level conversion method, the brightness level f (L1) obtained by converting the target brightness level L1 included in the overrange is higher than the target brightness level L1. Anything that is converted so as to be higher than the luminance level f (L2) obtained by converting the luminance level L2 may be used. That is, when the threshold value (upper limit value of the output dynamic range) is Lth,
If Lth <L1 <L2, then Lth> f (L1)> f (L2)
It is preferable that the brightness level conversion method satisfies the above.

4 to 7 show conversion processing so that the input and output luminance levels are equal in the range where the input luminance level is lower than the upper limit value (threshold value), that is, the range where the normalized input luminance is less than 1.0. An example (that is, no brightness level conversion is performed) is shown. However, the present invention is not limited to this, and conversion processing may be performed so that the input luminance and the output luminance are different in the range where the input luminance level is lower than the upper limit value (threshold value). In the brightness level conversion method, the brightness level f (L3) obtained by converting the target brightness level L3 not included in the overrange is the brightness obtained by converting the brightness level L4 higher than the target brightness level L3. Anything may be converted so as to be lower than the level f (L4). That is, when the threshold value (upper limit value of the output dynamic range) is Lth,
If L3 <L4 <Lth, then f (L3) <f (L4) <Lth
It is preferable that the brightness level conversion method satisfies the above.
The specific conversion method can be changed according to the content and visibility of the image.

When the pixels of the input image 12 are composed of a plurality of color components such as the three primary colors of RGB (Red, Green, Blue), a more suitable luminance level conversion method can be considered. The simplest method is to perform the luminance level conversion process independently for each of the plurality of color components. In another method, the above-mentioned luminance level conversion processing is performed only on the color component having the highest ratio in the pixel among the plurality of color components, and the ratio of the plurality of color components in the pixel is maintained before and after the conversion. It is a method of calculating the converted value of the remaining color components so as to be. The latter method has an advantage that the impression of appearance is maintained because the colors of the pixels before conversion and the pixels after conversion are almost the same.

Further, the pixels that are overranged may be converted to monochrome in addition to the luminance level conversion. Specifically, the image processing unit 103 may perform brightness level conversion on the overranged pixels, and then replace the pixels with monochrome pixels according to the converted brightness level. In the luminance level conversion process at this time, the above-mentioned method of determining the values of the remaining color components based on the conversion result of the maximum ratio color components is used so that the ratio of the color components is maintained before and after the conversion. Is preferable. Further, the pixels that are not overrange are expressed in the same manner as the input image 12. That is, when the input image 12 is a color image, the pixels that are not overranged are converted into color pixels. By displaying the overranged pixels in monochrome in this way, it becomes extremely easy to distinguish between the pixels in the output dynamic range and the overranged pixels.

Further, the image processing unit 103 may superimpose a predetermined image pattern (for example, a striped pattern called a zebra pattern) on the pixel region of the overrange. Even with such a display, it becomes extremely easy to distinguish between the pixels in the output dynamic range and the pixels in the overrange.

FIG. 8 is a schematic diagram showing an example of an output image generated by the conversion process according to the present embodiment when the input image is a gradation image.

FIG. 8A is a schematic diagram showing an input image. The input image is a gradation image that gradually changes from black to white. The horizontal axis indicates the brightness level at each position of the input image. The intermediate position indicates a position of 100 nits in the gradation image of the input image. The dynamic range of the input image is assumed to be 0 to 10000 nits. It is assumed that the upper limit value (threshold value) of the output dynamic range is 100 nits.

FIG. 8B shows an output image when the dynamic range (0 to 10000 nits) of the input image is compressed to the output dynamic range (0 to 100 nits) and converted. In this case, the brightness level of the entire image is reduced so that the region where the brightness level is 10000 nits in the input image (the right end portion of FIG. 8A) becomes 100 nits in the output image. In this case, the brightness of the region of the input image whose luminance level is included in the output dynamic range (that is, the pixel region of the luminance level of 0 to 100 nits in the input image) becomes dark as a whole, and the visibility of the user deteriorates. .. When processed in this way, the user can recognize whether or not there is a difference in the luminance level in the input image. Therefore, the user can recognize the objects included in the input image by looking at the output image. However, since the image becomes dark as a whole, it becomes difficult for the user to confirm the detailed representation of the image.

FIG. 8C shows an output image generated by converting a region (overrange region) of the input image whose brightness level is not included in the output dynamic range so as to clip it to the upper limit of the output dynamic range. Shown. In this case, in the input image, the region where the brightness level exceeds the threshold value (100 nits) is converted (clip) to the same brightness level. In the case of (c) of FIG. 8, the luminance level is converted to the brightness level corresponding to the threshold value in the input image. When processed in this way, the user cannot recognize an object or the like included in a region where the brightness level is higher than the threshold value (100 nits) in the input image by looking at the output image. However, in the input image, the region where the brightness level is equal to or less than the threshold value (100 nit) is expressed by the same brightness level as the input image. Therefore, the user can confirm the detailed representation of the image in the region where the brightness level is equal to or less than the threshold value (100 nits) in the input image.

FIG. 8D shows an output image when the positive / negative processing is executed on the overrange region of the input image as shown in the present embodiment. The positive / negative process is a process of converting so that the higher the luminance level of the input image is, the lower the luminance level of the output image is, and is also called an inversion process. For example, it is assumed that the relationship between the brightness level of the input image and the brightness level of the output image is the relationship shown in FIG. In this case, the region where the brightness level is equal to or less than the threshold value (100 nits) in the input image is expressed in the output image at the same brightness level as the input image. On the other hand, in the region where the luminance level is higher than the threshold value (100 nits) in the input image, the higher the luminance level of the input image, the lower the luminance level of the output image is converted. As a result, the user can confirm the detailed representation of the image in the region where the brightness level is equal to or less than the threshold value (100 nits) in the input image. Further, with respect to the overrange region of the input image, since the brightness level differs according to the difference in the brightness level of the input image, it is possible to recognize the objects included in the input image by looking at the output image.

FIG. 8 (e) shows an output image when the positive / negative process is executed on the overrange region of the input image as shown in the present embodiment. It is assumed that the relationship between the brightness level of the input image and the brightness level of the output image is the relationship shown in FIG. In this case as well, as in the example shown in FIG. 8D, the user can check the presence / absence of an object, the state of the pattern, etc. in the entire dynamic range of the input image by viewing the output image. It becomes.

FIG. 8 (f) shows an output image when a zebra pattern is further added to the overrange with respect to the output image shown in FIG. 8 (d). When the input image is a monochrome image, it may be difficult to determine at first glance whether or not it is in the overrange region when the brightness level is inverted. By superimposing the zebra pattern on the overrange area, the user can recognize the object in the overrange area and can easily recognize whether or not the object is in the overrange area.

According to the present embodiment described above, when an HDR image is confirmed by an image display device having a limited dynamic range, the image within the dynamic range of the image display device is faithfully reproduced and at the same time overranged. The image can be preferably visually recognized by the user. For example, according to the method of the present embodiment, it is possible to display all gradations up to 10000 nits of the PQ method by using a liquid crystal monitor having a contrast ratio of 1000: 1 and a maximum display brightness of 100 nits. Further, since it is not necessary to change the gradation conversion method according to the maximum brightness of the input image, it is possible to display HDR images of various methods such as PQ method, HLG method, and Log method.

Further, since the overranged image can be preferably visually recognized in the present invention, it is conceivable to use it in combination with a conventional image conversion method. One example is peaking, which is useful for adjusting the focus when shooting with a camera. Peaking is an image conversion method that makes it easier to visually recognize the degree of focusing by detecting the edge of an image and replacing the pixels in the corresponding region with a predetermined color (also referred to as "coloring the pixels"). However, for coloring by peaking, a color having high saturation and lightness is generally used in order to improve visibility. Since pixels of such colors have high gradation values, the brightness level may exceed the dynamic range of the display device. In that case, when the gradation conversion according to the present embodiment is performed on the image subjected to the image conversion by peaking, the gradation value of the colored pixel is replaced with a low gradation value, and the colored pixel becomes inconspicuous. May arise. Therefore, it is preferable to set the exclusion luminance range as a set value and replace the pixel whose luminance level before conversion corresponds to the exclusion luminance range with a predetermined luminance level (for example, the upper limit value of the output dynamic range). As a result, it is possible to maintain high visibility of coloring by peaking. This method can be used not only for peaking but also for superimposing an OSD (on-screen display) on an image.

(Other)
In the above embodiment, the display device is exemplified as an output device that outputs the converted image, but the scope of application of the present invention is not limited to the display device. As the output device, for example, a printing device (printer, multifunction device, etc.), a projector, or the like is also assumed.
Further, the threshold value can be set by the user regardless of the characteristics of the output device. A threshold value corresponding to the output device and display environment assumed by the user in image editing may be set.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

10: Image processing device 11: Image display device 12: Input image 13: Output dynamic range information 14: Output image 101: Image acquisition unit 102: Setting unit 103: Image processing unit

Claims (27)

A setting unit that sets the brightness level threshold, and
An image processing unit that converts the brightness level of at least a part of the first image to generate a second image,
With
In the image processing unit, the brightness level of the pixel of the second image corresponding to the pixel having the first brightness level larger than the threshold value of the first image is higher than that of the first brightness level of the first image. An image processing apparatus characterized in that the second image is generated so as to be higher than the brightness level of the pixel of the second image corresponding to the pixel having the high second brightness level. In the image processing unit, the brightness level of the pixels of the second image corresponding to the pixels having the third brightness level lower than the threshold value of the first image is higher than the third brightness level of the first image. The image processing apparatus according to claim 1, wherein the second image is generated so as to be lower than the brightness level of the pixel of the second image corresponding to the pixel having the high fourth brightness level. The image processing unit has a range in which the brightness level after conversion is equal to or lower than the threshold value and the brightness level before conversion is larger than the threshold value for pixels having a brightness level larger than the threshold value in the first image. A claim characterized in that the second image is generated by performing conversion processing so that the brightness level after conversion decreases with respect to the increase in the brightness level before conversion with respect to at least a part of the range. Item 2. The image processing apparatus according to Item 1 or 2. The image processing unit changes the brightness level after conversion from the brightness level before conversion to the brightness level after conversion for the pixels having a brightness level larger than the threshold value in the first image. The image processing apparatus according to claim 1 or 2, wherein the second image is generated by performing a conversion process such that the mapping of the image is monotonically reduced. The image processing unit performs conversion processing on the pixels having a brightness level lower than the threshold value in the first image so that the higher the brightness level before conversion, the higher the brightness level after conversion. The image processing apparatus according to claim 1, wherein the second image is generated. The image processing unit is characterized in that the pixel having a brightness level lower than the threshold value in the first image is subjected to conversion processing so that the brightness level before conversion and the brightness level after conversion are equal to each other. The image processing apparatus according to any one of claims 1 to 5. The second image is an image used for display on a display device.
The image processing device according to any one of claims 1 to 6, wherein the first image is an image having a dynamic range wider than the dynamic range of the display device. The image processing apparatus according to claim 7, wherein the first image is an image of a high dynamic range (HDR) standard. The image processing apparatus according to claim 7, wherein the luminance level threshold value is set to a value corresponding to the maximum luminance that can be displayed on the display device. The image processing apparatus according to claim 4, wherein the mapping is represented or approximated by a linear function with respect to the logarithm of the luminance level. The image processing apparatus according to claim 10, wherein the slope of the linear function is -1. The image processing apparatus according to claim 10, wherein the slope of the linear function is smaller than -1. The image processing apparatus according to claim 10, wherein the slope of the linear function is larger than -1. The image processing apparatus according to claim 4, wherein the mapping is represented or approximated by a piecewise linear function with respect to the logarithm of the luminance level. The image processing apparatus according to claim 4, wherein the mapping is represented by a curve with respect to a logarithm of a luminance level. The first image processing unit performs the first conversion so that the brightness level after conversion is continuous between a range in which the brightness level before conversion is below the threshold value and a range in which the brightness level before conversion is larger than the threshold value. The image processing apparatus according to any one of claims 1 to 15, wherein the brightness level of an image is converted. The image processing unit said that the brightness level after conversion is discontinuous between a range in which the brightness level before conversion is below the threshold value and a range in which the brightness level before conversion is larger than the threshold value. 1. The image processing apparatus according to any one of claims 1 to 15, wherein the brightness level of an image is converted. Each pixel of the first image has a plurality of color components, and has a plurality of color components.
The image processing apparatus according to claim 3 or 4, wherein the image processing unit performs the conversion processing on each of the plurality of color components. Each pixel of the first image has a plurality of color components, and has a plurality of color components.
The image processing unit
The conversion process is performed only on the color component having the highest ratio in the pixel among the plurality of color components.
The image processing apparatus according to claim 3 or 4, wherein the converted values of the remaining color components are calculated so that the ratio of the plurality of color components in the pixel is maintained. Any of claims 1 to 19, wherein the second image is an image in which pixels having a brightness level before conversion larger than the threshold value are replaced with monochrome pixels corresponding to the brightness level after conversion. The image processing apparatus according to item 1. The second image according to any one of claims 1 to 20, wherein the second image is an image in which a predetermined image pattern is superimposed on a region where the brightness level before conversion is larger than the threshold value. Image processing device. The image processing apparatus according to claim 21, wherein the predetermined image pattern is a striped pattern. The image processing unit performs conversion from the first image to the second image by using a data table in which the correspondence between the input value and the output value after conversion is defined for a plurality of input values. The image processing apparatus according to any one of claims 1 to 22. The image processing apparatus according to claim 23, wherein one of the plurality of correspondences defined in the data table is a correspondence relation regarding an input value corresponding to the threshold value. The image processing unit performs conversion processing so that the brightness level after conversion becomes a predetermined brightness level for the pixels having the brightness level included in the predetermined exclusion brightness range in the first image. The image processing apparatus according to any one of claims 1 to 24. Steps to set the brightness level threshold and
It has a generation step of converting the luminance level of at least a part of the first image to generate the second image.
In the generation step, the brightness level of the pixel of the second image corresponding to the pixel having the first brightness level larger than the threshold value of the first image is higher than that of the first brightness level of the first image. An image processing method characterized in that the second image is generated so as to be higher than the brightness level of the pixel of the second image corresponding to the pixel having a high second brightness level. A program for causing a computer to function as each part of the image processing apparatus according to any one of claims 1 to 25.

Images