JP2018007133A - Image processing device, control method therefor and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve visibility of an HDR region when displaying an HDR image on a monitor of which the gradation representation capability is low.SOLUTION: An image processing device comprises: a first luminance conversion part 104 and a second luminance conversion part 105 as conversion means for performing luminance conversion on an input image 101; and an output part 150 for outputting a luminance converted image to a display device 106. In the image processing device, the conversion means generates two images on which the luminance conversion has been performed by dividing a luminance region of the input image 101 into an SDR region and an HDR region and using a first gamma curve for performing the luminance conversion in such a manner that the SDR region is settled within a range of a gradation representation capability of the display device 106, and a second gamma curve for performing the luminance conversion in such a manner that the HDR region is settled within the range of the gradation representation capability of the display device 106.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing device, a control method thereof, and a program, and more particularly, to a technique for displaying a high dynamic range image on a display device.

  In recent years, an image processing apparatus having a high dynamic range (hereinafter referred to as “HDR”) function has been developed. In an imaging apparatus including such an image processing apparatus and a display device such as an HDR-compatible monitor, all processes from imaging to development and display can be performed in HDR.

  The HDR image can express a luminance range about 100 times that of an SDR (Standard Dynamic Range) image. On the other hand, not all monitors (display devices) that display images support HDR, and conventional SDR type monitors are also widely used. An SDR type monitor has a smaller gradation representation capability (the number of gradations that can be represented is smaller) than an HDR-compatible monitor. For this reason, when an HDR image is displayed on an SDR type monitor, whiteout, tone jump, or the like may occur in a high-luminance region of the HDR image, which may appear different from the actual image.

  Therefore, even for an SDR type monitor, there is a demand for grasping whether an HDR image at an intended luminance level is obtained in the process from shooting to storage of a shot image. In response to this demand, a technique has been proposed in which an input image is once converted into a luminance signal, an area having a predetermined luminance or higher is converted into a color signal, and the area is displayed with the color signal. (See Patent Document 1).

Japanese Patent No. 4181625

  However, in the method described in Patent Document 1, since the image displayed on the monitor is converted from the luminance signal to the color signal, the color appearance may be different between the image being captured and the actually displayed image. Yes, further improvement is desired.

  An object of the present invention is to provide an image processing apparatus capable of improving the visibility of an HDR area when an HDR image is displayed on a display apparatus having a small gradation expression capability.

  An image processing apparatus according to the present invention is an image processing apparatus comprising: a conversion unit that performs predetermined luminance conversion on an input image; and an output unit that outputs an image after luminance conversion by the conversion unit to a display device. When the gradation expression capability of the display device corresponds to the first luminance range and does not correspond to the second luminance range wider than the first luminance range, the conversion means The first image is obtained by performing luminance conversion using a first gamma curve in which a first region having a predetermined pixel value and a smaller pixel value in the input image is associated with the first luminance range. And performing a luminance conversion using a second gamma curve in which a second region having a pixel value larger than the predetermined pixel value in the input image is associated with the first luminance range. Generate two images, before Output means, and outputs the first image and the second image on the display device.

  ADVANTAGE OF THE INVENTION According to this invention, the visibility of an HDR area | region when displaying an HDR image with a display apparatus with small gradation expression capability can be improved, and, thereby, convenience can be improved.

1 is a schematic block diagram of an image processing apparatus according to a first embodiment of the present invention. It is a figure explaining the luminance conversion performed in the luminance conversion part with which an image processing apparatus is provided. It is a figure which shows the 1st example of the input image to an image processing apparatus, and the output image from an image processing apparatus. It is a figure which shows the 2nd example of the output image from the image processing apparatus which concerns on 1st Embodiment. It is a schematic block diagram of the image processing apparatus which concerns on 3rd Embodiment of this invention. It is a figure which shows the example of the image alternately displayed on a display apparatus by the image processing apparatus which concerns on 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, a first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to the first embodiment of the present invention. The image processing apparatus includes a codec unit 102, a first luminance conversion unit 104, a second luminance conversion unit 105, a display device 106, and an output unit 150. The output unit 150 includes an image analysis unit 111, a highlight unit 112, a region extraction unit 113, a resolution conversion unit 114, an image synthesis unit 115, and an output selection unit 116.

  The input image 101 is input to the codec unit 102, the first luminance conversion unit 104, the second luminance conversion unit 105, the image analysis unit 111, and the output selection unit 116. The input image 101 is an image that is captured by an image sensor (not shown), subjected to luminance conversion using a gamma curve corresponding to HDR, from which noise or the like due to the image sensor is removed. The input image 101 may be an image stored in a storage medium included in the image processing apparatus after shooting, or an image input from an external device (various electronic devices including an imaging device such as a digital camera). There may be. When the input image 101 is input from an external device, the input image may be an image stored in a memory (a storage medium such as a memory card or a hard disk) of the external device, or in real time when the image is captured by the imaging device. It may be an acquirable image. Note that the input image 101 is desirably data of 10 bits or more corresponding to HDR. This is because a tone jump occurs in luminance conversion at 8 bits, which is a standard number of gradations, and visibility may be lowered. By providing a bit depth of 10 bits or more, a dynamic range corresponding to HDR can be maintained.

  The codec unit 102 compresses and encodes the input image 101 in order to reduce and store the information amount of the input image 101. The compression encoding method is not particularly limited, and a well-known method can be used. H.264 (MPEG-4AVC) or the like is used. The stored image 103 compression-encoded by the codec unit 102 is once written in a memory (not shown) and then stored in a storage medium (not shown). Each of the first luminance conversion unit 104 and the second luminance conversion unit 105 performs luminance conversion by applying different gamma corrections to the input image 101, and outputs the image after luminance conversion to the output unit 150. Details of the first luminance conversion unit 104 and the second luminance conversion unit 105 will be described later.

The output unit 150 performs predetermined processing on the image output from each of the first luminance conversion unit 104 and the second luminance conversion unit 105 and compares the image with the input image 101 and outputs the image to the display device 106. To decide. The display device 106 is a known display device such as a TV monitor or an LCD panel. In the present embodiment, it is assumed that the display device 106 is an SDR type display device having a maximum luminance value of about 100 [Cd / m ² ]. For data transfer from the output unit 150 to the display device 106, a known standard such as HDMI (registered trademark) can be used. Internal processing in the output unit 150 will be described later.

  With reference to FIG. 2, the luminance conversion processing by the first luminance conversion unit 104 and the second luminance conversion unit 105 will be described. FIG. 2 is a diagram for explaining the luminance conversion performed by each of the first luminance conversion unit 104 and the second luminance conversion unit 105. 2A and 2B, pixel values (luminance values) that can be taken by the input image 101 are taken on the horizontal axes, and the pixel value 201 indicates the maximum luminance value of the SDR. In the dynamic range of the input image 101, an area that takes a pixel value less than or equal to the pixel value 201 is an SDR area (first area), and an area that takes a pixel value greater than the pixel value 201 is an HDR area (second area). Become. In each of the vertical axes of FIGS. 2A and 2B, a luminance range that can be displayed by the display device 106 is taken. In the present embodiment, since the gradation expression capability of the display device 106 is SDR (first luminance range), the luminance range is only the SDR, and the HDR range (the first luminance range) having a wider gradation expression capability than the SDR. 2 brightness range).

  FIG. 2A shows a gamma curve used for luminance conversion in the first luminance conversion unit 104. As described above, the pixel value of the input image 101 is divided into a plurality of luminance areas (here, two areas of an SDR area and an HDR area) with the pixel value 201 as a boundary. The first luminance conversion unit 104 performs luminance conversion using the first gamma curve 202 so that the dynamic range of the SDR region can be correctly displayed, and generates a first image. Specifically, the first luminance conversion unit 104 multiplies the pixel value of the input image 101 by a gamma curve that converts from HDR OETF to SDR OETF in the SDR region of the input image 101. As a result, the input image 101 can be converted into a pixel value suitable for the SDR EOTF characteristic of the display device 106, and can be displayed on the display device 106 at a correct luminance level. OETF is an abbreviation for Optical-Electro Transfer Function. EOTF is an abbreviation for Electro-Optical Transfer Function.

  For the HDR region in the input image 101, highlight processing is performed in the highlight unit 112 in the subsequent stage. Therefore, the first luminance conversion unit 104 converts the HDR region in the input image 101 to the maximum luminance value in the luminance range that can be expressed by the display device 106 so as to be saturated with respect to the luminance range of the display device 106.

  FIG. 2B shows a gamma curve used for luminance conversion in the second luminance conversion unit 105. The second luminance conversion unit 105 uses the second gamma curve 203 as a threshold value for the pixel value 201, which is the maximum luminance value of the display device 106, and performs luminance conversion so that the gradation of the HDR region can be expressed. To generate a second image. Specifically, the gradation of the SDR area of the input image 101 is compressed, and the pixel value of the input image 101 is multiplied by a gamma curve that converts the pixel value 201 so as to redraw the OETF in the HDR area. Thereby, the gradation of the HDR region in the input image 101 can be expressed by the display device 106. Note that the pixel value 201 of the input image 101 is converted into a pixel value 204. In the present embodiment, the compression is performed so that the gradation of the SDR region remains to some extent, but the OETF may be redrawn from the zero value (0) without leaving the gradation of the SDR region. In this case, the image in the SDR area is blacked out (blacked out) so that the subject is not visible in the SDR area, but more gradations can be assigned to the HDR area.

  Since the dynamic range of the HDR region is generally wider than the dynamic range of the SDR region, the second gamma curve 203 is set to draw a curve for compressing gradation. At this time, it may be possible to adjust which portion of the gradation in the HDR region remains. For example, when emphasizing the gradation in the region close to the SDR region, the curve in the HDR region of the second gamma curve 203 is a curve with a steep rise from the pixel value 201. On the other hand, when the gradation on the high luminance side is emphasized in the HDR region, the curve in the HDR region of the second gamma curve 203 is a curve with a gentle rise from the pixel value 201.

  In this embodiment, the luminance conversion of the input image 101 is performed by the first luminance conversion unit 104 and the second luminance conversion unit 105. However, the luminance conversion of the input image 101 is performed by one luminance conversion unit. Also good. In other words, one luminance conversion unit performs luminance conversion using each of the first gamma curve 202 and the second gamma curve 203, thereby generating two images (first image and second image). It is good also as a structure.

  Turning to the description of the output unit 150. The image analysis unit 111 analyzes the input image 101 and extracts a dynamic range, positions and sizes of main subjects, and the like. In addition, the image analysis unit 111 calculates an optimal synthesis position for the image synthesis unit 115 to perform image synthesis. The highlight unit 112 performs a highlight process such as replacing an HDR region that cannot be expressed by the display device 106 in the image output from the first luminance conversion unit 104 with, for example, diagonal display. Thereby, the visibility of the HDR region that cannot be expressed by the display device 106 can be improved.

  The region extraction unit 113 extracts an HDR region from the image output from the second luminance conversion unit 105 and outputs the HDR region to the resolution conversion unit 114. In the present embodiment, the area extraction unit 113 extracts a rectangular area including an area whose input pixel value is equal to or greater than the pixel value 204 (see FIG. 2B). However, when displaying the entire image output from the second luminance conversion unit 105 on the display device 106, the region extraction unit 113 performs a through process without extracting a rectangular region. The resolution conversion unit 114 performs resolution conversion on the image output from the region extraction unit 113 so that the converted resolution falls within the region calculated by the image analysis unit 111. The image synthesis unit 115 synthesizes the output from the highlight unit 112 and the output from the resolution conversion unit 114 to generate a synthesized image. The composite position is the image area calculated by the image analysis unit 111, and the image analysis unit 111 calculates, as the composite position, an area with the smallest main subject or HDR area among the four corners of the image. The output selection unit 116 can select an image to be output to the display device 106 from the input image 101 and the synthesized image by the image synthesis unit 115. In this embodiment, the output selection unit 116 outputs the composite image by the image composition unit 115 to the display device 106, and a mode of selecting and outputting the input image 101 will be described in a second embodiment to be described later. .

  Various processing units such as the first luminance conversion unit 104, the second luminance conversion unit 105, and the output unit 150 can be implemented by software (program) or hardware. It may be implemented in combination with wear. Each unit constituting the image processing apparatus may be a dedicated processor such as an ASIC that implements all or part of the processing by a logic circuit.

  Next, control executed by the image processing apparatus when the output unit 150 outputs an image to the display device 106 will be described by comparing the input image 101 and the output image. FIG. 3A is a diagram illustrating the input image 101. FIG. 3A is not an image displayed on the display device 106. The input image 101 is an HDR image, and gradation is maintained throughout the image. FIG. 3B is a diagram illustrating a display image 301 according to the first example output from the output unit 150 to the display device 106. The display image 301 is an example of an image in which the output image from the highlight unit 112 and the output image from the resolution conversion unit 114 are simultaneously displayed on the display device 106. As described above, in the present embodiment, since the output selection unit 116 displays the output image from the image synthesis unit 115 on the display device 106, the display image 301 is also an output image from the image synthesis unit 115.

  An area 303 in the display image 301 is an area in which gradation display cannot be performed because the display device 106 does not support HDR, and is an area that is displayed with diagonal lines as highlight processing by the highlight unit 112. A region 304 in the display image 301 is a rectangular region extracted by the region extraction unit 113 and subjected to resolution conversion by the resolution conversion unit 114, and the image analysis unit 111 sets the lower right corner to the combined position (overlapping of the rectangular regions). Display position). In the region 304, since the luminance conversion is performed in the second luminance conversion unit 105 so that the gradation of the HDR region remains, a subject (such as a bird and a branch) is displayed. Note that the area other than the area 303 and the area 304 in the display image 301 is an SDR area output through the highlight unit 112.

  FIG. 4 is a diagram illustrating a display image 401 according to the second example output from the output unit 150 to the display device 106. A display image 401 is another example of an image in which an output image from the highlight unit 112 and an output image from the resolution conversion unit 114 are simultaneously displayed on the display device 106. An area 403 in the display image 401 is an area in which gradation display cannot be performed because the display device 106 does not support HDR, similarly to the area 303 in the display image 301. It is an area | region where the diagonal line display was given. An area 404 in the display image 401 indicates an output image from the resolution conversion unit 114. The area 404 has not undergone area extraction processing in the area extraction unit 113 for the output image from the second luminance conversion unit 105, and the resolution conversion unit for the entire output image from the second luminance conversion unit 105. The output image after performing the resolution conversion process in 114 is shown. Since the area extraction processing by the area extraction unit 113 is not performed, the ratio of the area 404 in the display image 401 is larger than the ratio of the area 304 in the display image 301. Therefore, although the area hidden by the synthesis process with respect to the subject in the input image 101 increases, it is easy to grasp the positional relationship of the surrounding subjects. As described above, the image composition unit 115 may perform image composition by performing only the resolution conversion process by the resolution conversion unit 114 without performing the region extraction process by the region extraction unit 113.

  As described above, according to the present embodiment, the images subjected to luminance conversion suitable for the HDR region and the SDR region of the input image 101 are combined and displayed on the display device 106. Thereby, when an HDR image is displayed on the SDR type display device 106, the visibility of the HDR region can be improved, and the convenience for the user can be improved.

  Next, a second embodiment of the present invention will be described. In the first embodiment, the output selection unit 116 selects the output image from the image synthesis unit 115 and displays it on the display device 106. On the other hand, in 2nd Embodiment, it is set as the structure which switches the display image displayed on the display apparatus 106 according to the gradation performance which the display apparatus 106 can display. Therefore, the overall configuration of the image processing apparatus in the second embodiment is the same as that of the image processing apparatus (FIG. 1) described in the first embodiment, but the following points are different from the first embodiment.

  First, the display device 106 may be an SDR type or an HDR type, and outputs information including its own resolution or the like (hereinafter referred to as “auxiliary information”) to the output selection unit 116. It is assumed that it has a function of transmitting to the unit 116. Second, the output selection unit 116 acquires auxiliary information from the display device 106, and determines whether or not the display device 106 supports HDR based on the acquired auxiliary information. Third, the output selection unit 116 determines whether to output the input image 101 as it is to the display device 106 or to output the output image from the image composition unit 115 to the display device 106 based on the acquired auxiliary information. The image is output based on the determination result.

  Specifically, the output selection unit 116 acquires auxiliary information from the display device 106 in order to detect whether or not the display device 106 supports HDR. A well-known method can be used as a method for acquiring auxiliary information. For example, when communication based on the HDMI (registered trademark) standard is used, an auxiliary information transfer function based on EDID (Extended Display Identification Data) is used. The output selection unit 116 selects to output an output image from the image synthesis unit 115 when the gradation expression capability of the display device 106 is SDR (first luminance range) narrower than HDR. On the other hand, when the gradation expression capability of the display device 106 corresponds to HDR (second luminance range) wider than SDR, the output selection unit 116 selects the input image 101 to be output as it is. In this manner, by switching the image displayed on the display device 106 according to whether or not the display device 106 supports HDR, it is possible to display an image in accordance with the gradation performance of the display device 106, and the HDR region Visibility can be improved.

  Next, a third embodiment of the present invention will be described. In the first embodiment, the configuration in which the image synthesized by the image synthesis unit 115 is displayed on the display device 106 has been described. In contrast, in the third embodiment, two images are alternately displayed on the display device 106 at regular time intervals. FIG. 5 is a block diagram showing a basic configuration of an image processing apparatus according to the third embodiment of the present invention. The image processing apparatus according to the third embodiment is different from the image processing apparatus according to the first embodiment in the configuration of the output unit, but the other configurations are the same. Therefore, the same components (blocks) of the image processing apparatus according to the third embodiment as those of the image processing apparatus according to the first embodiment are denoted by the same reference numerals and described. Omitted.

  The image processing apparatus according to the third embodiment includes an output unit 510 having a highlight unit 112 and an output selection unit 516. The input image 101 is input to the codec unit 102, the first luminance conversion unit 104, the second luminance conversion unit 105, and the output selection unit 516. The processes of the first luminance conversion unit 104, the second luminance conversion unit 105, and the highlight unit 112 are as described in the first embodiment.

  When the display device 106 is known in advance to be of the SDR type as in the first embodiment, the output selection unit 516 outputs the output image from the highlight unit 112 and the output image from the second luminance conversion unit 105. Are selected at predetermined intervals and alternately output to the display device 106. Control similar to this is also performed when the output selection unit 516 acquires information related to the resolution performance of the display device 106 and, as a result, determines that the display device 106 is the SDR type. When the display device 106 is the SDR type, the output image switching timing in the output selection unit 516 can be set to a time that allows the user to confirm the subject in each image, for example, in units of several seconds (3 seconds or the like). . On the other hand, the output selection unit 516 outputs the input image 101 to the display device 106 when the display device 106 is an HDR type as a result of acquiring information on the resolution performance of the display device 106.

  FIG. 6 is a diagram illustrating an example of images displayed alternately on the display device 106. 6A shows a display image 601 output from the highlight unit 112 and displayed on the display device 106, and FIG. 6B is output from the second luminance conversion unit 105 and displayed on the display device 106. A display image 602 displayed in FIG. By alternately displaying the display images 601 and 602 every predetermined time, the user can compare the SDR area and the HDR area at the same angle of view, so that the positional relationship between the SDR area and the HDR area can be grasped. It becomes easy.

  As described above, in the present embodiment, images subjected to luminance conversion suitable for the HDR region and the SDR region are alternately displayed on the display device 106 at predetermined time intervals. Thereby, when an HDR image is displayed on the SDR type display device 106, the visibility of the HDR region can be improved, and the convenience for the user can be improved.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included. Furthermore, each embodiment mentioned above shows only one embodiment of this invention, and it is also possible to combine each embodiment suitably. The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

104 first luminance conversion unit 105 second luminance conversion unit 106 display device 111 image analysis unit 112 highlight unit 113 region extraction unit 114 resolution conversion unit 115 image synthesis unit 116,516 output selection unit 150,510 output unit 202 first 1 gamma curve 203 second gamma curve

Claims (11)

Conversion means for performing predetermined luminance conversion on the input image;
Output means for outputting an image after luminance conversion by the converting means to a display device,
When the gradation expression capability of the display device corresponds to the first luminance range and does not correspond to the second luminance range wider than the first luminance range,
The conversion means performs luminance conversion by using a first gamma curve in which a first region having a predetermined pixel value and a smaller pixel value in the input image is associated with the first luminance range. In addition to generating a first image, luminance conversion is performed using a second gamma curve in which a second region having a pixel value larger than the predetermined pixel value in the input image is associated with the first luminance range. To generate a second image,
The image processing apparatus, wherein the output means outputs the first image and the second image to the display device.   The image processing apparatus according to claim 1, wherein the output unit simultaneously displays the first image and the second image on the display device.   The image processing apparatus according to claim 2, wherein the output unit converts the resolution of the second image and displays the converted image on the first image.   The output means cuts out an area generated by luminance conversion of the second area in the input image of the second image from the second image and displays the area on the first image. The image processing apparatus according to claim 3, wherein:   The output means superimposes the second image on an image area after the luminance of the first area in the input image is converted using the first gamma curve in the first image. The image processing apparatus according to claim 3 or 4.   The image processing apparatus according to claim 1, wherein the output unit displays the first image and the second image alternately on the display device. The converting means, when generating the first image using the first gamma curve, converts the second region in the input image to a maximum luminance value of the first luminance range,
The output means, when displaying the first image on the display device, performs a highlight process on an image area of the first image whose luminance has been converted to the maximum luminance value. The image processing apparatus according to any one of 1 to 6.   The output means acquires information relating to the gradation representation capability of the display device, and when the gradation representation capability of the display device corresponds to the second luminance range, the input image is displayed as it is. The image processing apparatus according to claim 1, wherein the image processing apparatus outputs the image to an apparatus. A method for controlling an image processing apparatus for outputting an image to a display device, comprising:
Obtaining information on the gradation representation capability of the display device;
Outputting the input image to the display device when the gradation representation capability of the display device corresponds to a pixel value of the input image input to the image processing device;
In the case where the gradation expression capability of the display device corresponds to a luminance range narrower than the pixel value of the input image, a first region that takes a predetermined pixel value and a smaller pixel value in the input image A first image is generated by performing luminance conversion using a first gamma curve corresponding to the luminance range, and a second region having a pixel value larger than the predetermined pixel value in the input image Generating a second image by performing luminance conversion using a second gamma curve corresponding to the luminance range;
A method of controlling the image processing apparatus, comprising: outputting the first image and the second image to the display device.   A program that causes a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 8. Conversion means for performing predetermined luminance conversion on the input image;
Output means for outputting an image after luminance conversion by the converting means to a display device,
The converting means divides a luminance region of the input image into a plurality of luminances, and one of the plurality of divided luminance regions with respect to the entire input image falls within a range of gradation expression capability of the display device. An image processing apparatus that generates a plurality of images to be output to the display device by performing conversion on all of the plurality of divided luminance regions.