JP2018067758A - Image encoding device and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To adaptively switch whether or not to use a thing obtained by local decoding as a LDR image to be used in creating auxiliary data for returning accuracy of the LDR image to the accuracy of a HDR image when the HDR image is encoded.SOLUTION: An image encoding device that encodes a HDR image and that creates auxiliary data for returning accuracy of a LDR image to the accuracy of the HDR image comprises: a LDR image creation unit that creates the LDR image from the encoding target HDR image; an encoding unit that irreversibly encodes the created LDR image; a decoding unit that decodes the encoded LDR image; an auxiliary data creation unit that creates the auxiliary data for returning the accuracy of the LDR image to the accuracy of the HDR image on the basis of the HDR image and the LDR image; and a determination unit that determines which of the LDR image created by the LDR image creation unit or the LDR image obtained using the decoding by the decoding unit is used as the LDR image used by the auxiliary data creation unit.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image encoding technique, and more particularly to an encoding technique for a digital image (hereinafter, HDR image) having a high dynamic range (HDR).

  A representative data format for the distribution of still images is JPEG. JPEG is sometimes used as a data format of an output file of a digital camera, and there are many softwares that use it, including web browsers. A still image to be subjected to JPEG compression is 8-bit (256 gradation) image data per component.

  On the other hand, since human vision can recognize a contrast ratio of up to 1: 10000, the gradation expression of the conventional JPEG image can make a great difference from the actual appearance. Accordingly, attention has been paid to an HDR image represented by a bit number exceeding 8 bits per component (for example, 32 bits). Even recent digital cameras and smartphones are generally equipped with an HDR mode. Software applications that can easily create HDR images having 32-bit data per component are also increasing.

  ISO / IEC 18477-2 (JPEG XT Part 2) stipulates a technique capable of viewing an HDR image having 32-bit data per component with a conventional JPEG viewer. Patent Document 1 also discloses a similar technique. The HDR image is tone-mapped to a smaller dynamic range image (Low Dynamic Range: LDR image) of 8 [bit / component], and JPEG compression is performed. Then, the ratio of the original HDR image and LDR image is separately created as HDR auxiliary data, and the HDR auxiliary data and the JPEG code data of the LDR image are saved in one file. The place where the HDR auxiliary data is stored is in the JPEG application marker (APP11). A general JPEG viewer ignores unsupported APP markers and decodes them. Therefore, for example, when a JPEG viewer that cannot recognize APP11 as a marker storing HDR auxiliary data receives a JPEG file having HDR auxiliary data, only the LDR image is decoded and displayed. On the other hand, the HDR image viewer capable of recognizing that the HDR auxiliary data is stored in the APP 11 can decode and display the original HDR image by combining the LDR image and the HDR auxiliary data in the APP 11 marker.

Japanese Patent No. 05086067

  Here, a method of creating HDR auxiliary data will be considered. First, an HDR image encoding process compliant with ISO / IEC 18477-2 will be described with reference to FIG. 1A. Since each process is described in ISO / IEC 18477-2, detailed description is omitted and only a brief description is given here.

  First, the process up to creation of JPEG code data of an LDR image will be described. When the HDR image 101 is given as an input, the HDR image 101 is converted by the tone mapping unit 102 into an LDR image in a linear color space of 256 gradations of 0 to 255 per component. The color conversion unit 112 converts the linear LDR image in the linear color space into the sRGB color space. The image data in the sRGB space is processed by the RGB / YCC conversion unit 113 and the sub-sampling unit 114, supplied to the JPEG compression unit 115, and JPEG-encoded.

  Next, a procedure for creating HDR auxiliary data will be described. The Y calculation unit 103 calculates a luminance component (Y component) from the HDR image 101. On the other hand, the Y calculation unit 105 also calculates a luminance component (Y component) from the linear LDR image created by the tone mapping 102. The noise adding unit 106 adds noise to each luminance component calculated by the Y calculating unit 105. Then, the Yratio calculating unit 107 calculates the luminance component ratio (Yratio) by dividing (dividing) the HDR Y component by the LDR Y component to which noise is added. The Yratio mapping unit 108 maps the calculated Yratio to the range 0 to 255 and supplies it to the JPEG compression unit 109 as the Y component of the HDR auxiliary data. Further, the Yratio calculation unit 107 supplies the calculated Yratio to the RGB scale conversion unit 104. The RGB scale conversion unit 104 converts the RGB color components of the HDR image using Yratio. The difference acquisition unit 110 acquires a difference value between the RGB value of the scale-converted HDR image and the RGB value of the linear LDR image output from the tone mapping 102. The color difference calculation 111 calculates the color difference component of the HDR auxiliary data based on the difference value, and supplies it to the JPEG compression unit 109. The JPEG compression unit 109 creates JPEG code data of the HDR auxiliary data using the luminance component input from the Yratio mapping unit 108 and the color difference component input from the color difference calculation 111.

  As described above, JPEG code data for LDR and JPEG code data for HDR auxiliary data are generated. The output unit 116 creates one JPEG file together with both code data and information necessary for decompression from the LDR image to the HDR image, and outputs it as JPEG XT data 117.

  The above is encoding processing of an HDR image by the method of ISO / IEC 18477-2. The LDR image used when creating the HDR auxiliary data is data obtained from the tone mapping unit 102. However, the LDR image that is finally output as the JPEG XT data 117 is output from the JPEG compression unit 115 that is irreversible compression, and is data after deterioration. Therefore, high accuracy cannot be expected for an HDR image obtained by receiving and decoding JPEG XT data 117.

  For such a problem, Patent Document 1 discloses a method of using an LDR image obtained by once decoding an LDR image encoded by the JPEG compression unit 115 for creating HDR auxiliary data. . FIG. 1B shows the configuration disclosed in Patent Document 1. The same components as those in FIG. 1A are denoted by the same reference numerals, and the description thereof is omitted. The difference from FIG. 1A is that the result of the JPEG compression unit 115 of the LDR image is decoded by the JPEG decoding unit 119. The decoded LDR image is converted into an LDR image in a linear color space by the color conversion unit 118, and the result is supplied to the Y calculation unit 105 and the difference acquisition unit 110.

  In this way, on the HDR image encoding side, a JPEG-compressed LDR image is locally decoded by the JPEG decoding unit 119, so that the difference between the LDR image at the time of HDR auxiliary data creation and the LDR image used at the time of HDR decoding can be reduced. It will be resolved. Therefore, a more accurate HDR image can be decoded on the JPEG XT data reception side (decoded device for encoded HDR image).

  However, local decoding in this way causes a problem of increasing the processing load when creating an HDR image file. In particular, when considering a use case of synthesizing and compressing an HDR image in a mobile device such as a digital camera or a smartphone, it is a big problem to reduce the processing load and generate a high-quality HDR image file.

  The present invention has been made in view of the above, and intends to provide a technique for more efficiently encoding an HDR image in consideration of the balance between the load of encoding processing and the image quality of the decoded HDR image. Is.

In order to solve this problem, for example, an image encoding device of the present invention has the following configuration. That is,
An image encoding device for generating auxiliary data for encoding an HDR image and returning an LDR image to the accuracy of the HDR image,
LDR image generation means for generating an LDR image from an HDR image to be encoded;
Encoding means for irreversibly encoding the generated LDR image;
Decoding means for decoding the encoded LDR image;
Auxiliary data generating means for generating auxiliary data for returning the LDR image to the accuracy of the HDR image based on the HDR image and the LDR image;
The LDR image used by the auxiliary data generation unit includes a determination unit that determines which one of the LDR image generated by the LDR image generation unit and the LDR image obtained by decoding by the decoding unit is used.

  According to the present invention, whether or not to use an LDR image obtained by local decoding as an LDR image used when creating auxiliary data for returning the LDR image to the HDR image accuracy when encoding the HDR image is used. Switch adaptively. As a result, it is possible to cope with both the processing load and the image quality related to encoding.

The block diagram of a HDR image encoding process. The block diagram of a HDR image encoding process. The block diagram of the HDR image coding apparatus in 1st Embodiment. The figure which shows the compression flow of the HDR image in 1st Embodiment. The figure which shows the creation flow of the linear LDR image in 1st Embodiment. The figure which shows the creation flow of HDR auxiliary data in 1st Embodiment. The figure which shows the tone curve in 1st Embodiment. The figure which shows the compression flow of the HDR image in the modification of 1st Embodiment. The figure which shows the LDR image compression flow of the macroblock unit in the modification of 1st Embodiment. The figure which shows the creation flow of the linear LDR image of the macroblock unit in the modification of 1st Embodiment. The figure which shows the compression flow of the HDR image in 2nd Embodiment. The figure which shows the compression flow of the HDR image in 3rd Embodiment. The figure which shows the compression flow of the HDR image in the modification of 3rd Embodiment. The figure which shows the LDR image compression flow of the macroblock unit in the modification of 3rd Embodiment. The figure which shows the compression flow of the HDR image in 4th Embodiment. The figure which shows the compression flow of the HDR image in the modification of 4th Embodiment. The figure which shows the LDR image compression flow of the macroblock unit in 5th Embodiment. The figure which shows the comparison range of the DCT coefficient in the modification of 5th Embodiment. The figure which shows the LDR image compression flow of the macroblock unit in the modification of 5th Embodiment.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the embodiment, for ease of explanation, it is assumed that the HDR image data to be encoded is 1920 × 1080 pixel RGB color data, and each component (color) is represented by a 16-bit integer value. . However, the configuration of components other than RGB (for example, gray single color or YMC) may be used, and the type of color space and the number of components are not limited. The number of bits of one component is not limited to 16 bits, and the number of bits is not particularly limited as long as the number of bits exceeds 8 bits, such as 12 bits or 32 bits. Further, the value is not limited to an integer value, and may be a 32-bit real value. In other words, any format of the input image may be used as long as the HDR contrast ratio can be expressed. It should be understood that the above numerical values are only specific examples that facilitate understanding.

[First Embodiment]
FIG. 2 is a block diagram of an HDR image coding apparatus according to the present embodiment. In the figure, reference numeral 201 denotes a CPU, which is involved in all processing of each configuration. Reference numeral 202 denotes an input unit that inputs an instruction from a user, HDR image data to be encoded, and the like. This includes pointing systems such as keyboards and mice. The generation source of the HDR image data to be encoded is an imaging device, but the imaging device may be connected to the input unit 202, or a storage medium storing the HDR image data to be encoded is connected to the input unit 202. It does not matter. In some cases, a server that holds HDR image data exists on the network, and the input unit 202 may download from the server. Reference numeral 203 denotes an HDR auxiliary data generation unit, and 204 denotes an LDR image generation unit. Reference numeral 205 denotes a JPEG codec unit (lossy encoding unit) that converts 8-bit non-compressed image data per component into JPEG code data, or decodes JPEG code data and converts it into uncompressed image data. A memory 206 includes a ROM and a RAM and provides the CPU 201 with programs, data, work areas, and the like necessary for processing. Reference numeral 207 denotes a display unit, which typically uses a liquid crystal display or the like. An accumulation unit 208 accumulates image data, programs, and the like. Usually, a hard disk or the like is used. Further, it is assumed that a control program necessary for the processing of the subsequent flowcharts is stored in the storage unit 208 or stored in the ROM of the memory 206. If the data is stored in the storage unit 208, the data is once read into the RAM in the memory 206 and then executed. In addition to the above, there are various components of the system configuration, but since it is not the main point of the present embodiment, the description thereof is omitted.

  The method for generating HDR auxiliary data in the present embodiment is based on the method defined in ISO / IEC 18477-2. In other words, this is a method that uses the luminance component ratio (Y ratio) between HDR image data and LDR image data. Using the Y ratio, the LDR image data is returned to the HDR range, and a difference of the color difference component from the original HDR image data is generated. Then, the Y ratio and the color difference component are converted into 8 bits and JPEG encoded.

  A method of compressing the HDR image data input to the encoding apparatus by dividing it into an LDR image and HDR auxiliary data will be described with reference to FIG. In addition, detailed description here is abbreviate | omitted about the part similar to the process prescribed | regulated by ISO / IEC 18477-2.

  In step S <b> 301, the CPU 201 inputs the HDR image data to be encoded via the input unit 202 and stores it in the memory 206. In step S302, the CPU 201 controls the LDR image generation unit 204 to perform tone mapping processing on the input HDR image data. The LDR image generation unit 204 stores the generated linear LDR image data in the memory 206. The LDR image generation unit 204 counts and holds the number of low-luminance pixels when creating a linear LDR image. Details of the low luminance pixel count process will be described later.

  In step S303, the CPU 201 calculates the ratio RLP of the number of low luminance pixels from the number of low luminance pixels obtained above. That is, if the number of low-luminance pixels of the entire image is C, the width of the input HDR image is w pixels, and the height is h pixels, RLP = C / (w × h) can be calculated (in the embodiment, w = 1920, h = 1080).

  In step S304, the CPU 201 compares the calculated RLP with a predetermined threshold Th_RLP. If RLP is larger than threshold Th_RLP, the process proceeds to step S305; otherwise, the process proceeds to step S313. In the embodiment, the threshold Th_RLP is set to “0.5”. That is, when the low luminance pixels occupy more than half of the entire image, the HDR image to be encoded is determined as an image having a large proportion of low luminance pixels, and the process proceeds to step S305. In steps S305 and S313, information indicating whether or not to use the locally decoded LDR image when creating the HDR auxiliary data is set in a flag Flag reserved in the memory 206 in advance. Specifically, in step S305, the CPU 201 sets, to the flag Flag, a value “1” indicating that the locally decoded LDR image is used when creating the HDR auxiliary data. In step S313, the CPU 201 sets, to the flag Flag, a value “0” indicating that HDR auxiliary data is to be created using the linear LDR image created in step S302. Therefore, when the process proceeds to S313, local decoding of the LDR image is not performed.

  Next, the processing after step S305 when the LDR image is locally decoded will be described. In step S305, when the CPU 201 sets the flag Flag to “1”, the process proceeds to step S306. In step S306, the CPU 201 acquires sub-sampling information and information related to the quantization step as an LDR image encoding condition. Next, in step S307, the CPU 201 performs color conversion of the linear LDR image. That is, the CPU 201 performs gamma processing on the LDR image in the linear color space, converts it into image data in the sRGB color space, and then performs RGB / YCC conversion. Further, the CPU 201 performs sub-sampling of the color difference components Cb and Cr when sub-sampling is necessary according to the information acquired in step S306. Since these processes are general processes and are described in ISO / IEC 18477-2, detailed description thereof is omitted here. In step S308, the CPU 201 controls the JPEG codec unit 205, divides the YCC data of the LDR image obtained in step S307 into macroblocks, and performs DCT conversion and quantization processing. Since these processes are also general processes, a detailed description thereof is omitted here. In step S309, the CPU 201 controls the JPEG codec unit 205 to entropy-encode the quantized DCT coefficient using a method defined in JPEG (ISO / IEC 10918). The JPEG codec unit 205 temporarily stores the generated encoded data in the memory 206.

  Next, in step S310, the CPU 201 determines whether or not the value of the flag Flag is “1”. That is, if it is determined in step S304 that the LDR image that has been locally decoded is used, the value of Flag is 1, so the CPU 201 advances the process to step S311. If the Flag value is 0, the CPU 201 advances the process to step S317. In this description, since step S305 has been performed, the processing proceeds to step S311.

  In step S311, the CPU 201 controls the JPEG codec unit 206, and obtains YCC data of the LDR image after local decoding by performing inverse quantization and inverse DCT on the quantized DCT coefficient obtained in step S308. To do. In local decoding, although it may start from encoded data, since entropy (Huffman encoding) itself in JPEG is reversible, local decoding is performed from a coefficient after quantization in the middle. A detailed description of this process is omitted as in the case of DCT / quantization. In step S312, the CPU 201 performs reverse color conversion on the YCC data acquired in step S311 to generate a linear LDR image, stores it in the memory 206, and advances the processing to step S314. That is, the CPU 201 acquires RGB image data that has been locally decoded by YCC / RGB conversion, and further acquires linear RGB image data that has been locally decoded by performing inverse gamma processing. The memory 206 already stores the linear LDR image created in step S302, but the data is overwritten with the result of local decoding. In step S314, the CPU 201 controls the HDR auxiliary data generation unit 203 to input the local decoded linear RGB image obtained in step S312 and return from LDR to HDR accuracy (8 bits to 16 bits). Create HDR auxiliary data. Details of this processing will be described later with reference to FIG. Next, in step S315, the CPU 201 outputs the HDR auxiliary data created in step S314 by the method described in ISO / IEC 18477-2. In step S316, it is determined whether or not the flag Flag is “1” (indicating that local decoding is to be performed). If the value of the flag Flag is “1”, it is determined that JPEG code data of the LDR image has already been created in the memory 206, and the CPU 201 advances the process to step S317. If the value of the flag Flag is “0”, the process proceeds to step S306 to proceed to a process for creating JPEG code data of an LDR image. In step S317, the CPU 201 outputs encoded data of the LDR image by a method described in ISO / IEC 18477-1, and ends this flow.

  Next, a case will be described in which it is determined in step S304 that the HDR image to be encoded is an image with a low ratio of low luminance pixels.

  In step S313, the CPU 201 sets “0” in the flag Flag, that is, sets not to perform local decoding. In steps S314 and 315, the CPU 201 controls the HDR auxiliary data generation unit 203 to generate and output HDR auxiliary data using the linear LDR image obtained in step S302 as an input. Next, in step S316, the CPU 201 determines whether or not the value of the flag Flag is “1”. If “1”, the process proceeds to step S317, and if “0”, the process proceeds to step S306. Since the description here is from step S313, the process proceeds to step S306, and the CPU 201 performs LDR image encoding processing. Steps S306 to S309 are the same as described above. In step S310, the flag Flag is referred again. Since the process of step S313 is performed this time, the flag Flag is “0”. Accordingly, since there is no need to perform local decoding, the process proceeds to step S317, the encoded data of the LDR image is output, and this flow ends.

  Next, the linear LDR image creation processing by the LDR image generation unit 204 in step S302 will be described with reference to FIG.

  In step S401, the LDR image generation unit 204 acquires a tone curve for generating LDR image data from the HDR image data to be encoded. The tone curve is a calculation formula or table used for tone mapping processing for compressing an HDR image to a predetermined dynamic range. Various methods have been proposed for tone mapping processing, including methods that take into account human visual characteristics and display device characteristics. In the present embodiment, for each color, a mathematical expression that draws a tone curve as shown in FIG. 6A is acquired. Any other information can be used as long as the information can specify the tone curve, such as a look-up table or the values of several points sampled on the tone curve shown in FIG.

  In step S402, the LDR image generation unit 204 acquires the width w and height h of the image. In this embodiment, the width w = 1920 [pixel] and the height h = 1080 [pixel]. In step S403, the LDR image generation unit 204 initializes the counter C for counting low luminance pixel values by substituting zero. In step S <b> 404, the LDR image generation unit 204 initializes the variable y for counting the height direction of the image by substituting zero. In step S405, the LDR image generation unit 204 initializes the variable x for counting the width direction of the image by substituting zero. In the present embodiment, the following processing is performed in raster order from the upper left corner (0, 0) of the image.

  In step S406, the LDR image generation unit 204 acquires the pixel value PH at the pixel position represented by (x, y) in the HDR image. In the present embodiment, each pixel value PH is composed of 16-bit R, G, and B values. In step S407, the pixel value PH is mapped to the pixel value PL (8 bits) of the LDR image using the tone curve acquired in step S401. Then, the LDR image generation unit 204 stores the pixel value PL at the pixel position (x, y) in the area for the linear LDR image secured in the memory 206. Next, in step S408, the LDR image generation unit 204 calculates the luminance value Y_hdr at the pixel position (x, y) from the value of the pixel value PH. In the present embodiment, the luminance Y_hdr is calculated by a conversion formula defined by ISO / IEC 18477-1 as in the case of 8-bit JPEG compression. In step S409, the LDR image generation unit 204 compares Y_hdr calculated in step S407 with a predetermined brightness threshold Th_Y. If the luminance value Y_hdr is smaller than the threshold value Th_Y, the process proceeds to step S410; otherwise, the process proceeds to step S411. In the present embodiment, as shown in FIG. 6, a value “4550” at which the mapped LDR pixel value is “96” is set as the threshold Th_Y. Therefore, if the luminance value Y_hdr is smaller than “4550”, the process proceeds to step S410, and the LDR image generation unit 204 adds “1” to the counter C that counts the low luminance pixels. If the luminance value Y_hdr is 4550 or more, the process proceeds to step S411 without performing the process in step S410. In step S411, the LDR image generation unit 204 completes tone map processing and low-luminance pixel determination for the pixel position (x, y), and thus adds “1” to the horizontal counter x of the image. To do. In step S412, it is determined whether the horizontal counter x of the image has the same value as the image width w. If it is the same value, it is determined that processing has been performed up to the right end of the image, and the process proceeds to step S413. Otherwise, it is determined that the right end has not been reached yet, and the process returns to step S406. In step S413, the LDR image generation unit 204 adds “1” to the variable y in the height direction of the image. In step S414, the LDR image generation unit 204 compares the variable y with the height h of the image. If they are the same, the process is terminated assuming that all the pixels of the image have been processed. . If they are different, the process returns to step S405 to continue the remaining pixels.

  Through the above processing, the LDR image (linear LDR image) in the linear color space having the same size as the HDR image, the luminance component Y_hdr of the HDR image, and the low luminance pixel number C can be obtained. The linear LDR image is 8-bit integer value (0 to 255) image data per component.

  Next, with reference to FIG. 5, the HDR auxiliary data creation processing by the HDR auxiliary data generation unit 203 in step S314 will be described. As a method for creating HDR auxiliary data, the method described in ISO / IEC 18477-2 may be used as it is, and a detailed description thereof will be omitted.

First, in step S501, the HDR auxiliary data generation unit 203 acquires the luminance value Y_hdr of the HDR image acquired in step S301. Since the luminance value Y_hdr has already been calculated at the time of creating the linear LDR image in step S302, if the value is stored, it is not necessary to calculate here. In step S502, the HDR auxiliary data generation unit 203 acquires linear LDR image data from the region for the linear LDR image in the memory 206. When “0” is set in the flag Flag, the linear LDR image created in step S302 is stored in the memory 203. On the other hand, when the flag Flag is set to “1”, an image in which the linear LDR image created in step S302 is overwritten with the linear LDR image obtained as a result of local decoding in the process of step S312 is stored in the memory 203. Saved. In step S503, the HDR auxiliary data generation unit 203 calculates a luminance value Y_ldr for each pixel of the linear LDR image acquired in step S502. As a calculation method of the luminance value, a conversion formula defined by ISO / IEC 18477-2 is used. In step S504, the HDR auxiliary data generation unit 203 uses the Y_hdr acquired in step S501 and the Y_ldr acquired in step S503 in accordance with the method specified in ISO / IEC 18477-2, and all the pixels according to the following formula: Calculate the Y ratio (Y_ratio).
Y_ratio (x, y) = Y_hdr (x, y) / (Y_ldr (x, y) + nf)
Here, nf is noise added by the noise adding unit 106. Thereby, zero percent can be prevented.

  In step S505, the HDR auxiliary data generation unit 203 calculates a difference DifRGB of RGB values for each pixel position of the HDR image and the linear LDR image according to a method defined in ISO / IEC 18477-2. In step S506, the HDR auxiliary data generation unit 203 performs scale conversion by dividing DifRGB for each pixel position calculated in step S505 by Y_ratio of the same pixel position calculated in step S504. Then, the HDR auxiliary data generation unit 203 calculates the color difference component CC_hdr of the HDR auxiliary data corresponding to each pixel position in accordance with a method specified by ISO / IEC 18477-2. In step S507, the HDR auxiliary data generation unit 203 normalizes CC_hdr with Y_ldr calculated in step S503, and acquires CC_norm. In step S508, the HDR auxiliary data generation unit 203 takes the logarithm of Y_ratio acquired in step S504 and calculates Y_log. In step S509, the HDR auxiliary data generation unit 203 acquires the maximum value Max_CC (Max_Cb and Max_Cr) and the minimum value Min_CC (Min_Cb and Min_Cr) from the CC_norm values of all the pixels. Then, the HDR auxiliary data generation unit 203 quantizes CC_norm in the range of Max_CC and Min_CC, and obtains CC_xt obtained by converting CC_norm to 8 bits. In step S511, the HDR auxiliary data generation unit 203 acquires the maximum value Max_log and the minimum value Min_log of all the pixels for Y_log acquired in step S508. Then, the HDR auxiliary data generation unit 203 quantizes Y_log within the range of Max_log and Min_log, and acquires Y_xt converted to 8 bits. In step S513, Y_xt and CC_xt are encoded by the JPEG method, and this flow ends.

  In this embodiment, when the number of pixels having a luminance value of 96 or less in an LDR image exceeds half of the entire image, HDR auxiliary data is created using the result of local decoding of the LDR image. That is, only when the number of low-luminance pixels is very large, local decoding processing with a high processing load is performed to improve the image quality of the HDR image at the time of decoding. This is because the sensitivity to low luminance is large due to human visual characteristics. Therefore, even if the processing load is high, the effect of improving the image quality of the HDR image at the time of decoding is great. On the other hand, in the case of an image in which the ratio of low-luminance pixels is not large, the effect of improving the image quality of the HDR image at the time of decoding is small, so local decoding processing is not performed and the processing load related to encoding can be reduced . Therefore, according to the present embodiment, it is easy to execute local decoding processing that increases the processing load during encoding only when the effect of improving the image quality during decoding is large.

[Modification of First Embodiment]
In the first embodiment described above, the presence / absence of the local decoding process is determined using the ratio of the low luminance pixel value to the entire HDR image to be encoded. In step S302 of FIG. 3, a linear LDR image of the entire image, that is, 1920 × 1080 pixels is created, and low luminance pixel values are counted. Therefore, it was possible to determine the presence or absence of local decoding after creating the linear LDR image. However, since information on the entire image is required, parallel processing cannot be performed, and the amount of memory used increases.

  For example, consider dividing an image into 8 lines each having the same height as a macroblock (8 × 8 pixels), executing an HDR image compression flow, and increasing the speed by parallel processing. Encoding for eight lines after the creation of the LDR image is performed, and at the same time, the next eight lines of LDR images are created in parallel. However, if the ratio of low-luminance pixels in the entire image is not known, the presence or absence of local decoding cannot be determined. For this reason, the creation of HDR auxiliary data cannot be started until the processing in step S302 is completed, and parallel processing cannot be performed. Further, it is necessary to save all the quantized values of the LDR image obtained in step S308 until it is determined whether or not local decoding is performed. Therefore, all quantization values of the LDR image must be retained. Alternatively, only the LDR image code data can be retained and decoded when local decoding is required to create a linear LDR image. In this case, the memory usage of the LDR image may be only for the code data and the memory for 8 lines, but it takes time to decode the entropy coding, and the entire processing becomes slow.

  As a method for solving the above-described problem, in this modification, the presence / absence of local decoding is switched in units of macroblocks. Hereinafter, specific processing of this modification will be described with reference to the flowchart of FIG. In this modification, the same reference numerals are given to the same operation processes as those in the first embodiment, and the detailed description thereof is omitted. In the following description, an example in which various processing units such as the HDR auxiliary data generation unit 203, the LDR image generation unit 204, and the JPEG codec unit 205 are executed by the CPU 201 will be described.

  In step S306, the CPU 201 acquires the coding conditions for the LDR image. Next, in step S701, the CPU 201 acquires the size of the HDR image, that is, the width w and the height h. In this embodiment, the width is w = 1920 pixels and the height is h = 1080 pixels. In step S702, the CPU 201 calculates the number N of macroblocks to be processed. In this embodiment, since the width w of the image is 1920 pixels, it can be calculated that N = 1920 ÷ 8 = 240. In subsequent step S703, the CPU 201 acquires eight lines of HDR image data in order from the top. Eight lines is the number of lines that matches the height of the macroblock. In step S704, the CPU 201 initializes the counter M for counting processed macroblocks by substituting zero. In step S705, the CPU 201 performs LDR image compression processing in units of macroblocks. This process will be described later with reference to FIG. In step S706, the CPU 201 creates macro blocks CC_norm and Y_log necessary for creating HDR auxiliary data. This process is a process in which steps S501 to S508 of FIG. 5 described in the first embodiment are applied to a macroblock. Since it has already been described in the first embodiment, the processing flow and its description in this modification are omitted. In step S707, the CPU 201 adds “1” to the counter M of the macroblock. In step S708, the CPU 201 compares the counter M with the number N of macroblocks to be processed calculated in step S703. If M = N, the CPU 201 determines that the processing of the HDR image for eight lines input at the present time has ended, and the CPU 201 advances the processing to step S709. Otherwise, the CPU 201 returns the process to step S705. In step S709, the CPU 201 determines whether the processing has been completed up to the last line of the HDR image. In this embodiment, the number of lines input in step S703 is added, and when the height of the HDR image acquired in step S701 is the same, it is determined that the process has been completed up to the last line. When the process reaches the last line, the CPU 201 advances the process to step S710. Otherwise, the CPU 201 returns the process to step S703 and acquires the next eight lines.

  In step S710, the CPU 201 uses the CC_norm and Y_log obtained in step S706 to perform HDR auxiliary data compression processing. Since this process is the same as the process from step S509 to step S513 in the flowchart of FIG. 5 described in the first embodiment, the description thereof is omitted. In the processing after step S509 in the flow of FIG. 5, it is necessary to acquire the maximum value and the minimum value in the entire image. For this reason, in this modification in which LDR images are created every eight lines, it is necessary to divide the processing up to step S508 and thereafter. Next, in step S315, the CPU 201 outputs the HDR auxiliary data created in step S710. In step S317, the CPU 201 outputs the LDR compressed data created in step S705 as LDR information. Since the processing in step S315 and step S317 is the same as that in the first embodiment, description thereof is omitted here.

  Next, the macroblock unit LDR image compression processing in step S705 will be described with reference to the flowchart of FIG.

  In step S801, the CPU 201 creates a linear LDR image in units of macro blocks. This method will be described later with reference to FIG. In step S802, the CPU 201 calculates the ratio RLP_m of the low luminance pixels of the macro block of interest from the number of low luminance pixels counted when creating the linear LDR image in step S801. The size of the macroblock is 8 × 8 pixels. Therefore, if the number of low luminance pixels obtained in step S801 is C, RLP_m = C / 64. In step S803, the CPU 201 compares RLP_m with a predetermined threshold Th_RLP. If RLP_m is larger than the threshold Th_RLP, the CPU 201 advances the process to step S305. In this modification, the threshold Th_RLP is set to 0.5. Therefore, in the case of this modification, if there are 33 or more low-luminance pixels in one macroblock, the process proceeds to step S305. In step S305, the CPU 201 assigns “1” to the flag Flag to indicate that local decoding is to be performed. On the other hand, when RLP_m becomes equal to or less than the threshold value and the process proceeds to S313, the CPU 201 assigns “0” to the flag Flag to indicate that local decoding is not performed. Steps S307 to S310 perform the same operation as in the first embodiment. In step S310, the CPU 201 determines whether or not the flag Flag is “1”. If the flag Flag is “1”, the CPU 201 advances the process to step S311. Otherwise, the flow ends. Steps S311 and S312 operate in the same manner as in the first embodiment. However, the processing target in S311 and 312 is the target macroblock. In step S804, the image data after local decoding obtained in step S312 is overwritten with the linear LDR image of the macro block of interest created in step S801, and this flow ends.

  Next, the creation processing of the linear LDR image for each macroblock in step S801 will be described with reference to the flow of FIG.

  Basically, the processing performed on the entire image in FIG. 4 of the first embodiment is performed on one macroblock in accordance with this modification.

  In step S401, the CPU 201 acquires a tone curve used for tone mapping processing, and proceeds to step S403. Steps S403 to S411 are the same as those in the first embodiment. In step S901, the CPU 201 compares the variable x with the macroblock width = 8. If the variable x = 8, the CPU 201 determines that the process has been completed up to the right end of the macroblock, and advances the process to step S413. If the variable x has not reached the macroblock width, the CPU 201 returns the process to step S406 in order to process the pixel on the right side of the pixel of interest. In step S413, the CPU 201 adds “1” to the variable y in the height direction of the macroblock. In step S902, the CPU 201 compares the variable y with the height of the macroblock = 8. If y is 8, the process ends. If not, the process returns to step S405 to proceed from the beginning of the next line.

  According to the process of this modification, the presence / absence of local decoding can be switched in units of macroblocks. This increases the risk of artifacts occurring at the macroblock boundaries. However, it can be expected that the amount of memory used for encoding the LDR can be reduced, and the speed can be easily increased by parallel processing.

  In the present embodiment, data is read and processed every 8 lines having the same height as the macroblock. However, any number of lines may be used as long as it is a multiple of the macroblock height (= 8).

[Second Embodiment]
In the first embodiment, a tone curve that covers the entire possible range of HDR image data as shown in FIG. 6A is used. As described above, the tone mapping process that applies a tone curve that is uniquely determined regardless of an image is preferred when the calculation cost is low and real-time performance is required. However, the input HDR image may not use all of this range. In fact, the dynamic range of individual HDR images varies, and most images use only a part of the possible data range. For example, in the first embodiment, if the pixel value of the input HDR image is a value smaller than “4550”, it is counted as a low luminance pixel, but if the minimum pixel value of the input HDR image is “4550”, There is no determination that local decoding is necessary.

  On the other hand, it is also conceivable to perform tone mapping using a tone curve depending on the dynamic range of the HDR image to be encoded. For example, when the minimum value of the pixel value of the input HDR image is “4550” and the maximum value is “50530”, a tone curve as shown in FIG. 6B is generated to create an LDR image. Using a tone curve that matches the dynamic range of each HDR image to be encoded in this way is computationally expensive, but it is possible to create code data that can decode a more beautiful (high quality) LDR image. it can. That is, it can be determined that image quality is more important than calculation cost.

  In the second embodiment, encoding is performed using a fixed tone curve (FIG. 6A) corresponding to the entire possible range of pixel values of the HDR image, or a tone curve depending on the HDR image to be encoded An example will be described in which the user sets via the input unit 202 whether or not encoding is performed. Then, the presence or absence of local decoding is switched according to the setting.

  Note that the LDR image generation unit 204 in the second embodiment generates an LDR image using the tone curve shown in FIG. 6A when it is set to use a fixed tone curve. On the other hand, when it is set to use a tone curve that depends on the HDR image to be encoded, the LDR image generation unit 204 obtains the maximum value and the minimum value of the pixels of the HDR image to be encoded, and It is assumed that an LDR image is generated after a curve having an output value of 0 to 255 is set.

  Hereinafter, the processing of the second embodiment will be described with reference to the flowchart of FIG. In the second embodiment, the same processes as those in the first embodiment are given the same numbers, and the description thereof is omitted.

  In step S <b> 301, the CPU 201 acquires HDR image data and stores it in the memory 206. Next, in step S302, the CPU 201 controls the LDR image generation unit 204 to create a linear LDR image. The generated linear LDR image is stored in the memory 206. The LDR image generation procedure is as described above. Next, in step S1001, the CPU 201 determines whether the tone curve used in step S302 is a tone curve depending on the HDR image to be encoded. This determination may be made based on user settings.

  When it is determined that the tone curve depending on the HDR image to be encoded is used, the CPU 201 determines that the image quality is important and advances the process to step S305. In step S305, the CPU 201 sets “1” to the flag Flag so that the locally decoded LDR image is used when creating the HDR auxiliary data. On the other hand, if the tone curve used in step S302 is a fixed tone curve that does not depend on the HDR image to be encoded, the CPU 201 advances the process from step S1001 to step S313. In step S313, the CPU 201 sets “0” to the flag Flag to indicate that local decoding of the LDR image is not performed. Since step S306 and subsequent operations are the same as those in the first embodiment, description thereof is omitted here.

  According to the second embodiment described above, it is possible to determine a tone curve to be applied according to whether or not priority is given to image quality, and to determine whether or not to locally decode an LDR image.

[Modification of Second Embodiment]
In the second embodiment, whether to perform local decoding is switched depending on whether a tone curve depending on the HDR image to be encoded is used or a fixed tone curve is used. However, local decoding may be switched between the minimum value and the maximum value of the luminance value of the HDR image to be encoded. That is, when the maximum luminance value of the input HDR image is smaller than a preset threshold value, it is determined that the image is a low luminance image and local decoding is performed. Conversely, when the minimum luminance value of the input HDR image is equal to or greater than a preset threshold value, local decoding is not performed. In the case of this modification, the ratio of low-luminance pixels is not calculated as in the first embodiment. For this reason, local decoding is not performed when the maximum luminance value of one pixel increases due to noise or the like even though the luminance of most pixels is low. However, the presence or absence of local decoding can be determined more easily. Furthermore, while using a tone curve corresponding to the dynamic range of the input HDR image, the calculation cost for creating the HDR auxiliary data can be kept small.

[Third Embodiment]
There are two major tone mapping methods. One is global tone mapping that performs uniform luminance conversion on the entire image as shown in the first and second embodiments. The other is local tone mapping that performs different luminance conversion for each local region of the image.

  Since global tone mapping uses a uniform conversion function for the entire image, the visibility of texture information in the image decreases. On the other hand, the local tone mapping has a higher calculation cost than the global tone mapping. However, it is possible to maintain the contrast in the local region, and it is possible to prevent a decrease in the visibility of the texture information.

  A method of switching between local decoding and non-decoding when using the two tone mapping methods will be described with reference to FIG. Note that it is assumed that the user sets by the input unit 202 whether to use global tone mapping or local tone mapping when tone mapping is performed.

  In FIG. 11, the same processes as those in FIG. 10 are given the same numbers, and description thereof is omitted.

  In step S <b> 301, the CPU 201 acquires HDR image data to be encoded and stores it in the memory 206. Next, in step S <b> 302, the CPU 201 controls the LDR image generation unit 204 to create a linear LDR image from the HDR image stored in the memory 206 and store the created linear LDR image in the memory 206. In step S1101, it is determined whether the tone mapping method used in step S302 is local tone mapping. This determination is made based on a setting from the user. For example, when a plurality of tone curves are used for one HDR image, it may be determined that the local tone mapping is performed. If it is local tone mapping, the CPU 201 determines that image quality is important, and advances the processing to step S305. If it is not local tone mapping, the CPU 201 determines that calculation cost is important, and the process proceeds to step S313. The following processing is the same as in the second embodiment.

  According to the third embodiment, it is possible to easily determine whether the image quality is important or the calculation cost is important by the tone mapping method, and it is possible to easily determine whether or not to locally decode the LDR image.

[Modification of Third Embodiment]
In the third embodiment, when local tone mapping is selected, the result of local decoding for the entire region of the LDR image is used.

  In this modification, local tone mapping is performed for all macroblocks, but a case is considered where local decoding is switched for each macroblock. In this case, considering that human visual characteristics are highly sensitive to low luminance, only macroblocks classified as dark regions are locally decoded, thereby reducing the calculation cost. Such processing will be described with reference to FIG. The same processes as those in the first to third embodiments are given the same numbers, and detailed descriptions thereof are omitted.

  In step S <b> 301, the CPU 201 acquires HDR image data to be encoded and stores it in the memory 206. Next, in step S1201, the CPU 201 classifies the HDR image into two areas, a dark area and a bright area, and divides the area. Any method may be used as long as the region can be divided into a bright region and a dark region.

  In the present modification, the luminance value of each pixel is viewed, and if it is less than the middle (43678) of the luminance range (0 to 65535) that can be obtained in 16 bits, the pixel is classified as a dark region pixel. Otherwise, it is classified as a bright region pixel. Thereafter, for example, an area of less than 128 pixels × 128 pixels is integrated with an adjacent area, and the area division ends.

  In step S702, the CPU 201 calculates the number N of macroblocks included in the entire image. In the embodiment, since the image size is 1920 × 1080 pixels, it can be calculated as N = (1920/8) × (1080/8) = 32400. In step S704, the CPU 201 initializes the macroblock counter M by substituting zero. In step S1202, the CPU 201 performs LDR image compression in units of macro blocks. This process will be described later with reference to FIG.

  In step S707, the CPU 201 adds “1” to the macroblock counter M. In step S708, the CPU 201 compares the value of the macroblock counter M with the number N of macroblocks. If they are the same, the CPU 201 determines that the LDR image compression has been completed for all the macroblocks, and advances the processing to step S314. Otherwise, the CPU 201 returns the process to step S1202. Steps S314, S315, and S317 perform the same operation as in the first embodiment, and the flow is terminated.

  Next, the process of step S1202 will be described with reference to FIG. In FIG. 13, the same operations as those in FIG. 9 are denoted by the same reference numerals, and the description thereof is omitted.

  In step S801, the CPU 201 creates a linear LDR image in units of macroblocks. In step S1301, the CPU 201 determines whether or not the macro block currently being processed includes 32 pixels or more (half or more) of pixels determined as a dark region as a result of step S1202. If the area is darker than 32 pixels, the CPU 201 advances the processing to step S305, and substitutes “1” for the flag Flag, assuming that this macroblock is to be locally decoded. On the other hand, when the process proceeds to step S313, the CPU 201 assigns “0” to the flag Flag so that local decoding is not performed. From step S307 to step S804, processing similar to that described in the first embodiment and its modifications is performed, and this flow is terminated.

  According to the present modification, local decoding is performed only on macroblocks classified into the low luminance area. For this reason, there is a possibility that an artifact may appear at the boundary portion of the macro block where the processing is switched at the time of decoding the HDR image. However, even when local tone mapping is performed, the calculation cost for creating the HDR auxiliary data can be reduced by limiting the macro blocks to be locally decoded.

  In this modification, the method of performing local tone mapping for all macroblocks has been described. However, this modification is applied to the case of selecting either global tone mapping or local tone mapping as in the third embodiment. May be.

[Fourth Embodiment]
If it is better to perform local decoding, the LDR image obtained by tone mapping the HDR image to be encoded and the decoded LDR image finally obtained by decoding the encoded data written in the JPEG XT file This is the case when the difference is large. It can be considered that the larger the quantization step of the LDR image, the larger the error due to the quantization, and thus the above difference becomes large. Therefore, in the fourth embodiment, a method of switching between local decoding and non-decoding based on the LDR image quantization step will be described.

  Hereinafter, processing in the fourth embodiment will be described with reference to FIG.

  In steps S301 and S302, the CPU 201 acquires HDR image data to be encoded, and creates a linear LDR image, as in the first embodiment. In step S306, the CPU 201 acquires the coding conditions for the LDR image. The encoding condition includes information on the Q parameter of the LDR image. The Q parameter takes an integer value from 1 to 100. When 100 is specified, the quantization step is the finest, and when 1 is specified, the quantization step is the coarsest. In step S1401, the CPU 201 refers to the value of the Q parameter from the coding condition of the LDR image acquired in step S306. Then, the CPU 201 compares the Q parameter with a predetermined value Th_q. If the Q parameter is less than Th_q, the CPU 201 advances the process to step S305, otherwise proceeds to step S313. In the present embodiment, Th_q is 60. Therefore, if the Q parameter of the LDR image is less than 60, the process proceeds to step S305; otherwise, the process proceeds to step S313. Since the process after step S305 is the same process as the first embodiment, a detailed description thereof is omitted here.

  As a result of the above, the degree of deterioration of the LDR image is predicted from the Q parameter at the time of encoding the LDR image, and when it is determined that the degree of deterioration is large, local decoding is performed. Processing that does not perform local decoding can be realized.

[Modification of Fourth Embodiment]
In JPEG XT, luminance sub-sampling is not performed when JPEG compression is performed, and sub-sampling is defined only for color difference components. If the sampling ratios of Cb and Cr in the x-axis direction and y-axis direction are Sbx, Sby, Srx, and Sry, respectively, the following four sub-sampling ratios are defined in JPEG XT.
(1) (Sbx, Sby, Srx, Sry) = (1, 1, 1, 1)
(2) (Sbx, Sby, Srx, Sry) = (1, 2, 1, 2)
(3) (Sbx, Sby, Srx, Sry) = (2, 1, 2, 1)
(4) (Sbx, Sby, Srx, Sry) = (2, 2, 2, 2)
In the case of (1) above, since there are no color difference components to be thinned out, it can be considered that there is little deterioration in image quality. Therefore, in the fourth modification, local decoding is performed when the subsampling of the components Cb and Cr is any one of the above (2) to (4). The HDR image compression processing according to this modification will be described with reference to the flowchart of FIG.

  The difference between FIG. 15 and FIG. 14 is that step S1401 in FIG. 14 is replaced with step S1501. In step S1501 in FIG. 15, the CPU 201 refers to the sub-sampling information from the LDR image encoding condition acquired in step S306. If the CPU 201 determines that the sub-sampling is any one of (2) to (4), the CPU 201 advances the process to step S305. Otherwise, the process advances to step S313. In this way, when sub-sampling is performed, HDR auxiliary data is created using the result of local decoding of the LDR image.

[Fifth Embodiment]
In the first to third embodiments and the modifications thereof, switching between local decoding and non-decoding is performed based on HDR image information, LDR image encoding conditions, and the like. However, local decoding may be switched by directly evaluating deterioration of the LDR image. Therefore, in the fifth embodiment, an example will be described in which values before and after quantization are directly compared to evaluate the degree of deterioration and whether to perform local decoding is determined in units of macroblocks.

  In the case of the fifth embodiment, the operation is almost the same as that of the modification of the first embodiment, but the method of LDR image compression in units of macroblocks in step S705 in FIG. 7 is different. Therefore, a macroblock unit LDR image compression method according to the fifth embodiment will be described with reference to FIG. In FIG. 16, processes having the same operations as those in FIG. 8 are given the same numbers, and descriptions thereof are omitted.

  In step S801, the CPU 201 creates a linear LDR image in units of macroblocks. This process is the same as the modification of the first embodiment. Next, in step S307, the CPU 201 performs color conversion on the linear LDR image created in step S801 as in the first embodiment. In step S1601, the CPU 201 executes DCT on the macroblock of the LDR image obtained in step S307, and stores the DCT coefficient of the luminance at that time in the memory 206, for example. In step S1602, the CPU 201 performs a quantization process on the DCT coefficients of Y, Cb, and Cr obtained in step S1601. In step S1603, the CPU 201 inversely quantizes only the luminance Y component. In step S309, the CPU 201 entropy encodes the quantized DCT coefficient obtained in step S1602. In step S1604, the CPU 201 calculates a difference between the DCT coefficient before luminance stored in step S1601 and the DCT coefficient after luminance obtained in step S1603 for the 64 coefficients of the macroblock. Calculate the value. In step S1605, the CPU 201 counts the number K of coefficients having an absolute value equal to or greater than Th_dc determined in advance among the 64 difference values obtained in step S1604. In this embodiment, Th_dc = 64. In step S1606, the CPU 201 compares K obtained in step S1605 with a predetermined value Th_diff. In this embodiment, Th_diff = 32. If K is larger than Th_diff, the CPU 201 determines that the difference between the DCT coefficients before and after quantization in this macroblock is large, advances the processing to step S1607, and performs local decoding. Otherwise, this flow is terminated without performing local decoding. That is, when a difference of Th_dc = 64 or more is obtained due to quantization of more than half of the DCT coefficients of one macroblock, local decoding is performed.

  In step S1607, the CPU 201 inversely quantizes the quantized DCT coefficient obtained in step S1602. Since the luminance Y is inversely quantized in step S1603, only the color differences Cb and Cr need to be inversely quantized here. In step S312, the CPU 201 performs reverse color conversion to obtain a linear LDR image. In step S804, the CPU 201 overwrites the linear LDR image of the macroblock with the LDR image obtained by local decoding, and ends this flow.

  According to the fifth embodiment, since the values before and after quantization are compared for all DCT coefficients, the calculation cost increases. However, since the degree of deterioration due to encoding of the LDR image can be seen, the HDR auxiliary data can be created using the locally decoded image without fail when the deterioration is large.

[Modification of Fifth Embodiment]
In the fifth embodiment, all DCT coefficients are uniformly compared and whether local decoding is performed or not is switched. However, the number of DCT coefficients to be compared and Th_diff are set according to the Q parameter that is the encoding condition of the LDR image. You may switch. For example, Th_diff may be half of the DCT coefficients to be compared. FIG. 17 schematically shows DCT coefficients. It is assumed that the upper left corner is a DC component and the lower right corner is an AC component. When the Q parameter of the LDR image is large, only six DCT coefficients on the high frequency side indicated by reference numeral 1701 in FIG. 17 are compared. Conversely, when the Q parameter of the LDR image is small, 43 DCT coefficients that are combined with reference numerals 1701 to 1704 in FIG.

  The processing flow of this modification is shown in FIG. Note that steps that perform the same processing as in FIG. 16 are assigned the same numbers as in FIG. 16, and descriptions thereof are omitted.

In step S801, the CPU 201 creates a linear LDR image in units of macroblocks. In step S307, the CPU 201 performs color conversion of the linear LDR image created in step S801. In step S1601, the CPU 201 applies DCT to the color-converted LDR image and stores the luminance coefficient in the memory 206. In step S1602, the CPU 201 quantizes the DCT coefficient according to the Q parameter of the LDR image. In step S309, the CPU 201 performs entropy encoding. In step S1801, the CPU 201 determines whether the Q parameter of the LDR image is 10 or more. If it is 10 or more, the process advances to step S1802. If it is less than 10, the CPU 201 determines that the difference due to quantization is large (the quantization step is large), advances the processing to step S1607, and executes local decoding processing. In step S1702, the CPU 201 determines whether the Q parameter of the LDR image is less than 90. If it is less than 90, the CPU 201 advances the process to step S1803. If it is 90 or more, the CPU 201 determines that the difference due to quantization is small, does not perform local decoding, and ends this processing flow. In step S1803, the CPU 201 acquires the number of DCT coefficients to be compared according to the Q parameter and a threshold value Th_diff. In the present embodiment, the following values are acquired.
(1) When Q is 70 or more and less than 90 The comparison target DCT coefficient is 6 coefficients Th_diff = 3 indicated by 1701 in FIG.
(2) When Q is 50 or more and less than 70 The comparison target DCT coefficients are 15 coefficients Th_diff = 8 indicated by 1701 and 1702 in FIG.
(3) When Q is 30 or more and less than 50 The DCT coefficients to be compared are 28 coefficients Th_diff = 14 shown by 1701 to 1703 in FIG.
(4) When Q is 10 or more and less than 30 The DCT coefficients to be compared are 43 coefficients Th_diff = 22 indicated by 1701 to 1704 in FIG.
Next, the CPU 201 advances the process to step S1603. Since step S1603 and subsequent steps are the same as those in the fifth embodiment, description thereof is omitted here.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  201 ... CPU, 202 ... input unit, 203 ... HDR auxiliary data generation unit, 204 ... LDR image generation unit, 205 ... JPEG codec unit, 206 ... memory, 207 ... display unit, 208 ... storage unit

Claims (12)

An image encoding device for generating auxiliary data for encoding an HDR image and returning an LDR image to the accuracy of the HDR image,
LDR image generation means for generating an LDR image from an HDR image to be encoded;
Encoding means for irreversibly encoding the generated LDR image;
Decoding means for decoding the encoded LDR image;
Auxiliary data generating means for generating auxiliary data for returning the LDR image to the accuracy of the HDR image based on the HDR image and the LDR image;
Determining means for determining whether to use an LDR image generated by the LDR image generating means or an LDR image obtained by decoding by the decoding means as an LDR image used by the auxiliary data generating means; An image encoding device. The encoding means is a JPEG codec means for performing color conversion, DCT conversion, quantization, and entropy encoding in units of macroblocks,
The image decoding apparatus according to claim 1, wherein the decoding unit is a unit that performs inverse quantization, inverse DCT, and inverse color conversion on a coefficient before the entropy encoding. The LDR image generation means generates an LDR image in units of macroblocks,
The decoding means decodes macroblocks as units,
The image coding apparatus according to claim 2, wherein the determination unit determines the image indicated by the macroblock in units. The determination means includes
Find the proportion of pixels in the HDR image that are occupied by pixels with a luminance lower than a preset luminance value,
If the ratio exceeds a preset threshold, the LDR image obtained by the decoding unit is determined to be used by the auxiliary data generation unit,
The LDR image obtained by the LDR image generation unit is determined to be used by the auxiliary data generation unit when the ratio is equal to or less than a preset threshold value. The image encoding device described in 1. Whether to use a preset fixed tone curve or a tone curve depending on the maximum luminance and the minimum luminance in the HDR image to be encoded as the tone curve used when the LDR image generation means generates It further has setting means for setting,
The determination means includes
When it is set by the setting means that the tone curve depending on the maximum luminance and the minimum luminance in the HDR image to be encoded is used, the LDR image obtained by the decoding means is determined to be used by the auxiliary data generation means And
4. If the setting unit sets the fixed tone curve to be used, the LDR image obtained by the LDR image generation unit is determined to be used by the auxiliary data generation unit. The image encoding device according to any one of the above. A setting means for setting whether to use global tone mapping or local tone mapping as tone mapping when the LDR image generating means generates;
The determination means includes
If it is set by the setting means to use local tone mapping, the LDR image obtained by the decoding means is determined to be used by the auxiliary data generation means,
4. The LDR image obtained by the LDR image generation unit is determined to be used by the auxiliary data generation unit when it is set by the setting unit to use global tone mapping. 5. The image encoding device according to claim 1. The determination means includes
If the quantization step used for quantization in the encoding means exceeds a preset value, the LDR image obtained by the decoding means is determined to be used by the auxiliary data generation means,
When the quantization step used for quantization in the encoding means is equal to or less than the preset value, the LDR image obtained by the LDR image generation means is determined to be used by the auxiliary data generation means. The image encoding device according to claim 2 or 3. The determination means includes
If there is a thinning out of the color difference component due to color conversion in the encoding means, the LDR image obtained by the decoding means is determined to be used by the auxiliary data generating means,
The LDR image obtained by the LDR image generation unit is determined to be used by the auxiliary data generation unit when there is no thinning out of the color difference component due to color conversion in the encoding unit. The image encoding device described in 1. The determination means includes
The difference between the coefficient obtained by performing the DCT and the coefficient after inverse quantization by the decoding means is obtained, the ratio of the number of the absolute value of the difference exceeding a preset value, and a preset threshold value, Compare
If the ratio is greater than the threshold, the LDR image obtained by the decoding means is determined to be used by the auxiliary data generation means,
The image coding apparatus according to claim 2 or 3, wherein when the ratio is equal to or less than the threshold value, the LDR image obtained by the LDR image generation unit is determined to be used by the auxiliary data generation unit. The image coding apparatus according to claim 9, further comprising means for determining a range for obtaining the ratio in a macroblock performed by the DCT based on a quantization step used in the quantization. A control method of an image encoding device for generating auxiliary data for encoding an HDR image and returning an LDR image to the accuracy of the HDR image,
An LDR image generation step of generating an LDR image from the HDR image to be encoded;
An encoding step of irreversibly encoding the generated LDR image;
A decoding step of decoding the encoded LDR image;
An auxiliary data generation step for generating auxiliary data for returning the LDR image to the accuracy of the HDR image based on the HDR image and the LDR image;
A determination step of determining which one of the LDR image generated by the LDR image generation means and the LDR image obtained by decoding in the decoding step is used as the LDR image used in the auxiliary data generation step. Control method for an image encoding device.   The program for functioning the said computer as each means of any one of Claims 1 thru | or 10 when a computer reads and executes.

Images