JP6315069B2 - Image processing apparatus and program - Google Patents

Description

  The present disclosure relates to an image processing apparatus and a program.

  A JPEG (Joint Photographic Experts Group) method is widely used as an image encoding method capable of compressing the data size of an image. According to the JPEG method, the data size of an image having a bit depth of 8 bits for each color is compressed using elemental techniques such as orthogonal transform, quantization, and entropy coding. The JPEG method is supported by various applications such as currently popular browsers, image viewers, photo editors, and authoring tools.

  In contrast, a RAW image that is originally acquired by an imaging device such as a digital camera or an image scanner generally has a greater bit depth, such as 12 bits. Therefore, in order to finally encode the RAW image by the JPEG method, after the RAW image is developed, a process of reducing the bit depth of each pixel of the developed image is performed. As a result, the image quality of the image once encoded by the JPEG method is deteriorated as compared with the image quality of the initially captured image.

  The deterioration of the image quality due to the reduction of the bit depth appears, for example, as a decrease in gradation. Typical examples of gradation reduction are overexposure in which gradation is lost in a high luminance area, and blackout in which gradation is lost in a low luminance area. Patent Document 1 below proposes a technique for dynamically controlling the imaging sensitivity and brightness correction amount of an imaging apparatus to reduce whiteout of an image.

JP 2011-055171 A

  However, once the gradation has been reduced or lost, it has been difficult to regain the high gradation that the image originally had. In addition, if multi-bit encoding methods such as the existing extended JPEG method, JPEG 2000 method, and JPEG XR method are used, the need to reduce the bit depth is avoided, but these multi-bit encoding methods are different from the JPEG method. Not compatible between. For this reason, the solution of handling a high gradation image by a multi-bit encoding method has been difficult to be practically used.

Therefore, it is desirable to realize a mechanism that can restore image quality that is degraded by image encoding while ensuring compatibility with a standard encoding method for still images such as the JPEG method.

According to the present disclosure, a post-correction generated by performing correction processing including one or more of EV (Exposure Value) correction, WB (White Balance) correction, and gamma correction on the input full-color image data of a still image. A main image decoding unit that decodes a main image encoded stream generated by encoding image data using a predetermined standard encoding method; and the correction processing for the input full-color image data and at least the corrected image data. A differential image decoding unit that decodes the differential image encoded stream generated by encoding the differential image data between the reverse-processed image data generated by performing the reverse processing using the standard encoding method; An image processing apparatus is provided that includes a combining unit that combines the output of the main image decoding unit and the output of the difference image decoding unit . An image processing apparatus that performs corresponding encoding-side processing is also provided.

Further, according to the present disclosure, it is generated by performing correction processing including one or more of EV (Exposure Value) correction, WB (White Balance) correction, and gamma correction on the input full-color image data of a still image. Decoding the main image encoded stream generated by encoding the corrected image data using a predetermined standard encoding method, and outputting the corrected image data; and at least the input full-color image data and the correction The standard encoding method is used to convert the differential image encoded stream generated by encoding the differential image data between the post-processed image data generated by performing the reverse process of the correction process to the post-image data. And outputting the difference image data, and synthesizing the corrected image data and the difference image data. A method is provided. A corresponding encoding image processing method is also provided.

Further, according to the present disclosure, the computer performs a correction process including one or more of EV (Exposure Value) correction, WB (White Balance) correction, and gamma correction on the input full-color image data of the still image. A main image decoding unit that decodes a main image encoded stream generated by encoding the generated corrected image data using a predetermined standard encoding method; the input full-color image data; and at least the corrected image data The difference which decodes the difference image coding stream produced | generated by encoding the difference image data between the image data after reverse processing produced | generated by performing the reverse process of the said correction | amendment process with the said standard encoding system A program for functioning as an image decoding unit, and a combining unit that combines the output of the main image decoding unit and the output of the difference image decoding unit There is provided. A program for executing the corresponding encoding process is also provided.

According to the technology according to the present disclosure, it is possible to recover the image quality that is deteriorated by encoding an image while ensuring compatibility with a standard encoding method for still images .

It is a block diagram which shows an example of a structure of the image coding apparatus which concerns on one Embodiment. It is a block diagram which shows the 1st example of a detailed structure of the 2nd encoding branch of the image coding apparatus illustrated in FIG. It is a block diagram which shows the 2nd example of a detailed structure of the 2nd coding branch of the image coding apparatus illustrated in FIG. FIG. 12 is a block diagram illustrating a third example of a detailed configuration of a second encoding branch of the image encoding device illustrated in FIG. 1. FIG. 10 is a block diagram illustrating a fourth example of a detailed configuration of a second encoding branch of the image encoding device illustrated in FIG. 1. FIG. 10 is a block diagram illustrating a fifth example of a detailed configuration of the second encoding branch of the image encoding device illustrated in FIG. 1. It is explanatory drawing for demonstrating isolation | separation of the difference image in the 5th example of a detailed structure of a 2nd encoding branch. It is explanatory drawing for demonstrating an example of a structure of the encoding stream output by the image coding apparatus illustrated in FIG. It is explanatory drawing for demonstrating the other example of a structure of the encoding stream output by the image coding apparatus illustrated in FIG. It is a flowchart which shows an example of the flow of the image coding process which concerns on one Embodiment. It is a flowchart which shows an example of the detailed flow of the difference encoding process shown in FIG. It is a flowchart which shows an example of the detailed flow of the differential process shown in FIG. It is a flowchart which shows the other example of the flow of the image coding process which concerns on one Embodiment. It is a flowchart which shows an example of the detailed flow of the difference image separation process shown in FIG. It is a block diagram which shows an example of a structure of the image decoding apparatus which concerns on one Embodiment. It is a block diagram which shows the 1st example of a detailed structure of the 2nd decoding branch of the image decoding apparatus illustrated in FIG. It is a block diagram which shows the 2nd example of a detailed structure of the 2nd decoding branch of the image decoding apparatus illustrated in FIG. It is a block diagram which shows the 3rd example of a detailed structure of the 2nd decoding branch of the image decoding apparatus illustrated in FIG. It is a block diagram which shows the 4th example of a detailed structure of the 2nd decoding branch of the image decoding apparatus illustrated in FIG. It is a block diagram which shows the 5th example of a detailed structure of the 2nd decoding branch of the image decoding apparatus illustrated in FIG. It is a flowchart which shows an example of the flow of the image decoding process which concerns on one Embodiment. It is a flowchart which shows an example of the detailed flow of the differential decoding process shown in FIG. It is a flowchart which shows the other example of the flow of the image decoding process which concerns on one Embodiment. It is a flowchart which shows an example of the detailed flow of the difference synthetic | combination process shown in FIG. It is a block diagram which shows an example of a structure of the image processing apparatus which concerns on a 1st application example. It is a block diagram which shows an example of a structure of the image processing apparatus which concerns on a 2nd application example.

  Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be given in the following order.
1. 1. Configuration example of image encoding device according to embodiment 1-1. Overall configuration 1-2. Details of the second branch 1-3. Stream configuration 1-4. Flow of processing 2. Configuration example of image decoding device according to one embodiment 2-1. Overall configuration 2-2. Details of the second branch 2-3. Flow of processing Application example 4. Summary

<1. Configuration Example of Image Encoding Device According to One Embodiment>
The technology according to the present disclosure includes an image encoding device that generates an encoded stream from a full-color image generated by demosaicing a RAW image, and an image decoding device that reconstructs a full-color image by decoding the encoded stream Including. An image reconstructed by the image decoding apparatus according to the present disclosure does not necessarily completely reproduce a full-color image, but can express a higher gradation image than an image reconstructed by a JPEG decoder.

  In this specification, the term “JPEG system” means a baseline JPEG system that supports a bit depth of 8 bits, and does not include a multi-bit encoding system such as the extended JPEG system, JPEG2000 system, or JPEG XR system. .

  In this section, an image encoding apparatus according to an embodiment will be described. Thereafter, in the next section, an image decoding apparatus according to an embodiment will be described.

[1-1. Overall configuration]
FIG. 1 is a block diagram illustrating an example of a configuration of an image encoding device 100 according to an embodiment. Referring to FIG. 1, the image encoding apparatus 100 includes a RAW image acquisition unit 110, a development unit 120, a correction processing unit 130, an inverse processing unit 140, a difference calculation unit 150, a first encoding branch 160, and a second encoding branch. 170 and a stream output unit 190.

(1) RAW Image Acquisition Unit The RAW image acquisition unit 110 acquires RAW image data that is an input of encoding processing by the image encoding device 100. The RAW image acquisition unit 110 may acquire RAW image data from an image sensor including an image sensor such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or a contact image sensor (CIS). Instead, the RAW image acquisition unit 110 may acquire the RAW image data from another device or storage medium having the RAW image data, for example. A RAW image is generally an image having only a single color component for each pixel. In the present embodiment, it is assumed that the RAW image data acquired by the RAW image acquisition unit 110 has a bit depth of N bits for each color (N is greater than 8). The RAW image acquisition unit 110 outputs the acquired RAW image data to the development unit 120.

(2) Developing unit The developing unit 120 performs three-color components for each pixel (or for each predetermined unit depending on the image encoding method) by performing demosaic processing on the RAW image data input from the RAW image acquisition unit 110. The first full-color image data IM0 having is generated. The demosaic process typically refers to a process for complementing a color component that is insufficient for each pixel. In this specification, a full-color image means an image in which color components that are lacking in a RAW image are complemented and all color components are arranged for each pixel. The developing unit 120 may further perform additional processing such as defect correction, digital clamping, and digital gain. In general, the term “development” of a digital image may also include various correction processes performed on the full color image data after demosaicing. However, in this specification, the correction processing is performed by the correction processing unit 130 described below. The developing unit 120 outputs the generated first full-color image data IM0 to the correction processing unit 130 and the difference calculation unit 150.

(3) Correction Processing Unit The correction processing unit 130 corrects the first full-color image data IM0 input from the developing unit 120, and generates high gradation image data IM1. The bit depth of the high gradation image data IM1 is equal to the first full-color image data IM0 (and RAW image data). The correction processing by the correction processing unit 130 may include, for example, EV (Exposure Value) correction, WB (White Balance) correction, or gamma correction. Then, the correction processing unit 130 outputs the generated high gradation image data IM1 to the inverse processing unit 140 and the first encoding branch 160.

(4) Inverse Processing Unit The inverse processing unit 140 applies the inverse processing of the correction processing described above to the high gradation image data IM1 input from the correction processing unit 130, and generates the second full-color image data IM0 ′. The reverse processing of the correction processing by the reverse processing unit 140 can include, for example, EV reverse correction, WB reverse correction, or gamma reverse correction. Then, the inverse processing unit 140 outputs the generated second full-color image data IM0 ′ to the difference calculation unit 150. In this specification, an example in which the inverse processing unit 140 applies the inverse process of the correction process to the high gradation image data IM1 will be mainly described. However, the present invention is not limited to this example, and the inverse processing unit 140 may apply, for example, inverse processing of processing executed until the data is generated to first image data IMd1 described later. Further, the inverse processing unit 140 may apply, to a first encoded stream ST1 described later, an inverse process of the process executed until the stream is generated.

(5) Difference Calculation Unit The difference calculation unit 150 calculates a difference image between the first full-color image data IM0 input from the development unit 120 and the second full-color image data IM0 ′ input from the inverse processing unit 140. . Then, the difference calculation unit 150 outputs the second image data IM2 based on the calculated difference image to the second encoding branch 170. The second image data IM2 output directly from the difference calculation unit 150 has the same bit depth as the full-color image data IM0 (however, may have a positive or negative sign) representing the difference image. It is. Various processes, which will be described later, may be added to the second image data IM2 before encoding in the second encoding branch 170.

(6) First Encoding Branch The first encoding branch 160 is a processing branch for encoding the high gradation image data IM1. In the example of FIG. 1, the first encoding branch 160 includes a depth reduction / YC conversion unit 162 and a first encoding unit 168.

  The depth reduction / YC conversion unit 162 reduces the bit depth of the high gradation image data IM1 to 8 bits for each color, and converts the color space of the high gradation image data IM1 from RGB space to YC (luminance-color difference) space, First image data IMd1 is generated. Then, the depth reduction / YC conversion unit 162 outputs the generated first image data IMd1 to the first encoding unit 168.

  The first encoding unit 168 encodes the first image data IMd1 input from the depth reduction / YC conversion unit 162 by the JPEG method, and generates a first encoded stream ST1. That is, the first encoding unit 168 may be a general JPEG encoder. The encoding process by the first encoding unit 168 typically includes orthogonal transform, quantization, and entropy encoding that is executed for each block arranged in the image. By the encoding process here, the data size of the first image data IMd1 is compressed. The first encoding unit 168 outputs the first encoded stream ST1 including the first image data IMd1 encoded in this way to the stream output unit 190.

(7) Second Encoding Branch The second encoding branch 170 is a processing branch for encoding the second image data IM2. In the example of FIG. 1, the second encoding branch 170 includes a second encoding unit 179. The second encoding unit 179 encodes the second image data IM2 based on the above-described difference image by an arbitrary image encoding method, and generates a second encoded stream ST2. The image encoding method used by the second encoding unit 179 may be, for example, the JPEG method, or the extended JPEG method, JPEG2000 method, JPEG XR method, or the like. The second encoding unit 179 outputs the second encoded stream ST2 including the second image data IM2 encoded in this way to the stream output unit 190.

  The second encoding branch 170 may include a further processing unit that applies preprocessing to the second image data IM2 before the second encoding unit 179. The preprocessing applied to the second image data IM2 includes, for example, color space conversion, pixel value normalization, image size reduction, high-frequency component removal, and smoothing of a portion where whiteout and blackout are not generated. May include one or more of: Some specific embodiments of such a second encoding branch 170 will be further described later.

(8) Stream Output Unit The stream output unit 190 associates the second encoded stream ST2 input from the second encoding branch 170 with the first encoded stream ST1 input from the first encoding branch 160. These two encoded streams are output. For example, the stream output unit 190 may generate a reference parameter for accessing the second encoded stream ST2, and insert the generated reference parameter into the header area of the first encoded stream.

  For example, when the first and second encoded streams ST1 and ST2 are multiplexed in one file, the reference parameter includes the stream identifier, the start address, or the first address of the second encoded stream ST2. It may be a flag indicating that the stream following the encoded stream ST1 is the second encoded stream ST2. In addition, for example, when the first and second encoded streams ST1 and ST2 are stored in different files, the reference parameter is the file name of the file in which the second encoded stream ST2 is stored. It may be. The header area into which such a reference parameter is inserted may exist in the file in which the first encoded stream is stored, or a file different from the file in which the first encoded stream is stored. (For example, a header file).

  The first and second encoded streams ST1 and ST2 output from the stream output unit 190 may be stored in a storage medium connected to the image encoding device 100. Instead, the first and second encoded streams ST1 and ST2 may be transmitted from the image encoding device 100 to another device and stored or decoded in the other device.

[1-2. Details of the second branch]
In this section, five detailed embodiments of the second encoding branch 170 of the image encoding apparatus 100 illustrated in FIG. 1 will be described.

(1) First Example FIG. 2A is a block diagram showing a first example of a detailed configuration of the second encoding branch 170. Referring to FIG. 2A, the second encoding branch 170 includes a normalization unit 175, a YC conversion unit 177, and a second encoding unit 179a.

The normalizing unit 175 normalizes the pixel value of the second image data IM2 according to the pixel value range supported by the second encoding branch 170. In the first embodiment, the second encoding unit 179a uses the JPEG method. In this case, the pixel value range supported by the second encoding unit 179a is 0 to 255. On the other hand, the minimum value and the maximum value of the pixel value (difference pixel value) of the second image data IM2 input from the difference calculation unit 150 are defined as _Dmin and _Dmax , respectively. The minimum value D _min and the maximum value D _max theoretically have a bit depth equivalent to that of the full-color image data IM0, and may have a positive or negative sign. Therefore, for example, the normalization unit 175 normalizes the pixel value of the second image data IM2 so that the minimum value D _min of the difference pixel value is equal to zero and the maximum value D _max is equal to 255, respectively. Then, the normalization unit 175 outputs the second image data IM2 having the normalized pixel value to the YC conversion unit 177.

Further, the normalization unit 175 outputs the normalization parameter to the stream output unit 190 so that the decoder can denormalize the normalized pixel value to the original difference pixel value. The normalization parameter may be the aforementioned minimum value _Dmin and maximum value _Dmax . Instead, the normalization parameter may be a range ratio R _range and a shift amount S _range . The range ratio R _range indicates the ratio of the range width before and after normalization. The shift amount S _range indicates the amount of deviation of the center position of the range before and after normalization. The range ratio R _range and the shift amount S _range can be calculated from the minimum value D _min and the maximum value D _max described above.

  The YC conversion unit 177 converts the color space of the second image data IM2 input from the normalization unit 175 from the RGB space to the YC space. Then, the YC conversion unit 177 outputs the second image data IM2 having pixel values expressed in the YC space to the second encoding unit 179a.

  The second encoding unit 179a encodes the second image data IM2 input from the YC conversion unit 177 using the JPEG method, and generates a second encoded stream ST2. In other words, in the present embodiment, the second encoding unit 179a may be a general JPEG encoder, like the first encoding unit 168.

  According to the first embodiment, since both the first and second encoded streams ST1 and ST2 are encoded by using the JPEG encoder, the widely used JPEG encoder can be used to reduce the cost. Thus, the technology according to the present disclosure can be realized.

(2) Second Example FIG. 2B is a block diagram showing a second example of a detailed configuration of the second encoding branch 170. Referring to FIG. 2B, the second encoding branch 170 includes a reduction unit 173, a normalization unit 175, and a second encoding unit 179b.

  The reduction unit 173 reduces the size of the second image data IM2. The size reduction ratio used by the reduction unit 173 may be any value. For example, the reduction unit 173 may reduce the size of the second image data IM2 to 1/4 as a whole by reducing the size in the vertical direction and the horizontal direction to 1/2. Then, the reduction unit 173 outputs the second image data IM <b> 2 whose size has been reduced to the normalization unit 175.

The normalization unit 175 normalizes the pixel value of the second image data IM2 input from the reduction unit 173 according to the pixel value range supported by the second encoding branch 170. For example, when the second encoding unit 179b uses a 10-bit image encoding method, the pixel value range supported by the second encoding unit 179a is 0 to 1023. Therefore, the normalization unit 175 normalizes the pixel value of the second image data IM2 so that the minimum value D _min of the difference pixel value is zero and the maximum value D _max is equal to 1023. Then, the normalizing unit 175 outputs the second image data IM2 having the normalized pixel value to the second encoding unit 179b. Further, the normalization unit 175 outputs the normalization parameter to the stream output unit 190.

  The second encoding unit 179b encodes the second image data IM2 input from the normalization unit 175 using an arbitrary image encoding method, and generates a second encoded stream ST2. Before the second image data IM2 is encoded, the color space of the second image data IM2 may be converted from the RGB space to the YC space. When the JPEG method is used by the second encoding unit 179b, the technique according to the present disclosure can be realized at a lower cost by using the JPEG encoder also in the differential image encoding as in the first embodiment. can do.

  According to the second embodiment, since the size of the second image data IM2 to be encoded is reduced, the data amount of the second encoded stream ST2 can be reduced. As will be described later, the second image data IM2 is image data that is used in a complementary manner in order to compensate for the image quality that is deteriorated by encoding the full-color image data. Therefore, when reproduction of a nearly full color image is not required, it is beneficial to reduce the data amount of the encoded stream by reducing the size of the second image data IM2 in this way.

(3) Third Example FIG. 2C is a block diagram illustrating a third example of a detailed configuration of the second encoding branch 170. Referring to FIG. 2C, the second encoding branch 170 includes a difference processing unit 171, a normalizing unit 175, and a second encoding unit 179b.

  For example, the difference processing unit 171 can replace the pixel value of the second image data IM2 at the pixel position where the high-frequency component shows a strong value in the first image data IMd1 color-converted into the YC space with a constant value. As an example, a high-frequency component can show a strong value in a pixel in which noise occurs or a pixel whose pixel value changes abruptly between adjacent pixels. In general, image data including a strong high-frequency component tends to cause a decrease in encoding efficiency of an image encoding method that compresses the data size through quantization of transform coefficients in the frequency domain. On the other hand, the high frequency component of the image is difficult to be detected through human vision. Therefore, by replacing the pixel value at the pixel position where the high-frequency component shows a strong value with a constant (that is, flat) value, the high-frequency component can be removed without causing a human to perceive a reduction in image quality. The encoding efficiency of the 2-encoding branch 170 can be increased. Note that, by setting the constant value here as the center value of the pixel value range of the second image data IM2, the energy after orthogonal transformation (for example, discrete cosine transformation) is concentrated in a low frequency range, and coding efficiency is increased. Can be further enhanced. If normalization is not performed on the second image data IM2, the center value of the pixel value range is zero.

  Further, the difference processing unit 171 sets the pixel value of the second image data IM2 at a pixel position whose luminance belongs to the central portion of the pixel value range in the first image data IMd1 color-converted to the YC space to a constant value. Can be replaced. The central portion of the pixel value range here refers to a portion excluding the upper end portion and the lower end portion of the pixel value range in which overexposure or blackout is likely to occur as a result of the encoding process. As an example, the pixel value range of the first image data IMd1 having a bit depth of 8 bits is 0 to 255. In this case, assuming that the range of the central portion is R1 to R2, for example, R1 = 30, R2 = 230, and the like. By such pixel value replacement, while maintaining the gradation of the difference image at the pixel position that is likely to cause overexposure or blackout, the gradation of the difference image at the pixel position that is unlikely to be smoothed is smoothed. The Accordingly, while enhancing the encoding efficiency of the second encoding branch 170, even if a whiteout or blackout occurs in a part of the image in the first encoding branch 160, the gradation of the part can be supplemented with the difference image. it can. Note that the constant value here may also be the center value of the pixel value range of the second image data IM2.

  The normalization unit 175 normalizes the pixel value of the second image data IM2 processed by the difference processing unit 171 according to the pixel value range supported by the second encoding branch 170. Then, the normalizing unit 175 outputs the second image data IM2 having the normalized pixel value to the second encoding unit 179b. The normalization unit 175 outputs the normalization parameter to the stream output unit 190.

  The second encoding unit 179b encodes the second image data IM2 input from the normalization unit 175 using an arbitrary image encoding method, and generates a second encoded stream ST2. Before the second image data IM2 is encoded, the color space of the second image data IM2 may be converted from the RGB space to the YC space. When the JPEG method is used by the second encoding unit 179b, the technique according to the present disclosure can be realized at a lower cost by using the JPEG encoder also in the differential image encoding as in the first embodiment. can do.

(4) Fourth Example FIG. 2D is a block diagram showing a fourth example of a detailed configuration of the second encoding branch 170. The fourth embodiment is a combination of the first, second, and third embodiments described above. Referring to FIG. 2D, the second encoding branch 170 includes a difference processing unit 171, a reduction unit 173, a normalization unit 175, a YC conversion unit 177, and a second encoding unit 179a.

  For example, in the first image data IMd1 color-converted to the YC space, the difference processing unit 171 includes the second image data at the pixel position where the high frequency component has a strong value or the pixel position at which the luminance belongs to the central portion of the pixel value range. The pixel value of IM2 is replaced with a constant value. The reduction unit 173 reduces the size of the second image data IM2. The normalizing unit 175 normalizes the pixel value of the second image data IM2 in accordance with the pixel value range supported by the second encoding unit 179a. The YC conversion unit 177 converts the color space of the second image data IM2 from the RGB space to the YC space. Then, the second encoding unit 179a encodes the second image data IM2 by the JPEG method to generate a second encoded stream ST2.

(5) Fifth Embodiment FIG. 2E is a block diagram illustrating a fifth example of a detailed configuration of the second encoding branch 170. Referring to FIG. 2E, the second encoding branch 170 includes a classification unit 180, a difference processing unit 171, a reduction unit 173, a normalization unit 175, a YC conversion unit 177, a second encoding unit 179a, a difference processing unit 181 and a reduction. 183, a normalization unit 185, a YC conversion unit 187, and a second encoding unit 189a.

  The classification unit 180 uses a predetermined luminance threshold value to set each pixel of the second image data IM2 to a first group for pixels that may cause overexposure and pixels that may cause overexposure. Therefore, it is classified into one of a plurality of groups including the second group.

  More specifically, the classification unit 180 acquires the luminance value of the first image data IMd1 from the first encoding branch 160. Then, the classification unit 180 compares the acquired luminance value with a predetermined luminance threshold. Typically, the luminance threshold value may be a central value in the luminance value range. In the first image data IMd1, there is a possibility that overexposure occurs as a result of encoding by the JPEG method at a pixel position showing a luminance value exceeding the luminance threshold. Further, in the first image data IMd1, there is a possibility that blackout may occur similarly at a pixel position that shows a luminance value lower than the luminance threshold value. Therefore, the classification unit 180 can classify pixels whose luminance values are above the luminance threshold value into the first group and classify pixels whose luminance values are below the luminance threshold value into the second group. The classification unit 180 uses two different luminance thresholds, the first group for pixels that may cause overexposure, the second group for pixels that may cause overexposure, and other Each pixel may be classified into any of the third group for pixels.

  The classification unit 180 separates the second image data IM2 into the overexposure difference image data IM2a and the underexposure difference image data IM2b according to the above-described classification result. The overexposed difference image data IM2a is data having a difference pixel value of the second image data IM2 in the first group of pixels and a pixel value of zero in the other pixels. The blackout difference image data IM2b is data having a difference pixel value of the second image data IM2 in the second group of pixels and a pixel value of zero in the other pixels. Then, the classification unit 180 outputs the difference image data IM2a for overexposure to the difference processing unit 171 and the difference image data IM2b for blackout to the difference processing unit 181.

  FIG. 3 is an explanatory diagram for explaining separation of difference images by the classification unit 180. In the upper left of FIG. 3, an example image represented by the first image data IMd1 is shown. The image shows mountains and trees. In the image, the brightness of part of the mountain surface and part of the trees is high, and the brightness of the other parts is low. An isoluminance line EL in the figure indicates a pixel position having a luminance value equal to the luminance threshold value used for the group classification described above. Based on such luminance analysis, the classification unit 180 performs whiteout difference image data IM2a having a difference pixel value inside the isoluminance line EL and black having a difference pixel value outside the isoluminance line EL. The second image data IM2 can be separated from the crushing difference image data IM2b.

  The difference processing unit 171 is, for example, a difference for overexposure of a pixel position where the high frequency component has a strong value in the first image data IMd1 color-converted to the YC space, or a pixel position whose luminance belongs to the central portion of the pixel value range. The pixel value of the image data IM2a is replaced with a constant value. The reduction unit 173 reduces the size of the difference image data IM2a for whiteout. The normalizing unit 175 normalizes the pixel value of the overexposed difference image data IM2a according to the pixel value range supported by the second encoding unit 179a. The YC conversion unit 177 converts the color space of the overexposed difference image data IM2a from the RGB space to the YC space. Then, the second encoding unit 179a encodes the difference image data IM2a for overexposure by the JPEG method, and generates an encoded stream ST2a for overexposure.

  The difference processing unit 181 is, for example, a blackout difference between a pixel position where the high frequency component has a strong value or a pixel position where the luminance belongs to the center part of the pixel value range in the first image data IMd1 color-converted to the YC space. The pixel value of the image data IM2b is replaced with a constant value. The reduction unit 183 reduces the size of the blackout difference image data IM2b. The normalizing unit 185 normalizes the pixel value of the blackout difference image data IM2b according to the pixel value range supported by the second encoding unit 189a. The YC converter 187 converts the color space of the blackout difference image data IM2b from the RGB space to the YC space. Then, the second encoding unit 189a encodes the blackout difference image data IM2b by the JPEG method to generate a blackout encoded stream ST2b.

  When the fifth embodiment is employed, the stream output unit 190 illustrated in FIG. 1 associates the two encoded streams ST2a and ST2b input from the second encoding branch 170 with the first encoded stream ST1. Thus, these three encoded streams are output. For example, the stream output unit 190 may generate reference parameters for accessing the encoded streams ST2a and ST2b, and insert the generated reference parameters into the header area of the first encoded stream.

  According to the fifth embodiment, differential image data for complementing gradations lost due to overexposure and differential image data for complementing gradations lost due to blackout are encoded separately. In general, the correlation between images between a region where whiteout occurs and a region where blackout occurs is low. Therefore, by separately encoding the above-described two difference image data, the image quality compensated with the second image data in these regions can be improved.

[1-3. Stream configuration]
FIG. 4A is an explanatory diagram for describing an example of a configuration of an encoded stream output by the image encoding device 100. Referring to FIG. 4A, a first encoded stream ST1 and a second encoded stream ST2 associated with the first encoded stream ST1 are shown.

  The first encoded stream ST1 is generated according to the JPEG format, and includes a start marker (SOI: Start Of Image), a JPEG header area, a JPEG data area, and an end marker (EOI: End Of Image). The JPEG header area further includes a user-defined area USER_SEC. The user-defined area USER_SEC may be an APP segment or a COM segment, for example. In the user-defined area USER_SEC, the above-described reference parameter used for accessing the second encoded stream ST2 is inserted. Furthermore, additional parameters such as an identifier (encoding scheme ID) for identifying the encoding scheme used when encoding the second encoded stream ST2 are inserted in the user-defined area USER_SEC. Also good.

  Therefore, the decoder can access the second encoded stream ST2 using the reference parameter inserted in the header area of the first encoded stream ST1. The second encoded stream ST2 includes at least a header area and a data area. Parameters (for example, the normalization parameter, the quantization table, and the encoding table described above) necessary for decoding the second encoded stream ST2 are inserted into the header area of the second encoded stream ST2. Can be done. Note that the normalization parameter may be inserted into the header area of the first encoded stream ST1.

  FIG. 4B is an explanatory diagram for describing another example of the configuration of the encoded stream output by the image encoding device 100. The example of FIG. 4B corresponds to the fifth example of the second encoding branch 170 described with reference to FIG. 2E. Referring to FIG. 4B, a first encoded stream ST1 and encoded streams ST2a and ST2b of two difference images associated with the first encoded stream ST1 are shown.

  The configuration of the first encoded stream ST1 is the same as the configuration described with reference to FIG. 4A. However, a reference parameter used for accessing the encoded streams ST2a and ST2b of two difference images is inserted into the user-defined area USER_SEC.

  Therefore, the decoder can access the encoded streams ST2a and ST2b of the two difference images using the reference parameter inserted in the header area of the first encoded stream ST1. Each of the encoded streams ST2a and ST2b includes at least a header area and a data area. Parameters required for decoding each encoded stream may be inserted into each header area of the encoded streams ST2a and ST2b.

  For example, the reference parameter for accessing the difference image encoded stream ST2b (or ST2a) is not the header area of the first encoded stream ST1, but the difference image encoded stream ST2a (or ST2b). It may be inserted in the header area.

[1-4. Process flow]
In this section, the flow of image encoding processing by the image encoding device 100 will be described with reference to FIGS.

(1) Overall Flow FIG. 5 is a flowchart showing an example of the flow of image encoding processing according to the present embodiment.

  Referring to FIG. 5, first, the RAW image acquisition unit 110 acquires RAW image data generated by an image sensor or input from another device or a storage medium (Step S100). Then, the RAW image acquisition unit 110 outputs the acquired RAW image data to the development unit 120.

  Next, the developing unit 120 generates first full-color image data by performing demosaic processing on the RAW image data input from the RAW image acquisition unit 110 (step S110). Then, the developing unit 120 outputs the generated first full color image data to the correction processing unit 130.

  Next, the correction processing unit 130 corrects the first full-color image data input from the developing unit 120 to generate high gradation image data (step S120). Then, the correction processing unit 130 outputs the generated high gradation image data to the inverse processing unit 140 and the first encoding branch 160.

  Next, in the first encoding branch 160, the depth reduction / YC conversion unit 162 reduces the bit depth of the high gradation image data input from the correction processing unit 130 to 8 bits for each color, and the color space is converted to the RGB space. To YC space to generate first image data (step S130). Then, the depth reduction / YC conversion unit 162 outputs the generated first image data to the first encoding unit 168.

  Next, the first encoding unit 168 encodes the first image data input from the depth reduction / YC conversion unit 162 using the JPEG method to generate a first encoded stream (step S135). Then, the first encoding unit 168 outputs the first encoded stream including the encoded first image data to the stream output unit 190.

  In addition, the inverse processing unit 140 applies reverse processing (of correction processing) such as EV reverse correction, WB reverse correction, or gamma reverse correction to the high-gradation image data input from the correction processing unit 130 to perform the second processing. Full color image data is generated (step S140). Then, the inverse processing unit 140 outputs the generated second full-color image data to the difference calculation unit 150.

  Next, the difference calculation unit 150 calculates a difference between the first full-color image data input from the developing unit 120 and the second full-color image data input from the inverse processing unit 140 (step S145). Then, the difference calculation unit 150 outputs second image data based on the difference image to the second encoding branch 170.

  Next, in the second encoding branch 170, differential encoding processing is executed, and the second encoded stream is output to the stream output unit 190 (step S160). The differential encoding process here will be further described with reference to FIG.

  Next, the stream output unit 190 associates the second encoded stream input from the second encoding branch 170 with the first encoded stream input from the first encoding branch (step S180). The association here can be performed using the reference parameters described above. Then, the stream output unit 190 outputs the first and second encoded streams to a storage medium connected to the image encoding device 100 or another device (step S190).

(2) Differential Encoding Process FIG. 6 is a flowchart illustrating an example of a detailed flow of the differential encoding process illustrated in FIG.

  Referring to FIG. 6, it is first determined whether or not the second image data (difference image) should be processed before encoding the second image data (step S161). For example, in the first and second embodiments described above, the second image data is not processed. On the other hand, in the third and fourth embodiments described above, the second image data can be processed. If it is determined that the second image data should be processed, the difference processing unit 171 executes a difference processing process on the second image data (step S162). The differential machining process here will be further described with reference to FIG.

  Next, it is determined whether or not the size of the second image data (difference image) should be reduced (step S163). For example, in the first and third embodiments described above, the size of the second image data is not reduced. On the other hand, in the second and fourth embodiments described above, the size of the second image data can be reduced. If it is determined that the size of the second image data should be reduced, the size of the second image data is reduced by the reduction unit 173 (step S164).

Next, the normalization unit 175 normalizes the pixel value of the second image data according to the pixel value range supported by the second encoding branch 170 (step S165). Thereby, each pixel value of the second image data may have a positive or negative sign, 0 to 2 N -1 ^(N bit depth) is normalized to the pixel value range. Further, the normalization unit 175 outputs the normalization parameter to the stream output unit 190.

  Next, it is determined whether or not the color space of the second image data should be converted (step S166). For example, when the JPEG method is used to encode the second image data, it can be determined that the color space of the second image data should be converted to the YC space. When it is determined that the color space of the second image data should be converted, the YC conversion unit 177 changes the color space of the second image data from the RGB space to another color space (for example, YC space). Conversion is performed (step S167).

  Next, the second encoding unit 179 encodes the second image data using the JPEG method or another image encoding method to generate a second encoded stream (step S168). Then, the second encoding unit 179 outputs the generated second encoded stream to the stream output unit 190.

(3) Differential Machining Process FIG. 7 is a flowchart showing an example of a detailed flow of the differential machining process shown in FIG.

  The processing in steps S172 to S175 included in the difference processing shown in FIG. 7 is repeated for each pixel of the second image data (step S171).

  In the repetition using each pixel as the target pixel, the difference processing unit 171 first determines whether the high-frequency component of the first image data shows a strong value at the pixel position of the target pixel (step S172). Here, when the high frequency component does not indicate a strong value, the process of the next step S173 is skipped. If the high-frequency component indicates a strong value, the difference processing unit 171 replaces the pixel value of the target pixel with a certain value (step S173).

  Next, the difference processing unit 171 determines whether or not the luminance value of the first image data belongs to the central portion of the pixel value range at the pixel position of the target pixel (step S174). Here, if the luminance value does not belong to the central portion of the pixel value range, the process of the next step S175 is skipped. When the luminance value belongs to the central portion of the pixel value range, the difference processing unit 171 replaces the pixel value of the target pixel with a certain value (step S175).

  When these processes are executed for all the pixels of the second image data, the difference processing process shown in FIG. 7 ends.

(4) Another Example of Overall Flow FIG. 8 is a flowchart showing another example of the flow of image encoding processing according to this embodiment. The processing from step S100 to step S145 shown in FIG. 8 may be the same as the processing described with reference to FIG.

  When the second image data based on the difference image between the first full-color image data and the second full-color image data is generated by the difference calculation unit 150, the difference image separation is performed by the classification unit 180 in the second encoding branch 170. Processing is executed (step S150). Thereby, the difference image data for overexposure and the difference image data for underexposure are separated from the second image data. The difference image separation process here will be further described with reference to FIG.

  Next, the differential coding process as described with reference to FIGS. 6 and 7 is performed on the whiteout difference image data, and an encoded stream including the whiteout difference image data is streamed. The data is output to the output unit 190 (step S160a).

  Also, the differential image processing described with reference to FIGS. 6 and 7 is performed on the blackout difference image data, and an encoded stream including the blackened difference image data is output as a stream. Is output to the unit 190 (step S160b).

  Next, the stream output unit 190 associates the two encoded streams of the difference image input from the second encoding branch 170 with the first encoded stream input from the first encoding branch (step S180). The association here can be performed using the reference parameters described above. Then, the stream output unit 190 outputs these encoded streams to a storage medium connected to the image encoding device 100 or another device (step S190).

(5) Difference Image Separation Processing FIG. 9 is a flowchart showing an example of a detailed flow of the difference image separation processing shown in FIG.

  The processes in steps S152 to S155 included in the difference image separation process shown in FIG. 9 are repeated for each pixel of the second image data (step S151).

  In the repetition using each pixel as the target pixel, the classification unit 180 first acquires the luminance value of the target pixel of the first image data (step S152). Next, the classification unit 180 compares the acquired luminance value with a luminance threshold value (typically, the central value of the pixel value range) (step S153). Here, for example, when the luminance value is not smaller than the luminance threshold, the classification unit 180 classifies the target pixel into the first group for overexposure (step S154). On the other hand, when the luminance value is smaller than the luminance threshold, the classification unit 180 classifies the target pixel into the second group for blackout (step S155).

  Then, the classification unit 180 separates the difference image data for overexposure from the second image data in accordance with the classification result (step S156), and the difference image data for underexposure from the second image data. Separate (step S157).

  Note that the order of the processing steps described in this section is not limited to the order shown in the drawings. For example, in the image encoding process of FIG. 5, the encoding of the first image data in step S135 and the differential encoding process in step S160 may be performed in parallel.

<2. Configuration Example of Image Decoding Device According to One Embodiment>
[2-1. Overall configuration]
FIG. 10 is a block diagram illustrating an example of a configuration of an image decoding device 200 according to an embodiment that decodes an encoded stream output from the image encoding device 100 described above. Referring to FIG. 10, the image decoding apparatus 200 includes a stream acquisition unit 210, a first decoding branch 220, a second decoding branch 230, an inverse processing unit 260, a synthesis unit 270, and an image output unit 280.

(1) Stream Acquisition Unit The stream acquisition unit 210 acquires an encoded stream that is an input for decoding processing by the image decoding device 200. The encoded stream acquired by the stream acquisition unit 210 includes a first encoded stream ST1 encoded by reducing the bit depth of full-color image data having a bit depth larger than 8 bits to 8 bits, and the first encoded stream ST1. The second encoded stream ST2 associated with the encoded stream ST1 may be included. The full-color image data here is image data generated by demosaicing the RAW image data when the image is encoded. The second encoded stream ST2 is generated by applying the first full-color image data and the inverse process of the process applied to the first full-color image data to the high gradation image data that is the result of the process. Second image data IM2 based on the difference image between the second full-color image data and the second full-color image data.

  For example, the stream acquisition unit 210 acquires the first encoded stream ST1, and a reference parameter for accessing the second encoded stream ST2 is inserted in the header area of the first encoded stream ST1. In this case, the second encoded stream ST2 is further acquired using the reference parameter. Then, the stream acquisition unit 210 outputs the acquired first encoded stream ST1 to the first decoding branch 220 and the acquired second encoded stream ST2 to the second decoding branch 230, respectively.

(2) First decoding branch The first decoding branch 220 is a processing branch for decoding the first encoded stream ST1. In the example of FIG. 10, the first decoding branch 220 includes a first decoding unit 222 and an RGB conversion / depth extension unit 228.

  The first decoding unit 222 decodes the first encoded stream ST1 input from the stream acquisition unit 210 by the JPEG method, and generates first image data IMd1 having a bit depth of 8 bits. That is, the first decoding unit 222 may be a general JPEG decoder. The decoding process by the first decoding unit 222 typically includes entropy decoding, inverse quantization, and inverse orthogonal transform, which are executed for each block arranged in the image. The first decoding unit 222 outputs the first image data IMd1 generated as a result of such decoding to the RGB conversion / depth extension unit 228.

  The RGB conversion / depth extension unit 228 converts the color space of the first image data IMd1 from the YC space to the RGB space, and extends the bit depth to each color M (M> 8, for example, M = 12) bits. The high gradation image data IM1 is restored. Then, the RGB conversion / depth extension unit 228 outputs the restored high gradation image data IM1 to the inverse processing unit 260.

(3) Inverse processing unit The inverse processing unit 260 applies the inverse processing of the processing applied at the time of encoding (for example, correction processing by the correction processing unit 130 shown in FIG. 1) to the high gradation image data IM1. First full color image data IM0 ′ is generated. The reverse processing by the reverse processing unit 260 can include, for example, EV reverse correction, WB reverse correction, or gamma reverse correction. Then, the inverse processing unit 260 outputs the generated first full-color image data IM0 ′ to the synthesizing unit 270.

(4) Second decoding branch The second decoding branch 230 is a processing branch for decoding the second encoded stream ST2. In the example of FIG. 10, the second decoding branch 230 includes a second decoding unit 231. The second decoding unit 231 decodes the second encoded stream ST2 input from the stream acquisition unit 210 by the image encoding method used when encoding the second encoded stream ST2, Second image data IM2 is generated. The image encoding method used when encoding the second encoded stream ST2 can be, for example, the JPEG method, the extended JPEG method, the JPEG2000 method, or the JPEG XR method. Then, the second decoding unit 231 outputs the second image data IM2 generated as a result of the decoding to the synthesis unit 270.

  The second decoding branch 230 may include a further processing unit that applies post-processing to the second image data IM2 after the second decoding unit 231. The post-processing applied to the second image data IM2 may include, for example, one or more of color space conversion, pixel value denormalization, and image size expansion. Some specific embodiments of the second decoding branch 230 will be further described later.

(5) Synthesizer The synthesizer 270 synthesizes the first full-color image data IM0 ′ input from the inverse processor 260 and the second image data IM2 input from the second decoding branch 230 to obtain the original. Full color image data IMrec, which is an image, is reconstructed. Then, the synthesis unit 270 outputs the reconstructed full color image data IMrec to the image output unit 280.

(6) Image Output Unit The image output unit 280 outputs the full color image data IMrec input from the synthesizing unit 270 to a device that stores the full color image data IMrec or displays a full color image.

[2-2. Details of the second branch]
In this section, five detailed examples of the second decoding branch 230 of the image decoding apparatus 200 illustrated in FIG. 10 will be described.

(1) First Example FIG. 11A is a block diagram showing a first example of a detailed configuration of the second decoding branch 230. Referring to FIG. 11A, the second decoding branch 230 includes a second decoding unit 231a, an RGB conversion unit 233, and a denormalization unit 235.

  The second decoding unit 231a decodes the second encoded stream ST2 input from the stream acquisition unit 210 using the JPEG method, and generates second image data IM2. That is, in the present embodiment, the second decoding unit 231a may be a general JPEG decoder, like the first decoding unit 222. Then, the second decoding unit 231a outputs the generated second image data IM2 to the RGB conversion unit 233.

  The RGB conversion unit 233 converts the color space of the second image data IM2 input from the second decoding unit 231a from the YC space to the RGB space. Then, the RGB conversion unit 233 outputs the second image data IM2 having pixel values expressed in the RGB space to the denormalization unit 235.

The denormalization unit 235 inserts the pixel value of the second image data normalized at the time of encoding in accordance with the pixel value range supported by the second decoding unit 231a into the header area of the encoded stream. Is denormalized to the difference pixel value using the normalization parameter. In the first embodiment, the second decoding unit 231a uses the JPEG method. In this case, the pixel value range supported by the second decoding unit 231a is 0 to 255. On the other hand, the minimum value D _min and the maximum value D _max of the difference pixel value may be indicated by a normalization parameter. Therefore, the inverse normalization unit 235 inversely normalizes each pixel value to the difference pixel value so that the pixel value zero of the second image data becomes the difference pixel value _Dmin and the pixel value 255 becomes the difference pixel value _Dmax. Turn into. Then, the denormalization unit 235 outputs the second image data IM2 having the denormalized pixel value to the synthesis unit 270.

  According to the first embodiment, since both the first and second encoded streams ST1 and ST2 are decoded using the JPEG decoder, the widely used JPEG decoder can be used at a lower cost. The technology according to the present disclosure can be realized.

(2) Second Example FIG. 11B is a block diagram showing a second example of a detailed configuration of the second decoding branch 230. Referring to FIG. 11B, the second decoding branch 230 includes a second decoding unit 231b, an inverse normalization unit 235, and an expansion unit 237.

  The second decoding unit 231b decodes the second encoded stream ST2 input from the stream acquisition unit 210 using the image encoding method used when encoding the second encoded stream ST2, Second image data IM2 is generated. The denormalization unit 235 denormalizes the pixel value of the second image data normalized at the time of encoding in accordance with the pixel value range supported by the second decoding unit 231b into a difference pixel value.

  The enlargement unit 237 enlarges the size of the second image data IM2 in accordance with the size of the full-color image data to be restored when the size of the second image data IM2 is reduced from the size of the full-color image data. . Note that when the size is increased, the pixel values that are insufficient may be interpolated by linear interpolation from surrounding pixel values or application of an interpolation filter. Then, the enlargement unit 237 outputs the second image data IM <b> 2 whose size has been enlarged to the synthesis unit 270.

  According to the second embodiment, since the size of the second image data IM2 is reduced in the encoded stream ST2, the data amount of the second encoded stream ST2 can be reduced. As described above, the second image data IM2 is image data that is used complementarily in order to compensate for the image quality that is deteriorated by encoding the full-color image data. Therefore, when reproduction of a nearly full color image is not required, it is beneficial to reduce the data amount of the encoded stream by reducing the size of the second image data IM2 in this way.

(3) Third Example FIG. 11C is a block diagram illustrating a third example of a detailed configuration of the second decoding branch 230. Referring to FIG. 11C, the second decoding branch 230 includes a second decoding unit 231b and a denormalization unit 235.

  The second decoding unit 231b decodes the second encoded stream ST2 input from the stream acquisition unit 210 using the image encoding method used when encoding the second encoded stream ST2, Second image data IM2 is generated. The denormalization unit 235 denormalizes the pixel value of the second image data normalized at the time of encoding in accordance with the pixel value range supported by the second decoding unit 231b into a difference pixel value. Then, the denormalization unit 235 outputs the second image data IM2 having the denormalized pixel value to the synthesis unit 270.

  For example, when an image encoding method that can encode and decode pixel values in the RGB space without undergoing color space conversion is used, it is possible to employ such a simple configuration as in the third embodiment. it can.

(4) Fourth Example FIG. 11D is a block diagram showing a fourth example of a detailed configuration of the second decoding branch 230. The fourth embodiment is a combination of the first and second embodiments described above. Referring to FIG. 11D, the second decoding branch 230 includes a second decoding unit 231a, an RGB conversion unit 233, an inverse normalization unit 235, and an enlargement unit 237.

  The second decoding unit 231a decodes the second encoded stream ST2 input from the stream acquisition unit 210 using the JPEG method, and generates second image data IM2. The RGB conversion unit 233 converts the color space of the second image data IM2 input from the second decoding unit 231a from the YC space to the RGB space. The denormalization unit 235 denormalizes the pixel value of the second image data IM2 normalized at the time of encoding in accordance with the pixel value range supported by the second decoding unit 231a into a difference pixel value. . When the size of the second image data IM2 is reduced from the size of the full color image data, the enlargement unit 237 enlarges the size of the second image data IM2 in accordance with the size of the full color image data. Then, the enlargement unit 237 outputs the second image data IM <b> 2 whose size has been enlarged to the synthesis unit 270.

(5) Fifth Example FIG. 11E is a block diagram illustrating a fifth example of a detailed configuration of the second decoding branch 230. Referring to FIG. 11E, the second decoding branch 230 includes a second decoding unit 231a, an RGB conversion unit 233, a denormalization unit 235, an enlargement unit 237, a second decoding unit 241a, an RGB conversion unit 243, and a denormalization unit 245. , An enlargement unit 247 and a difference synthesis unit 240.

  When the fifth embodiment is employed, the stream acquisition unit 210 illustrated in FIG. 10 causes blackout with the stream ST2a including the difference pixel value of the first group for pixels that may cause overexposure. A stream ST2b including a second group of difference pixel values for the possible pixels is obtained. Among these, the coded stream ST2a of the difference image for overexposure is input to the second decoding unit 231a. Also, the blacked-out difference image encoded stream ST2b is input to the second decoding unit 241a.

  The second decoding unit 231a decodes the coded stream ST2a of the whiteout difference image input from the stream acquisition unit 210 using the JPEG method, and generates whiteout difference image data IM2a. The RGB conversion unit 233 converts the color space of the overexposed difference image data IM2a from the YC space to the RGB space. The inverse normalization unit 235 performs inverse normalization on the pixel value of the differential image data IM2a for overexposure normalized at the time of encoding according to the pixel value range supported by the second decoding unit 231a. Turn into. The enlargement unit 237 enlarges the size of the difference image data IM2a for overexposure to the size of the full color image data when the size of the difference image data IM2a for overexposure is reduced from the size of the full color image data. To do. Then, the enlargement unit 237 outputs the oversized difference image data IM2a for overexposure to the difference composition unit 240.

  The second decoding unit 241a decodes the blacked-out difference image encoded stream ST2b input from the stream acquisition unit 210 using the JPEG method, and generates blacked-out difference image data IM2b. The RGB conversion unit 243 converts the color space of the blackout difference image data IM2b from the YC space to the RGB space. The inverse normalization unit 245 inversely normalizes the pixel value of the blackout difference image data IM2b normalized at the time of encoding according to the pixel value range supported by the second decoding unit 241a to the difference pixel value. Turn into. When the size of the blackout difference image data IM2b is reduced from the size of the full color image data, the enlargement unit 247 enlarges the size of the blackout difference image data IM2b according to the size of the full color image data. To do. Then, the enlarging unit 247 outputs the blacked-out difference image data IM2b whose size has been increased to the difference combining unit 240.

  The difference composition unit 240 converts the pixel value of the overexposed difference image data IM2a input from the enlargement unit 237 or the pixel value of the undercut difference image data IM2b input from the enlargement unit 247 according to the pixel position. By selectively adopting, the second image data IM2 is generated. More specifically, for example, the difference synthesizer 240 acquires the luminance value of the first image data IMd1 from the first encoding branch 160. Then, the difference synthesis unit 240 adopts the pixel value of the overexposed difference image data IM2a at the pixel position where the luminance value of the first image data IMd1 exceeds the predetermined luminance threshold, and the luminance value is predetermined. At the pixel position below the luminance threshold, the pixel value of the blackout difference image data IM2b is adopted. The difference combining unit 240 outputs the second image data IM2 combined in this way to the combining unit 270.

  According to the fifth embodiment, a stream in which differential image data for complementing gradations lost due to overexposure and differential image data for complementing gradations lost due to blackout are separately encoded. Is decrypted from In general, the correlation between images between a region where whiteout occurs and a region where blackout occurs is low. Therefore, by encoding the above-described two difference image data separately, it is possible to mitigate the deterioration of the image quality of these areas caused by orthogonal transformation and quantization.

[2-3. Process flow]
In this section, the flow of image decoding processing by the image decoding apparatus 200 will be described with reference to FIGS.

(1) Overall Flow FIG. 12 is a flowchart illustrating an example of the flow of image decoding processing according to the present embodiment.

  Referring to FIG. 12, first, the stream acquisition unit 210 acquires a first encoded stream that is encoded by reducing the bit depth of full-color image data having a bit depth of more than 8 bits to 8 bits ( Step S200). Then, the stream acquisition unit 210 outputs the acquired first encoded stream to the first decoding branch 220.

  Next, in the first decoding branch 220, the first decoding unit 222 decodes the first encoded stream input from the stream acquisition unit 210 by the JPEG method to generate first image data (step S210). ). Then, the first decoding unit 222 outputs the generated first image data to the RGB conversion / depth extension unit 228.

  Further, the stream acquisition unit 210 determines whether the second encoded stream is associated with the first encoded stream (step S215). For example, when a reference parameter for accessing the second encoded stream is inserted in the header area of the first encoded stream, the stream acquisition unit 210 adds the second parameter to the first encoded stream. May be determined to be associated with each other. Here, when it is determined that the second encoded stream is associated with the first encoded stream, the process proceeds to step S220. On the other hand, when it is determined that the second encoded stream is not associated with the first encoded stream, the process proceeds to step S290.

  In step S220, the RGB conversion / depth extension unit 228 converts the color space of the first image data from the YC space to the RGB space, extends the bit depth to each color M bits, and restores the high gradation image data. (Step S220). Then, the RGB conversion / depth extension unit 228 outputs the restored high gradation image data to the inverse processing unit 260.

  Next, the inverse processing unit 260 restores the first full-color image data by applying the above-described inverse processing to the high gradation image data input from the RGB conversion / depth extension unit 228 (step S225). Then, the inverse processing unit 260 outputs the restored first full-color image data IM0 ′ to the synthesis unit 270.

  Further, the stream acquisition unit 210 acquires the second encoded stream using the reference parameter described above (step S230). Then, the stream acquisition unit 210 outputs the acquired second encoded stream to the second decoding branch 230.

  Next, in the second decoding branch 230, differential decoding processing is executed, and second image data is output to the synthesizing unit 270 (step S240). The differential decoding process here will be further described with reference to FIG.

  Next, the synthesizing unit 270 synthesizes the first full-color image data input from the inverse processing unit 260 and the second image data input from the second decoding branch 230 (step S270). Then, the synthesis unit 270 outputs the reconstructed full color image data to the image output unit 280.

  Thereafter, for example, the reconstructed full-color image data is output from the image output unit 280 to the display device, and a full-color image can be displayed (step S280).

  On the other hand, when it is determined in step S215 that the second encoded stream is not associated with the first encoded stream, the first image data generated by the first decoding unit 222 is the image output unit. 280 can be output to the display device. Then, an image can be displayed using the first image data (step S290).

(2) Differential Decoding Process FIG. 13 is a flowchart illustrating an example of a detailed flow of the differential decoding process illustrated in FIG.

  Referring to FIG. 13, first, the second decoding unit 231 uses the second encoded stream input from the stream acquisition unit 210 as the image encoding used when the second encoded stream is encoded. The second image data is generated by decoding using the method (step S241).

  Next, it is determined whether or not the color space of the second image data should be converted (step S242). For example, when the JPEG method is used to decode the second image data, it can be determined that the color space of the second image data should be converted from the YC space to the RGB space. If it is determined that the color space of the second image data should be converted, the RGB conversion unit 233 converts the color space of the second image data into the RGB space (step S243).

  Next, the inverse normalization unit 235 uses the normalization parameter inserted into the header area of the encoded stream to calculate the difference pixel value of the pixel value of the second image data normalized at the time of encoding. Is denormalized (step S244).

  Next, it is determined whether or not the size of the second image data should be enlarged (step S245). For example, in the first and third embodiments described above, the size of the second image data is not enlarged. On the other hand, in the second and fourth embodiments described above, the size of the second image data can be enlarged. If it is determined that the size of the second image data should be enlarged, the size of the second image data is enlarged by the enlargement unit 237 (step S246).

(3) Another Example of Overall Flow FIG. 14 is a flowchart showing another example of the flow of image encoding processing according to the present embodiment. The process from step S200 to step S225 and the process of step S290 illustrated in FIG. 14 may be the same as the process described with reference to FIG.

  When it is determined that the second encoded stream is associated with the first encoded stream, the stream acquisition unit 210 acquires the encoded streams of the two difference images using the reference parameters described above. (Step S230). Then, the stream acquisition unit 210 outputs the acquired encoded stream to the second decoding branch 230.

  Next, in the second decoding branch 230, a differential decoding process is performed on the encoded stream of the whiteout difference image, and whiteout difference image data is generated (step S240a). The process here may be the same as the differential decoding process described with reference to FIG.

  Further, in the second decoding branch 230, a differential decoding process is performed on the encoded stream of the blackout difference image, and blackout difference image data is generated (step S240b). The process here may also be the same as the differential decoding process described with reference to FIG.

  Next, the difference synthesis unit 240 executes a difference synthesis process for synthesizing the two difference image data (step S260). Thereby, second image data is generated. Then, the difference synthesis unit 240 outputs the generated second image data to the synthesis unit 270. The difference synthesis process here will be further described with reference to FIG.

  Next, the synthesizing unit 270 synthesizes the first full-color image data input from the inverse processing unit 260 and the second image data input from the difference synthesizing unit 240 (step S270). Then, the synthesis unit 270 outputs the reconstructed full color image data to the image output unit 280.

  Thereafter, for example, the reconstructed full-color image data is output from the image output unit 280 to the display device, and a full-color image can be displayed (step S280).

(4) Difference Synthesis Process FIG. 15 is a flowchart showing an example of a detailed flow of the difference synthesis process shown in FIG.

  The processes in steps S262 to S265 included in the difference synthesis process shown in FIG. 15 are repeated for each pixel of the second image data (step S261).

  In the repetition using each pixel as the target pixel, the difference synthesis unit 240 first acquires the luminance value of the target pixel of the first image data (step S262). Next, the difference synthesizer 240 compares the acquired luminance value with a luminance threshold value (typically, the central value of the pixel value range) (step S263). Here, for example, when the luminance value is not smaller than the luminance threshold value, the difference synthesis unit 240 adopts the pixel value of the overexposed difference image data as the pixel value of the target pixel of the second image data ( Step S264). On the other hand, when the luminance value is smaller than the luminance threshold, the difference synthesis unit 240 adopts the pixel value of the blackout difference image data as the pixel value of the target pixel of the second image data (step S265). .

  When these processes are executed for all the pixels of the second image data, the difference synthesis process shown in FIG. 15 ends.

  Note that the order of the processing steps described in this section is not limited to the order shown in the drawings. For example, in the image decoding process of FIG. 14, the two differential decoding processes of step S240a and step S240b may be performed in parallel.

<3. Application example>
The technology according to the present disclosure can be applied to various products that handle full-color images developed from RAW images. In this section, two typical applications will be described with reference to FIGS. 16A and 16B. In the first application example, an image processing apparatus 300 that can correspond to a digital camera is realized. In the second application example, an image processing apparatus 400 that can correspond to a PC (Personal Computer) on which a photo editor operates is realized.

(1) First Application Example FIG. 16A is a block diagram illustrating an example of a configuration of an image processing apparatus 300 according to a first application example. Referring to FIG. 16A, the image processing apparatus 300 includes an operation unit 310, a control unit 320, an image sensor 330, an encoding unit 100, a storage medium 340, a decoding unit 200, and a display 350.

  The operation unit 310 provides an operation interface for the user to operate various functions of the image processing apparatus 300.

  The control unit 320 controls the overall functions of the image processing apparatus 300. For example, the control unit 320 causes the image sensor 330 to capture a RAW image in response to a first user operation received by the operation unit 310. Then, the control unit 320 causes the storage medium 340 to store the encoded stream generated by the encoding unit 100 based on the captured RAW image. Further, the control unit 320 inputs the encoded stream stored in the storage medium 340 to the decoding unit 200 in accordance with the second operation of the user, and causes the decoding unit 200 to reconstruct full-color image data. Then, the control unit 320 causes the display 350 to display a full color image using the reconstructed full color image data.

  The image sensor 330 includes an image sensor such as a CCD or a CMOS, and senses real world light to capture a RAW image.

  The encoding unit 100 has a configuration equivalent to that of the image encoding device 100 described above. The encoding unit 100 acquires the RAW image data generated by the image sensor 330, executes the above-described image encoding process, and outputs the first and second encoded streams to the storage medium 340. The second encoded stream is associated with the first encoded stream.

  The storage medium 340 stores the first and second encoded streams output from the encoding unit 100.

  The decoding unit 200 has a configuration equivalent to that of the image decoding device 200 described above. The decoding unit 200 acquires the first and second encoded streams from the storage medium 340, executes the above-described image decoding process, and reconstructs full-color image data having a bit depth equivalent to RAW image data for each color. .

  The display 350 includes a display element such as liquid crystal or OLED (Organic Light Emitting Diode), and displays a full-color image using the full-color image data reconstructed by the decoding unit 200.

(2) Second Application Example FIG. 16B is a block diagram illustrating an example of a configuration of an image processing apparatus 400 according to a second application example. Referring to FIG. 16B, the image processing apparatus 400 includes an operation unit 410, a control unit 420, a storage medium 440, a communication interface 450, and a display 460.

  The operation unit 410 provides an operation interface for the user to operate various functions of the image processing apparatus 400.

  The control unit 420 controls overall functions of the image processing apparatus 400. In this application example, the control unit 420 includes an encoding unit 100, a decoding unit 200, and an application unit 430.

  The application unit 430 is an application module that implements a photo editor function. The application unit 430, for example, from the first and second encoded streams stored in the storage medium 440 or received via the communication interface 450 according to the first operation of the user received by the operation unit 410. Full-color image data is reconstructed by the decoding unit 200. Then, the application unit 430 causes the display 460 to display a full color image using the reconstructed full color image data. In addition, after the displayed full-color image is edited by the user, the application unit 430 causes the encoding unit 100 to encode the edited full-color image data in accordance with the user's second operation. Then, the application unit 430 stores the first and second encoded streams generated by the encoding unit 100 in the storage medium 440.

  The encoding unit 100 is an encoding module having a configuration equivalent to that of the image encoding device 100 described above. The encoding unit 100 acquires full-color image data edited by the user, executes the above-described image encoding processing, and outputs the first and second encoded streams to the storage medium 440. The second encoded stream is associated with the first encoded stream.

  The decoding unit 200 is a decoding module having a configuration equivalent to that of the image decoding device 200 described above. The decoding unit 200 acquires the first and second encoded streams specified by the application unit 430 and executes the above-described image decoding process, and obtains full-color image data having a bit depth equivalent to the RAW image data for each color. Reconfigure.

  The storage medium 440 stores the first and second encoded streams that are output from the encoding unit 100 or input to the decoding unit 200.

  The communication interface 450 connects the image processing apparatus 400 to another apparatus according to an arbitrary wired communication system or wireless communication system.

  The display 460 includes a display element such as a liquid crystal or an OLED, and displays a full color image using the full color image data reconstructed by the decoding unit 200. The display 460 may also display a graphical user interface (GUI) for a user to edit a full color image under the control of the application unit 430.

<4. Summary>
Up to this point, an embodiment of the technology according to the present disclosure and an application example thereof have been described in detail with reference to FIGS. According to the above-described embodiment, when encoding full-color image data, the first image data encoded by the JPEG method through various processes is stored in the first encoded stream, and is generated by these processes. Second image data based on the difference image representing the error is stored in the second encoded stream, and the second encoded stream is associated with the first encoded stream. At the time of decoding, the first encoded stream is decoded by the JPEG method to generate first image data, the second encoded stream is decoded to generate second image data, and the decoding is performed. Full-color image data is reconstructed using the first and second image data. Therefore, it is possible to reconstruct an image with a higher gradation by compensating the fine gradation that is not expressed in the first image data encoded and decoded by the 8-bit JPEG method with the second image data. . In addition, since the first image data itself is encoded by the JPEG method, at least the first image data can be decoded by a widely used JPEG decoder, that is, compatibility with the JPEG decoder is possible. Secured.

  In addition, when the second image data based on the difference image is also encoded by the JPEG method, the existing JPEG engine (encoder / decoder) can be used for both the first and second image data. The technology according to the present disclosure can be realized at a low cost.

  Further, the gradation of the difference image is usually small and easily lost through various processes after the demosaic process. However, before encoding the difference image, the pixel value is normalized according to the pixel value range supported by the image encoding method, so that the difference image can be encoded without losing a small gradation. it can.

  In addition, the size of the second image data may be reduced before encoding the second image data. Further, the pixel value of the second image data at the pixel position where the high frequency component shows a strong value may be replaced with a constant value. Further, the pixel value of the second image data at the pixel position indicating the luminance value belonging to the central portion of the pixel value range may be replaced with a constant value. Accordingly, it is possible to increase the encoding efficiency of the second encoded stream without greatly reducing the image quality compensated with the second image data.

  The encoding of the second image data may be performed separately for an area where overexposure may occur and an area where blackout may occur. In general, the correlation between images between a region where whiteout occurs and a region where blackout occurs is low. Therefore, by separately encoding and decoding the difference image for these regions, the image quality compensated with the second image data in these regions can be improved.

  Further, according to the above-described embodiment, a parameter for accessing the second encoded stream is inserted into the header area of the first encoded stream. According to such a configuration, a decoder that intends to decode an image first determines whether the first encoded stream is the same as the general image decoding process regardless of whether or not the second encoded stream is associated. Read the header. Then, when the second encoded stream is not associated, the decoder can decode only the first image data that has been JPEG encoded. On the other hand, when the second encoded stream is associated, the decoder can further decode the second image data to reconstruct full-color image data. In this case, since the processing flow when only JPEG-encoded image data is decoded is not modified, it is easy to extend the existing JPEG engine and implement the technology according to the present disclosure.

  Note that a series of control processing by each device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. For example, a program constituting the software is stored in advance in a storage medium provided inside or outside each device. Each program is read into a RAM (Random Access Memory) at the time of execution and executed by a processor such as a CPU (Central Processing Unit).

  The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

The following configurations also belong to the technical scope of the present disclosure.
(1)
A first encoded stream encoded by reducing the bit depth of full-color image data having a bit depth of more than 8 bits to 8 bits, and a second encoded stream associated with the first encoded stream A stream acquisition unit for acquiring
A first decoding unit that decodes the first encoded stream by a JPEG (Joint Photographic Experts Group) method to generate first image data;
A second decoding unit for decoding the second encoded stream to generate second image data;
A combining unit that combines the full-color image data restored using the first image data and the second image data;
An image processing apparatus comprising:
(2)
The second image data includes first full-color image data and second full-color image data generated by applying a reverse process of the process applied to the first full-color image data to the result of the process. The image processing apparatus according to (1), based on a difference image of
(3)
The image processing apparatus according to (2), wherein the second decoding unit decodes the second encoded stream by a JPEG method.
(4)
The image processing apparatus further includes a denormalization unit that denormalizes a pixel value of the second image data normalized according to a pixel value range supported by the second decoding unit. The image processing apparatus according to 2) or (3).
(5)
The image processing apparatus further includes an enlargement unit that enlarges the size of the second image data generated by the second decoding unit in accordance with the size of full-color image data to be restored,
The combining unit combines the full-color image data restored using the first image data and the second image data enlarged by the enlargement unit;
The image processing apparatus according to (2) or (3).
(6)
The second encoded stream includes a stream including a first group of difference pixel values and a stream including a second group of difference pixel values;
The first group is a group for pixels in which overexposure may occur,
The second group is a group for pixels in which blackout may occur,
The second decoding unit decodes two streams of the first group and the second group, respectively.
The second image data is generated by selectively adopting pixel values decoded from the first group of streams or pixel values decoded from the second group of streams according to pixel positions.
The image processing apparatus according to any one of (2) to (5).
(7)
The stream acquisition unit acquires the second encoded stream using a parameter inserted in a header added to the first encoded stream, and any one of (2) to (6) The image processing apparatus according to item.
(8)
The image processing apparatus according to any one of (2) to (7), wherein the first full-color image data is image data generated by demosaicing RAW image data.
(9)
The said image processing apparatus is further provided with the display control part which displays a full-color image on a display apparatus using the full-color image data reconfigure | reconstructed by the said synthetic | combination part, The said any one of (1)-(8) The image processing apparatus described.
(10)
A processing unit for generating first image data from first full-color image data having a bit depth greater than 8 bits;
A first encoder that encodes the first image data with a bit depth reduced to 8 bits according to a JPEG (Joint Photographic Experts Group) method to generate a first encoded stream;
Applying an inverse process of the process applied to the first full-color image data by the processing unit to the result of the process to generate a second full-color image data;
A second encoding unit that encodes second image data based on a difference image between the first full-color image data and the second full-color image data to generate a second encoded stream;
A stream output unit for outputting the second encoded stream in association with the first encoded stream;
An image processing apparatus comprising:
(11)
The image processing apparatus according to (10), wherein the second encoding unit encodes the second image data by a JPEG method.
(12)
The image processing apparatus further includes a normalization unit that normalizes a pixel value of the second image data according to a pixel value range supported by the second encoding unit. The image processing apparatus according to 11).
(13)
The image processing apparatus further includes a reduction unit that reduces the size of the second image data,
The second encoding unit encodes the second image data reduced by the reduction unit;
The image processing apparatus according to any one of (10) to (12).
(14)
The image processing apparatus further includes a difference processing unit that replaces a pixel value of the second image data at a pixel position where the high-frequency component shows a strong value in the first full-color image data with a constant value. The image processing apparatus according to any one of 10) to (13).
(15)
The image processing apparatus further includes a difference processing unit that replaces the pixel value of the second image data at a pixel position whose luminance belongs to the central portion of the pixel value range in the first full-color image data with a constant value. The image processing apparatus according to any one of (10) to (14).
(16)
The image processing apparatus includes a plurality of groups including a first group for pixels that may cause overexposure and a second group for pixels that may cause blackout using a predetermined luminance threshold. A classification unit that classifies each pixel into any one of
The second encoding unit separately encodes the second image data for the first group of pixels and the second image data for the second group of pixels.
The image processing apparatus according to any one of (10) to (15).
(17)
The stream output unit inserts a parameter for accessing the second encoded stream into a header added to the first encoded stream, according to any one of (10) to (16). The image processing apparatus described.
(18)
The image processing apparatus according to any one of (10) to (17), wherein the first full-color image data is image data generated by demosaicing the RAW image data.
(19)
The image processing device according to (18), further including an image sensor that captures a RAW image.
(20)
Computer
A first encoded stream encoded by reducing the bit depth of full-color image data having a bit depth of more than 8 bits to 8 bits, and a second encoded stream associated with the first encoded stream A stream acquisition unit for acquiring
A first decoding unit that decodes the first encoded stream by a JPEG (Joint Photographic Experts Group) method to generate first image data;
A second decoding unit for decoding the second encoded stream to generate second image data;
A combining unit that combines the full-color image data restored using the first image data and the second image data;
Program to function as.
(21)
Computer
A processing unit for generating first image data from first full-color image data having a bit depth greater than 8 bits;
A first encoder that encodes the first image data with a bit depth reduced to 8 bits according to a JPEG (Joint Photographic Experts Group) method to generate a first encoded stream;
Applying an inverse process of the process applied to the first full-color image data by the processing unit to the result of the process to generate a second full-color image data;
A second encoding unit that encodes second image data based on a difference image between the first full-color image data and the second full-color image data to generate a second encoded stream;
A stream output unit for outputting the second encoded stream in association with the first encoded stream;
Program to function as.
(22)
A first encoded stream encoded by reducing the bit depth of full-color image data having a bit depth of more than 8 bits to 8 bits, and a second encoded stream associated with the first encoded stream And getting
Decoding the first encoded stream using a JPEG (Joint Photographic Experts Group) method to generate first image data;
Decoding the second encoded stream to generate second image data;
Combining full-color image data restored using the first image data and the second image data;
An image processing method including:
(23)
Generating first image data from first full color image data having a bit depth greater than 8 bits;
Encoding the first image data with a bit depth reduced to 8 bits by a JPEG (Joint Photographic Experts Group) method to generate a first encoded stream;
Applying a reverse process of the process applied to the first full-color image data to the result of the process to generate second full-color image data;
Encoding second image data based on a difference image between the first full-color image data and the second full-color image data to generate a second encoded stream;
Outputting the second encoded stream in association with the first encoded stream;
An image processing method including:

DESCRIPTION OF SYMBOLS 100 Image coding apparatus 110 Raw image acquisition part 120 Development part 130 Correction | amendment process part 140 Inverse process part 150 Difference calculating part 162 Depth reduction and YC conversion part 168 1st encoding part 180 Classification | category part 171,181 Difference process part 173,183 Reduction unit 175, 185 Normalization unit 179, 179a, 179b, 189a Second encoding unit 190 Stream output unit 200 Image decoding device 210 Stream acquisition unit 222 First decoding unit 228 RGB conversion / depth extension unit 231, 231a, 231b, 241a Second decoding unit 235, 245 Inverse normalization unit 237, 247 Enlargement unit 240 Difference synthesis unit 260 Inverse processing unit 270 Synthesis unit 280 Image output unit 300, 400 Image processing device 320, 420 Control unit 330 Image sensor 340, 440 Storage Medium

Claims (20)

The main image decoder for decoding the main picture coded stream generated by encoding the corrected image data generated by performing the pair and compensation processing to the input color image data in a predetermined standard encoding scheme When,
A difference image generated by encoding difference image data between the input full-color image data and at least the image data after reverse processing generated by performing reverse processing of the correction processing on the image data after correction A differential image decoding unit for decoding an encoded stream by the standard encoding method;
The pixel value of the difference image data that has been normalized according to the pixel value range supported by the standard encoding method is denormalized using a normalization parameter associated with the difference image encoded stream. A denormalization unit;
A combining unit that combines the output of the main image decoding unit and the difference image data after denormalization ;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the input full-color image data is still image data. The image processing apparatus according to claim 2 , wherein the standard encoding method is a JPEG (Joint Photographic Experts Group) method. The main image encoded stream is generated by encoding the corrected image data after reducing the bit depth of the corrected image data to a bit depth smaller than the bit depth of the input full-color image data . The image processing apparatus according to claim 1, wherein the image processing apparatus is a stream . An enlargement unit for enlarging the size of the difference image data output by the difference image decoding unit according to the size of the full-color image data to be restored;
The synthesizing unit synthesizes the full-color image data restored using the corrected image data and the differential image data after denormalization enlarged by the enlarging unit;
The image processing apparatus of any one of Claims 1-4.   The image processing apparatus according to claim 1, wherein the input full-color image data is image data generated from RAW image data.   The image processing apparatus according to claim 1, wherein the correction process includes one or more of EV (Exposure Value) correction, WB (White Balance) correction, and gamma correction. It decodes the main picture coded stream generated by encoding the corrected image data generated by performing the pair and compensation processing to the input color image data in a predetermined standard encoding scheme, the correction Output post-image data,
A difference image generated by encoding difference image data between the input full-color image data and at least the image data after reverse processing generated by performing reverse processing of the correction processing on the image data after correction Decoding an encoded stream with the standard encoding method and outputting the difference image data;
The pixel value of the difference image data that has been normalized according to the pixel value range supported by the standard encoding method is denormalized using a normalization parameter associated with the difference image encoded stream. And
Combining the corrected image data and the denormalized difference image data;
An image processing method including: Computer
The main image decoder for decoding the main picture coded stream generated by encoding the corrected image data generated by performing the pair and compensation processing to the input color image data in a predetermined standard encoding scheme When,
A difference image generated by encoding difference image data between the input full-color image data and at least the image data after reverse processing generated by performing reverse processing of the correction processing on the image data after correction A differential image decoding unit for decoding an encoded stream by the standard encoding method;
The pixel value of the difference image data that has been normalized according to the pixel value range supported by the standard encoding method is denormalized using a normalization parameter associated with the difference image encoded stream. A denormalization unit;
A combining unit that combines the output of the main image decoding unit and the difference image data after denormalization ;
Program to function as. The corrected image data generated by performing the pair and compensation processing to the input color image data is encoded at a predetermined standard encoding scheme, the main image coding unit for outputting a main image encoded stream,
Supports pixel values of difference image data between the input full-color image data and at least the post-correction image data generated by performing reverse processing of the correction processing on the post-correction image data by the standard encoding method. A normalization unit that normalizes the pixel value range and generates a normalization parameter based on the pixel value range of the difference image data;
A differential image encoding unit that encodes the normalized differential image data using the standard encoding method and outputs a differential image encoded stream;
An associating unit for associating the difference image encoded stream and the normalization parameter with the main image encoded stream;
An image processing apparatus comprising:   The image processing apparatus according to claim 10, wherein the input full-color image data is still image data. The image processing apparatus according to claim 11 , wherein the standard encoding method is a JPEG (Joint Photographic Experts Group) method. A depth reduction unit that reduces the bit depth of the corrected image data;
The main image encoding unit encodes the corrected image data having the bit depth reduced by the depth reduction unit;
The image processing apparatus according to claim 10 . A reduction unit for reducing the size of the difference image data,
The difference image encoding unit encodes the reduced difference image data after normalization reduced by the reduction unit as the difference image data.
The image processing apparatus according to claim 10 . The image processing apparatus according to claim 10 , wherein the associating unit inserts a parameter for accessing the difference image encoded stream into a header added to the main image encoded stream. The image processing apparatus according to claim 10 , wherein the input full-color image data is image data generated from RAW image data. The image processing apparatus according to claim 16 , further comprising an image sensor that captures a RAW image.   The image processing apparatus according to claim 10, wherein the correction process includes one or more of EV (Exposure Value) correction, WB (White Balance) correction, and gamma correction. And that the corrected image data generated by performing the pair and compensation processing to the input color image data is encoded at a predetermined standard encoding scheme, and outputs the main image encoded stream,
A normalization parameter is set based on a pixel value range of difference image data between the input full-color image data and at least the post-correction image data generated by performing reverse processing of the correction processing on the post-correction image data. Generating,
Normalizing the pixel values of the difference image data to a pixel value range supported by the standard encoding method;
Encoding the difference image data after normalization by the standard encoding method and outputting a difference image encoded stream;
Associating the difference image encoded stream and the normalization parameter with the main image encoded stream;
An image processing method including: Computer
The corrected image data generated by performing the pair and compensation processing to the input color image data is encoded at a predetermined standard encoding scheme, the main image coding unit for outputting a main image encoded stream,
Supports pixel values of difference image data between the input full-color image data and at least the post-correction image data generated by performing reverse processing of the correction processing on the post-correction image data by the standard encoding method. A normalization unit that generates a normalization parameter based on the pixel value range of the difference image data and normalization according to the pixel value range to be performed;
A differential image encoding unit that encodes the normalized differential image data using the standard encoding method and outputs a differential image encoded stream;
An associating unit for associating the difference image encoded stream and the normalization parameter with the main image encoded stream;
Program to function as.

Images