JP2019022080A - Coding apparatus, coding method, and program - Google Patents

Abstract

To provide a coding apparatus, a coding method, and a program that can perform efficient coding according to an amount of noise.SOLUTION: A coding apparatus includes determination means that estimates a noise amount included in image data and determines whether the noise amount is greater than a threshold value, frequency conversion means that generates a conversion coefficient by frequency-converting the image data when the noise amount is larger than a threshold value, and coding means that encodes the conversion coefficient when the noise amount is greater than the threshold value and encode the image data when the noise amount is less than the threshold value.SELECTED DRAWING: Figure 5

Description

  The present invention relates to an encoding device, an encoding method, and a program.

  Digital imaging devices such as digital still cameras and digital video cameras are widely used and are generally used. These digital imaging devices generate digital image data by converting light received by an imaging device such as a CCD or CMOS sensor into a digital signal. In the process of generating digital image data, dark current noise, thermal noise, shot noise, and the like are generated due to the characteristics of the image sensor and the circuit, and random noise (random noise) is mixed into the digital image data.

  With recent miniaturization of imaging devices and higher pixel counts, the pixel pitch has been minimized, so that random noise is conspicuous. Especially when imaging sensitivity is increased, random noise is noticeably generated. From this point of view, when recording imaging data, it is desirable to perform encoding in consideration of a random noise component and record it. In view of this, a method is disclosed in which encoding is performed by selectively switching quantization tables in accordance with imaging conditions (see, for example, Patent Document 1).

Japanese Patent No. 4765240

  However, when RAW data is encoded and recorded, lossless encoding without information loss is important because the materiality is important, and quantization may not be used. Thus, a method for efficiently performing lossless encoding in consideration of the amount of noise mixed in the image data is desired.

  An object of the present invention is to provide an encoding device, an encoding method, and a program that can perform efficient encoding according to the amount of noise.

  The encoding apparatus of the present invention estimates a noise amount included in image data, determines whether or not the noise amount is greater than a threshold value, and if the noise amount is greater than a threshold value, the image data Frequency conversion means for generating a transform coefficient by frequency transforming, and when the amount of noise is greater than a threshold, encoding is performed on the transform coefficient, and when the amount of noise is less than a threshold, Coding means for coding image data.

  According to the present invention, efficient encoding according to the amount of noise can be performed.

It is a block diagram which shows the structural example of the imaging device by 1st Embodiment. It is a figure which shows the structural example of RAW data. It is a figure which shows subband division | segmentation. It is a figure which shows a MED prediction method. It is a flowchart which shows the encoding method by 1st Embodiment. It is a graph which shows the relationship between ISO sensitivity and the amount of generated codes. It is a graph which shows the relationship between ISO sensitivity and a standard deviation. It is a flowchart which shows the encoding method by 2nd Embodiment. It is a figure which shows the example of a division | segmentation of RAW data. It is a flowchart which shows the encoding method by 3rd Embodiment.

(First embodiment)
FIG. 1A is a block diagram illustrating a configuration example of the imaging apparatus 100 according to the first embodiment of the present invention. The imaging device 100 can be applied to a smart phone, a tablet, an industrial camera, a medical camera, and the like in addition to a digital camera and a video camera. The imaging device 100 is an encoding device capable of lossless encoding of RAW data, and includes a control unit 101, an imaging unit 102, a memory I / F (interface) unit 103, a memory 104, a RAW data encoding unit 105, and a recording processing unit. 106 and a recording medium 107.

  The control unit 101 controls each processing unit that configures the imaging apparatus 100. The imaging unit 102 is a lens optical system capable of optical zoom including an optical lens, an aperture, a focus control and a lens driving unit, and an imaging such as a CCD image sensor or a CMOS sensor that converts optical information from the lens optical system into an electrical signal. Including elements. The imaging unit 102 generates RAW data (image data) by converting an electrical signal obtained by the imaging element into a digital signal. The control unit 101 writes the RAW data generated by the imaging unit 102 to the memory 104 via the memory I / F unit 103.

  FIG. 2 is a diagram illustrating a configuration example of RAW data. The RAW data has an effective pixel area 201 and an optical black area 202. The optical black area 202 includes a horizontal optical black area 203 and a vertical optical black area 204. The imaging element of the imaging unit 102 includes a plurality of optical black pixels that are shielded from light and a plurality of effective pixels that are not shielded from light. The effective pixel area 201 is an area of RAW data generated by effective pixels. The optical black area 202 is an area of RAW data generated by optical black pixels. The RAW data is composed of pixel data of four color elements of R (red), G1 (green), G2 (green), and B (blue) in a Bayer array. Note that the arrangement and color elements of the RAW data are not limited to this configuration, and other methods may be used.

  In FIG. 1A, the memory I / F unit 103 arbitrates memory access requests from the respective processing units, and controls reading or writing with respect to the memory 104. The memory 104 is, for example, a volatile memory, and stores various data output from each processing unit included in the imaging device 100.

  The RAW data encoding unit 105 includes a frequency conversion unit 105a, a prediction difference generation unit 105b, and an entropy encoding unit 105c. The control unit 101 reads the Bayer array RAW data from the memory 104 and outputs the RAW data to the RAW data encoding unit 105. The frequency conversion unit 105a separates the Bayer array RAW data into color element planes, performs frequency conversion processing by reversible discrete wavelet transform at a desired separation level in units of the separated color element planes, and generates a prediction difference generation unit The conversion coefficient is output to 105b.

  FIG. 3A is a diagram illustrating an example of a filter bank configuration for realizing discrete wavelet transform related to the decomposition level 1 subband division processing. In step S301, the frequency conversion unit 105a performs horizontal low-pass filter processing on the input RAW data with the transfer function lpf (z) of the low-pass filter of Expression (1). Next, in step S302, the frequency conversion unit 105a performs 2: 1 decimation on the data after the processing in step S301.

  Next, in step S303, the frequency conversion unit 105a performs a low-pass filter process in the vertical direction on the data after the process in step S302, using the transfer function lpf (z) of the low-pass filter of Expression (1). Next, in step S304, the frequency conversion unit 105a generates the conversion coefficient of the LL subband in FIG. 3B by performing 2: 1 decimation on the data after the processing in step S303.

  In step S305, the frequency conversion unit 105a performs vertical high-pass filter processing on the data after processing in step S302, using the high-pass filter transfer function hpf (z) of Expression (2). Next, in step S306, the frequency conversion unit 105a generates a conversion coefficient of the HL subband in FIG. 3B by performing 2: 1 decimation on the data after the processing in step S305.

  In step S311, the frequency conversion unit 105a performs a high-pass filter process in the horizontal direction on the input RAW data by using the transfer function hpf (z) of the high-pass filter of Expression (2). Next, in step S312, the frequency conversion unit 105a performs 2: 1 decimation on the data after the processing in step S311.

  Next, in step S313, the frequency conversion unit 105a performs a low-pass filter process in the vertical direction on the data after the process in step S312 by the transfer function lpf (z) of the low-pass filter of Expression (1). Next, in step S314, the frequency conversion unit 105a performs 2: 1 decimation on the data after the processing in step S313, thereby generating the conversion coefficient of the LH subband in FIG.

  In step S315, the frequency conversion unit 105a performs a high-pass filter process in the vertical direction on the data after the process in step S312 using the high-pass filter transfer function hpf (z) of Expression (2). Next, in step S316, the frequency conversion unit 105a generates the conversion coefficient of the HH subband in FIG. 3B by performing 2: 1 decimation on the data after the process in step S315.

  As described above, the frequency conversion unit 105a performs discrete wavelet transform on the RAW data in FIG. 2, thereby converting the conversion coefficients of the LL subband LL, the HL subband, the LH subband, and the HH subband in FIG. Generate.

  When the decomposition level is higher than 1, the frequency conversion unit 105a performs subband division hierarchically on the LL subband. For example, when the decomposition level is 3, the frequency conversion unit 105a performs subband division as illustrated in FIG. When the decomposition level is 0, the frequency conversion unit 105a does not perform the discrete wavelet conversion process and outputs the input RAW data as it is to the prediction difference generation unit 105b. Here, the discrete wavelet transform uses a 5-tap low-pass filter and a 3-tap high-pass filter as shown in equations (1) and (2), but uses a filter with a different number of taps and different coefficients. As long as it is a reversible type, other frequency conversion methods may be used.

  In FIG. 1A, the prediction difference generation unit 105b generates a prediction value pd by MED (Media Edge Detector) prediction for the pixel value or conversion coefficient of the RAW data input from the frequency conversion unit 105a.

  FIG. 4 is a diagram illustrating a MED prediction method. The prediction difference generation unit 105b obtains pixel difference or conversion coefficient prediction difference data. Hereinafter, a method for generating prediction difference data for pixel values will be described as an example, but a method for generating prediction difference data for transform coefficients is also the same. The prediction difference generation unit 105b generates a prediction value pd for the value of the encoding target pixel from the values of the surrounding pixels a to d of the encoding target pixel x by MED prediction. When the smaller value of the values of the pixels a and b is greater than or equal to the value of the pixel c, the larger value of the values of the pixels a and b is set to the predicted value pd of the encoding target pixel x. To do. In addition, when the larger value of the values of the pixels a and b is less than or equal to the value of the pixel c, the smaller value of the values of the pixels a and b is used as the predicted value of the encoding target pixel x. Let pd. In other cases, a value obtained by subtracting the value of the pixel c from the addition value of the values of the pixels a and b is set as the predicted value pd of the encoding target pixel x.

  Next, the prediction difference generation unit 105b generates difference data between the value of the encoding target pixel x and the prediction value pd, and outputs the difference data to the entropy encoding unit 105c. Note that the prediction difference generated by the prediction difference generation unit 105b is generated only in the low frequency component subband (LL subband) generated by the frequency conversion unit 105a, and the high frequency component subband (HL subband, LH). This does not apply to subbands and HH subbands.

  The entropy encoding unit 105c performs entropy encoding on the data input from the prediction difference generation unit 105b by, for example, Huffman encoding or Golomb encoding, and generates encoded data. Note that the prediction method and the entropy coding method may be other methods. The control unit 101 writes the encoded data generated by the entropy encoding unit 105 c into the memory 104 via the memory I / F unit 103.

  The control unit 101 reads the encoded data from the memory 104 and outputs it to the recording processing unit 106. The recording processing unit 106 performs a predetermined recording format on the encoded data and records it on the recording medium 107. The recording medium 107 is, for example, a nonvolatile memory.

  Here, since general RAW data has high similarity between adjacent pixel values and high spatial correlation, predictive differential encoding using predictive differences as described above is effective for efficient data compression. It is. However, since RAW data imaged under high ISO sensitivity conditions contain a lot of noise components, the spatial correlation is reduced and the compression efficiency in predictive differential encoding is reduced. Therefore, it is considered that the RAW data encoding unit 105 can improve compression efficiency by performing frequency conversion by frequency conversion processing and performing encoding as data with increased correlation for each frequency band.

  Here, the amount of noise mixed into the RAW data depending on the imaging conditions such as ISO sensitivity is considered to be the same in the effective pixel area 201 and the optical black area 202. Therefore, the imaging apparatus 100 estimates the amount of noise included in the RAW data by analyzing the optical black region 202, determines the effectiveness of frequency conversion in accordance with the amount of noise, and selectively encodes it. Thereby, the imaging device 100 can perform compression efficiently.

  FIG. 5 is a flowchart illustrating the encoding method of the imaging apparatus 100 according to the first embodiment. In step S501, the RAW data encoding unit 105 analyzes the optical black area 202 of the RAW data. Specifically, the RAW data encoding unit 105 encodes a part or the entire pixel value of the optical black area 202, acquires the code amount generated by the encoding, and converts the generated RAW data into the RAW data based on the generated code amount. Estimate the amount of noise contained.

  Next, in step S502, the RAW data encoding unit 105 determines the effectiveness of frequency conversion based on the estimated noise amount. The RAW data encoding unit 105 determines that the frequency conversion is not effective when the noise amount is smaller than the threshold by the determination unit, and advances the process to step S505. In step S505, the frequency conversion unit 105a does not perform frequency conversion, and the prediction difference generation unit 105b generates prediction difference data from pixels around the encoding target pixel in units of color element planes of the input RAW data. Specifically, similarly to FIG. 4, the prediction difference generation unit 105 b generates a prediction value pd by MED prediction from the surrounding pixels a to d of the encoding target pixel x, and the value of the encoding target pixel x and the prediction value Difference data with respect to pd is generated, and the difference data is output to the entropy encoding unit 105c. Next, in step S506, the entropy encoding unit 105c performs entropy encoding on the difference data input from the prediction difference generation unit 105b by, for example, Huffman encoding or Golomb encoding, and the encoded data is obtained. Generate.

  In step S502, the RAW data encoding unit 105 determines that the frequency conversion is effective when the amount of noise is larger than the threshold by the determination unit, and advances the process to step S503. In step S503, the frequency conversion unit 105a performs frequency conversion processing in units of color element planes of input RAW data. Specifically, the frequency conversion unit 105a separates the RAW data of the Bayer array into each color element plane in the same manner as in FIGS. 3A to 3C, and a desired separation level in units of the separated color element planes. Then, frequency conversion processing by discrete wavelet transform is performed to generate transform coefficients. For example, the frequency conversion unit 105a generates conversion coefficients for a low-frequency component subband (LL subband) and a high-frequency component subband (HL subband, LH subband, and HH subband). Next, in step S504, the RAW data encoding unit 105 advances the process to step S505 for the transform coefficient of the low frequency component subband (LL subband). Then, the RAW data encoding unit 105 proceeds to step S506 for the transform coefficients of the high frequency component subbands (HL subband, LH subband, and HH subband). In step S505, the prediction difference generation unit 105b generates prediction difference data from the transform coefficients around the encoding target transform coefficient for the transform coefficient of the low frequency component subband (LL subband). Specifically, the prediction difference generation unit 105b generates a prediction value pd for the encoding target transform coefficient from the peripheral transform coefficients a to d of the encoding target transform coefficient x by MED prediction, as in FIG. Then, the prediction difference generation unit 105b generates difference data between the encoding target transform coefficient x and the prediction value pd, and outputs the difference data to the entropy encoding unit 105c. Next, in step S506, the entropy encoding unit 105c converts the difference data of the low frequency component subband (LL subband) and the transform coefficient of the high frequency component subband (HL subband, LH subband, HH subband). On the other hand, encoded data is generated. For example, the entropy encoding unit 105c performs entropy encoding by Huffman encoding or Golomb encoding, and generates encoded data.

  Next, a method for determining the effectiveness of frequency conversion will be specifically described using an example of actual RAW data. FIG. 6 is a graph showing the relationship between the ISO sensitivity and the generated code amount of the optical black area 202. FIG. 6 shows the relationship between the ISO sensitivity and the generated code amount of the optical black area 202 for RAW data picked up by changing the ISO sensitivity from 100 to 204800 one step at a time in the same scene. “OB” indicates the amount of code generated when the RAW data encoding unit 105 encodes the optical black area 202. “HOB” indicates a code amount generated when the RAW data encoding unit 105 encodes the horizontal optical black area 203. “VOB” indicates a code amount generated when the RAW data encoding unit 105 encodes the vertical optical black area 204. The abscissa indicates the number of steps of ISO sensitivity. “1” indicates “ISO sensitivity = 100”. Thereafter, the ISO sensitivity value doubles every time one step proceeds, and “12” indicates “ISO sensitivity = 100”. 204800 ". The vertical axis represents the ratio of the generated code amount when the generated code amount (reference value) with an ISO sensitivity of 100 is 1.

  In this example of RAW data, the ISO sensitivity = 100 to ISO sensitivity = 400 are almost the same amount of generated codes. After ISO sensitivity 800, the amount of generated code increases in a linear function. This indicates that the amount of generated code increases corresponding to the amount of mixed noise. Therefore, when applied to the flowchart shown in FIG. 5, for example, when the ratio is 2 times or more on the basis of the generated code amount of the optical black area 202 with ISO sensitivity = 100, the amount of mixed noise is extremely large. The spatial correlation is also considered to be extremely low. Therefore, the RAW data encoding unit 105 can determine that the frequency conversion is effective when the ratio of the generated code amount is twice or more. For example, the RAW data encoding unit 105 can determine that the frequency conversion is effective when the ISO sensitivity is 25600 or more based on the ratio of the generated code amount of VOB shown in FIG.

  As described above, the RAW data encoding unit 105 determines that the frequency conversion is effective when the generated code amount of all or part of the optical black region 202 is larger than the threshold. The RAW data encoding unit 105 determines that the frequency conversion is not effective when the generated code amount of the entire or part of the optical black region 202 is smaller than the threshold value.

  The RAW data encoding unit 105 determines that the frequency conversion is effective when the ratio of the generated code amount in the entire optical black region 202 or a part of the generated code amount to the reference value is larger than the threshold value. Also, the RAW data encoding unit 105 determines that the frequency conversion is not effective when the ratio of the generated code amount of the entire optical black region 202 or a part of the generated code amount to the reference value is smaller than the threshold value.

  Here, for simplification of explanation, a method for determining the effectiveness of frequency conversion is shown. However, when the frequency conversion is effective, the decomposition level may be changed according to the generated code amount. Good.

FIG. 1B is a block diagram illustrating another configuration example of the imaging apparatus 100. The imaging apparatus 100 in FIG. 1B is obtained by adding an optical black analysis unit 108 to the imaging apparatus 100 in FIG. The optical black analysis unit 108 is provided outside the RAW data encoding unit 105. In the imaging apparatus 100 in FIG. 1A, as an analysis of the optical black region 202, an example in which the RAW data encoding unit 105 calculates the generated code amount of the optical black region 202 is shown. On the other hand, in the imaging apparatus 100 of FIG. 1B, the optical black analysis unit 108 is provided exclusively, and the optical black analysis unit 108 analyzes the optical black region 202. Specifically, the optical black analysis unit 108 calculates the variance σ ² shown in Expression (3) or the standard deviation σ shown in Expression (4) with respect to the pixel value of the optical black region 202, thereby obtaining the optical black. Variations in pixel values in the black area 202 are analyzed. Here, σ ² is the variance, σ is the standard deviation, n is the number of pixels to be measured, x is the pixel value, and μ is the average value of the pixels to be measured. The optical black analysis unit 108 determines the effectiveness of frequency conversion based on the variance σ ² or the standard deviation σ.

  FIG. 7A is a graph showing the relationship between the ISO sensitivity and the standard deviation of the optical black area 202. FIG. 7A shows the ISO sensitivity and the pixel value of the optical black area 202 with respect to the RAW data obtained by using the same raw data as in the example of FIG. The standard deviation relationship is shown. “OB” indicates the standard deviation of the pixel value of the optical black area 202. “HOB” indicates the standard deviation of the pixel value of the horizontal optical black area 203. “VOB” indicates the standard deviation of the pixel value of the vertical optical black area 204. The horizontal axis indicates the number of steps of ISO sensitivity. “1” indicates “ISO sensitivity = 100”, and the value of ISO sensitivity is doubled every time one step is advanced, and “12” is “ISO sensitivity = 204800”. It is. The vertical axis indicates the ratio of the standard deviation when the standard deviation (reference value) with an ISO sensitivity of 100 is 1.

  In the example of the RAW data, as shown in FIG. 7A, from ISO sensitivity = 100 to ISO sensitivity = 400 shows almost the same standard deviation. After ISO sensitivity 800, the standard deviation increases as a quadratic function, indicating that the standard deviation increases corresponding to the amount of mixed noise.

  FIG. 7B is a graph in which the standard deviation on the vertical axis in FIG. 7A is replaced with a logarithmic scale, and shows the relationship between the ISO sensitivity and the standard deviation of the optical black region 202. The graph showing the ratio of the standard deviation in FIG. 7B can be expressed in a linear function like the graph showing the ratio of the generated code amount in FIG. The optical black analysis unit 108 can determine the effectiveness of frequency conversion using the standard deviation in the same way as when the generated code amount in FIG. 6 is used.

  In other words, the optical black analysis unit 108 determines that the frequency conversion is effective when the standard deviation or variance of all or some of the pixel values in the optical black region 202 is greater than the threshold value. Further, the optical black analysis unit 108 determines that the frequency conversion is not effective when the standard deviation or variance of all or some of the pixel values of the optical black region 202 is smaller than the threshold value.

  Further, the optical black analysis unit 108 determines that the frequency conversion is effective when the ratio of the whole or a part of the pixel values of the optical black region 202 to the standard deviation or variance reference value is larger than the threshold value. Further, the optical black analysis unit 108 determines that the frequency conversion is not effective when the ratio of the whole or a part of the pixel values of the optical black region 202 to the standard deviation or dispersion reference value is smaller than the threshold value.

  Note that the analysis result of the optical black region 202 when the standard ISO sensitivity is 100 may be stored in advance, or may be analyzed and acquired, for example, every time the power is turned on. Good. Further, as can be seen from the graphs of FIG. 6 and FIG. 7B, the generated code amount or the standard deviation ratio of the optical black area 202 shows the same tendency. Therefore, the optical black area to be analyzed may be the optical black area 202, the horizontal optical black area 203, the vertical optical black area 204, or a part of a smaller area where an equivalent result is obtained.

  As described above, the imaging apparatus 100 can accurately estimate the amount of noise mixed in the RAW data by calculating the generated code amount, standard deviation, or variance of the optical black area 202 included in the RAW data. . The imaging apparatus 100 can determine the effectiveness of frequency conversion based on the amount of noise, and can efficiently encode and record it.

(Second Embodiment)
FIG. 8 is a flowchart illustrating an encoding method of the imaging apparatus 100 according to the second embodiment of the present invention, and FIG. 9 is a diagram illustrating an example of dividing RAW data. The imaging device 100 according to the present embodiment has the same configuration as the imaging device 100 of FIG. 1 shown in the first embodiment. The imaging apparatus 100 according to the present embodiment is different from the imaging apparatus 100 according to the first embodiment in the RAW data encoding method. Therefore, a common description is omitted, and the RAW data encoding method is illustrated in FIG. An explanation will be given based on the flowchart. As shown in FIG. 9, the RAW data is divided into an effective pixel area 201, a horizontal optical black area 203, and a vertical optical black area 204 and encoded. Note that the way of division and the number of divisions are not limited to this. For example, the division may be divided into two areas, that is, a vertical optical black area 204 and other areas including the effective pixel area 201.

  In step S801, the RAW data encoding unit 105 encodes the pixel values of the horizontal optical black region 203 and the vertical optical black region 204 of the RAW data, and generates a code. Next, in step S802, the RAW data encoding unit 105 acquires a code amount generated by the encoding. Next, in step S803, the RAW data encoding unit 105 estimates a noise amount based on the generated code amount, and determines the effectiveness of frequency conversion for encoding the effective pixel region 201 based on the noise amount. . For example, when the generated code amount is larger than the threshold value, the RAW data encoding unit 105 determines that the noise amount is larger than the threshold value, and determines that the frequency conversion for the encoding of the effective pixel region 201 is effective. . Further, when the generated code amount is less than the threshold value, the RAW data encoding unit 105 determines that the noise amount is less than the threshold value, and determines that the frequency conversion for encoding the effective pixel region 201 is not effective.

  In step S803, if the RAW data encoding unit 105 determines that the frequency conversion is not effective, the process proceeds to step S806. In step S806, the frequency conversion unit 105a does not perform frequency conversion, and the prediction difference generation unit 105b performs prediction differences from pixels around the encoding target pixel in units of color element planes in the effective pixel region 201 of the input RAW data. Generate data. Specifically, similarly to FIG. 4, the prediction difference generation unit 105 b generates a prediction value pd by MED prediction from the surrounding pixels a to d of the encoding target pixel x, and the value of the encoding target pixel x and the prediction value Difference data with respect to pd is generated, and the difference data is output to the entropy encoding unit 105c. Next, in step S807, the entropy encoding unit 105c performs entropy encoding on the difference data input from the prediction difference generation unit 105b by, for example, Huffman encoding or Golomb encoding, and the encoded data is obtained. Generate.

  In step S803, if the RAW data encoding unit 105 determines that the frequency conversion is valid, the process proceeds to step S804. In step S804, the frequency conversion unit 105a performs frequency conversion processing in units of color element planes in the effective pixel area 201 of the input RAW data. Similarly to FIGS. 3A to 3C, the frequency conversion unit 105a separates the effective pixel area 201 of the RAW data into each color element plane, and performs discrete wavelet at a desired separation level in units of the separated color element planes. A frequency conversion process is performed by conversion to generate a conversion coefficient. For example, the frequency conversion unit 105a generates conversion coefficients for a low-frequency component subband (LL subband) and a high-frequency component subband (HL subband, LH subband, and HH subband). Next, in step S805, the RAW data encoding unit 105 proceeds to step S806 for the transform coefficient of the low frequency component subband (LL subband). Then, the RAW data encoding unit 105 proceeds to step S807 for the transform coefficients of the high frequency component subbands (HL subband, LH subband, and HH subband). In step S806, the prediction difference generation unit 105b generates prediction difference data from the transform coefficients around the encoding target transform coefficient for the transform coefficient of the low frequency component subband (LL subband). As in FIG. 4, the prediction difference generation unit 105b generates a prediction value pd by the MED prediction from the peripheral conversion coefficients a to d of the encoding target conversion coefficient x, and the difference between the encoding target conversion coefficient x and the prediction value pd. Data is generated and the difference data is output to the entropy encoding unit 105c. Next, in step S807, the entropy encoding unit 105c converts the difference data of the low frequency component subband (LL subband) and the transform coefficient of the high frequency component subband (HL subband, LH subband, HH subband). On the other hand, encoded data is generated. For example, the entropy encoding unit 105c performs entropy encoding by Huffman encoding or Golomb encoding, and generates encoded data.

  As described above, the imaging apparatus 100 divides the RAW data into the optical black area 202 and the effective pixel area 201, and performs coding of the optical black area 202 in step S801, so that the noise amount is based on the generated code quantity. Can be estimated accurately. The imaging apparatus 100 can determine the effectiveness of frequency conversion for the effective pixel region 201 based on the amount of noise, and can efficiently encode and record it.

(Third embodiment)
FIG. 10 is a flowchart showing an encoding method of the imaging apparatus 100 according to the third embodiment of the present invention. The imaging device 100 according to the present embodiment has the same configuration as the imaging device 100 of FIG. 1 shown in the first embodiment. The imaging apparatus 100 according to the present embodiment is different from the imaging apparatus 100 according to the first embodiment in the RAW data encoding method. Therefore, a common description is omitted, and the RAW data encoding method is illustrated in FIG. An explanation will be given based on the flowchart.

  In step S1001, the RAW data encoding unit 105 acquires imaging information. Specifically, the RAW data encoding unit 105 mainly uses ISO sensitivity information or exposure time information at the time of imaging as imaging information in order to estimate noise amount due to ISO sensitivity (imaging sensitivity) and exposure time. get. Next, in step S1002, the RAW data encoding unit 105 estimates the amount of noise included in the RAW data based on the imaging information, and determines the effectiveness of frequency conversion based on the amount of noise. This determination condition is a condition determined by the characteristics of the imaging unit 102, and is determined based on a value statistically obtained in advance through experiments. For example, when the ISO sensitivity or the exposure time is larger than the threshold, the RAW data encoding unit 105 determines that the amount of noise is larger than the threshold, and determines that the frequency conversion is effective. Further, when the ISO sensitivity or the exposure time is smaller than the threshold, the RAW data encoding unit 105 determines that the noise amount is smaller than the threshold, and determines that the frequency conversion is not effective.

  If the RAW data encoding unit 105 determines in step S1002 that the frequency conversion is not valid, the process proceeds to step S1005. In step S1005, the frequency conversion unit 105a does not perform frequency conversion, and the prediction difference generation unit 105b generates prediction difference data from pixels around the encoding target pixel in units of color element planes of the input RAW data. Specifically, similarly to FIG. 4, the prediction difference generation unit 105 b generates a prediction value pd by MED prediction from the surrounding pixels a to d of the encoding target pixel x, and the value of the encoding target pixel x and the prediction value Difference data with respect to pd is generated, and the difference data is output to the entropy encoding unit 105c. Next, in step S1006, the entropy encoding unit 105c performs entropy encoding on the difference data input from the prediction difference generation unit 105b by, for example, Huffman encoding or Golomb encoding, and the encoded data is obtained. Generate.

  In step S1002, when the RAW data encoding unit 105 determines that the frequency conversion is effective, the process proceeds to step S1003. In step S1003, the frequency conversion unit 105a performs frequency conversion processing in units of color element planes of input RAW data. Specifically, the frequency conversion unit 105a separates the RAW data of the Bayer array into each color element plane in the same manner as in FIGS. 3A to 3C, and a desired separation level in units of the separated color element planes. Then, frequency conversion processing by discrete wavelet transform is performed to generate transform coefficients. For example, the frequency conversion unit 105a generates conversion coefficients for a low-frequency component subband (LL subband) and a high-frequency component subband (HL subband, LH subband, and HH subband). Next, in step S1004, the RAW data encoding unit 105 advances the process to step S1005 for the transform coefficient of the low frequency component subband (LL subband). Then, the RAW data encoding unit 105 proceeds to step S1006 for the transform coefficients of the high frequency component subbands (HL subband, LH subband, and HH subband). In step S1005, the prediction difference generation unit 105b generates prediction difference data from the transform coefficients around the encoding target transform coefficient for the transform coefficient of the low frequency component subband (LL subband). As in FIG. 4, the prediction difference generation unit 105b generates a prediction value pd by the MED prediction from the peripheral conversion coefficients a to d of the encoding target conversion coefficient x, and the difference between the encoding target conversion coefficient x and the prediction value pd. Data is generated and the difference data is output to the entropy encoding unit 105c. Next, in step S1006, the entropy encoding unit 105c converts the difference data of the low frequency component subband (LL subband) and the transform coefficient of the high frequency component subband (HL subband, LH subband, HH subband). On the other hand, encoded data is generated. For example, the entropy encoding unit 105c performs entropy encoding by Huffman encoding or Golomb encoding, and generates encoded data.

  As described above, the imaging apparatus 100 estimates the amount of noise mixed in the RAW data based on imaging information such as ISO sensitivity information and exposure time information for the RAW data. The imaging apparatus 100 can determine the effectiveness of frequency conversion based on the amount of noise, and can efficiently encode and record it.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

105: RAW data encoding unit, 105a: frequency converting unit, 105b: prediction difference generating unit, 105c: entropy encoding unit

Claims (17)

A determination unit that estimates a noise amount included in the image data and determines whether the noise amount is greater than a threshold;
When the amount of noise is greater than a threshold value, a frequency conversion unit that generates a conversion coefficient by frequency converting the image data;
Encoding means for encoding the transform coefficient when the noise amount is greater than a threshold value, and encoding the image data when the noise amount is less than the threshold value. An encoding device characterized by the above. Further, when the amount of noise is larger than a threshold value, a prediction value for the encoding target transform coefficient among the generated transform coefficients is generated, and difference data between the encoding target transform coefficient and the prediction value is generated. If the amount of noise is greater than a threshold value, a predicted value for the encoding target pixel value in the image data is generated, and difference data between the encoding target pixel value and the predicted value is generated. A prediction difference generation means;
The encoding apparatus according to claim 1, wherein the encoding unit encodes the difference data. The frequency conversion means generates a low-frequency component conversion coefficient and a high-frequency component conversion coefficient,
The prediction difference generation means generates the difference data for the conversion coefficient of the low frequency component when the amount of noise is greater than a threshold value,
The encoding unit performs encoding on the difference data of the low-frequency component conversion coefficient and the high-frequency component conversion coefficient when the amount of noise is greater than a threshold value. 2. The encoding device according to 2.   The encoding apparatus according to claim 1, wherein the frequency conversion unit performs wavelet transform.   The said determination means estimates the noise amount contained in the said image data based on the pixel value of all or one part of the optical black area | region contained in the said image data, The one of Claims 1-4 characterized by the above-mentioned. The encoding device according to item 1.   The determination unit encodes all or a part of pixel values of an optical black area included in the image data, and determines whether the noise amount is larger than a threshold based on a code amount generated by the encoding. The encoding apparatus according to claim 1, wherein:   The determination unit encodes all or a part of pixel values of an optical black area included in the image data, and when the amount of code generated by the encoding is larger than a threshold, the amount of noise is larger than the threshold. The encoding apparatus according to claim 1, wherein if the amount of code generated by the encoding is less than a threshold, it is determined that the amount of noise is less than the threshold. .   The determination unit encodes all or a part of pixel values of the optical black area included in the image data, and when the ratio of the code amount generated by the encoding to a reference value is larger than a threshold value, the noise amount The noise amount is determined to be less than the threshold value when the ratio of the code amount generated by the encoding to the reference value is less than the threshold value. The encoding device according to any one of claims. The image data has an optical black area and an effective pixel area,
The determination means performs encoding on the optical black region,
The encoding apparatus according to claim 6, wherein the encoding unit performs encoding on the effective pixel area.   The determination unit calculates a standard deviation or variance of all or some of the pixel values of the optical black area included in the image data, and based on the standard deviation or variance, whether or not the noise amount is greater than a threshold value. The encoding apparatus according to any one of claims 1 to 5, wherein:   The determination unit calculates a standard deviation or variance of all or some of the pixel values of the optical black area included in the image data, and when the standard deviation or variance is greater than a threshold, the amount of noise is greater than the threshold. 6. The encoding apparatus according to claim 1, wherein it is determined that the amount of noise is less than a threshold value when the standard deviation or variance is less than a threshold value.   The determination means calculates the standard deviation or variance of all or part of the pixel values of the optical black area included in the image data, and when the ratio of the standard deviation or variance to the reference value is greater than a threshold value, The noise amount is determined to be greater than a threshold value, and when the ratio of the standard deviation or variance to a reference value is less than the threshold value, the noise amount is determined to be less than the threshold value. The encoding device according to claim 1.   The encoding apparatus according to claim 1, wherein the determination unit estimates a noise amount included in the image data based on imaging information of the image data.   The encoding apparatus according to claim 1, wherein the determination unit estimates an amount of noise included in the image data based on an imaging sensitivity or an exposure time of the image data. .   The determination means determines that the amount of noise is greater than a threshold when the imaging sensitivity or exposure time of the image data is greater than a threshold, and if the imaging sensitivity or exposure time of the image data is less than a threshold, The encoding apparatus according to claim 1, wherein the amount of noise is determined to be less than a threshold value. A determination step of determining a noise amount included in the image data by a determination unit and determining whether the noise amount is greater than a threshold;
When the amount of noise is greater than a threshold by a frequency conversion means, a frequency conversion step of generating a conversion coefficient by frequency converting the image data;
Encoding means for encoding the transform coefficient when the amount of noise is greater than a threshold, and encoding the image data when the amount of noise is less than the threshold. And an encoding method.   The program for functioning a computer as each means of the encoding apparatus described in any one of Claims 1-15.

Images