JP2022043080A - Image encoding device, image encoding method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to record RAW data compactly by suppressing a data amount of quantization parameters to be included in encoded data while implementing irreversible compression through flexible quantization control in compression of the RAW data.

SOLUTION: An image encoding device includes: generation means for generating multiple pieces of sub-band data from RAW data; quantization parameter generation means which generates a first quantization parameter and generates a second quantization parameter from the first quantization parameter; quantization means which uses the second quantization parameter to quantize the multiple pieces of sub-band data for each segment; and encoding means for encoding a quantization result that is obtained by the quantization, with sub-band data defined as a unit and encoding the first quantization parameter, thereby generating encoded data.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an image coding apparatus, an image coding method, and a program.

CCD or CMOS image sensors are often used in the image sensors of digital cameras these days, and in single-panel image sensors, green, blue, and red pixel data (hereinafter referred to as RAW data) are obtained by passing through a so-called Bayer-arranged color filter. Called) is obtained. In a digital camera, RAW data is subjected to development processing including demosaic processing, noise removal processing, optical distortion correction, color correction processing, etc. to generate final image data. JPEG for still images and JPEG for moving images. H. Compressed image data can be recorded by a coding method such as 264. On the other hand, it is also equipped with a function to record RAW data before the development process so that the development process can be executed according to the user's own preference.

In many cases, an uncompressed or lossless compressed format is adopted as the recording format of the RAW data, and the size of the RAW data generally recorded is larger than that of the compressed image data after development. On the other hand, as the density and the number of pixels of the image sensor increase, it is becoming important to record RAW data compactly. Therefore, as a visually lossless lossy compression format, multiple color channels are generated from the RAW data, two-dimensional discrete wavelet transform is performed for each channel, and each generated subband is individually quantized. A method of compressing and recording as encoded data has been proposed (see Patent Document 1).

Special Table 2002-516540

However, when 4 color channels are generated from RAW data and 3 layers of octave division are performed by 2D discrete wavelet transform, each channel is divided into 10 subbands, so 4 channels × 10 subbands are generated. Will be done. Quantizing these subbands individually requires 4 × 10 = 40 quantization parameters. Further, assuming that finer quantization control in a predetermined block unit is performed according to the features in the screen, the quantization parameters for 4 × 10 × the number of block divisions are required. Since all of these quantization parameters are included in the coded data, the amount of data of the quantization parameters accompanying the compression of the RAW data becomes bloated. In this case, it becomes difficult to compactly record RAW data while maintaining visually lossless image quality.

Therefore, the present invention realizes lossy compression by flexible quantization control in the compression of RAW data, and suppresses the amount of data of the quantization parameter included in the coded data to compactly record the RAW data. Providing the technology that makes it possible.

In order to achieve the above object, the image coding apparatus according to the present invention is
A generation means that performs discrete wavelet transform of two or more decomposition levels and generates multiple subband data from RAW data.
Common quantization in which the plurality of subband data is divided into the same number of quantization units, individually set for each quantization unit, and commonly assigned to a plurality of subband data included in the same quantization unit. A quantization parameter generation means for generating a parameter and generating a quantization parameter for quantizing each of the plurality of subband data based on the common quantization parameter.
A quantization means for quantizing each of the plurality of subband data based on the quantization parameters for quantizing the plurality of subband data, respectively.
It is provided with a coding means for coding the quantization result obtained by the quantization for each subband.

According to the present invention, it is possible to record RAW data compactly by suppressing the amount of quantization parameter data included in the coded data while realizing lossy compression by flexible quantization control in compression of RAW data. It will be possible.

The block diagram which shows the structural example of the image processing apparatus corresponding to the embodiment of an invention, and the figure which shows the RAW data of the Bayer array. The block diagram which shows the structural example of the RAW data coding / decoding unit 103 corresponding to the embodiment of an invention. The figure which shows the example of the channel conversion corresponding to the embodiment of an invention. The figure for demonstrating the frequency conversion (subband division) corresponding to the embodiment of an invention, and the predictive coding method (MED prediction). The figure which shows an example of the quantization control unit corresponding to the embodiment of an invention. The figure which shows an example of the quantization control unit based on the image quality characteristic corresponding to the embodiment of an invention. The flowchart corresponding to an example of the quantization control processing based on the image quality characteristic corresponding to the embodiment of the invention. The figure which shows the relationship between the quantization control unit and RAW data which are related to the image quality control corresponding to the embodiment of an invention. The figure which shows an example of the data structure of the coded data corresponding to the embodiment of an invention. The figure which shows an example of the syntax element of the header information corresponding to the embodiment of an invention. The figure which shows the mapping example to the 2D coordinate system of the common quantization parameter corresponding to the embodiment of an invention. The figure which shows an example of the quantization control unit based on the image quality characteristic corresponding to the 2nd Embodiment of an invention. The figure which shows the comparison with the conventional method of the amount of data contained in the coded data corresponding to Embodiment 2 of the invention.

Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[Embodiment 1]
Hereinafter, embodiments of the invention will be described. FIG. 1A shows an image pickup apparatus 100 corresponding to an embodiment of the invention. The image pickup apparatus 100 can be realized as, for example, a digital camera or a digital video camera. In addition, it can also be realized as an arbitrary information processing terminal or image processing device such as a personal computer, a mobile phone, a smartphone, a PDA, a tablet terminal, and a portable media player. Note that FIG. 1A shows a configuration including an image pickup unit 102 in consideration of a case where it functions as a digital camera or the like as an image pickup device. However, as an embodiment of the invention, the image pickup unit 102 is provided as an image coding device / image decoding device, an image recording device, an image compression device, an image restoration device, etc. capable of compressing, recording, and reproducing a RAW image. It may be realized with a configuration that does not.

In the image pickup apparatus 100, the control unit 101 controls each processing unit constituting the image pickup apparatus 100. The image pickup unit 102 is a lens optical system capable of optical zoom including an optical lens, an aperture, a focus control, and a lens drive unit, and a CCD image sensor or a CMOS image sensor that converts optical information from the lens optical system into an electric signal. Includes an image pickup element. The image pickup unit 102 outputs the RAW data obtained by converting the electric signal obtained by the image pickup element into a digital signal to the RAW data coding / decoding unit 103.

Here, the RAW data is composed of an effective imaging region, which is a light-receiving pixel region, and an optical black region, which is a light-shielded pixel region, as shown in FIG. 1 (b). Each pixel is composed of a periodic pattern of R (red), G1 (green), G2 (green), and B (blue), and human visual sensitivity is more sensitive to the luminance component than the color component. The area of green, which contains a lot of luminance components, is twice as large as that of red and blue. In the present embodiment, the RAW data is composed of four color elements of the Bayer arrangement, but the arrangement and the color elements are not limited to this configuration and may be another method.

The RAW data coding / decoding unit 103 encodes the RAW data acquired by the imaging unit 102 and writes the generated encoded data to the memory 105. Further, the unencoded RAW data may be acquired from the recording medium 107 and the coding process corresponding to the present embodiment may be executed to generate the encoded data. Further, the RAW data coding / decoding unit 103 can decode the coded data and restore the RAW data. The detailed configuration and operation related to the RAW data coding / decoding unit 103 will be described later.

The memory I / F unit 104 arbitrates the memory access request from each processing unit, and performs read / write control for the memory 105. The memory 105 is a storage area composed of, for example, a volatile memory for storing various data output from each processing unit constituting the image pickup apparatus 100. The recording processing unit 106 reads out the coded data stored in the memory 105, performs a predetermined recording format, and records the data on the recording medium 107. The recording medium 107 is, for example, a recording medium composed of a non-volatile memory, and is configured to be removable from the image pickup apparatus 100. The image pickup apparatus is composed of the above processing units.

Subsequently, the detailed configuration and processing flow of the RAW data coding / decoding unit 103 will be described with reference to the block diagrams shown in FIGS. 2 (a) and 2 (b). FIG. 2A shows the configuration of the RAW data coding unit 103E of the RAW data coding / decoding unit 103, and FIG. 2B shows the configuration of the RAW data decoding unit 103D of the RAW data coding / decoding unit 103. The configuration is shown.

In FIG. 2A, the RAW data coding unit 103E mainly includes a channel conversion unit 201, a frequency conversion unit 202, a quantization parameter generation unit 203, a quantization unit 204, a coding unit 205, and a quantization parameter coding unit 206. It is composed of. The channel conversion unit 201 converts the RAW data of the Bayer array input as shown in FIG. 3 into a plurality of channels. The frequency conversion unit 202 and the quantization unit 204 are prepared with blocks corresponding to the number of channels obtained by the conversion, and are processed in parallel. In the present embodiment, for example, the channel conversion unit 201 converts RAW data into four channels for each of R, G1, G2, and B of the Bayer array, and further converts the following conversion formulas for R, G1, G2, and B. 1 can be converted from C0 to 4 channels of C3.

C0 = a + c
C1 = B-G2 formula 1
C2 = R-G1
C3 = ba
However, a = G2 + [C1 / 2], b = G1 + [C2 / 2], c = a + [C3 / 2]
Further, in equation 1, the floor function [x] represents the maximum integer less than or equal to x.

Although a configuration example for converting to four channels is shown here as shown in FIG. 3A, R, G1 and G2 are combined as shown in FIGS. 3B and 3C. It may be converted into three channels for each of G and B, and the number of channels and the conversion method may be a method other than the above.

Next, the frequency conversion unit 202 performs frequency conversion processing by discrete wavelet conversion at a predetermined decomposition level (hereinafter, simply referred to as “level” or “lev”) for each channel, and a plurality of subband data (hereinafter, simply referred to as “level” or “lev”) are generated. The conversion coefficient) is output to the quantization parameter generation unit 203 and the quantization unit 204.

FIG. 4A shows a filter bank configuration for realizing the discrete wavelet transform related to the subband division process of lev = 1, and as a result of executing the discrete wavelet transform process in the horizontal and vertical directions, FIG. 4 (a) shows. As shown in b), it is first divided into one low frequency subband (LL) and three high frequency subbands (HL, LH, HH). The transfer functions of the low-pass filter (hereinafter referred to as lpf) and the high-pass filter (hereinafter referred to as hpf) shown in FIG. 4A are shown in the following equations 2 and 3, respectively.
lpf (z) = (-z ^-2 + 2z ^-1 + 6 + 2z-z ² ) / 8 Equation 2
hpf (z) = (-z ^-1 + 2-z) / 2 Equation 3

Further, "↓ 2" represents 2 to 1 downsampling. Therefore, since downsampling is performed once in the horizontal direction and once in the vertical direction, the size of the subband becomes 1/4 of the original size. When the lev is larger than 1, subband division is performed hierarchically for the low frequency subband (LL). For example, when lev = 3, 10 pieces are performed as shown in FIG. 4 (c). Subband data corresponding to the subband is generated. Here, the discrete wavelet transform is composed of a 5-tap lpf and a 3-tap hpf as shown in the above equations 2 and 3, but a filter configuration having a different number of taps and different coefficients may be used. ..

The quantization parameter generation unit 203 generates a first quantization parameter common to all channels and all subbands for performing quantization processing on the subband data (conversion coefficient) generated by the frequency conversion unit 202. Then, it is output to the quantization parameter coding unit 206. In addition, a separate second quantization parameter for each channel and each subband is generated from the first quantization parameter and supplied to the quantization unit 204. The generation unit of the first quantization parameter and the second quantization parameter and the generation method thereof will be described later with reference to FIGS. 5 to 7 and the like.

The quantization unit 204 performs a quantization process on the subband data (conversion coefficient) input from the frequency conversion unit 202 based on the second quantization parameter supplied from the quantization parameter generation unit 203, and quantizes the subband data (conversion coefficient). As a result, the quantized subband data (conversion coefficient) is output to the coding unit 205.

The coding unit 205 performs predictive difference type entropy coding for each subband in the raster scan order for the quantized subband data (conversion coefficient) input from the quantization unit 204. As a result, the quantized sub-band data is encoded in sub-band units. Here, as shown in FIG. 4D, the predicted value pd is generated from the peripheral data of the coded target data (conversion coefficient) by MED (Media Edge Detector) prediction, and the value xd and the predicted value of the coded target data x are generated. The difference data from the pd is entropy-encoded by, for example, Huffman coding, Golomb coding, or the like, and the generated coded data is stored in the memory 105. At this time, since there is no peripheral data in the first data in the raster scan order of each subband, the value xd of the coded target data x is encoded as it is. Then, the difference data is encoded for the subsequent coding target data. The predictive coding method and the entropy coding method may be other methods. Further, the generated code amount generated in line units for each subband is supplied to the quantization parameter generation unit 203. The specific data structure of the coded data generated here will be described later with reference to FIGS. 9 and 10.

The quantization parameter coding unit 206 is a processing unit that encodes the first quantization parameter input from the quantization parameter generation unit 203. The quantization parameter coding unit 206 encodes the quantization parameter by a coding method common to the coding unit 205, and stores the generated encoded quantization parameter in the memory 105. FIG. 2A shows a case where the quantization parameter coding unit 206 is provided independently of the coding unit 205, but since both adopt a common coding method, the function of the quantization parameter coding unit 206 is shown. Can be configured to be common in the coding unit 205. The RAW data coding unit 103E is configured as described above.

Next, the configuration of the RAW data decoding unit 103D will be described. The RAW data decoding unit 103D is mainly composed of a decoding unit 211, a quantization parameter decoding unit 212, a quantization parameter generation unit 213, an inverse quantization unit 214, a frequency inverse conversion unit 215, and a channel inverse conversion unit 216.

First, the decoding unit 211 decodes the coded data input from the memory 105 by a decoding method corresponding to the coding method adopted by the RAW data coding unit 103E, and the quantized subband data (conversion coefficient). To restore. For example, when the predicted difference type entropy coding method is used, the corresponding predicted difference type entropy decoding method is adopted. At this time, as shown in FIG. 4D, the coded data is the entropy of the difference data between the predicted value pd generated from the peripheral data of the coded target data (conversion coefficient) and the value xd of the coded target data x. It is encoded. Therefore, when the decoding unit 211 decodes the coded data by the entropy decoding method and restores the difference data, the predicted value pd is generated from the decoded data around the decoding target data, and the difference data is added to the coded target data. Restore x. Since the coded data is composed of data encoded in subband units, the decoding unit 211 also restores the conversion coefficient in subband units.

The quantization parameter decoding unit 212 decodes the coded first quantization parameter included in the coded data input from the memory 105 by the same decoding method as the decoding unit 211, and the first quantization parameter. To restore. Further, the quantization parameter decoding unit 212 also decodes a predetermined coefficient (α, β in the formula 5 described later) for calculating the second quantization parameter in units of channels and subbands. The decoding result is output to the quantization parameter generation unit 213. The quantization parameter generation unit 213 generates a channel and sub-band unit second quantization parameter from the first quantization parameter provided by the quantization parameter decoding unit 212 and a predetermined coefficient, and dequantizes. Provided to unit 214.

The dequantization unit 214 dequantizes the channel-based quantized sub-band data output from the decoding unit 211 using the second quantization parameter provided by the quantization parameter generation unit 213, and sub-quantizes the sub-quantization. Restore the band data. The frequency inverse transform unit 215 performs frequency inverse transform processing by discrete wavelet inverse transform, and restores a channel from subband data (conversion coefficient) of a predetermined division level. The channel inverse conversion unit 216 reversely converts the input channel data and restores the original RAW data. For example, when the channel conversion unit 201 of the RAW data coding unit 103E converts R, G1, G2, and B of the Bayer array into four channels C0 to C3 according to the formula 1, the inverse conversion process corresponding to the formula 1 According to this, the RAW data of R, G1, G2, and B are restored from the four channels C0 to C3. The RAW data decoding unit 103D is configured as described above.

Next, the quantization parameter generation process in the quantization parameter generation unit 203 will be described. In this embodiment, a first quantization parameter common to all channels and all subbands is generated, and a separate second quantization parameter for each channel and each subband is obtained from the first quantization parameter. Generate. The quantization parameter generation unit 203 performs a quantization parameter generation process related to code amount control in a predetermined subband data unit based on a target code amount set in advance before coding, and first performs all channels and all subbands. Generate a common first quantization parameter.

In this embodiment, a case where the first quantization parameter is not directly generated but is generated in two stages will be described. First, a common quantization parameter (third quantization parameter) for all channels and all subbands is generated for each subband data in a predetermined line unit. Then, the predetermined line is divided into units called segments, and the third quantization parameter is modified according to the image quality characteristics of the subband data in the segment to generate the first quantization parameter. Hereinafter, the procedure for generating the first quantization parameter will be described in detail.

FIG. 5 shows a unit for updating the code amount evaluation and the quantization parameter related to the code amount control when each channel is subbanded at lev = 3. Level 3 subbands (2HL, 2LH, 2HH) for 1 × N (N is an integer) line in the vertical direction of the level 3 subbands (3LL, 3HL, 3LH, 3HH) of the four channels from C0 to C3. 2 x N lines in the vertical direction and 4 x N lines in the vertical direction of the level 1 subband (1HL, 1LH, 1HH) are combined into one processing unit (first processing unit). do. As shown in FIG. 5, the first processing unit is set as a region (band) in which each subband is vertically divided into predetermined lines. In particular, FIG. 5 shows the case where N = 1.

The quantization parameter generation unit 203 compares the target code amount corresponding to the first processing unit with the generated code amount, and determines the generated code amount for the data of all channels and all subbands of the next processing unit. Feedback control is performed so as to approach the quantity, and a third quantization parameter (QpBr) common to all channels and all subbands is generated. When the code amount control in the screen is not performed for the RAW data, QpBr common to all channels and all subbands that are fixed to the entire screen is set or set regardless of the code amount control unit as shown above. Just generate it. The control of the code amount at this time can be executed according to the following equation 4.

QpBr (i) = QpBr (0) + r × Σ {S (i-1) -T (i-1)} Equation 4
QpBr (0): Initial quantization parameter QpBr (i): i-th quantization parameter (i> 0)
r: Control sensitivity
S: Generated code amount
T: Target code amount

In Equation 4, the difference between the generated code amount and the target code amount is based on the initial quantization parameter set for the first band of each band (first processing unit) set in the subband data. The initial quantization parameter is adjusted according to the magnitude. Specifically, the initial quantization parameter is such that the code amount difference between the total code amount generated after the first band (total generated code amount) and the total of the corresponding target code amounts (total target code amount) becomes small. Adjust the value of to determine the third quantization parameter for the band to be processed. As the initial quantization parameter set for the first band, a parameter determined by the quantization parameter generation unit 203 based on the compression rate or the like can be used.

In this way, after the third quantization parameter (QpBr) is generated for each band, the quantization parameter generation unit 203 further subdivides (segments) the band, and for each segment, which is the second processing unit. The third quantization parameter (QpBr) is adjusted according to the image quality evaluation result of the subband data to generate the first quantization parameter (QpBs). This makes it possible to generate quantization parameters related to image quality control that adjusts the strength of quantization according to the characteristics of RAW data.

FIG. 6A shows a second processing unit (segment) for updating the evaluation and quantization parameters of the subband data related to the image quality control when each channel is subbanded at lev = 3. In each segment, level 3 subbands (3LL, 3HL, 3LH, 3HH) of all channels are (1 × M) × (1 × N) (M and N are integers) and level 2 in the horizontal and vertical directions, respectively. Subbands (2HL, 2LH, 2HH) horizontally and vertically (2 × M) × (2 × N), and level 1 subbands (1HL, 1LH, 1HH) horizontally and vertically (4), respectively. × M) × (4 × N).

The segment as the second processing unit shown here is a horizontally subdivided band as the first processing unit. In the present embodiment, segmentation is performed so that the number of segments included in each subband matches. When M = 1 and N = 1, the segment corresponds to level 1 and the band can be regarded as horizontally divided into 4 pixel units. The quantization parameter generation unit 203 generates the first quantization parameter (QpBs) for each segment by modifying the third quantization parameter QpBr calculated for each band according to the image quality characteristics for each segment. ..

FIG. 6B shows the relationship between the first quantization parameter (QpBs) and the subband data when M = 1 and N = 1. As shown here, one first quantization parameter (QpBs) is generated for a plurality of subband data of each channel contained in one segment.

Regarding the image quality evaluation for each segment, for example, among the subband data included in each segment, the subband data of 3LL is evaluated as a low frequency component, and the high frequency component is evaluated by the subband data of 1HL, 1LH, and 1HH. At this time, the smaller the amplitude of the low-frequency component, the finer the quantization, and the larger the amplitude of the high-frequency component, the coarser the quantization is controlled. Feed forward control may be performed so as to adjust the offset.

FIG. 7 shows a flowchart corresponding to an example of the adjustment process of the third quantization parameter (QpBr) based on the image quality evaluation executed by the quantization parameter generation unit 203. The process corresponding to the flowchart can be realized, for example, by executing a program (stored in ROM or the like) corresponding to one or more processors functioning as the quantization parameter generation unit 203.

In S701, the quantization parameter generation unit 203 determines the low frequency component. The conversion coefficient of 3LL is compared with the threshold value Th1 set for determination, and if the conversion coefficient of 3LL is greater than or equal to the threshold value (“YES” in S701), the process proceeds to S702. In S702, the quantization parameter generation unit 203 reduces the value of the third quantization parameter QpBr set for the band to which the segment to be processed belongs. For example, QpBr is multiplied by a coefficient less than 1, or a predetermined value is subtracted from QpBr.

In the determination of S701, when the conversion coefficient of 3LL is larger than the threshold value Th1 (“NO” in S701), the process proceeds to S703. In S703, the quantization parameter generation unit 203 determines the high frequency component. The average value of the conversion coefficients of 1HL, 1LH, and 1HH is compared with the threshold value Th2 set for determination, and if the average value is smaller than the threshold value Th2 (“NO” in S703), the process proceeds to S704. If the average value is greater than or equal to the threshold value Th2 (“YES” in S703), the process proceeds to S705. In S704, the value of QpBr is maintained. In S705, the value of QpBr is increased. For example, QpBr is multiplied by a coefficient of 1 or more, or a predetermined value is added to QpBr.

As described above, the result obtained by adjusting the third quantization parameter QpBr for each segment is used as the first quantization parameter QpBs of the segment. When the quantization control related to the image quality control is not performed, QpBr may be used as it is as QpBs.

The first quantization parameter QpBs common to all channels and all subbands generated as described above is, for example, 8 × 8 pixels of each channel as shown in FIG. 8 when the above M = 1 and N = 1. Correspondingly, it is possible to control the quantization of the RAW data corresponding to 16 × 16 pixels of the original RAW data in block units.

Next, a process of generating a separate second quantization parameter QpSb for each channel and each subband using the first quantization parameter QpBs will be described. In this embodiment, as described above, one first quantization parameter QpBs is generated for each segment, but the quantization of each segment of each channel does not use QpBs as it is, but individually for each channel and segment. The corresponding second quantization parameter QpSb is generated from QpBs and the quantization process is performed. Hereinafter, the QpSb generation process will be described.

Equation 5 is a calculation equation for generating QpSb.
QpSb [i] [j] = QpBs × α [i] [j] ＋ β [i] [j] Equation 5
QpSb: Second quantization parameter for each channel and each subband
QpBs: First quantization parameter common to all channels and all subbands α: Slope β: Intercept
i: Channel index (0-3)
j: Subband index (0-9)

In Equation 5, the slope α and the intercept β are coefficients (variables) individually given to each channel and each subband, and can be predetermined. The value of α increases for channels C0 to C3 in the relationship of C0> C1 = C2> C3, and the larger α, the larger the value of QpSb. In addition, for each subband, the values of α and β are made smaller in order to make the value of the quantization parameter smaller in the subband closer to the low frequency component, and the value of the quantization parameter is made larger in the subband closer to the high frequency component. Increase the values of α and β.

Further, although each value of α [i] [j] and β [i] [j] is set in advance, a plurality of combinations may be prepared according to the set compression rate. For example, when the compression rate is high, information is lost if the value of QpSb is too large, so the values of α and β can be adjusted to be small so that the value of QpSb does not become too large.

In this way, the first quantization parameter QpBs is modified by the individual weight coefficients α and β of each channel and each subband to generate the second quantization parameter QpSb, so that each channel and each subband On the other hand, it becomes possible to flexibly control the quantization. The quantization unit 204 quantizes the conversion coefficient obtained from the frequency conversion unit 202 using the second quantization parameter QpSb of each channel and each subband generated based on the equation 5.

Further, the method of generating the second quantization parameter QpSb in the RAW data decoding unit 103 can also be performed according to the above equation 5. At that time, the first quantization parameter QpSr and each value of α and β are provided by the quantization parameter decoding unit 212.

The subband data of each channel encoded by the RAW data coding / decoding unit 103 as described above and the first quantization parameter QpBs are multiplexed by the recording processing unit 106 based on a predetermined data format. , Recorded as coded data. FIG. 9A shows an example of a data structure for recording encoded data. As shown in FIG. 9A, the coded data has a hierarchical structure, starting with "main_header" indicating information related to the entire coded data, and the RAW data is tiled and coded in units of a plurality of pixel blocks. It is possible to store data in tile units by "tile_header" and "tile_data" on the assumption that the data will be changed. In the above description, the case where the RAW data is directly encoded in the channel and subband data units has been described, but the present invention can be applied not only to tiles but also to a predetermined set in the RAW data as a unit. That is, when the RAW data is divided into tiles and the tiles are encoded in channel and subband data units, the same processing as above may be executed in the tile units obtained by dividing the RAW data into a plurality of tiles. .. In this case, the subband data of each channel and the first quantization parameter QpBs are encoded for each tile and included in the encoded data. When tile division is not performed, there is only one "tile_header" and "tile_data".

"Tile_data" includes the following elements. First, "qp_header" indicating information related to the coded first quantization parameter QpBs and "coded_qp_data" which is the coded first quantization parameter itself are arranged. The following "coded_raw_data" is arranged side by side in channel units, and the coded data for each channel is stored in the order of "channel_header" indicating information related to each channel and "channel_data" which is the main body data for each channel. Will be done. "Channel_data" is composed of a set of coded data for each subband, and "sb_header" indicating information related to each subband and "sb_data" which is coded data for each subband are subs. They are arranged in band index order. The subband index is as shown in FIG. 9B.

Subsequently, the syntax elements of each header information will be described with reference to FIG. "Main_header" is composed of the following elements. "Code_data_size" indicates the amount of data of the entire encoded data. "Width" indicates the width of the RAW data. "Height" indicates the height of the RAW data. "Dept" indicates the bit depth of the RAW data. “Channel” indicates the number of channels when encoding RAW data. "Type" indicates the type of channel conversion. “Lev” indicates the subband level of each channel.

"Tile_header" is composed of the following elements. "Tile_index" indicates the index of the tile for identifying the tile division position. "Tile_data_size" indicates the amount of data contained in the tile. "Tile_width" indicates the width of the tile. "Tile_height" indicates the height of the tile.

"Qp_header" is composed of the following elements. “Qp_data_size” indicates the amount of data of the first coded quantization parameter. “Qp_width” indicates the width of the first quantization parameter, that is, the number of horizontal first quantization parameters corresponding to the RAW data. “Qp_height” indicates the height of the first quantization parameter, that is, the number of the first quantization parameters in the vertical direction corresponding to the RAW data.

"Channel_header" is composed of the following elements. "Channel_index" indicates the index of the channel for identifying the channel. "Channel_data_size" indicates the amount of data in the channel. "Sb_header" is composed of the following elements. “Sb_index” indicates a subband index for identifying a subband. “Sb_data_size” indicates the amount of coded data of the subband. “Sb_qp_a” indicates the α value shown in Equation 5 for generating the quantization parameter for each subband. “Sb_qp_b” indicates the β value shown in Equation 5 for generating the quantization parameter for each subband.

According to the invention corresponding to the above embodiment, instead of recording all the quantization parameters of each channel and each subband, the common quantization parameters of all channels and all subbands are recorded and the common quantization is performed. Quantization parameters for each channel and each subband can be generated from the parameters. As a result, the effect of suppressing the amount of data of the quantization parameter as described below can be achieved.

Compare the amount of data between the case of recording all the individual quantization parameters of each channel and each subband (conventional example) and the case of recording the quantization parameters common to all channels and all subbands (this embodiment). think. Here, one quantization parameter is recorded for each subband from level 1 to level 3. Therefore, the data size of the quantization parameter to be recorded for one segment is (1 × 4 subband + 1 × 3 subband + 1 × 3 subband) × 4 channels, which is 40 in total. On the other hand, since only one common quantization parameter is required, it becomes 1, and the size of the quantization parameter can be reduced to 1/40 by setting the quantization parameter to be recorded as the common quantization parameter. ..

Further, as shown in FIG. 11, since the common quantization parameter to be recorded can be regarded as screen data mapped to the two-dimensional coordinate system, the predicted difference coding method (MED) used for coding the quantization parameter can be regarded. By applying the prediction) in the same way, it is possible to further reduce the amount of data of the quantization parameter. In FIG. 11 (b), QpBs [p, q] indicate common quantization parameters assigned to each segment, p takes a value from 1 to the number of divisions P of the vertical segment, and q is. Values from 1 to the number of divisions Q of the horizontal segment can be taken. Each value of [p, q] corresponds to the position of each segment in the subband.

When RAW data is converted into multiple channels as described above and subband division is performed for each channel, all channels are divided into segments, and each channel is divided by a quantization parameter and a weighting function common to all subbands. Quantization parameters for individual subbands can be generated. As a result, while realizing flexible quantization control, the amount of data of the quantization parameter is greatly reduced by performing predictive coding using the common quantization parameter as two-dimensional data, and the RAW data can be obtained with high image quality and compactness. It becomes possible to record.

[Embodiment 2]
Hereinafter, the second embodiment of the invention will be described. The image pickup device of the present embodiment can have the same configuration as the image pickup device of the first embodiment. However, the generation unit of the first quantization parameter in the quantization parameter generation unit 203 of the RAW data coding / decoding unit 103 is different from the process of generating the second quantization parameter for each channel and each subband. .. The description of the configuration common to the first embodiment will be omitted in the present embodiment, and the configuration related to the difference in the quantization parameter generation method will be mainly described.

FIG. 12A shows a unit for updating the evaluation and quantization parameters of the subband data related to image quality control when each channel is subbanded at lev = 3, and is a level 3 subband of all channels. (3LL, 3HL, 3LH, 3HH) horizontal and vertical directions (1 × M) × 1 line (M is an integer), while level 2 subbands (2HL, 2LH, 2HH) are horizontal and vertical. 1st common 1st line corresponding to (4 × M) × 1 line of subband data of (4 × M) × 1 line in the horizontal and vertical directions of the level 1 subband (1HL, 1LH, 1HH), respectively. Generate quantization parameters (QpBs).

Compared with the segment related to image quality control shown in FIG. 6A, in FIG. 12A, the segment size at level 1 is one-fourth in the vertical direction. Therefore, the relationship between the first quantization parameter (QpBs) and the subband data when M = 1 is as shown in FIG. 12 (b). Here, since the segment size is 4 × 1 at level 1, there are four first quantization parameters (QpBs) when converted into a 4 × 4 segment. This enables finer quantization control in the vertical direction as compared with the first embodiment. Similarly, it is possible to make the size finer in the horizontal direction.

The evaluation method of the subband data related to the image quality control is substantially the same as that of the first embodiment, but in the present embodiment, the target to be evaluated as the low frequency component is generated intermediately in the subband division step instead of 3LL. Subband data equivalent to 1LL. The processing flow is the same as the flowchart shown in FIG. 7, but the determination target in S701 is not 3LL but a conversion coefficient of 1LL.

In the present embodiment, for example, when converted into a 4 × 4 segment of M = 1, the number of quantization parameters for each line subject to image quality control differs from level 1 to level 3, so that the level is the same as in the first embodiment. The first quantization parameter QpBs from 1 to level 3 cannot be used in common. Therefore, in the present embodiment, QpLv1 to QpLv3 of each level are generated from the four QpBs at the level 1 based on the calculation formula shown in the following formula 6, and these are used to individually correspond to each channel and each subband. The second quantization parameter QpSb is generated.
QpLv1 [m] = QpBs [m]
QpLv2 [n] = (QpBs [n × 2] + QpBs [n × 2 + 1]) >> 1 Equation 6
QpLv3 [n] = (QpBs [0] + QpBs [1] + QpBs [2] + QpBs [3]) >> 2
m: Number of lines for level 1 subband in segment (0 to 3)
n: Number of lines for level 2 subbands in segment units (0 to 1)
>>: Right bit shift

As shown in Equation 6, the level 2 and level 3 quantization parameters are calculated by the average value of the corresponding QpBs, but the maximum value or the minimum value of QpBs may be selected. Further, QpLv3 may be calculated as an average value, a maximum value or a minimum value of QpLv2. The second quantization parameter (QpSb) of each channel and each subband used for the final quantization may be calculated by replacing QpBs in the above equation 5 with QpLv1 to QpLv3, and each quantization parameter and quantization may be performed. The relationship with the target subband is shown in FIG. 12 (c).

Next, with reference to FIG. 13, the degree of reduction in the amount of recorded quantization parameter data in the present embodiment will be described. In the conventional example to be compared, assuming that the quantization parameter is recorded in units of one line for each level, when the quantization parameter of the subband of level 3 is 1, level 2 is twice level 3 and level 1 is. Quantization parameters that are four times higher than level 3 will be recorded. Therefore, the data size of the quantization parameter to be recorded for one segment is (1 × 4 subband + 2 × 3 subband + 4 × 3 subband) × 4 channels, which is 88 in total. On the other hand, the common quantization parameter is 4 because it corresponds to the level 1 subband, and the size of the quantization parameter is reduced to 1/22 by using the common quantization parameter as the recording target. Can be done.

As described above, even when finer quantization control is performed in the vertical direction as in the present embodiment, a sufficient data reduction effect can be expected as compared with the conventional example. Further, as in the first embodiment, the present embodiment also realizes flexible quantization control, and the amount of data of the quantization parameter is significantly reduced by performing predictive coding using the common quantization parameter as two-dimensional data. However, it is possible to record RAW data with high image quality and compactness.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Imaging device, 101: Control unit, 102: Imaging unit, 103: RAW data coding / decoding unit, 104: Memory I / F unit, 105: Memory, 106: Recording processing unit, 107: Recording medium

Claims (20)

A generation means that performs discrete wavelet transform of two or more decomposition levels and generates multiple subband data from RAW data.
Common quantization in which the plurality of subband data is divided into the same number of quantization units, individually set for each quantization unit, and commonly assigned to a plurality of subband data included in the same quantization unit. A quantization parameter generation means for generating a parameter and generating a quantization parameter for quantizing each of the plurality of subband data based on the common quantization parameter.
A quantization means for quantizing each of the plurality of subband data based on the quantization parameters for quantizing the plurality of subband data, respectively.
A coding means for coding the quantization result obtained by the above-mentioned quantization for each subband,
An image coding apparatus comprising: The image coding apparatus according to claim 1, wherein the coding means encodes the common quantization parameter. The invention according to claim 1 or 2, wherein the recording processing means for recording the plurality of subband data encoded by the coding means and the common quantization parameter in a predetermined data structure is provided. Image coding device. The quantization parameter generation means generates a plurality of common quantization parameters for each quantization unit, and quantizes a plurality of subband data included in the quantization unit from the plurality of common quantization parameters. Generate quantization parameters,
The image coding apparatus according to claim 1. The quantization parameter generation means is
As a plurality of common quantization parameters for each quantization unit, a level 1 common quantization parameter is generated for each predetermined line of level 1 of the subband of decomposition level 1.
For each quantization unit, each level common quantization parameter for each predetermined line of each decomposition level is generated based on the plurality of the level 1 common quantization parameters.
Based on the common quantization parameter for each level and the coefficient set for each of the plurality of subbands, the quantization parameter for quantizing the plurality of subband data is generated.
The image coding apparatus according to claim 4. The image coding apparatus according to claim 5, wherein the predetermined line is one line. It is characterized by comprising a recording processing means for recording the plurality of subband data encoded by the coding means and the level 1 common quantization parameter which is the common quantization parameter in a predetermined data structure. The image coding apparatus according to claim 5 or 6. The coding means encodes the level 1 common quantization parameter, which is the common quantization parameter.
The recording processing means is characterized in that a level 1 common quantization parameter encoded by the coding means is recorded in a predetermined data structure together with the plurality of subband data encoded by the coding means. The image coding apparatus according to claim 7. The quantization parameter generation means is
Based on the third quantization parameter determined based on the difference between the target code amount and the generated code amount as a result of coding when the plurality of subband data are quantized and coded in the predetermined line unit. The image coding apparatus according to any one of claims 5 to 8, wherein the common quantization parameter is generated. The ninth aspect of claim 9, wherein the quantization parameter generation means generates the common quantization parameter by modifying the third quantization parameter according to the image quality characteristics of each quantization unit. Image encoder. The quantization parameter generation means modifies the third quantization parameter according to the image quality characteristics of each predetermined line of the quantization unit of the level 1 subband data, thereby modifying the common quantization parameter. The image coding apparatus according to claim 10, wherein the image coding apparatus is generated. The quantization parameter generation means is
In the case of the image quality characteristic indicating that the amplitude of the low frequency component is small, the third quantization parameter is reduced to generate the common quantization parameter.
The invention according to any one of claims 9 to 11, wherein the common quantization parameter is generated by increasing the third quantization parameter in the case of the image quality characteristic indicating that the amplitude of the high frequency component is large. The image encoding device of the description. The image coding apparatus according to claim 5, wherein the coefficient is set according to a compression rate set for the RAW data. The image coding apparatus according to claim 5, wherein the coefficient is commonly assigned to a common quantization parameter of a plurality of quantization units. The image coding apparatus according to claim 5, wherein the coefficient set for each of the plurality of subbands is set to a coefficient in which the quantization parameter becomes larger as the subband of the high frequency component becomes larger. The image coding apparatus according to claim 8, wherein the recording processing means records the coefficient together with the plurality of subband data in the predetermined data structure. The generation means generates the plurality of subband data for each of the plurality of channels obtained by converting the RAW data.
The image code according to any one of claims 1 to 16, wherein the common quantization parameter is commonly assigned to the plurality of subband data of each of the plurality of channels of the quantization unit. Quantization device. The generation means generates the plurality of subband data for each of the plurality of channels obtained by converting the RAW data.
The common quantization parameter is commonly assigned to the plurality of subband data of each of the plurality of channels of the quantization unit.
The coefficients are set for each of the plurality of subbands and the plurality of channels.
The image coding apparatus according to claim 5. A process of generating multiple subband data from RAW data by performing discrete wavelet transforms with two or more decomposition levels,
Common quantization in which the plurality of subband data is divided into the same number of quantization units, individually set for each quantization unit, and commonly assigned to a plurality of subband data included in the same quantization unit. The process of generating parameters and
A step of generating a quantization parameter for quantizing each of the plurality of subband data based on the common quantization parameter, and a step of generating the quantization parameter.
A step of quantizing each of the plurality of subband data based on the quantization parameter for quantizing the plurality of subband data, respectively.
The step of encoding the quantization result obtained by the above-mentioned quantization for each subband,
Image encoding methods, including. A program for operating a computer as each means of the image coding apparatus according to any one of claims 1 to 18.

Images