JP6792370B2 - Image processing equipment, its control method, and computer programs - Google Patents

Description

The present invention relates to a technique for encoding image data of a Bayer array.

In general, an image pickup device represented by a digital camera is equipped with a Bayer array image pickup element. The Bayer array has a structure in which red (R), green (G), and blue (B) pixels are arranged in a mosaic pattern. The 2 × 2 pixels in the Bayer array are composed of one red (R), two green (G0, G1) and one blue (B) pixels. Therefore, the array of pixels of the image data immediately after being imaged by such an image sensor is also a Bayer array.

As described above, each pixel of the image data of the Bayer array has only information on one color component. Therefore, in general, an interpolation process called demosaic is applied to the image data of the Bayer array, and image data in which one pixel has a plurality of components of R (red), G (green), and B (blue) is generated. Then, from the viewpoint of recording and transfer efficiency, the image data obtained by the demosaic processing is encoded and the amount of the data is compressed. JPEG (Joint Photographic Experts Group), which is a representative of compression coding, converts image data in RGB color space into image data in YUV color space and then compresses and encodes it. However, if the number of bits per component is the same, the amount of image data after demosaic is three times the amount of image data of the Bayer array before demosaic. That is, it can be said that JPEG targets image data having a data amount three times as much as the image data of the Bayer array.

On the other hand, there is known a technique of classifying the image data of the Bayer array into each component (R, G0, B, G1) without performing demosaic processing and encoding the image data of each component independently ( For example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2003-125209 JP-A-2006-121669

Half of the total number of pixels included in the image data of the Bayer array is the pixels of the G component. Therefore, it is important how to efficiently encode the pixels of the G component. In the method of Patent Document 1, the G0 component and the G1 component, which are originally close to each other in pixel position and have the same color and high correlation, are classified into different components. Therefore, since the wavelet transform in the coding process is performed in the state where the image data of the Bayer array is sub-sampled, folding noise is applied when the high frequency component and the low frequency component are separated, and the compression rate is lowered. Connect.

On the other hand, a technique of performing color space conversion on the image data of the Bayer array to generate one luminance component (Y) and three color difference components (Dg, Co, Cg) and encoding the image data of each component. Is known (Patent Document 2). This utilizes the luminosity factor characteristic that the human eye has high sensitivity to the luminance component, and is one of the compression efficiency improvement methods aiming at reducing redundant data for each color component. The Dg component of this method is represented by G1-G0 (high-pass filter by differentiation), and the high frequency component with respect to the G component is calculated. However, in Patent Document 2, since the low frequency component with respect to the G component is not calculated, there is room for improvement in terms of compression efficiency.

The present invention is intended to provide a technique for more efficiently compressing and encoding image data of a Bayer array.

In order to solve this problem, for example, the image coding apparatus of the present invention has the following configuration. That is,
An image coding device that encodes image data in a Bayer array.
A generation means for generating GL component data composed of low frequency components of G component and GH component data composed of high frequency components of G component from G0 component data and G1 component data of image data of Bayer array.
From the R component data, B component data, and the GL component data of the image data of the Bayer array, the luminance component data composed of the luminance component, the first luminance component data composed of the first luminance component, and the second Luminance color difference conversion means for generating the second color difference component data composed of the color difference components of
The luminance component data, the first color difference component data, the second color difference component data, and the GH component data, have a coding means for coding each component,
In the coding means, the code amount of the brightness component data is larger than the code amount of the first color difference component data and the code amount of the second color difference component data, and the code amount of the GH component data is increased. It is characterized in that each component data is encoded so as to be smaller than the code amount of the first color difference component data and the code amount of the second color difference component data .

According to the present invention, the image data of the Bayer array can be compressed and encoded more efficiently than before.

The block block diagram which the coding apparatus which concerns on 1st Embodiment has. The figure explaining the color separation and the plane formation method of the image data of a Bayer array. The flowchart of the coding process which concerns on 1st Embodiment. The figure for demonstrating the wavelet transform. The figure explaining the generation method of the low frequency component plane and the high frequency component plane using the G0 plane and the G1 plane. The figure which shows an example of setting of the target code amount for each subband of each plane. The block block diagram which the coding apparatus which concerns on 2nd Embodiment has. The flowchart of the coding process which concerns on 2nd Embodiment. The figure which shows another example of the color separation and plane formation method of the image data of a Bayer array. The block block diagram of the information processing apparatus in the modification of 1st Embodiment. The figure which shows an example of the setting of the target code amount for each subband of each plane in the modification of 1st Embodiment.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

[First Embodiment]
The first embodiment is an example applied to an imaging device typified by a digital camera. FIG. 1 is a block configuration diagram of a main part related to image coding in the image pickup apparatus 100. The imaging device 100 includes an imaging unit 101, a plane forming unit 102, a memory I / F unit 103, a memory 104, a plane conversion unit 105, a color conversion unit 106, a frequency conversion unit 107, a control unit 108, a quantization unit 109, and an entropy code. It has a conversion unit 110 and an output unit 111. The storage medium 112 is, for example, a removable memory card. In addition, the present device has an operation unit 115 that functions as a user interface.

The imaging unit 101 has a structure in which sensors that convert light into electrical signals are arranged two-dimensionally. A red (R), green (G), or blue (B) color filter is arranged on the front surface of each sensor. The arrangement of these color filters is a Bayer arrangement. The imaging unit 101 supplies the electrical signals of each coloration obtained by each sensor to the plane forming unit 202 as digital image data. Since the color filter has a Bayer array, each pixel of the image data supplied to the plane forming unit 202 also has a Bayer array. Since the image data to be encoded in the first embodiment is the image data of the Bayer array in the stage before the demosaic processing is performed, the image data of the Bayer array is hereinafter referred to as RAW image data.

The plane forming unit 102 receives the RAW image data from the imaging unit 101. Further, the plane forming unit 102 is composed of an R plane composed of R component data, a G0 plane composed of G0 component data, a G1 plane composed of G1 component data, and a B component data from RAW image data. Form (separate) the B plane. FIG. 2 shows the relationship between the RAW image data and the R, G0, G1, and B planes. When the number of pixels in the horizontal direction of the RAW image data is W and the number of pixels in the vertical direction is H, the number of pixels in the horizontal direction of each of the R, G0, G1, and B planes is W / 2, and the number of pixels in the vertical direction is W. It becomes H / 2. The plane forming unit 102 temporarily stores the formed R plane, G0 plane, G1 plane, and B plane in the memory 104 via the memory I / F unit 103.

The memory I / F unit 103 mediates access requests to the memory 104 from each processing unit, and performs read / write control for the memory 104.

The memory 104 is provided to temporarily store and hold various data output from each processing unit, and is composed of a RAM.

The plane conversion unit 105 reads the G0 plane and the G1 plane from the memory 104 via the memory I / F unit 103. Then, the plane conversion unit 105 performs a predetermined pixel calculation using the correlation between these two planes, and generates high frequency component (hereinafter referred to as GH) data and low frequency component (hereinafter referred to as GL) data of the G component. .. Then, the plane conversion unit 105 writes the GL plane composed of the generated GL data and the GH plane composed of the GH data to the memory 104 via the memory I / F unit 103. The details of the pixel calculation performed by the plane conversion unit 105 will be described later.

The color conversion unit 106 reads out the GL plane, the R plane, and the B plane stored in the memory 104 via the memory I / F unit 103, and performs luminance color difference conversion on the three planes. Then, the color conversion unit 106 writes each plane obtained by the luminance color difference conversion to the memory 104 again via the memory I / F unit 103. At this time, the generated planes are a Y plane showing the brightness Y, a U plane showing the color differences U and V, and a V plane.

The frequency conversion unit 107 executes wavelet transform on the Y plane, U plane, V plane, and GH plane read from the memory 104 via the memory I / F unit 103. Then, the frequency conversion unit 107 sends the conversion coefficient data of a plurality of types of subbands obtained by this wavelet conversion to the quantization unit 109. Here, the wavelet transform is one of the frequency analysis methods, and refers to a process of separating the frequency component of the image data into a low frequency band (a low pass filter process is performed) and a high frequency band (a high pass filter process is performed). FIG. 4 shows the results of subband formation when the two-dimensional wavelet transform was performed once on the input image data (decomposition level 1). “L” shown in FIG. 4 means a low frequency component, and “H” means a high frequency component. For example, the HH subband is a set of conversion coefficient data obtained by performing high-pass filtering in the vertical direction and the horizontal direction. Also, the wavelet transform may be performed recursively many times. The target for performing the second and subsequent wavelet transforms (decomposition level 2 and later) is the LL subband of the wavelet transform (here, the wavelet transform of decomposition level 1) executed immediately before.

The control unit 108 controls the entire device. Then, the control unit 108 receives an instruction from the user via the operation unit 115. This instruction includes a recording instruction, a recording quality, a total target code amount of RAW image data, and the like. When the control unit 108 receives an instruction input from the user via the operation unit 115, the control unit 108 stores the instructed information in a memory (not shown) in the control unit 108. Then, the control unit 108 sets the plane target code amount assigned to the Y plane, the U plane, the V plane, and the GH plane among the total target code amounts in the quantization unit 109 based on the information stored in the memory. To do.

The quantization unit 109 executes quantization on the conversion coefficient data supplied from the wavelet conversion unit 108, and sends the quantization coefficient data after quantization to the entropy coding unit 110. The quantization parameter (quantization step) used when performing quantization is determined based on the plane target code amount set by the control unit 108.

The entropy coding unit 110 entropy-encodes the conversion coefficient after quantization to generate coded data. Then, the entropy coding unit 110 supplies the generated coded data to the output unit 111.

The output unit 111 concatenates the coded data of the Y plane, the U plane, the V plane, and the GH plane supplied from the entropy coding unit 110 according to a preset format. Then, the output unit 111 generates a file header including information necessary for decoding, and writes the file header and the encoded data as one file in the storage medium 112. The storage medium 111 is a removable non-volatile memory such as an SD card.

Next, the coding process for the RAW image data (1 frame) in the first embodiment will be described with reference to the flowchart of FIG. In the first embodiment, the processing in units of one frame is shown as an example, but one frame may be divided into a plurality of tiles, and the coding processing described later may be executed for each tile.

In step S301, the plane forming unit 102 inputs the RAW image data of the Bayer array output by the imaging unit 101 to form the R plane, the G0 plane, the G1 plane, and the B plane. Then, the plane forming unit 102 writes the formed R plane, G0 plane, G1 plane, and B plane to the memory 104 via the memory I / F unit 103.

In step S302, the plane conversion unit 105 reads the G0 plane and the G1 plane from the memory 104 via the memory I / F unit 103, and generates a GL plane composed of GL component data.

The method of generating the GL component data will be described with reference to FIG. The plane conversion unit 105 adds and averages the pixel data at the same coordinate positions of the input G0 plane and G1 plane, and generates the added and averaged value as GL component data. That is, the GL plane is a set of averaging of the G0 plane and the G1 plane. The additive averaging process is a moving average (integral) process between the pixels, and is synonymous with the low-pass filter process. Then, the added average value corresponds to the low frequency component data for the G component. The plane conversion unit 105 writes the generated GL plane to the memory 104 via the memory I / F unit 103.

In step S303, the plane conversion unit 105 reads the G0 plane and the G1 plane from the memory 104 via the memory I / F unit 103, and generates a GH plane composed of GH component data. Specifically, as shown in FIG. 5, the plane conversion unit 105 calculates the difference average of the pixel data at the same coordinate positions of the input G0 plane and G1 plane, and generates the difference average value as GH component data. .. That is, the GH plane is a set of the difference average values of the G0 plane and the G1 plane. The difference averaging process is a moving difference (derivative) operation between two G0 and G1 planes, and is consistent with the high-pass filter process. And the difference average value corresponds to the high frequency component data with respect to the G component. Then, the plane conversion unit 105 writes the generated GH plane to the memory 104 via the memory I / F unit 103.

In step S304, the color conversion unit 106 reads the GL plane, the R plane, and the B plane from the memory 104 via the memory I / F unit 103, performs luminance color difference conversion, and performs luminance color difference conversion on one luminance plane and two luminance planes. Generate a plane. Specifically, the color conversion unit 106 performs luminance color difference conversion according to the equation (1), and a Y plane composed of luminance component data, a U plane composed of color difference component data of color difference U, and a color difference of color difference V. Generate a V-plane composed of component data. In the luminance color difference conversion of the first embodiment, the integer type reversible component conversion formula will be described as an example, but other similar conversion formulas can also be applied.

In step S305, the frequency conversion unit 107 reads the Y plane, the U plane, the V plane, and the GH plane from the memory 104 via the memory I / F unit 103. Then, the frequency conversion unit 107 executes wavelet transform on each read plane to form a subband. The frequency conversion of the first embodiment may be replaced by the discrete cosine transform used in a coding technique such as JPEG, and is not limited to the wavelet transform.

In step S306, the control unit 108 determines the ratio of the plane target code amount of each plane with reference to the information stored in its own memory. The control unit 108 determines the ratio of each plane target code amount so that the plane target code amount of the Y plane becomes larger than the target code amount of the other planes in consideration of the human visibility characteristic. The ratio of the plane target code amounts of the U plane, the V plane, and the GH plane excluding the Y plane may be allocated so that the target code amount ratios of the planes are equal to each other, or may be changed respectively.

The control unit 108 also determines the subband target code amount of each subband generated by executing the wavelet transform by the frequency conversion unit 107 for each plane. In general, among the subbands obtained by the wavelet transform, the energy of the image is more concentrated in the low frequency subband. Therefore, as for the ratio of each subband target code amount within the same decomposition level, it is necessary to set a large ratio of the subband target code amount of the low frequency subband in order to suppress deterioration of image quality. On the other hand, the GH plane contains a large amount of high frequency components with respect to the G component. For this reason, with respect to the GH plane, there is a high possibility that deterioration of the edges and the like of the image will be large only by the target code amount setting method in which only the subband target code amount ratio of the low frequency subband is given preferential treatment. Therefore, the ratio of the subband target code amount of the HH subband of the GH plane is set to be larger than the ratio of the subband target code amount of the HH subband set in the other planes. By setting in this way, it is possible to leave information on high-frequency components and suppress deterioration of edges and the like of the image.

Here, FIG. 6 shows an example of setting the subband target code amount ratio of each plane and each subband. In the figure, the ratio of the plane target code amount of each plane is Y: U: V: GH = 40: 15: 15: 30 (percentage). The subband target code amount ratio of each subband in the GH plane is LL: HL: LH: HH = 40: 20: 20: 20. Then, the sub-band target code amount ratio of each sub-band other than the GH plane is LL: HL: LH: HH = 45: 22: 22: 11. The control unit 108 determines the distribution ratio of the plane target code amount for each plane so that the total target code amount of the entire RAW data is 100%. Then, the control unit 108 determines the subband target code amount so that the plane target code amount of one plane becomes 100%.

In addition, in order to simplify the explanation of the subband formation result shown in FIG. 6, the number of subband divisions by the wavelet transform is set to one. Then, the control unit 108 obtains each sub-band target code amount when the total target code amount of the entire RAW data is T and the plane target code amount ratio is set under the conditions of FIG. 6 as follows.
[Y plane]
Subband target code amount of LL subband = T × (40/100) × (45/100)
Subband target code amount of HL subband = T × (40/100) × (22/100)
Subband target code amount of LH subband = T × (40/100) × (22/100)
Subband target code amount of HH subband = T × (40/100) × (11/100)
[U plane]
Subband target code amount of LL subband = T × (15/100) × (45/100)
Subband target code amount of HL subband = T × (15/100) × (22/100)
Subband target code amount of LH subband = T × (15/100) × (22/100)
Subband target code amount of HH subband = T × (15/100) × (11/100)
[V plane]
Subband target code amount of LL subband = T × (15/100) × (45/100)
Subband target code amount of HL subband = T × (15/100) × (22/100)
Subband target code amount of LH subband = T × (15/100) × (22/100)
Subband target code amount of HH subband = T × (15/100) × (11/100)
[GH plane]
Subband target code amount of LL subband = T × (30/100) × (40/100)
Subband target code amount of HL subband = T × (30/100) × (20/100)
Subband target code amount of LH subband = T × (30/100) × (20/100)
Subband target code amount of HH subband = T × (30/100) × (20/100)
In FIG. 6, when the total target code amount of the entire RAW data is 100%, the target code amount assigned to each subband is shown in parentheses.

In step S307, the quantization unit 109 quantizes the conversion coefficient data after frequency conversion supplied from the frequency conversion unit 107 according to the type of plane and subband to which the conversion coefficient data belongs. Further, the quantization parameter used by the quantization unit 109 is determined based on the subband target code amount calculated in step S306.

In step S308, the entropy encoding unit 110 compresses and encodes the conversion coefficient data of each subband after quantization, and supplies the encoded data to the output unit 111. The output unit 111 concatenates the coded data of each subband of each plane in a preset order. Further, the output unit 111 creates a file header including information including information necessary for decoding. The information stored in the file header includes the number of pixels in the horizontal and vertical directions of the RAW image data, the number of bits per pixel, the quantization parameters for each type of plane and subband, and the like. Then, the output unit 111 writes a file composed of the file header and the subsequent coded data to the storage medium 112.

As described above, according to the first embodiment, the image pickup apparatus 100 generates independent planes for the color component data of R, G0, G1, and B from the RAW image data of the Bayer array. Then, the image pickup apparatus 100 generates a GL plane by performing addition averaging processing of the G0 and G1 planes. Further, the image pickup apparatus 100 generates a GH plane by performing a subtraction averaging process of the G0 and G1 planes. Here, since the averaging is a low-pass filter operation, the GL plane generation process corresponds to the process of extracting the low frequency component with respect to the G component of the RAW data. Further, since the subtraction average is a high-pass filter operation, the GH plane generation process corresponds to the process of extracting the high frequency component with respect to the G component of the RAW data. Then, the image pickup apparatus 100 of the first embodiment converts the GL, R, and B planes into Y, U, and V planes having luminance color differences in order to further improve the coding efficiency. Then, the image pickup apparatus 100 performs frequency conversion, quantization, and entropy coding on each of the GH planes in addition to the YUV planes obtained by the conversion.

As described above, the image pickup apparatus 100 (encoding apparatus) of the first embodiment generates a high frequency component and a low frequency component with respect to the G component in advance before frequency conversion. As a result, it is possible to suppress the generation of folding noise generated by subsampling during the formation of the G0 and G1 planes, and to reduce the deterioration of the frequency conversion efficiency. Then, the luminance color difference conversion to YUV is performed using the R plane, the GL plane, and the B plane. Then, in order to weight the target code amount according to the luminosity factor characteristic, the target code amount of the Y plane is assigned more than the target code amount of the other planes. As a result, the compression efficiency of the coding process can be improved while suppressing the deterioration of the image quality.

Further, in the first embodiment, the control unit 108 adjusts the code amount by setting the quantization unit 109 with a subband target code amount according to the type of plane and the type of subband. And said. However, this does not limit the invention. For example, even in JPEG2000, the conversion coefficient data obtained by the wavelet transform is quantized. Then, in JPEG2000, the conversion coefficient data obtained by quantization is regarded as binary data in a bit plane composed of the same bit positions, and entropy coding (arithmetic coding) is performed in each bit plane as a unit. Now, it is assumed that the coded data of the bit plane of the bit i of the subband of interest of a certain color component of interest is expressed as Ci, and the code amount thereof is expressed as A (Ci). At this time, the total code amount C_total of the coded data of the subband of interest of the color component of interest is
C_total = ΣA (Ci) (i = 0,1,2, ..., MSB)
Is. Therefore, if the target code amount of the focus subband of the focus color component is A_Target, the minimum value of k that satisfies the following equation is obtained.
C_total-ΣA (Ck) ≤ A_Target
Then, the coded data of the bit plane from bit 0 to bit k may be discarded. In each of the embodiments described below, the code amount is adjusted by the quantization parameter as in the first embodiment, but the code is specified by the bit plane discarding process by taking advantage of the merits of adopting JPEG2000. The amount may be adjusted.

[Modification 1 of the first embodiment]
The first embodiment is an example of application to an imaging device. Hereinafter, an example realized by an application program executed by a general-purpose information processing device such as a personal computer will be described as a modified example of the first embodiment.

FIG. 10 is a block configuration diagram of the information processing device in this modified example. When the power of this device is turned on, the CPU 1001 executes a boot program stored in the ROM 1002, loads the OS (operating system) 1005 from the HDD (hard disk drive) 1004 into the RAM 1003, and executes the OS. As a result, the CPU 101 can accept instructions from the user via the keyboard 1009 and the mouse 1010, and control the display control unit 1011 to display menus and the like on the display device 1012. That is, this device functions as an information processing device used by the user. Then, when the user instructs the mouse 1010 or the like to start the application program 1006, the CPU 1001 loads the application program 1006 into the RAM 1002 and executes the application program 1006. As a result, this device functions as an image processing device that encodes RAW image data. Then, the unencoded RAW data included in the RAW image data file 1007 stored in the HDD 1004 is encoded and generated as an encoded file 1008.

In the above, the processing procedure of the CPU 1001 when the application program 1006 is executed is almost the same as the flowchart of FIG. The difference is that the CPU 1001 executes each step of FIG. Further, the RAM 1003 will be used as the memory 104 in FIG. 1 and the memory used for the temporary storage in each step.

[Modification 2 of the first embodiment]
In the first embodiment, the target code amount ratio is set as shown in FIG. 6, but a case where the target code amount is set by another method will be described as a modification 2.

In this modification, as shown in FIG. 11, the wavelet transform is performed up to the decomposition level 3, and the target code amount is assigned to each subband. In this modification, the operation unit 115 and the display unit (not shown) of the image pickup apparatus 100 are used to determine the target code amount for each plane and each subband according to the compression rate set by the user. The distribution ratio is changing. The compression rate is set by the user by the operation unit and the display unit, but the user may automatically set the compression rate according to the shooting mode of the imaging device 100.

FIG. 11A shows the ratio of the target code amount of each plane and each subband when 1/3 compression is set as the compression ratio, and FIG. 11B shows 1/5 compression. The ratio of the code amount of each plane and each subband when set is shown.

Assuming that the target code amount of the entire RAW image is 100%, in the case of 1/3 compression, the target code amount is assigned at a ratio of 35% to the Y plane, 23% to the U plane and the V plane, and 19% to the GH plane. Similarly, in the case of 1/5 compression, the target code amount is assigned at a ratio of 45% to the Y plane, 19% to the U plane and V plane, and 17% to the GH plane. That is, in this modification, the target code amount is set so that the target code amount of the Y plane> the target code amount of the U plane = the target code amount of the V plane> the target code amount of the GH plane.

Since the luminance component is an important component in image data, the Y plane is assigned a larger amount of code than the other planes. Further, since the Y plane, the U plane, and the V plane are components necessary for forming an RGB image, a larger amount of code is assigned than the GH plane.

Further, in the case of 1/5 compression, the total code amount is smaller than in the case of 1/3 compression, and if the ratio of each plane is the same as 1/3 compression, the code amount assigned to the Y plane becomes smaller. It ends up. Therefore, in 1/5 compression, the code amount of the Y plane is secured by increasing the distribution ratio of the target code amount of the Y plane as compared with the case of 1/3 compression.

The code amount assigned to each plane is further assigned to each subband. In FIG. 11, the ratio of distribution of the target code amount to each subband is shown with the target code amount of the entire RAW image as 100%.

In the image data, the low frequency component is more important than the high frequency component. Therefore, in the Y, U, and V planes, the target code amount is set so that HL subband = LH subband> HH subband at decomposition level 2 and decomposition level 1. On the other hand, in the GH plane, the target code amount is set so that HL subband = LH subband ≦ HH subband at decomposition level 2 and decomposition level 1. This is because the GH plane is the data corresponding to the green high frequency component, so the high frequency component data has a more important role than the Y, U, and V planes. Therefore, in the GH plane, the target code amount of the HH subband is set to be equal to or larger than the target code amount of the HL subband and the LH subband.

In this modification, in the case of 1/3 compression, HL subband = LH subband = HH subband at the decomposition levels 1 and 2 of the GH plane, and in the case of 1/5 compression, the decomposition levels 1 and 2 of the GH plane In, the target code amount was set so that HL subband = LH subband <HH subband.

However, not limited to this, even when HL subband = LH subband> HH subband, in the GH plane, the high frequency component is weighted more than other planes to set the target code amount. You may try to do so. For example, the target code amount of the HH subband to the HL subband or the LH subband in the GH plane is larger than the ratio of the target code amount of the HH subband to the HL subband or the LH subband in the Y, U, and V planes. By doing so, it is possible to set the target code amount by weighting the subbands of the high frequency components of the GH plane.

[Second Embodiment]
In the code amount control performed by the control unit 108 according to the first embodiment, a static target code amount setting method in which the plane target code amount of the Y plane is made larger than the target code amount of the other planes has been described. On the other hand, in the second embodiment, the generated code amount of the GH plane is estimated from the dispersion of each color plane output by the plane forming unit 102, and the plane target code amount of the GH plane, the R plane, and the B plane is dynamically set. The method of setting to is explained.

Further, the plane conversion unit 105 according to the first embodiment performs a predetermined pixel calculation between two pixels of G0 pixels and G1 pixels located in the diagonal direction constituting the Bayer array, and performs a predetermined pixel calculation with respect to the G component, and the low frequency component and the high frequency component. The components were calculated. On the other hand, in the second embodiment, a predetermined pixel calculation is performed in units of two adjacent upper and lower lines of G0 pixels and G1 pixels, or two adjacent left and right rows of G0 pixels and G1 pixels, and the G component is low. The frequency component and the method of calculating the high frequency component will also be described.

FIG. 7 is a block configuration diagram of a main part related to coding in the image pickup apparatus 700 according to the second embodiment. The difference from FIG. 1 in the first embodiment is that the feature analysis unit 701 is added. The feature analysis unit 701 reads the G0 plane stored in the memory 104 and calculates the variance of the G0 plane. Then, the feature analysis unit 701 supplies the calculated dispersion information to the code amount control unit 108. Although the color component used in the dispersion calculation in the second embodiment is the G plane, it may be the average of the dispersions of the R plane, the G plane, and the B plane. Further, at least one of the components of the image pickup apparatus 700 according to the second embodiment has a hardware configuration.

In general, an image having many edges has a large variance, and the amount of code generated by the GH plane representing the high frequency component with respect to the G component is large. Therefore, when the variance is large, it is easy to maintain the edge of the image by allocating a large amount of plane target code of the GH plane. This is the purpose of calculating the variance.

Hereinafter, the coding process for one frame of the RAW image data in the image pickup apparatus 100 in the second embodiment will be described with reference to the flowchart of FIG. Although the processing in units of one frame is shown here as an example, the tiles may be divided into tiles of arbitrary size and the coding processing described later may be executed independently for each tile.

In step S801, the plane forming unit 102 forms independent planes for each of the color components of R, G, and B by forming the RAW image data of the Bayer array from the imaging unit 101 as shown in FIG. 9, and the memory I. Write to the memory 104 via the / F unit 103. The G plane shown in FIG. 9 is arranged so as to be adjacent to the G0 pixel on the left and right by moving the G1 pixel upward between two lines adjacent to the top and bottom of the Bayer array. Therefore, the horizontal size of the G plane is the same as the number of pixels in the horizontal direction of the RAW image data, and is twice the number of pixels in the horizontal direction of the R plane and the B plane. Further, in step S801, the G0 pixel may be moved to the left between the two rows adjacent to the left and right of the Bayer array so as to be vertically adjacent to the G1 pixel. In this case, the vertical size of the G plane is double that of the R plane and the B plane.

In step S802, the feature analysis unit 701 reads the G plane from the memory 104 via the memory I / F unit 103 and calculates the variance. The formula for calculating the variance is as shown in formula (2). In the formula, σ ² is the variance, N is the total number of pixels that make up the G plane, Xave is the average value of all the pixel values that make up the G plane, and Xi is the value of the i-th pixel that makes up the G plane. Then, the feature analysis unit 701 supplies the calculated variance σ ² to the control unit 108.

In step S803, the plane conversion unit 105 reads the G plane from the memory 104 via the memory I / F unit 103 to generate the GL plane. Then, the plane conversion unit 105 writes the generated GL plane to the memory 104 again via the memory I / F unit 103. The GL plane may be generated, for example, by applying the low-pass filter processing of the wavelet transform performed by the frequency conversion unit 107 once in the horizontal direction. However, the plane transform unit 105 executes the wavelet transform filter process while subsampling the even-numbered pixels along the horizontal direction of the G plane. Therefore, the horizontal size of the generated GL plane is the same as that of the R plane and the B plane.

Further, it is assumed that in step S801, the plane forming portion 102 forms the G plane by arranging the G0 pixels and the G1 pixels vertically adjacent to each other between the two rows adjacent to the left and right of the Bayer array. In this case, the plane conversion unit 105 may apply the low-pass filter processing of the wavelet transform performed by the frequency conversion unit 107 once in the vertical direction. Then, the plane conversion unit 105 writes the generated GL plane to the memory 104 via the memory I / F unit 103.

In step S804, the plane conversion unit 105 reads the G plane from the memory 104 via the memory I / F unit 103 to generate the GH plane. Then, the plane conversion unit 105 writes the generated GH plane to the memory 104 again via the memory I / F unit 103. The GH plane may be generated, for example, by applying the wavelet transform high-pass filter processing performed by the frequency converter 107 once in the horizontal direction. However, the plane transform unit 105 executes the wavelet transform filter process while subsampling the odd-numbered pixel strings along the horizontal direction of the G plane. Therefore, the horizontal size of the generated GH plane is the same as that of the R plane and the B plane.

Further, it is assumed that in step S801, the plane forming portion 102 forms the G plane by arranging the G0 pixels and the G1 pixels vertically adjacent to each other between the two rows adjacent to the left and right of the Bayer array. In this case, the plane conversion unit 105 may apply the wavelet transform high-pass filter processing performed by the frequency conversion unit 107 once in the vertical direction. Then, the plane conversion unit 105 writes the generated GH plane to the memory 104 via the memory I / F unit 103.

In step S805, the color conversion unit 106 reads the GL plane, the R plane, and the B plane from the memory 104 via the memory I / F unit 103, and executes the luminance color difference conversion processing for these three planes. Then, the color conversion unit 106 writes the Y plane, the U plane, and the V plane obtained by the luminance color difference conversion process into the memory 104 via the memory I / F unit 103. The following equation (3) is an example of a real number type lossy conversion equation for luminance color difference conversion.
Y = 0.2126 x R + 0.7152 x GL + 0.0722 x B
U = -0.1146 x R-0.3854 x GL + 0.5 x B
V = 0.5 × R-0.4542 × GL-0.0458 × B… (3)

In step S806, the frequency conversion unit 107 reads the Y plane, the U plane, the V plane, and the GH plane from the memory 104 via the memory I / F unit 103, executes wavelet transform, and performs wavelet conversion of each plane. Form a subband. Then, the frequency conversion unit 107 supplies the formed subband to the quantization unit 109.

In step S807, the control unit 108 determines the plane target code amount of each plane. The control unit 108 weights the Y plane so that the plane target code amount is larger than the other plane target code amounts. The ratio of the plane target code amount set here is Y: U: V: GH = 40: 15: 15: 30, as in the first embodiment. The method of setting the sub-band target code amount for each sub-band is the same as that of the first embodiment, and the description thereof will be omitted.
In step S808, the control unit 108 includes a plurality of preset threshold values (in the embodiment, T1 and T2, and has a relationship of T1 <T2) and the variance σ ² supplied from the feature analysis unit 701. Make a comparative judgment. When the variance σ ² is equal to or greater than the threshold value T1 and equal to or less than the threshold value T2, the process proceeds to step S810 without changing the ratio of the plane target code amounts. If this is not the case, that is, if the variance σ ² <T1 or the variance σ ² > T2, the control unit 108 advances the process to step S809.

In step S809, the control unit 108 corrects the plane target code amounts of the GH plane, the U plane, and the V plane based on the variance σ ² output by the feature analysis unit 701. When σ ² <T1, the RAW image data to be encoded can be regarded as a flat image. Therefore, the control unit 108 modifies the GH plane so that the plane target code amount is small. Specifically, the control unit 108 corrects the ratio of the plane target code amount of the Y: U: V: GH plane to 40:20:20:20. Further, when the variance σ ² > T2, the RAW image data to be encoded can be regarded as an image having many undulations (many edges) in terms of brightness. Therefore, the control unit 108 modifies the plane target code amount of the GH plane to be even larger. Specifically, the control unit 108 corrects the ratio of the plane target code amount of Y: U: V: GH to 40:10:10:40. Here, assuming that the total target code amount of the entire RAW image data is T, the target code amount of each plane is obtained as follows.
[For σ ² <T1]
Plain target code amount of Y plane = T × (40/100)
U-plane plane target code amount = T × (20/100)
V plane plane target code amount = T × (20/100)
GH plane plane target code amount = T × (20/100)
[When T1 ≤ σ ² ≤ T2]
Plain target code amount of Y plane = T × (40/100)
U-plane plane target code amount = T × (15/100)
Plane target code amount of V plane = T × (15/100)
Plane target code amount of GH plane = T × (30/100)
[When T2 <σ ² ]
Plain target code amount of Y plane = T × (40/100)
U-plane plane target code amount = T × (10/100)
V-plane plane target code amount = T × (10/100)
Plane target code amount of GH plane = T × (40/100)
Then, the subband target code amount of each plane is determined. The method for setting the subband target code amount is the same as that in the first embodiment.

In step S810, the quantization unit 109 quantizes the conversion coefficient data supplied by the frequency conversion unit 107 with a quantization parameter (quantization step) corresponding to the subband target code amount set by the control unit 108. To do. The quantization parameter is determined based on the subband target code amount set in step S807 or step S809.

In step S811, the entropy encoding unit 110 compresses and encodes the conversion coefficient data of each subband after quantization, and supplies the encoded data to the output unit 111. The output unit 111 concatenates the coded data of each subband of each plane in a preset order. Further, the output unit 111 creates a file header including information including information necessary for decoding. Then, the file composed of the file header and the subsequent encoded data is written to the storage medium 112.

As described above, according to the second embodiment, at least the same effect as that of the first embodiment described above can be obtained. Then, in the second embodiment, in order to estimate the number of edges included in the RAW image data to be encoded, the feature analysis unit 701 determines the variance for the G plane. Then, the control unit 108 adaptively determines the ratio of the target code amounts of the U, V, and GH planes other than the Y plane based on the dispersion. Specifically, when the variance output by the feature analysis unit 701 is between the two threshold values T1 and T2, the control unit 108 considers the variance to be standard and has a default target code amount. Set the ratio. Then, when the variance exceeds the threshold value T2, it is determined that there are many edges (high frequency components), and the ratio of the target code amounts of the GH plane, the U plane, and the V plane is set so that the ratio of the target code amount of the GH plane increases. Fix it. On the contrary, when the variance output by the feature analysis unit 701 is lower than the threshold value T1, the control unit 108 estimates that the edge (high frequency component) is small, and the GH plane, so that the ratio of the target code amount of the GH plane is small. Correct the ratio of the target code amount of the U plane and the V plane. That is, the larger the variance, the larger the code amount assigned to the GH plane and the smaller the code amount assigned to the U and V planes. As a result, the compression efficiency can be improved even for an image having many edges (high frequency components), and as a result, deterioration of subjective image quality can be suppressed.

[Third Embodiment]
In the above embodiment, a target code amount is set for the entire RAW image data, and coding is performed so as to obtain the set target code amount. On the other hand, in the present embodiment, instead of setting the target code amount, the quantization parameter is set for each subband, and the quantization is performed with the set quantization parameter. Since the basic configuration is the same as that of the first embodiment and the second embodiment, only different parts will be described.

In the present embodiment, after the wavelet transform (S305, S806), the quantization parameter is set for each subband of each plane without calculating the target code amount (S306, S807). After that, quantization (S307, S810) and entropy coding (S308, S811) are performed.

The quantization parameters are set as follows.
Quantization step value of 3LL of Y plane: 3LLy
3HL quantization step value of Y plane: 3HLy
Quantization step value of 3LH of Y plane: 3LHy
3HH quantization step value of Y plane: 3HHy
2HL quantization step value of Y plane: 2HLy
2LH quantization step value of Y plane: 2LHy
2HH quantization step value of Y plane: 2HHy
1HL quantization step value of Y plane: 1HLy
1LH quantization step value of Y plane: 1LHy
1HH quantization step value of Y plane: 1HHy
Quantization step value of 3LL of U / V plane: 3LLuv
U / V plane 3HL quantization step value: 3HLuv
Quantization step value of 3LH of U / V plane: 3LHuv
U / V plane 3HH quantization step value: 3HHuv
2HL quantization step value of U / V plane: 2HLuv
2LH quantization step value of U / V plane: 2LHuv
2HH quantization step value of U / V plane: 2HHuv
1HL quantization step value of U / V plane: 1HLuv
1LH quantization step value of U / V plane: 1LHuv
1HH quantization step value of U / V plane: 1HHuv
GH plane 3LL quantization step value: 3LLgh
GH plane 3HL quantization step value: 3HLgh
GH plane 3LH quantization step value: 3LHgh
GH plane 3HH quantization step value: 3HHgh
2HL quantization step value of GH plane: 2HLgh
2LH quantization step value of GH plane: 2LHgh
2HH quantization step value of GH plane: 2HHgh
GH plane 1HL quantization step value: 1HLgh
GH plane 1LH quantization step value: 1LHgh
GH plane 1HH quantization step value: 1HHgh
Then
3LLy = 3HLy = 3LHy = 3HHy ≦ 2HLy = 2LHy <2HHy ≦ 1HLy = 1LHy <1HHy… (4)
3LLuv ≦ 3HLuv = 3LHuv <3HHuv <2HLuv = 2LHuv <2HHuv ≦ 1HLuv = 1LHuv <1HHuv… (5)
3LLgh = 3HLgh = 3LHgh ≦ 3HHgh ≦ 2HLgh = 2LHgh = 2HHgh <1HLgh = 1LHgh = 1HHgh… (6)
3LLy≤3LLuv <3LLgh ... (7)
1HHy <1HHuv <1HHgh ... (8)
Set the quantization parameters for the subbands of each plane to satisfy.

When setting the quantization parameter, the control unit 108 may calculate the quantization step value satisfying the above, or the quantization step value satisfying the above is stored in advance in a non-volatile memory (not shown). , It may be read from the non-volatile memory and set. Further, when the compression rate can be set as in the second modification of the first embodiment, different quantization step values may be set according to the compression rate.

By setting the quantization step value having the above relationship, usually, the generated code amount of the Y plane> the generated code amount of the U / V plane> the generated code amount of the GH plane.

This is because the luminance component is an important component in image data, so the Y plane has a smaller quantization step value and is quantized more finely than other planes where deterioration is relatively inconspicuous, so that deterioration is noticeable. I have lost it. Further, since the Y plane, the U plane, and the V plane are components necessary for forming an RGB image, the quantization step value is smaller than that of the GH plane, and the quantization is made finer than that of the GH plane to deteriorate the deterioration. It makes it inconspicuous. The GH plane has a larger quantization step value than the other planes and quantizes coarser than the other planes to suppress the amount of generated code.

The quantization step value in the decomposition level 2 subband is 2HL = 2LH <2HH in the Y, U, and V planes, whereas it is 2HL = 2LH = 2HH in the GH plane. Regarding the quantization step value in the subband of decomposition level 1, 1HL = 1LH <1HH in the Y, U, and V planes, whereas 1HL = 1LH = 1HH in the GH plane. Since the low frequency component is more important in the image, in the Y, U, and V planes, the subbands HL and LH have a smaller quantization step value than the subband HH and are deteriorated more than the subband HH. Is reduced. Since the subband HH has a low importance, the quantization step value is increased to coarsely quantize and suppress the generated code amount. On the other hand, since the GH plane is the data of the green high frequency component, the importance of the high frequency component is higher than that of the other planes. Therefore, unlike the Y, U, and V planes, the quantization step value is set to the same as that of the subband HL and the subband LH without increasing the quantization step value by the subband HH, and the deterioration of the subband HH is prevented. ..

(Other Examples)
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. This is the process to be executed.

101 ... Imaging unit, 102 ... Plane forming unit, 103 ... Memory I / F unit, 104 ... Memory, 105 ... Plane conversion unit, 106 ... Color conversion unit, 107 ... Frequency conversion unit, 108 ... Control unit, 109 ... Quantization Unit, 110 ... Entropy coding unit, 111 ... Output unit, 112 ... Storage medium, 115 ... Operation unit, 701 ... Feature analysis unit

Claims (13)

An image coding device that encodes image data in a Bayer array.
A generation means for generating GL component data composed of low frequency components of G component and GH component data composed of high frequency components of G component from G0 component data and G1 component data of image data of Bayer array.
From the R component data, B component data, and the GL component data of the image data of the Bayer array, the luminance component data composed of the luminance component, the first luminance component data composed of the first luminance component, and the second Luminance color difference conversion means for generating the second color difference component data composed of the color difference components of
The luminance component data, the first color difference component data, the second color difference component data, and the GH component data, have a coding means for coding each component,
In the coding means, the code amount of the brightness component data is larger than the code amount of the first color difference component data and the code amount of the second color difference component data, and the code amount of the GH component data is increased. An image coding apparatus, characterized in that each component data is encoded so as to be smaller than the code amount of the first color difference component data and the code amount of the second color difference component data . The coding means has such that the target code amount of the brightness component data> the target code amount of the first color difference component data ≈ the target code amount of the second color difference component data> the target code amount of the GH component data. a, sets a target code amount of each component data, the image encoding apparatus according to claim 1, characterized in that coding each component data so that the set target amount of codes. The image coding apparatus according to claim 1, wherein the coding means generates a plurality of sub-bands by wavelet transforming component data, and encodes each of the generated sub-bands. In the coding means, a target code amount is set for each subband, and coding is performed so as to obtain the set target code amount.
When encoding a subband of a predetermined decomposition level, for the brightness component data, the first color difference component data, and the second color difference component data, the target code amount of the HH subband is the HL sub. The target code amount is set so as to be smaller than the target code amount of the band and the LH subband, and for the GH component data, the ratio of the target code amount of the HH subband to the target code amount of the HL or LH subband is The third aspect of claim 3 , wherein the target code amount of the subband is set so as to be larger than the brightness component data, the first color difference component data, or the second color difference component data. Image encoding device. When the coding means encodes the subband of the predetermined decomposition level, the brightness component data, the first color difference component data, and the second color difference component data are referred to as HH subbands. The target code amount is set and encoded so that the target code amount is smaller than the target code amount of the HL subband and the LH subband, and for the GH component data , the target code amount of the HH subband is the HL subband. The image coding apparatus according to claim 4 , wherein the target code amount is set and coded so as to be equal to or larger than the target code amount of the band and the LH subband. The coding means determines the target code amount of the subband by distributing the target code amount assigned to the component data to each subband.
Further, it has a compression rate setting means for setting the compression rate of image data.
The image coding apparatus according to claim 3 , wherein the coding means has a different ratio of a target code amount to be distributed to each subband depending on a compression rate set by the compression rate setting means. The image coding apparatus according to claim 3 , wherein the coding means performs quantization based on a quantization parameter set for each subband and encodes the data after quantization. An image coding device that encodes image data in a Bayer array.
A generation means for generating GL component data composed of low frequency components of G component and GH component data composed of high frequency components of G component from G0 component data and G1 component data of image data of Bayer array.
From the R component data, B component data, and the GL component data of the image data of the Bayer array, the luminance component data composed of the luminance component, the first luminance component data composed of the first luminance component, and the second Luminance color difference conversion means for generating the second color difference component data composed of the color difference components of
It has a luminance component data, a first color difference component data, a second color difference component data, and a coding means for encoding the GH component data for each component.
The coding means wavelet transforms the component data to generate a plurality of subbands, quantizes each subband based on the quantization parameter set for each subband, and encodes the quantized data.
The coding means quantizes the quantization parameter of the subband of a predetermined decomposition level, and for the brightness component data, the first color difference component data, and the second color difference component data, the HH subband. The parameters are set to be larger than the quantization parameters of the HL subband and the LH subband, and for the GH component data, the quantization parameters of the HH subband are set to the quantization of the HL subband and the LH subband. picture image encoding apparatus and sets so that the above parameters. The coding means has the quantization parameter of the subband of the predetermined decomposition level, and for the GH component data, the quantization parameter of the HH subband is equal to the quantization of the HL subband and the LH subband. The image coding apparatus according to claim 8 , wherein the image coding apparatus is set so as to be such. The generation means
The GL component data is generated by calculating the addition average of the G0 component data and the G1 component data as low frequency component data.
The image coding apparatus according to claim 1 or 8 , wherein the GH component data is generated by calculating the subtraction average of the G0 component data and the G1 component data as high frequency component data. It is a control method of an image coding device that encodes image data of a Bayer array.
The generation means generates GL component data composed of low frequency components of G component and GH component data composed of high frequency components of G component from G0 component data and G1 component data of image data of Bayer array. Generation process and
A first color difference conversion means is composed of a brightness component data composed of a brightness component and a first color difference component from the R component data, the B component data, and the GL component data of the image data of the Bayer array. A brightness color difference conversion step for generating color difference component data and a second color difference component data composed of a second color difference component, and
Encoding means, said luminance component data, the first color difference component data, the second color difference component data, and the GH component data, possess an encoding step of encoding each component,
In the coding step, the code amount of the brightness component data becomes larger than the code amount of the first color difference component data and the code amount of the second color difference component data, and the code amount of the GH component data becomes larger. A control method of an image coding apparatus, characterized in that each component data is encoded so as to be smaller than the code amount of the first color difference component data and the code amount of the second color difference component data . It is a control method of an image coding device that encodes image data of a Bayer array.
The generation means generates GL component data composed of low frequency components of G component and GH component data composed of high frequency components of G component from G0 component data and G1 component data of image data of Bayer array. Generation process and
A first color difference conversion means is composed of a brightness component data composed of a brightness component and a first color difference component from the R component data, the B component data, and the GL component data of the image data of the Bayer array. A brightness color difference conversion step for generating color difference component data and a second color difference component data composed of a second color difference component, and
The coding means includes a coding step of coding the luminance component data, the first color difference component data, the second color difference component data, and the GH component data for each component.
In the coding step, the component data is wavelet-transformed to generate a plurality of sub-bands, each sub-band is quantized based on the quantization parameter set for each sub-band, and the quantized data is encoded.
In the coding step, the quantization parameter of the subband of a predetermined decomposition level is set, and the HH subband is quantized for the brightness component data, the first color difference component data, and the second color difference component data. The parameters are set to be larger than the quantization parameters of the HL subband and the LH subband, and for the GH component data, the quantization parameters of the HH subband are set to the quantization of the HL subband and the LH subband. A control method for an image encoding device, characterized in that it is set so as to be equal to or greater than the quantization parameter. A program for causing the computer to function as each means of the image coding apparatus according to any one of claims 1 to 10 by reading and executing the computer.

Images